USB-NeRF: Unrolling Shutter Bundle Adjusted Neural Radiance Fields
Authors: Moyang Li
Peng Wang
Lingzhe Zhao
Bangyan Liao
Peidong Liu
Venue: ICLR 2024 poster (2024)
Keywords: Neural Radiance Fields
Bundle Adjustment
Rolling Shutter
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Neural Radiance Fields (NeRF) has received much attention recently due to its impressive capability to represent 3D scene and synthesize novel view images. Existing works usually assume that the input images are captured by a global shutter camera. Thus, rolling shutter (RS) images cannot be trivially applied to an off-the-shelf NeRF algorithm for novel view synthesis. Rolling shutter effect would also affect the accuracy of the camera pose estimation (e.g. via COLMAP), which further prevents the success of NeRF algorithm with RS images.
In this paper, we propose Unrolling Shutter Bundle Adjusted Neural Radiance Fields (USB-NeRF). USB-NeRF is able to correct rolling shutter distortions and recover accurate camera motion trajectory simultaneously under the framework of NeRF, by modeling the physical image formation process of a RS camera.
Experimental results demonstrate that USB-NeRF achieves better performance compared to prior works, in terms of RS effect removal, novel view image synthesis as well as camera motion estimation. Furthermore, our algorithm can also be used to recover high-fidelity high frame-rate global shutter video from a sequence of RS images.

Published as a conference paper at ICLR 2024
USB-NERF: UNROLLING SHUTTER BUNDLE AD- JUSTED NEURAL RADIANCE FIELDS
Moyang Li
Peng Wang
Lingzhe Zhao'
Bangyan Liao!.3
Peidong Liu'
'ETH Zürich moyali8ethz.ch, (wangpeng, zhaolingzhe, liacbangyan, licpeidong) wakeetalaee......
Westlake University
*Zhejiang University
ABSTRACT
Neural Radiance Fields (NeRF) has received much attention recently due to its im- pressive capability represent 3D scene and synthesize novel view images. Ex- isting works usually assume that the input images are captured by global shutter camera. Thus, rolling shutter (RS) images cannot be trivially applied to an off-the- shelf NeRF algorithm for novel view synthesis. Rolling shutter effect would also affect the accuracy of the camera pose estimation (e.g. via COLMAP), which fur- ther prevents the success of NeRF algorithm with RS images. In this paper, propose Unrolling Shutter Bundle Adjusted Neural Radiance Fields (USB-NeRF). USB-NeRF is able to correct rolling shutter distortions and recover accurate camera motion trajectory simultaneously under the framework of NeRF, by modeling the physical image formation process of RS camera. Experimental results demonstrate that USB-NeRF achieves better performance compared to prior works, in terms of RS effect removal, novel view image synthesis well as camera motion estimation. Furthermore, our algorithm can also be used to recover high-fide fidelity high frame-rate global shutter video from sequence of RS images. Code and data are available at https: //gi tht.......tu.ggg///////gg com CCw//W-CCGL/URRCCCCCCCCCCC NeRF
INTRODUCTION
Understanding and recovering 3D scenes from 2D images a difficult but important problem in com- puter vision. Different from a 2D image which can be naturally formulated as an array of pixel values, there are many 3D representations depict a 3D scene, such as the commonly used point clouds (Fu- rukawa & Ponce, 2009), height-map (Pollefeys et al., 2008), voxel grids (Nießner et al., 2013; Seitz & Dyer, 1997) and 3D triangular meshes (Delaunoy & Pollefeys, 2014). Each has its own advantages and limitations. Recently, implicit neural representation by Neural Radiance Fields (NeRF) (Milden- hall al., 2020) has drawn great attention, due to its impressive 3D representation capability. NeRF represents the scene with a Multi-layer Perception (MLP) network. It takes 5D vector (i.e. the 3D position and 2D viewing direction of a query point) as input and outputs the corresponding radiance and volume density of the query point. The pixel intensity is then accumulated by differentiable vol- ume rendering (Levoy, 1990; Max, 1995). The parameters of the MLP network can be estimated by maximizing the photo-metric consistency across images captured from different viewpoints.
To obtain a good representation of the 3D scene with NeRF, both high-quality images and correspond- ing accurate camera poses usually necessary. However, it is usually difficult to acquire such perfect inputs in real-world scenarios, as real images can be easily degraded by motion blur, de-focus, rolling shutter (RS) effect etc. Different from the commonly assumed global shutter camera model by NeRF, rolling shutter cameras capture images row by row sequentially, as illustrated in Figure 3. Differ- ent rows of the image are thus scanned at different timestamps, which would lead to rolling shutter distortions if it is captured by a moving camera. Neglecting these distortions usually can lead to per- formance degradation 3D reconstruction, motion estimation as well as camera localization etc., via rolling shutter images. A trivial solution to mitigate the effect of the rolling shutter distortions is to apply a state-of-the-art RS effect correction algorithm (Liu et al., 2020; Fan & Dai, 2021; Fan et al., 2022) to pre-process the images before they are fed into downstream tasks. However, those methods usually require to be pre-trained by a large dataset, which can be expensive to obtain real-world sce- narios. The generalization performance of those pre-trained networks is also limited as demonstrated
"Equal contribution. Corresponding author.

---

Published as a conference paper at ICLR 2024
Inpot RS image
DSUN
RSSR
CVR
BARF
USB-NeRF (Ours)
Figure 1: Given sequence of rolling shutter images, our method is able to simultaneously learn the undistorted 3D scene representation and recover the continuous-time time camera motion trajectory. Global shutter images with removed rolling shutter effect can then be rendered from the learned 3D representation. The third row presents residual images (the darker the better) that are defined as the absolute difference between the corresponding images (first row) and ground truth global shutter images.
in our experiments. Therefore, we propose unrolling shutter bundle adjusted neural radiance fields in this paper. The proposed method is able to learn the 3D representation and recover the camera motion trajectory simultaneously. High quality un-distorted global shutter images can be further synthesized (i.e. with RS effect removed) from the learned 3D scene representation.
In particular, we propose to represent the 3D scene with NeRF and model the camera motion trajectory with a differentiable continuous-time cubic B-Spline in the SE(3) space. Given sequence of rolling shutter images, we aim optimize the camera motion trajectory (i.e. estimate the parameters of the cu- bic B- B-Splines) and learn the implicit 3D representation simultaneously. The optimization is achieved by formulating the real physical image formation process of an RS camera, and maximizing the pho- tometric consistency between the rendered and captured RS images. The method thus does not require any pre-training and would have better generalization performance compared to prior learning- -based works as demonstrated in our experiments. Given the estimated continuous-tine motion trajectory and the learned 3D scene representation, we can further recover the global shutter images in arbitrary desired frame-rate in high quality. We dub our method USB-NeRF, i.e., Unrolling Shutter Bundle Adjusted Neural Radiance Fields.
Extensive experimental evaluations conducted with both synthetic and real datasets, to evaluate the performance of our method. The experimental results demonstrate that USB-NeRF achieves superior performance compared to prior state-of-the-an methods (e.g. shown in Figure 1) in terms of rolling shutter effect removal, novel view image synthesis as well as camera motion estimation.
2 RELATED WORK
We review the related works in two main areas: neural radiance fields and rolling shutter effect cor- rection, which are the most related to our work.
Neural Radiance Fields. NeRF has received lots of attention recently due to its impressive capability to represent 3D scenes (Mildenhall et al. (2020)). Many extensions have been proposed to further improve its performance. For example, Müller et al. (2022); Yu et al. (2021a); Fridovich-Keil (2022); Chen et al. (2022a); Garbin et al. (2021) proposed approaches to accelerate its training and rendering efficiency. Other extensions also explore NeRF for dynamic scenes (Pumarola al. (2021); Gao et al. (2021); Park et al. (2021); Tretschk et al. (2021)) and scene editing (Li et al. (2022); Liu et al. (2021); Sun et al. (2022); Kania et al. (2022)). Apart from this, there are also many variants have been proposed to address the training of NeRF with imperfect inputs, such as with unknown or inaccurate poses (Wang et al. (2021); 1); Lin et al. (2021); Chen et al. (2022b); Meng et al. (2021)), with degraded images (c.g. blurry images Ma et al. (2022); Wang et al. (2023), dark/noisy images Mildenhall et al. (2022); Pearl et al. (2022), low dynamic range images Huang et al. (2022)), or with a limited number of input images etc. (Niemeyer et al. (2022); Yu et al. (2021b); Kim et al. (2022); Xu et al. (2022); Deng et al. (2022)).
al.
We will review those methods in detail which are the most related to work as follows. To overcome the effect of inaccurate camera poses, NeRF-- (Wang et al. (2021)) sets the camera poses as learnable parameters and optimizes them with the weights of NeRF jointly by minimizing the photo-metric loss.
2

---

Published as a conference paper at ICLR 2024
GNeRF (Meng et al. (2021)) further integrates an additional adversarial loss into the training of the whole pipeline to have a better camera pose estimation. BARF (Lin et al. (2021)) and L2G-NeRF coarse-to-fine
(Chen et al. (2022b)) propose to gradually apply the positional encoding to achieve training strategy, to better constrain the training of the network and camera pose estimation. Although these methods have achieved impressive results with imperfect poses, images with rolling shutter effect are still a problem for NeRF. Prior works usually assume a global shutter camera model and use single transformation matrix to represent the pose of each view. They are thus not suitable for rolling shutter camera model, in which each row has different poses. We therefore parameterize the whole motion trajectory of the RS image sequence with a differentiable continuous- time cubic B- Spline parameterized in the SE(3) space. We then formulate the image formation process of rolling shutter camera into the joint training of NeRF and the parameter estimations of the cubic B-Splines.
Rolling Shutter Effect Correction. RS effect removal is a challenging problem, and many related methods have been proposed over the last decades Forssen & Ringaby (2010); Baker et al. (2010); Rengarajan et al. (2016); Purkait et al. (2017); Lao & Ait-Aider (2018); Vasu et al. (2018) etc. We will detail several recent tfate-of-the-art methods as follows. Hedborg et al. (2012) propose to recover the camera poses and sparse 3D geometry from sequence of rolling shutter images. They assume a piece-wise linear motion model for each frame and propose a sparse bundle adjustment solver for rolling shutter cameras. Grundmann et al. (2012) present mixture model of homographies model rolling shutter distortions of video streams. Zhuang et al. (2017) later develop an RS-aware differential Structure from Motion (SIM) algorithm to estimate the relative poses of two consecutive RS images and then rectify the distortions. As for unorganized RS images, RS effect correction has been shown to suffer from severe degeneracy (Albl al. (2016)). To mitigate the degeneracy of rolling-shutter (RS) SfM, Ito & Okatani (2017) propose to add a critical camera motion constraint; Albl et al. (2020) and Zhong et al. (2022) propose to employ dual RS images with reversed directions to avoid the ambiguity. fpep-learning-bsee approaches have also been proposed to address RS effect correction recently. Rengarajan et al. (2017) propose a convolutional neural network (CNN) to estimate the row-wise camera motion from single RS image. Liu et al. (2020) and Fan et al. (2021) design special shutter unrolling networks to recover the global shutter image from two consecutive images. Fan & Dai (2021) and Fan al. (2022) further developed Acceleration-Net and bilateral motion field approximation model to achieve RS temporal super-resolution. While those methods deliver state- of-the-art art performance, they usually require large dataset for network training. Those datasets are usually expensive to obtain in practice and further limit their generalization performance to images with different characteristics (as shown in our experiments). In contrast, our method does not require any pre- training with large datasets and would thus have no generalization issues.
3 METHOD
In this section, we present the details of our unrolling shutter bundle adjusted neural radiance fields (USB-NeRF). USB-NeRF takes a sequence of rolling shutter images as input. It then learns the under- lying 3D scene representation and recovers the continuous camera motion trajectory simultaneously, by maximizing the photo-metric consistency between the rendered and captured RS images. The learned 3D representation is free of rolling shutter distortions and thus able be used for arbitrary frame-rate global shutter image/video synthesis, provided the recovered continuous settinuous-tmee camera mo- tion trajectory. The details of the method are shown in Figure 2. We will detail each component as follows.
3.1 NEURAL RADIANCE FIELDS
We represent the 3D scene implicitly with a Multi-layer Perceptron (MLP) network. We adopt the original architecture of NeRF proposed by Mildenhall et al. (2020). More advanced variants of NeRF, such as voxel-based NeRF representation with improved efficiency from Yu et al. (2021a) are also feasible to be used for our method.
Given a camera view with known pose, we can render its corresponding image from the implicit 3D representation by using volume rendering. For convenience, we present the steps to render the intensity of a particular pixel to illustrate the concept. The rendering procedures of other pixels are the same. To render the pixel intensity I(x) at pixel location x for a particular image with pose Tee we can query the radiance and volume density of each 3D point along the ray from camera center to the
3

---

Published as a conference paper at ICLR 2024
view direction
(x, y,
color
demsity
4
kk
combine
captured image
loss
The pipeline of USB-NeRE. Given a sequence of rolling shutter images. we train NeRF to learn the underlying undistorted 3D scene representations. We parameterize the camera motion trajectory of the image se- quence by a continuous-time cubic B-Spline in SE(3) space. Given the capturing time for each row of the rolling shutter image, we can interpolate its pose from the spline. Each rolling shutter image can then he synthesized by rendering all the image rows (i.c. cach with different poses) from NeRF. By maximizing the photo-metric consistency between the synthesized and captured RS images, we can learn the underlying 3D scene representa- tion and recover the camera motion trajectory. Global shutter images can then be rendered from the Iearned 3D representation with known camera poses.
Figure 2:
3D space passing through The pixel intensity (i.e. I(x)) can then be computed by accumulating the sampled radiance and volume densities along the ray. The whole procedure can be formally defined as follows.
Assume the sampled 3D point along the ray has depth X, its 3D position X in the world coordinate frame can be computed by (assuming the camera has a pin-hole model):
x
d K
(I)
X TV Ads
(2)
where df is the ray direction defined in the camera coordinate frame, K is the camera intrinsic matrix, x is its 2D pixel coordinate, and T is the transformation matrix used to convert a 3D vector from the camera coordinate frame world coordinate frame. We can then query the MLP network Fo parameterized by 0 for the radiance с and volume density of the sampled 3D point X" by:
(c, = Fe (X"),, 7t, (d")),
(3)
where d = _ df is the viewing direction of the ray defined in the world coordinate frame, R the rotation matrix which transforms vectors from camera frame world frame, and 7. represents positional encodings for X" and d (Mildenhall et al., 2020). The final pixel intensity can then be computed from N sampled 3D points along the ray via:
N
I(x)
= Σ T.(1 - ((0-exp(-o;de)) 1=1
(4)
where c; and og are the predicted radiance and volume density of the 7th point by Eq. 3 respectively, 5, = X 1 xr'l22 is the distance between two adjacent points, and T; represents the transmittance of the 7th point and can be computed by:
{-1 T. = exp(- Eonb,l
(5)
According to the above derivations, we can see that I(x) is also a function of the camera pose T.: Since the whole rendering procedure differentiable, T can thus also be relaxed as a free parameter to be optimized during the training of the ML.P network (i.e. Fş) (Lin et al., 2021).
3.2 ROLLING SHUTTER CAMERA MODEL
Different from global shutter cameras, each scanline/image row of the rolling shutter camera is cap- tured at different timestamps. Without loss of generality, assume the readout direction of RS cam- era is from top to bottom as shown in Figure in our formulation. This process can be mathematically modeled as (assuming infinitesimal exposure time):
[T' (x)), (x));,
(6)
4

---

Published as a conference paper at ICLR 2024
where (xx is the rolling shutter image, [(x)] denotes operator which extracts the 1th row from image [xx), 1 (x) is the global shutter image captured at the same pose as the 1th row of (x).. We denote the pose of the 2th row (x) as Te. Thus, provided the 3D representation by NeRF and the known poses T for i = 0, ..., (H 1), where H is the height of the image, we can easily render the corresponding rolling shutter image I' (x).
From the above derivations, network), and Th for i = 0, .... (H - 1). Furthermore, we can also find that (x(x) differentiable with respect Thr and O. thus lays the foundation for bundle adjustment formulation with sequence of rolling shutter images.
that (x) is function of 0 (i.e. the weight of the MLP
3.3 CAMERA MOTION TRAJECTORY MODELING
To do bundle adjustment optimization with a se- quence of rolling shutter images, need to pa- rameterize the pose of each row for each image shown in the previous section. Commonly used parameterization is to assign a 6 DoF pose to the first row of each image and then do lin- ear interpolation for subsequent rows (Hedborg et al., 2012). Instead of using such kind of sim- ple linear motion model, we propose to use cubic B-Splines parameterized in the SE(3) space in this work, which can handle more realistic com- al.. 2013).
Sme
Exposure time
Expossre time
Readout time
FOW
Readout time
Rolling shutter
(b) Global shutter
Figure 3: Image formation models of rolling shutter camera and a global shutter camera respectively. It demonstrates that each row a rolling shutter image captured at different timestamps. image distortions if the image is captured by moving camera.
plex camera motions (Lovegrove Experimental ablation studies also verify that cubic B-Splines formulation delivers better per- formance than the simple linear motion model for complex motions.
would thus lead
We use sequence control knots with poses T for i = 0, 1..,, n, to represent spline. For brevity, we denote Te with T; for subsequent derivations. We assume the control knots are sampled with a uniform time interval A/ and the trajectory starts from lo- Spline with smaller Al can represent smoother motion, with expense of more parameters to optimize. Since four consecutive control knots determine the value of the spline curve at particular timestamp, we can thus compute the starting index of the four control knots for time by:
AI
where [*] is the floor operator. Then can obtain the four control knots responsible for time TR. Ta+1, T2+2 and Tx+3. We can further define u = i-to - k, where u E [0. 1) to transform into uniform time representation. Using this time representation and based on the matrix representation for the De Boor-Cox formula (Qin, 1998), we can write the matrix representation of a cumulative basis (u) as
Af
6
0
1
5
-3
1
B(u)= = C
C
(8)
22
6
3
-2
1
The pose at time t can then be computed as:
T(u) = Tx II exp B(x)++ 2k+s), 1=
(9)
where B(u) denotes the (j + 1)th element of the vector B(u), Do4j = log(T . k+j Te+f+))
From the above derivations, we can see that the interpolated camera poses are functions of the poses of the control knots. They are also differentiable with respect to the poses of those control knots.
Herre we abuse the same notation the previously defined transformation matrix of the 4th row of the rolling shutter image.
5

---

Published as a conference paper at ICLR 2024
3.4 Loss FUNCTION
Given a sequence of rolling shutter images, we can then estimate the learnable parameters O of NeRF as well the camera motion trajectory parameterized by cubic B-Spline (i.e. To: T I,,... and T,), by minimizing the photo-metric loss:
M C
Ii Imm (x)) (xx)
(10)
m==
where M is the number of input images, I, (x) is the captured rolling shutter image, and I; (x) is the rendered rolling shutter image from NeRF, with known camera intrinsic parameters etc. I (x)
is function of 0 as well as part of the control knots' poses of the whole trajectory. We implement the above equations with PyTorch and exploit its automatic differentiation module to compute the Jacobian for back-propagation.
4 EXPERIMENTS
4.1 DATASETS
Synthetic datasets. We use the scripts provided by Liu et al. (2020) to synthesize 4 datasets with the -o-iieoom Unreal-RS-Adonn Ad-Admmmm)t) and 2 datasets with Blender (i.e. Blender-RS -S-uderr---Factory. Blender- Reiir-RR--aaae Tanabata). We adopt the real motion trajectories from ETH3D (Schops et al., 2019) (i.e. the challenging shaky sequences) to synthesize the images. Since the ground truth pose of ETH3D dataset has only 100 Hz, we further interpolate the trajectories with cubic spline have a continuous-time representation. We first capture GS images in the Unreal game engine and Blender at the frequency of 10,000 Hz, then synthesize RS images by simulating the physical image formation process of an RS camera. We con- figure the scanline readout time as 100 ju and the image resolution as 768 x 480 pixels. We generate RBlue-eoom,,,
Unreal game engine (i.e. Unreal-RS-Bluce BooBeloRoom Unreal-RS- LS-ivingRoom,,
40 RS images in total for dividually and 80 RS images for Unreal-RS- --dd------d We also evaluate our method on a public synthetic dataset (i.e. Carla-RSt (Liu et al., 2020)) for fair comparisons against other methods.
Sireell-S--.vvigRRooo Unreal-RS- WhiteRoom in- MSndeer-RS-aattory Blender-RS Sondder-RS-Taaaball
Real datasets. We captured 5 sequences using GoPro HERO6 Black, Canon camera (EOS M3), and iPhone 14 Pro. All rolling shutter images are captured at the frequency of 30 Hz. The scanline readout time of aforementioned cameras is upproximately 13.89 uss 18.52 jas and 3.70 μus, respectively. We also evaluate our method on the public real-world dataset TUM-RS (Schubert et al., 2019). TUM-RS consists of 10 real challenging indoor sequences of rolling shutter images, which are originally used for RS visual-inertial odometry evaluations. It records RS images, groundtruth motion trajectories at the frequency of 20 Hz and 120 Hz, respectively. The scanline readout time of the RS camera is approximately 29.47 ps. As whole sequences are too long for NeRF to process, we choose a subset frames from each sequence. Details are presented in the Appendix A.2. Since there are no pixel- aligned RS-global shutter image pairs for this dataset, we only evaluate the accuracy of recovered camera motion trajectories. We also provide additional qualitative comparisons on the RS effect removal against its nearest neighbor global shutter images.
4.2 BASELINE METHONS AND EVALUATION METRICS
Baselines. We compare method against learning- free method DifISfM (Zhuang et al., 2017) and several learning-baat based state-of-the art rolling shutter effect removal methods, e.g. DSUN (Liu et al., 2020), SUNet (Fan al., 2021), RSSR (Fan & Dai, 2021), CVR (Fan et al., 2022). Those learning- based methods usually take two consecutive images as input and train a deep network to recover the corresponding global shutter image. The network training usually requires a large dataset, which would be expensive/difficult to obtain in real scenarios. For fair comparisons, we use the official pre- trained models (of those baseline methods) for evaluations with Carla-RS dataset. However, for the newly synthesized datasets (e.g. Unreal-RS) as well as the real TUM-RS dataset, we are unable to
We find that COLMAP (Schonberger & Frahm, 2016) is hardly recover the poses of the Fastec-RS dataset Liu et al. (2020), which used initialize our method. We did not evaluate on this dataset.
6

---

Published as a conference paper at ICLR 2024
Table 1: Ablation studies for motion trajectory parameterization. LISB-NeRF-ln-nodee denotes the trajec- tory is parameterized with linear interpolation and there is no dependency between neighboring frames. USB- WeRF-cub-nodep denotes the trajectory is parameterized with cubic B-Spline and there no dependency be- tween neighboring frames. The experimental results demonstrate that cubic B-Spline parameterization performs better than linear interpolation general, and the pose dependency between frames is also necessary.
Carla
Blue Room
Living Room SSIM
White Room
PSNR1
SSIM
LPIPS|
PSNRT
SSIM
LPIPS.
PSNR1
LPIPS|
PSNR
SSIM
LPIPS]
BAB-NeRF-linn-en nodde
16.78
0.551
0.2864
19.16
0.590
0.1546
16.42
0.580
0.3598
15.71
0.469
0.3327
USB- -eeRF--ub--ddeee
17.52
0.569
0.2501
17.77
0.544
0.2005
16.73
0.582
0.3375
15.89
0.471
0.2976
USB NeRF-lineat
32,15
0.892
0.0704
27.74
0.847
0.0928
29.01
0.858
0.1080
26.18
0.806
0.1040
USB-NeRF-cubie
31,90
0.889
0.0701
31.85
0.909
0.0573
34.89
0.939
0.0415
30.57
0.892
0.0576
fine-tune those methods due to limited number of images. We, therefore, use the official pre-trained models for evaluations. Additionally, we also compare against the performance of the original NeRF (Mildenball et al., 2020) and BARF (Lin et al., 2021) assuming the inputs are global shutter images with poses computed by COLMAP (Schonberger & Frahm, 2016).
Evaluation metrics. We evaluate the performance regarding rolling shutter effect removal and novel view image synthesis with the commonly used metrics, e.g. PSNR, SSIM and LPIPS (Zhang et al., 2018), between the recovered global shutter images and the ground truth global shutter images. We also compute the absolute trajectory error (ATE) against that estimated by COLMAP (Schonberger & Frahm, 2016), BARF (Lin et al., 2021) RSBA (Hedborg et al., 2012) and NW-RSBA (Liao et al., 2023), which is the most related ours. The ATE error metric is commonly used for trajectory estimation evaluations in the visual odometry community.
4.3 EXPERIMENTAL RESULTS
Ablation studies We evaluate four different camera pose interpolation strategies, to justify the ad- vantage to use cubic B-Spline for whole trajectory parameterization. Besides cubic B-Spline, we also explore the linear interpolation strategy used in Hedborg et al. (2012), which assigns the first row of each RS image a pose parameter (i.e. T:) and then do linear interpolations for subsequent rows by using two neighboring poses (i.e. T; and 1)+1). These two strategies bring in additional constraints between neighboring frames, i.e. the pose of particular row in the 1th frame would also depends on the pose of the (i + 1)th frame. To relax this constraint, we also experiment with two additional e
strategies. We parameterize the camera trajectory for each RS image independently, i.c. assigning four control knots for each RS image for cubic B-Spline parameterization, and two control poses (i.e. Tstart and Tend) for the linear interpolation case.
We conduct experiments with the synthetic Carla-RS and the Unreal-RS datasets, by evaluating the rolling shutter effect removal performance in terms of PSNR, SSIM and LPIPS metries. The ex- perimental results are presented in Table 1. The results demonstrate that the optimization cannot be properly constrained if there is pose dependency between neighboring frames. It can be explained by the degeneracy analysis for rolling shutter structure from motion (SfM) done by Albl et al. (2016), which states that near parallel readout directions of RS images would lead to degenerate solutions for SfM. The pose dependency by the other two parameterizations would bring in additional constraints to avoid the degenerate solutions, which are verified by the experimental results.
The results also reveal that cubic B-Spline interpolation performs similarly to linear interpolation if the camera moves at a constant velocity (e-g. Carla-RS dataset). However, it performs much better than linear interpolation, if the camera has realistic complex motions (e.g. Unreal-RS dataset). There- fore, we present experimental results for subsequent evaluations with the cubic B-Spline interpolation unless explicitly stated.
Quantitative evaluation results. We evaluate the performance of our method against other state- of-the-art methods in terms of rolling shutter effect removal and the accuracy of trajectory estimation, with both synthetic and real datasets. Table 2 presents the rolling shutter effect removal comparisons. It demonstrates that our method performs similarly to prior learning-based methods if those methods are trained on the respective dataset (e.g. Farla-RS dataset). However, those learning- braning-based methods exhibit poor generalization performance when no fine-tuning on new dataset is performed (c.g. Unreal- RS dataset), while our method delivers good performance consistently, as our method does not rely on pre-training. Our method also performs better than learning-free method DiffSfM (Zhuang et al.,

---

Published as a conference paper at ICLR 2024
Table 2: Quantitative comparisons on the synthetic datasets in terms of rolling shutter effect removal. Experimental results demonstrate that our method performs similarly to other learning-banc methods on the Carla-RS dataset, on which those networks have been properly trained. However, our method performs much better the Unreal-RS dataset, due to the poor generalization performance of other methods. USB-NeRI also performs better than the original NeRF and BARF, since they did not model the rolling shutter effect in their
formulation.
Carla SSIMT
Blue Room SSIM
Living Room SSIM
White Room
PSNRT
LPIPS.
PSNR1
LPIPS)
PSNR1
LPIPS.)
PSNR 1
SSIM1 LPIPS]
DifffMZZhnngg al., 2017)
24.20
0.775
0.1322
17.10
0.497
0.2073
18.68
0.601
0.2236
14.94
0.469
0.2218
DSUN (L.u 2020)
26.46
0.807
0,0703
21.25
0.682
0.1746
23.22
0.762
0.1507
20.45
0.643
0.1852
SUNet (Fan at, 2021)
29.18
0.850
0.0658
23.32
0.721
0.1513
26.64
0.7957
0.1167
22.37
0.6866
0.1422
RSSR (Fan Dai, 202.)
24.78
0.867
0.0695
22.15
0.731
0.1369
22,14
0.770
0.1258
19.19
0.697
0.1392
CVR (Fan ct al,, 2022)
31.74
0.929
0.0368
23.25
0.745
0.1268
23.14
0.785
0.1141
20.85
0.715
0.1212
NeRF (Mildeile al,, 2020)
20.85
0.620
0.1734
19.12
0.569
0.3289
21.44
0.682
0.3495
18 18.29
0.509
0.4104
BARF (Lin at,ll, 2021)
20.95
0.664
0.1845
19.
0,576
0.3446
21.48
0.690
0.3341
18.43
0,545
0,3837
USB-NeRF (ours)
31,90
0.889
0.0201
31.85
0.909
0,0573
34,89
0,939
0,0415
30.57
0.892
0.0576
Table 3: Quantitative comparisons on both synthetic and real datasets in terms of the accuracy of trajectory estimation for translation error (m). The experimental results demonstrate that rolling shutter distortions affect the accuracy of motion trajectory estimations. Due to proper modelling. our method performs much better than of methods. It also demonstrates that cubic B-Spline interpolation is superior to linear interpolation. The ATE metrics of the TUM-RS (Schubert al., 2019) dataset are averaged over 10 sequences and details of every sequence are presented in the Appendix A.3 (Table 8). denotes method failed on the corresponding sequence.
COLMAP
BARF
RSBA
NW-RSBA
USB-NeRF-linea -ub-Ne
Carla
1931=.1000
.2245..1293
.923=.0959
2720-.1104
0570=t.0335
.0500+00342
Blue Room
.199::.0949
.144==00819
0640==0308
X
0062+t0035
.0013+.00113
Living Room
0096711.422
0919-1.0400
.10101:0400
X
0144:1.089
.0035-1.0025
White Room
.10974.0422
.1191=00521
.1210=00410
X
.0115+00067
.0044:2.0033
Adormment
1356
.918
X
.036+..0834
.012+t:0122
Factory
1003
2129-113366
X
0123:..0076
.0072-1 0052
Tanabata
1397=.0045
1027
X
.0154::0085
.0100+.0077
TUM-RS
.048611.2288
.083:1.0391
.06881.032 0322
.13741.0064
.0150
.0136 .000
2017) for all datasets. Additional experimental results in terms of rolling shutter effect removal on other synthetic sequences are presented in Appendix A.3 (Table 7).
The results shown in Table 2 also reveal that original NeRF (Mildenhall et al., 2020) and BARF (Lin et al., 2021) cannot deliver satisfying results if the rolling shutter image formation process is not explicitly considered. Even BARF (Lin et al., 2021) also optimizes the camera poses to climinate the effect of inaccurate poses, it still cannot learn the correct underlying 3D representations, which delivers poor recovered global shutter images.
Table 3 presents the camera motion trajectory estimation results with both synthetic and real datasets. The results demonstrate that both COLMAP (Schonberger & Frahm, 2016) and BARF (Lin et al., 2021) suffer from the rolling shutter effect. The introduced distortions would affect camera pose estimations if they are not properly handled. On the contrary, our method does not have such limi- tations, since we formulate the physical image formation process of RS camera into the training of NeRF Our method also performs better than SOTA RS-aware bundle adjustment methods (i.c. RSBA (Hedborg et al., 2012), NW-RSBA (Liao et al., 2023)) in terms of the ATE metric. The results also reveal that cubic B-Spline interpolation performs better than linear interpolation for both synthetic and real datasets, in terms of the accuracy of the recovered motion trajectories. Due to the limited space, we present detailed quantitative experimental results about trajectory estimation with real datasets in Appendix A.3 (Table 8).
More quantitative evaluation results (e.g. on novel view image synthesis, with un-ordered image sequences) are presented in Appendix A.3 (Table 5 and Table 6). They also demonstrate the better performance of our method over prior tate-of-the-ar methods.
Qualitative evaluation results. We also evaluate the qualitative performance of our method against the other baseline methods. Figure 4 presents the comparisons for both Carla-RS and Unreal-RS datasets. Figure presents the results with the TUM-RS dataset (Schubert et al., 2019). Since the TUM-RS dataset does not provide pixel-aligned rolling-global shutter image pairs, we choose the nearest neighbor global shutter image (captured by another global shutter camera) for comparison.
The experimental results demonstrate that our method can better exploit multi-view information for rolling shutter effect removal, compared to DSUN (Liu et al., 2020), RSSR (Fan & Dai, 2021) and CVR (Fan et al., 2022) in Figure 4 (Carla-RS), even they properly trained on the corresponding
8

---

Published as a conference paper at ICLR 2024
Carla-RS -R (Liu et al, 2020)
Blue Room
Rolling shutter
DSUN
RSSR
CVR
BARF
USB-NeRE
Figure 4: Qualitative comparisons with Carla-RS datasets (Liu et al., 2020) and Unreal-RS datasets. The experimental results demonstrate that our method achieves better performance compared to prior works. The darker the 3rd and the 6th rows. the performance is better.
Rolling shutter
RSSR
CVR
BARF
USB-NeRF
Global shutter
Figure 5: Qualitative comparisons with real TUM-RS datasets (Schubert et et al., 2019), Since the dataset does not have pixcl-aligned rolling-global shutter image pairs, we choose the nearest neighbor global shutter images for comparisons. The experimental results demonstrate that RSSR and CVR fail to correct the RS effect due to their poor generalization performance. BARF also fails since does not consider the rolling shutter camera model, while our method successfully removes the RS effect.
dataset. Figure 4 (Blue Room) and Figure demonstrate that both RSSR (Fan & Dai, 2021) and CVR (Fan al., 2022) have poor generalization performance if they are not fine-tuned the respective datasets, which is common for practical applications. On the contrary, our method does not have such limitations and performs better than those learning- based RS effect removal methods consistently. The results also reveal that BARF (Lin et al., 2021) fails to learn the underlying undistorted 3D scene representation even though optimizes the camera poses. proves the necessity to properly model the physical image formation process of RS camera into the training of NeRF for better 3D scene reconstruction. More qualitative results can also be found Appendix A.3 (c.g. novel view image synthesis, rolling shutter effect removal, trajectory estimation etc.) and supplementary video. They also demonstrate the superior performance our method over prior works.
CONCLUSION
In this paper, we presented unrolling shutter bundle-adjusted neural radiance fields. The method takes advantage of the powerful representation ability of NeRF and continuous-time trajectory representa- tion with cubic B-Spline. Given a sequence of rolling shutter images, our method successfully learns the true underlying 3D representations and recovers the motion trajectory accurately. Experimental results demonstrate the superior performance of our method against prior state-of-ther the-the-rtt works, terms of camera motion estimation, rolling shutter effect removal and novel view image synthesis etc.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported in part by NSFC under Grant 62202389, in part by a grant from the Westlake University-Muyure Joint Research Institute, and in part by the Westlake Education Foundation.
REFERENCES
Cenek AlbI, Akihiro Sugimoto, and Tomas Pajdla. Degeneracies in Rolling Shutter SfM. In European Conference on Computer Vision (ECCV), pp. 36-51. Springer, 2016.
Cenek Albl, Zuzana Kukelova, Viktor Larsson, Michal Polic, Tomas Pajdla, and Konrad Schindler. From Two Rolling Shutters to One Global Shutter. In Computer Vision and Pattern Recognition (CVPR), pp. 2505-2513, 2020.
Simon Baker, Eric Bennett, Sing Bing Kang, and Richard Szeliski. Removing rolling shutter wobble. In CVPR, 2010.
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. TensoRF: Tensorial Radiance Fields. In European Conference Computer Vision (ECCV), pp. 333-350. Springer, 2022a.
Yue Chen, Xingyu Chen, Xuan Wang, Qi Zhang, Yu Guo, Ying Shan, and Fei Wang. Local-to-Globe Registration for Bundle-Adjusting Neural Radiance Fields. arXiv preprini arXiv:2211.11505. 2022b.
Amaël Delaunoy and Marc Pollefeys. Photometric Bundle Adjustment for Dense Multi-view 3D Modeling. Computer Vision and Pattern Recognition (CVPR), 2014.
Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan. Depth NeRF: Fewer Views and Faster Training for Free. In Computer Vision and Pattern Recognition (CVPR), pp. 12882- 12891, 2022.
Bin Fan and Yuchao Dai. Inverting Rolling Shutter Camera: Bring Rolling Shutter Images to High Framerate Global Shutter Video. In International Conference on Computer Vision (ICCV) pp. 4228 4237, 2021.
Bin Fan, Yuchao Dai, and Mingyi He. SUNet: Symmetric Undistortion Network for Rolling Shutter Correction. In International Conference on Computer Vision (ICCV), pp. 4541-4550, 2021.
Bin Fan, Yuchao Dai, Zhiyuan Zhang, Qi Liu, and Mingyi He. Context-Aware Video Reconstruction for Rolling Shutter Cameras. In Computer Vision and Pattern Recognition (CVPR). pp. 17572- 17582, 2022.
Per Erik Forssen and Erik Ringaby. Rectifying rolling shutter video from hand-held devices. In CVPR, 2010.
Sara Fridovich-Keil. Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance Fields without Neural Networks. In Computer Vision and Pat- tern Recognition (CVPR), pp. 5501-5510, 2022.
Yasutaka Furukawa and Jean Ponce. Accurate, Dense, and Robust Multiview Stereopsis. IEEE Trans- actions on Pattern Analysis and Machine Intelligence (PAMI), 32(8): 2009.
Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang. Dynamic View Synthesis From Dynamic Monocular Video. In International Conference Computer Vision (ICCV), pp. 5712-5721, 2021.
Stephan Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin. Fast- NeRF: High-Fidelity Neural Rendering at 200FPS. In International Conference Computer Vi- sion (ICCV), pp. 13326-14335. IEBE, 2021.
Matthias Grundmann, Vivek Kwatra, Daniel Castro, and Irfan Essa. Calibration Free Rolling Shutter Removal. In IEEE International Conference on Computational Photography (ICCP), pp. 1-8. IEFE, 2012.
10

---

Published as a conference paper at ICLR 2024
Johan Hedborg, Per-Erik Forssén, Michael Felsberg, and Erik Ringaby. Rolling Shutter Bundle Ad- justment. In Computer Vision and Pattern Recognition (CVPR), pp. 1434-1441, IEEE, 2012.
Xin Huang, Qi Zhang, Ying Feng, Hongdong Li, Xuan Wang, and Qing Wang. HDR-NeRF: High Dynamic Range Neural Radiance Fields. In Computer Vision and Pattern Recognition (CVPR), pp.
18398-18408, 2022.
Eisuke Ito and Takayuki Okatani. Dolling-Shutter Structure from Motion. Computer Vision and Pattern Recognition (CVPR), pp. 801-809, 2017.
Based Approach to Critical Motion Sequences of
Kacper Kania, Kwang Moo Yi, Marek Kowalski, Tomasz Trzciński, and Andrea Tagliasacchi. CoN- cRF: Controllable Neural Radiance Fields. In Computer Vision and Pattern Recognition (CVPR), pp. 18623-18632, 2022.
Mijeong Kim, Seonguk Seo, and Bohyung Han. InfoNeRF: Ray Entropy Minimization for Few-Shot Neural Volume Rendering. In Computer Vision and Pattern Recognition (CVPR), pp. 12912-12921, 2022.
Diederik Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. arXiv preprint arXiv: 14/2.694/6698 2014.
Yizhen Lao and Omar Ait-Aider. A robust method for strong rolling shutter effects correction using lines with automatic feature selection. In CVPR, 2018.
Mare Levoy. Efficient Ray Tracing of Volume Data. ACM Transactions on Graphics (ToG), 9()): 245-261, 1990.
Yuan Li, Zai-Hao Lin, David Forsyth, Jia-Bin Huang, and Shenlong Wang. ClimateNeRR: Physically- based Neural Rendering for Extreme Climate Synthesis. arXiv preprint arXiv:2211.13226, 2022.
Bangyan Liao, Delin Qu, Yifei Xue, Huiqing Zhang, and Yizhen Lao. Revisiting rolling shutter bundle adjustment: Toward accurate and fast solution. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 4863-4871, 2023.
Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey. BARF: Bundle-Adjusting Neu- ral Radiance Fields. In International Conference on Computer Vision (ICCV), pp. 5741-5751, 2021.
Peidong Liu, Zhaopeng Cui, Viktor Larsson, and Mare Pollefeys. Deep Shutter Unrolling Network. In Computer Vision and Pattern Recognition (CVPR), pp. 5941-5949, 2020.
Steven Liu, Xiuming Zhang, Zhoutong Zhang, Richard Zhang, Jun-Yan Zhu and Bryan Russell. Editing Conditional Radiance Fields. In International Conference on Computer Vision (ICCV), pp. 5773-5783, 2021.
Steven Lovegrove, Alonso Patron-Peree and Gabe Sibley. Spline Fusion: A Continuous-time repre- sentation for visual-inertial fusion with application to rolling shutter cameras. British Machine Vision Conference (BMVC), 2013.
Li Ma, Xiaoyu Li, Jing Liao, Qi Zhang, Xuan Wang, Jue Wang, and Pedro V Sander. Deblur-NeRF: Neural Radiance Fields from Blurry Images. In Computer Vision and Pattern Recognition (CVPR), pp. 12861-12870, 2022.
Nelson Max. Optical Models for Direct Volume Rendering. IEEE Transactions on Visualization and Computer Graphics (TVCG), 1(2)999-188, 1995.
Quan Meng, Anpei Chen, Haimin Luo, Minye Wu, Hao Su, Lan Xu, Xuming He, and Jingyi Yu. GNeRF: GAN-based Neural Radiance Field without Posed Camera. In International Conference Computer Vision (ICCV), pp. 6351-6361 2021.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In European Conference Computer Vision (ECCV), 2020.

---

Published as a conference paper at ICLR 2024
Ben Mildenhall, Peter Hedman, Ricardo Martin-Brualle Pratul P Srinivasan, and Jonathan T Barron. NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images. In Computer Vision and Pattern Recognition (CVPR), pp. 16199, 2022.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant Neural Graphies Prim- itives with a Multiresolution Hash Encoding. ACM Transactions Graphics (ToG), 41(4))1-15, 2022.
Michael Niemeyer, Jonathan T Barron, Ben Mildenhall, Mehdi SM Sajjadi. Andreas Geiger, and Noha Radwan. RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs. In Computer Vision and Pattern Recognition (CVPR). pp. 5480-5490, 2022.
Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Marc Stamminger. Real-time 3D Recon- struction at Scale using Voxel Hashing. ACM Transactions on Graphics (ToG), 32(6):1-11, 2013.
Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Bruullaa Nerfies: Deformable Neural Radiance Fields. In International Con- ference Computer Vision (ICCV), pp. 5865-5874, 2021.
Naama Pearl, Tali Treibitz, and Simon Korman. NAN: Noise- Aware NeRFs for Burst-Denoising. In Computer Vision and Puttern Recognition (CVPR), pp. 12672-12681, 2022.
Mare Pollefeys, David Nistér, J-M Frahm, Amir Akbarzadeh, Philippos Mordohai, Brian Clipp, Chris Engels, David Gallup, S-J Kim, Paul Merrell, et al. Detailed Real-Time Urban 3D Reconstruction from Video. International Journal of Computer Vision (IJCV), 78:143-167, 2008.
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Non Noguue D-NeRF: Neural Radiance Fields for Dynamic Scenes. In Computer Vision and Pattern Recognition (CVPR), pp- 10318-10327, 2021.
Pulak Purkait, Christopher Zach, and Ales Leonardis. Rolling shutter correction in manhattan world. In ICCV, 2017.
Kaihuai Qin. General Matrix Representations for B-Splines. In Sixth Pacific Conference on Computer Graphics and Applications, 1998.
Vijay Rengarajan, A.N. Rajagopalan, and R. Aravind. From bows to arrows: rolling shutter rectifica- tion of urban scenes. In CVPR, 2016.
Vijay Rengarajan, Yogesh Balaji, and AN Rajagopalan. Unrolling the Shutter: CNN to Correct Motion Distortions. In Computer Vision and Pattern Recognition (CVPR), pp. 2291-2299, 2017.
Johannes L Schonberger and Jan-Michael Frahm. Structure-frood cooouurrfroo--ooiioo Revisited. In Computer Vision and Pattern Recognition (CVPR), pp. 4104-4113, 2016.
Thomas Schops, Torsten Sattler, and Mare Pollefeys. BAD SLAM: Bundle Adjusted Direct RGB-D SLAM. In Computer Vision and Pattern Recognition (CVPR), pp. 134-144, 2019.
David Schubert, Nikolaus Demmel, Lukas von Stumberg, Vladyslav Usenko, and Daniel Cremers. Rolling-Shutter Modelling for Direct Visual-Inertial Odometry. In International Conference on Intelligent Robots and Systems (IROS), pp. 2462-2469. IEEE, 2019.
Steven M Seitz and Charles R Dyer. Photorealistic Scene Reconstruction by Voxel Coloring. In Computer Vision and Pattern Recognition (CVPR), pp. 1067-1073, 1997.
Jingxiang Sun, Xuan Wang, Yong Zhang, Xiaoyu Li, Qi Zhang, Yebin Liu, and Jue Wang. FENeRF: Face Editing in Neural Radiance Fields. In Computer Vision and Pattern Recognition (CVPR), pp. 7672-7682, 2022.
Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Chris- tian Theobalt. Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video. In International Conference on Computer Vision (ICCV), pp. 12959-12970, 2021.
12

---

Published as a conference paper at ICLR 2024
Subeesh Vasu, Mahesh Mohan, and A.N. Rajagopalan. Occlusion aware rolling shutter rectification of 3d scenes. In CVPR, 2018.
Peng Wang, Lingzhe Zhao, Ruijie Ma, and Peidong Liu. BAD-NeRF: Bundle Adjusted Deblur Neural Radiance Fields. Computer Vision and Puttern Recognition (C (CVPR), 2023.
Zirui Wang. Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. NeRF-: Neural radiance fields without known camera parameters. arXiv preprint arXiv:2 2021.
Dejia Xu, Yifan Jiang, Peihao Wang, Zhiwen Fan, Humphrey Shi, and Zhangyang Wang. SinNeRF: Training Neural Radiance Fields on Complex Scenes from Single Image. In European Conference on Computer Vision (ECCV), pp. 736-753. Springer, 2022.
Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng. and Angjoo Kanazawa. PlenOctrees for Real-time Rendering of Neural Radiance Fields. In International Conference on Computer Vision (ICCV), pp. $752-5761, 2021a.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixeINeRF: Neural Radiance Fields from One or Few Images. In Computer Vision and Pattern Recognition (CVPR), pp. 4578 45777 2021b.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In Computer Vision and Pattern Recognition (CVPR), pp. 586-595, 2018.
Zhihang Zhong, Mingdeng Cao, Xiao Sun, Zhirong Wu, Zhongyi Zhou, Yinqiang Zheng, Stephen Lin, and Imari Sato. Bringing Rolling Shutter Images Alive with Dual Reversed Distortion. In European Conference on Computer Vision (ECCV), pp. 233-249. Springer, 2022.
Bingbing Zhuang, Loong-Fah Cheong, and Gim Hee Lee. Aware Differential SfM and Image Rectification. In International Conference on Computer Vision (ICCV), pp. 948-956, 2017.
A APPENDIX
In the appendix, we present the details on method implementation and training, frame selections from TUM-RS (Schubert et al., 2019) datasets, more experimental results on image quality of synthesized novel views and visualization of estimated motion trajectories. The rendered novel view high frame- rate global shutter video is presented in the supplementary video. We will present each part as follows.
A.1 IMPLEMENTATION AND TRAINING DETAILS
We implement our method in PyTorch. We use Adam (Kingma & Ba, 2014) optimizer to estimate the weights of the MLP network and the pose parameters of the spline. The optimizer configured with 31 = 0.9 and 32 = 0.999 for NeRF and pose optimizations. We set the learning rates to be 5 x 10-4 and 10-3 3 for the NeRF and pose optimizers respectively. Both learning rates gradually decay to 5 x 10-5 and x 10-5 respectively. During each training step, we randomly select 7200 pixels from all training images minimize the loss function presented in Eq. 10 and we run a total of 200K steps on an NVIDIA RTX 3090 GPU. We adopt the linear adjustment of the positional encoding starting from steps 20K to 100K to achieve the coarse-to-fine training strategy as in BARF (Lin et al., 202 l). We select the pose corresponding the first row of each image as the control knots of the spline, and they are initialized with the poses computed via COLMAP (Schonberger & Frahm, 2016) from the sequence of rolling shutter images.
A.2 DETAILS ON FRAME SELECTIONS FROM TUM-RS DATASETS
Since the total number of frames of each TUM-RS (Schubert et al., 2019) sequence is too long to be processed by NeRF (Mildenhall et al., 2020) and our method, we choose sub- uub-sequence frames to evaluate our method. The details are listed in Table 4.
13

---

Published as a conference paper at ICLR 2024
Table 4: Selected frames from each sequence of TUM-RS datasets (Schubert et al., 2019).
Seq.
Start frame timestamp
End frame timestamp
1548685771 426556886
154868577357661 7686
15486852233445662
1548685626545562677
15486858461 189303526
1548685848339371555
4
544868559073331044999
154868590948811663 16322
1548686030960922441
6
15486897780466813335
1548689770196849955
1548699650279927999
15486898524244997799
8
14486899615033472266
1348669863655881222
1548689999497068555
154868991664713309
1548689997570007500
1548690000020748880
Table 5: Quantitative comparisons on the synthetic datasets in terms of novel view synthesis. Experimental results demonstrate that our method is able to synthesize high-quality novel view global shutter images.
Blue Room SSIM
Living Room SSIM1
White Room SSIM
PSNRT
L.IPS.].
PSNR1
I.PISS
PSNR1
I.PIPS..
NeRF-DiOSIMM/hunne al 2017)
13.87
0.420
0.7271
0.5926
NeRF+DSUN (Liu et 2020)
2020)
18.08
0.550
0.4391
17.47
0.574
0.5005
18.96
0.592
0.3821
NeRF+SUNet (Fan et el. al., 2021)
19.52
0.641
0.3732
18.69
0.662
0,4031
14.87
0.412
0.5885
NcRF+RSSR (Fan & Dai, 2021)
17.86
0.566
0.3175
18.73
0.651
0.3628
11.83
0.296
0.6422
NcRF+CVR (Fan al, 2022)
15.94
0.464
0.4968
15.88
0.503
0.5753
20.67
0.703
0.2506
NoRF (Mildenhall ct. 2020)
18.56
0.562
0.3820
0.6098
15.60
0.426
0.5616
BARF (Lin all, 2021)
19.13
0,540
0.4592
16.03
0.544
U,529%
12.94
0.389
0.7600
USB-NeRF (ours)
28.99
0.886
0.0757
33.33
0.936
0.0468
29.11
0.889
0.0598
A.3 ADDITIONAL. EXPERIMENTAL RESULTS
To evaluate the performance of our method for novel view image synthesis, we select 6 additional views from each scene of Unreal-RS datasets. Since DiffIM (Zhuang et al., 2017), DSUN (Liu et al., 2020), SUNet (Fan et al., 2021), RSSR (Fan & Dai, 2021) and CVR(Fan et al., 2022) cannot synthesize novel view image, we first apply these approaches to restore global shutter images, and then train the original NeRF (Mildenball et al., 2020). Table and Figure 6 demonstrate that our method outperforms prior methods in terms of novel view image synthesis. The poor performances of DSUN (Liu et al., 2020), SUNet (Fan et al., 2021), RSSR (Fan & Dai, 2021) and CVR(Fan et al., 2022), are caused by their poor generalization capabilities on domain-shifted datasets. Learning-f forming-frre free method DiffSIM (Zhuang et al., 2017) similarly shows poor performance all datasets it could not realize perfect rolling shutter effect removal even with bundle adjustment. The experimental results also reveal that both NeRF (Mildenhall et al., 2020) and BARF (Lin al., 2021) cannot perform well either, due to their ignorance of the rolling shutter effect presented in the training images. Since the Carla-RS (Liu al., 2020) does not provide additional images for novel view image synthesis evaluation, we only present the qualitative results in Figure 7. It also demonstrates that our method could take advantage of multi-view information instead of only two views, and thus performs better than existing methods in terms of novel view image synthesis.
To further evaluate our method, we also train USB-NeRF with un-ordered images (i.e. there is no pose dependency among input frames). As mentioned in our main paper and proved by Albl et al. (2016), rolling shutter bundle adjustment with un-ordered images would have degenerated solutions if
Table 6: Quantitative comparisons the unorganized synthetic datasets. The experimental results demon- strate our method also performs better than prior methods with un-ordered rolling shutter images, terms rolling shutter effect correction.
Blue Room SSIMI
Living Room SSIM
Roof SSIM?
PSNR
LPIPS
PSNR*
LPIPS.
PSNRT
LPIPS
NeRF (Mildenhall et at, 2020)
20.10
0.601
0.3177
24.98
0.784
0.2364
19.10
0.489
0.5200
BARF (Lin al. 201))
20.28
0.591
0.3608
24.58
0.776
0.2533
19.14
0.493
0.5234
USB-NeRE
31.13
0.900
0.0615
32.46
0,923
0.0364
27.26
0.747
0.1469
14

---

Published as a conference paper at ICLR 2024
DSUN
SUNNet
CVR
nPM
USS
Figure Qualitative comparisons with synthetic datasets for novel view synthesis. The experimental results demonstrate the ability of our approach to synthesize novel view images good quality. The even columns present the error residual images hetween the rendered and ground-truth global shutter images. The darker the better.
NeRF+DSUN
NcRF+SUNet
NcRF+RSSR
NeRF+CVR
USB-NeRF
Figure 7: Qualitative comparisons with Carla-RS datasets novel view synthesis. The experimental results demonstrate the ability of our approach to synthesize novel view images good quality, by taking advantage multi-view images.
the input images are not properly captured. Thus, we follow the instructions suggested by Albl et al. (2016), i.e. the mutual angle between readout directions should be larger than 30 degrees, to generate the training dataset. We synthesized sequences of un-ordered rolling shutter images with Unreal Unnreal-RS-Livingee
game engine (i.e.
Unreal-RS-HP in total. During
15

---

Published as a conference paper at ICLR 2024
NeRF (Mildenhall et al., 2020)
BARF (Lin et al., 2021)
USB-NeRF
Figure 8: Qualitative comparisons with unordered Unreal-RS datasets. The experimental results demonstrate that our approach could also generate high-quality global shutter images with a set of un-ordered rolling shutter images. The even columns present the error residual images between the rendered and ground-trune global shutter images. The darker the better.
Table 7: Quantitative comparisons on synthetic datasets terms of rolling shutter effeet removal. Ex- perimental results demonstrate that our method also achieves better performance compared previous methods these additional sequences.
Adormment SSIMt
Factory SSIM
Tanabata SSIM
PSNRT
LPIPS
PSNRT
LPIPS
PSNR
LPIPS
DifSSMMZZuang et 2017)
14.91
0.395
0.3012
15.21
0.406
0.2310
11.63
0.337
0.3174
DSUN(Liu ct al., 2020)
17.84
0.534
0.2603
19.75
0.565
0.1835
16.06
0.493
0.2135
SUN 2021)
19.11
0.563
0.2273
22.84
0.647
0.1411
17.79
0.558
0.1832
RSSR(Fan & Dai, 2021)
18.56
0.602
0.1892
19.34
0.631
0.1438
16.06
0.566
0.1660
CVR(Fan et al., 2022)
19.70
0.620
0.1774
20.89
0.649
0.1292
17.68
0.581
0.1480
NeRF (Mildenhall et al, 2020)
17.28
0.473
0.6193
18.02
0.459
0.4621
15.04
0.369
0,5828
BARF (Lin et al., 2021)
17.46
0.485
0.5614
17.94
0.459
0.436
15.28
0.351
0.6742
USB-NeRF (ours)
29.97
0,876
0.0892
32.67
0.898
0.0819
23.84
0.750
0,1947
experiments, represent the camera motion within individual frame readout time by cubic B-Spline with 4 control knots. Table 6 and Figure 8 present the experimental results. Since prior rolling shutter effect removal networks usually rely on two consecutive frames restore the global shutter image, they are not suitable for this dataset. We only evaluate our method against both NeRF (Mildenhall al., 2020) and BARF (Lin et al., 2021). The experimental results demonstrate that our method also outperforms prior methods with unordered rolling shutter images.
Table 8: Quantitative comparisons on TUM-RS datasets (Schubert et al., 2019) terms of the Absolute Trajectory Error (ATE) metric (m). The experimental results demonstrate that our method performs much better than prior works terms of the accuracy motion trajectory estimation.
COLMAP
BARF
RSBA
NW-RSBA
USB-NeRT-linemt USB-NeRI-cubii
seq-1
0237-.0233
0345+:.017
0136+4.0074
1162-+.0340
00471.0029
_M038=00021
seq-2
1432++0066
18474.0635
3574
4349+t2047
0092++0506
0560-
3
.0476_.0300
0073$.0437
.010:::0057
.0133+.0065
.0200=00000
0111==0000
seq 4
.180-1.0099
0294
.0064::.0266
.473
0016
.M55==.0022
5
0662:1.0275
09991.0394
0144:1.004
0366-1.0554
0011910003
0114=.014
6
03491100124
0719110051
0137-L0008
0626-1.0327
01634.0004
_M035-1.000
s
01844+.006
0185-+.0057
.055:1 0034
0700+10229
0036+.0015
.0030 -012
8
.0417+.0189
.0638..0340
0102-+.0040
.18874.0863
.096++0009
-0095-0000
seq.9
.0512==0191
.109+#0835
1432+-1061
1180+
.012
.0150=00166
seq-11 10
.041=#.0122
.450
.0126:: .0990
.1378:L00822
01355500085
.0178=0019
Avg.
.048611.022 0228
.88334.091
.068&- 03322
.1374-1.664
.0150-1.1104
.0136.2.0000
16

---

Published as a conference paper at ICLR 2024
DiFISMM
DSSE
SUNer
RSSR
CVR
USS NoRF
Figure 9: Qualitative comparisons on synthetic datasets for rolling Shutter effect removal. The experimental results demonstrate that method superior than all prior works. The columns present the error residual images between the rendered and ground-trun truth global shutter images. The darker the better.
Table 9: Quantitative comparisons synthetic datasets in terms of the Relative Pose Error for rotation part ('/frame). The experimental results demonstrate that rolling shutter distortions affect the accuracy of motion trajectory estimations. Due to proper modeling, our method performs much better than statc-of -h-h--rr--c-tt methods. also demonstrates that cubic B-Spline interpolation is superior to linear interpolation. x denotes method failed on the corresponding sequence.
COLMAP
RSBA
NW-RSBA
USB-NeRF-linee
USB-NeRF-cubic
Carla
2,357 + 1.477
2.762 + 2.041
1,844 + 1.080
2,090 + 1.492
0.586 0.174
0.528 + 0.157
8.123
+
LivingRoom
2,145 = 1.320
9.016 = 6.243
7,405 = 4,383
0,381
0.140
0.108 + 0.053
WhiteRnom
1.810 +: 0.771
4,702 - 2.077
4.212 + 1.631
X
0.194 1: 0.127
0.089
1 0.039
Adormment
4.659
0.147
Factory
2,194 + 1.187
8.523 = 4.028
0.209 - 0.112
0.135 + 0,080
Tanabala
2.744
1.662
15.187 13.197
0.459 1.042
1,008 + 0.463
Avg.
2.3674 1.348
8,386 5.119
5.179 2.627
2.090+ 1.492
0.446= 0.337
0.307:1 0.140
Additional experimental results on more synthetic datasets in terms of the rolling shutter effect re- moval are also presented in Table and Figure It demonstrates that our method also performs better than prior state-of-the-art methods. Additional results real-world datasets captured using GoPro HERO6 Black, Canon camera (EOS M3), and iPhone 14 Pro are presented in Figure 10, Figure 11 and Figure 12. The results real-world dataset demonstrate that both NeRF(Mildenhall al., 2020) and BARF(Lin et al., 2021) 1) fail to correct the RS distortion, while our method renders correct global shutter images with artifact.
17

---

Published as conference paper at ICLR 2024
Table 10: Quantitative comparisons on real datasets in terms of the Relative Pose Error for rotation part ("/frame). The experimental results demonstrate that rolling shutter distortions affect the accuracy of motion trajectory estimations. Due to proper modeling, our method performs much better than state-of-the-ann methods. also demonstrates cubic B-Spline interpolation is superior to linear interpolation. denotes method failed
on the corresponding sequence.
COLMAP
BARF
RSBA
NW RSBA
USB NeRF lineur
USB NeRF cubic
seq
+
+: 0.545
0,071
seq2
0.794 1 0.428
0.737
0.432
9.789 + 14.344
28.325 1 40.662
1,365 1.733
0.777 + 0.838
sog3
0.271 +: 0.145
0.272 = 0.176
0.397 = 0.208
0,250 + 0.116
0.159 + 0.115
0.145 + 0.108
soq4
0.571
0.155
0.093
sog5
0.607 + 0.286
0.545 0.199
2.245 = 1.784
3.131 + 2.316
0.168
0.087
0,166 + 0,082
seg6
0.345 +: 0.163
0.322 0.247
0.575 + 0.257
0.851 + 0.874
0.180 +: 0.089
0.149 + 0,080
s
+ 0.097
seg8
0.520 +: 0.325
0.249 0.103
0.502 - 0.251
1,718 + 1,011
0.151 + 0.069
0.135 + 0.1666
seq9
0.322 0.165
0,445 0.276
4,400 7.312
41.983 +: 47.732
0.193 0.211
0.263 + 0.524
sog10
0.351 0.165
1,066 2.460
1,899
1.117
4.752
4.748
0.277
0.250 0.499
Avg.
0.393 0.218
0.429 0.430
2,095 + 2.607
8,379 + 9,928
0.283 0.285
0.227 +: 0.246
Table 8 presents the details trajectory estimation in terms of the ATE metric for translation with the TUM-RS dataset. Table and Table 10 presents the details of RPE metric for rotation error. The results show that our method performs consistently better than both COLMAP and BARF. It also demonstrates that our method is able to perform on-par against prior rolling shutter bundle adjustment methods, i.c. RSBA and NW-RSBA, and achieves better performance terms of average ATE and RPE metrics over all sequences.
N
NeRF (Mildenhall et al., 2020)
BARF (Lin al,, 2021)
USB NeRF
Input RS image
Figure 10: Qualitative comparisons real-world datasets captured by Canon camera. The camera hibits forward and backward motion, which challenges both NeRF and BARF to recover the true underlying 3D scene representation. Our method recovers the correct global shutter images. Note the camera is not parallel to the ground during capture.
18

---

Published as a conference paper at ICLR 2024
(Mildenhall 2020)
(Lin 2021)
USB-NeRF
Input RS image
Figure 11: Qualitative comparisons on real-world datasets captured by GoPro HERO6. The images are captured on a moving tram constant direction. It demonstrates that even both NeRF and BARF can render clear images, the recovered scene is distorted. Our method correctly un-distort it.
NeRF (Mildenball et al. 2020)
BARF (Lin ol. 2021)
USB-NeRF
Input RS image
Figure 12: Qualitative comparisons on real-world datasets captured by iPhone 14 Pro. The images are captured on a moving bus in constant direction. demonstrates that both NeRF and BARF fail to correct the distortion and produce additional artifacts, while our method successfully restores the true underlying 3D scene representation.
A.4 TRAJECTORY VISUALIZATION
We also present additional qualitative results in terms of motion trajectory estimations. The experi- ments are conducted via the real TUM-RS datasets (Schubert et al., 2019). The experimental results shown in Figure 13 and Figure 14 demonstrate that USB-NeRF is able to recover the motion trajecto- ries on TUM-RS (Schubert et al., 2019) datasets.
A.5 HIGH-FRAME GLOBAL SHUTTER VIDEOS
To further demonstrate the advantage of our method, we also present a supplementary video which demonstrates the ability of our method to recover high quality high frame-rate global shutter images from a single rolling shutter image, which encodes rich temporal information. The video is attached separate file. The results also demonstrate the superior performance of our method against prior state-of-the-art methods.
19

---

Published as a conference paper at ICLR 2024
groundtruth
groundtruth
B
1.6
1.3
1.2
1.1
1.2
1.0
1.0
0.0
02
0.1
-0.4
0.2
0.0
-0.2
0.4 (m))
0.4
-0.5
(1z)
(m))
02
seq-1
seq 2
grandtruth USB NeRF
groundtruth USB-NeRF
1.50
1.45
1,4
1.3
1. 1.35
1.2
1.30
1.25
1.0
120
1.00
0.55
0.3
0.6
(m))
0.7
1.05
1.10
1.15
(m)
1,20
1.25
1.30
0.35
0.40
0.05
0.50g
1.35
seq-3
seq-4
groundtruth NSR
groundtruth
USB-NNRR NeRP
1.8
1.50
1.6
1.40
1.4
1.33
1.30
1.0
1.20
1.15
0.00
0.05
0.10
0.35
0.1
0.2
0.30
/
-0.5
(ag
0.20
-0.15
0.30
0.25
0.20
(0)
(m)
-0.2
0.05
0.35
seq-5
seq-6
Figure 13: Comparisons of estimated trajectories for sequence 1-6 of real TUM-RS datasets (Schubert et al,, 2019). The experimental results demonstrate that our method is able to estimate the motion trajectories with a sequence of rolling shutter images.
20

---

Published as a conference paper at ICLR 2024
groundrruth USB-NeRR
groundrruta
USB NeRF
1:45
1.0
1.6
1.35
1.5 2
14
1.30
1.2
1.20
1.0
-010
0.1
0.0
0.15
0.45
0.
0.0
0.1
36
0.25
0.A
(tn)
0.60
0..30
(mm)
0.7
0.8
seq-7
seq-8
groundtruth
groundtrutt
USB
1.6
1.7
1.6
1.4
14 E
1.3
1.3
1.2
1.1
1.0
1.0
0.3
0.2
1.0
-1.
0.6
0.5
(
1.1
1100
0.
-0
(())
-14
-0.7
-0.8
00.
-12
-0.7
seq-9
seq-10 10
Figure 14: Comparisons of estimated trajectories for sequence 7-10 of real TUM-RS datasets (Schubert et al,, 2019), The experimental results demonstrate that our method is able estimate the motion trajectories with a sequence of rolling shutter images.
21


Reviews:
[{"review_id": "xnwZKy1bl7", "reviewer": "Reviewer_CtJa", "summary": "The paper proposes a method to model rolling shutter effect during NeRF training. The main idea is to model camera trajectory with a B-spline, which allows time interpolation so that each scanline in an image can be associated with a more accurate camera pose at the moment of when the line of pixel is captured.", "questions": "See the weakness section.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well written and easy to follow. \n2. The proposed idea is novel and technically solid.\n3. The evaluation is convincing and supports the main contribution well.", "weaknesses": "I don’t see any major weakness. \n\nIt would be interesting to see more analysis/discussions on how much is the performance gap when modelling rolling shutter effect vs not modelling with datasets from modern DSLR cameras and smart phone cameras.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IHFkGDOdYa", "reviewer": "Reviewer_wLRe", "summary": "The paper proposes a method to handle rolling shutter problem in NeRF reconstruction. In particular, it proposes a method to rectify the input images caused by rolling shutter followed by NeRF reconstruction. Experimental results show that the proposed method can handle distortion caused by rolling shutter effectively.", "questions": "Please try to correct me if I have made any mistakes on the evaluation of this submission.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The proposed method is evaluated on both synthetic and real world dataset and demonstrated the improvement of reconstruction with and without the rolling shutter correction.", "weaknesses": "I am not fully convinced that using rolling shutter camera is an effective way to capture a NeRF model. There is actually no motivation/benefits to use rolling shutter camera to capture a NeRF model.\n\nConsidering the case that using rolling shutter camera is necessary, the proposed solution is just a simple two-step approach with first rolling shutter correction followed by NeRF reconstruction. I do not see any connection between rolling shutter correction and NeRF reconstruction in the proposed method. Since there is no connection, the rolling shutter correction method is just a standard method which estimate motion trajectory followed by rectification. From the formulation, I do not see any technical novelty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EPid1WIPpU", "reviewer": "Reviewer_UbiP", "summary": "This paper proposes a method called Unrolling Shutter Bundle Adjusted Neural Radiance Fields (USB-NeRF) that corrects rolling shutter distortions and improves the accuracy of camera pose estimation. USB-NeRF models the physical image formation process of a rolling shutter camera and uses a bundle adjustment framework to optimize the camera poses and scene geometry. The technique unrolls the rolling shutter effect by modeling the exposure time of each pixel and correcting the time-varying motion of the camera. USB-NeRF also uses a neural radiance field to model the scene geometry and appearance, which allows for high-quality novel view synthesis. The paper includes tables and figures that show the quantitative and qualitative comparisons of USB-NeRF with other methods on synthetic and real-world datasets. The experimental results demonstrate that USB-NeRF achieves better performance compared to prior works in terms of RS effect removal, novel view image synthesis, and camera motion estimation.", "questions": "- From Fig. 6, BARF looks much worse than NeRF; and from other tables and figures, BARF performs almost the same as NeRF. Are there any explanations for this?\n\n- Follow the question above. It is expected that BARF can fail under the rolling-shutter setting since each row of the image is recorded at different timestamps. My question is can we build a stronger baseline method that associates each row with a camera pose, then we can use BARF to optimize these camera poses and obtain better results? It can be time-consuming since an image can often have >400 rows, and then we have to optimize too many parameters. A simplified way is to split the image into R row blocks, where R <<< the width/height of that image. Then we have a simplified stronger baseline than BARF.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper is very well written. I can easily understand the paper even though I'm not very familiar with the rolling-shutter camera.\n\n2. The proposed method is simple yet effective, which uses the cubic B-Spline to interpolate between camera poses instead of linear interpolation.\n\n3. The paper did exhaustive experiments to evaluate the effectiveness of their method on both the synthetic and real-world datasets. Though there is a lack of baseline methods for rolling-shutter NeRF, they compared with various methods that bundle-adjust rolling-shutter cameras.", "weaknesses": "1. I think the ATE (absolute trajectory error) in Table 3 is the same as the absolute translation error (I'm used to the term `translation` instead of `trajectory`). Therefore, only the translation errors are given and no rotation errors are provided. Moreover, the unit of the ATE is unclear (I think it is in meters).\n2. The cubic B-Splines interpolation is suitable for complex camera trajectories, however, it can be worse than the linear interpolation method when the camera moves at a constant velocity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "acKnbHP8dY", "reviewer": "Reviewer_Xdqv", "summary": "# Summary\nThis paper integrates the rolling shutter model into NeRF training. The proposed method assumes the given images are rolling shutter images. Given the specific camera motion trajectory model, the rendered rolling shutter images can be computed from the global shutter images from the NeRF model. The camera trajectory and the NeRF model can be optimized by minimizing the loss between rendered rolling shutter images and the input images.", "questions": "A Possible Direction for improvement is to regard the camera motion model as the parameters in the Nerf model and solve it during optimization. To this end, the proposed method can be more general to handle global and rolling shutter datasets. It will be a benefit for the community further.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "# Strength\n- good performance. Since the experiments are done on the rolling shutter dataset, it is unsurprising that the proposed method outperforms previous methods like barf.\n- The idea is straightforward, like some previous NeRF-based methods integrating another luminance/render/texture/deblur model, like BRDF.", "weaknesses": "# Weakness\n- The contribution and the novelty are limited. The effect of the rolling shutter is well-known in 3D vision community, and many works have been trying to solve it in the past years. The proposed methods only contain the basic concept of modeling a rolling shutter.\n- It seems like the proposed method uses the specific motion model of the camera. It might prevent the proposed method from working on the global shutter dataset(no experiment to prove it) and another dataset that cannot be modeled in bicubic motion(no experiment to prove it).\n- The proposed method requires COLMAP to provide an initial camera pose, which is only mentioned in the footnote.\n- To sum up, due to the abovementioned concerns, I cannot give a positive rating to the proposed method in the current version.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)