IMU-Assisted Learning of Single-View Rolling Shutter Correction
Authors: Jiawei Mo
Md Jahidul Islam
Junaed Sattar
Venue: CoRL2021 Poster (2021)
Keywords: Rolling Shutter Correction
IMU
Learning

Abstract: Rolling shutter distortion is highly undesirable for photography and computer vision algorithms (e.g., visual SLAM) because pixels can be potentially captured at different times and poses. In this paper, we propose a deep neural network to predict depth and row-wise pose from a single image for rolling shutter correction. Our contribution in this work is to incorporate inertial measurement unit (IMU) data into the pose refinement process, which, compared to the state-of-the-art, greatly enhances the pose prediction. The improved accuracy and robustness make it possible for numerous vision algorithms to use imagery captured by rolling shutter cameras and produce highly accurate results. We also extend a dataset to have real rolling shutter images, IMU data, depth maps, camera poses, and corresponding global shutter images for rolling shutter correction training. We demonstrate the efficacy of the proposed method by evaluating the performance of Direct Sparse Odometry (DSO) algorithm on rolling shutter imagery corrected using the proposed approach. Results show marked improvements of the DSO algorithm over using uncorrected imagery, validating the proposed approach.

IMU-Assisted Learning of Single-View Rolling Shutter Correction
Jiawei Mo, Md Jahidul Islam, Junaed Sattar Department of Computer Science and Engineering University of Minnesota, Twin Cities, United States of America [moxxx066, islam034, junaed] @umn edu
Rolling shutter distortion is highly undesirable for photography and computer vision algorithms (e.g. visual SLAM) because pixels can be potentially captured at different times and poses. In this paper, we propose a deep neural net- work to predict depth and row-wise pose from single image for rolling shutter correction. Our contribution
Abstract:
this work is to incorporate inertial measurement unit (IMU) data into the pose refinement process, which, compared the state- of-the-art, greatly enhances the pose prediction. The improved accuracy and ro- bustness make it possible for numerous vision algorithms use imagery captured by rolling shutter cameras and produce highly accurate results. We also extend a dataset to have real rolling shutter images, IMU data, depth maps, camera poses, and corresponding global shutter images for rolling shutter correction training. We demonstrate the efficacy of the proposed method by evaluating the performance of Direct Sparse Odometry (DSO) algorithm on rolling shutter imagery corrected using the proposed approach. Results show marked improvements of the DSO algorithm over using uncorrected imagery, validating the proposed approach.
Keywords: Rolling Shutter Correction, IMU, Learning
Input RS image
IMU
Corrected GS image
RsPoseNet
Poses
Projection
RsDepthNet
Depth
Figure 1: An overview of the proposed system. Given an RS image, the pixel-wiie mixel-wise depth is generated by RsDepthNet: the row-wise poses are predicted by RsPoseNet using the RS image and IMU data. The pose estimates and depth maps are subsequently used for geometric projection to recover the corresponding GS image.
1 Introduction
Computer vision is one of the most commonly-used sensing modalities in robotics, empowered by the low cost, low power consumption cameras, and the rich information they provide, For the un- derlying image readout mechanism, cameras either use global shutter (GS) or rolling shutter (RS) system. The GS cameras capture the entire image at once, while RS cameras capture the image row by row. As the camera move arbitrarily when capturing the image for RS cameras, the pixels in different rows can be recorded at different camera poses, which leads to the RS distortions. Most vision algorithms are incapable of accounting for RS distortions and consequently perform poorly in their presence. However, most consumer-gren -rrade cameras (e.g., smartphone cameras) are RS sys- tems, which creates challenge in making vision-based computing more ubiquitous. Even though
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

some vision algorithms can be tuned for RS cameras, the adaptation process is non- von-trivval. For vi- sual SLAM systems (e.g.. [I]), intermediate poses are usually interpolated for features in different image rows. RS distortions can thus be significantly detrimental to visual SLAM performance.
Alternatively, the RS images be rectified before routing into the vision algorithms. Such RS correction algorithms can be categorized into multi-view methods [2, 3] and single-view methods [4, 5J. Traditionally, multi-view methods explore the relative geometry between images for RS correction; while the single-view methods rely on special features (e.g.. straight lines in [5]) due to the lack of geometric information. With the emergence of deep learning, several research works have addressed the problem of single-view automatic RS correction and have reported inspiring results [6. 7]. However, the accuracy and robustness of the existing methods for single-view RS correction are limited by the ilo-pd nature of the problem. Besides, these approaches also adopt simplified formulations by making various assumptions on the camera motion such as restricting it in two-dimensions in [6] and assuming a constant velocity in [7]. Nevertheless, single-view RS correction more appealing in real-world applications for its input simplicity. Moreover, the RS two-view geometry degenerates when the camera motion is pure translational [7]. which is not uncommon in many robotic applications.
In this paper, we propose a novel deep neural network for correcting RS distortions; an overview of the system is illustrated in Fig. 1. In the proposed network, we estimate the depth of each pixel in an RS image and also recover its row-wise camera poses, which we subsequently use to reconstruct the corresponding GS image. This two-step process is analogous to the one by Zhuang et al. [7]. However, [7] incorporates constant velocity assumption on camera motion which is not always valid in natural RS images (see Sec. 4. 4.1); we propose to predict camera pose for each image row to eliminate such assumption. To improve the accuracy and robustness of single-view RS correction, we propose to integrate the inertial measurement unit (IMU) data for pose prediction. An IMU estimates angular velocity and linear accelerations and is widely used in SLAM domain (e.g.. [8, IJ) as a complementary sensor to cameras. IMUs provide high-freyuency motion cues that can help RS correction, and consequently, offers significantly more learning capacity for deep RS correction in more realistic scenarios. To train and test the proposed neural network, we generate a dataset from the TUM rolling shutter dataset [8], including RS images, IMU data, depth maps, row-wise camera poses, and corresponding GS images. Training the proposed neural network using this dataset, show that RS distortions are correctly removed and the vision algorithm (i.e., DSO (9]) works accurately on the resulting images.
In summary, the contributions of this work are the following: (i) a deep neural network that learns from single-view images and IMU data for accurate and robust RS correction; (ii) a novel dataset with real images and data for RS correction training; and (mii extensive experimental validations of the proposed dataset and deep neural network for RS correction. Our implementation and dataset are available at https: //g::://ttt //github..tt//t//// com htt/II/Laabunroollnnn
2 Related Work
Removing RS distortions from images has several benefits as discussed in the previous section, and it is useful in computer vision, robotics, consumer mobile photography, etc.
Several neural networks have been proposed recently addressing the problem of single view RS correction. Rengarajan el al. [6] proposed convolutional neural network architecture to predict the camera motion for RS correction. The predicted motions are up-sampled using a polynomial function to get row-wise components of camera motion, which is subsequently used to correct the RS distortion. This approach is feasible because the camera motion is limited to two dimensions (i.e., -r-xxs translation and z-axis rotation). The mapping function relating the coordinates of the RS pixel to its undistorted correspondence is fully determined by this 2D motion. The work by Zhuang et al. [7] eliminates the 2D motion assumption above and enables full (6D) camera motion. They proposed the VelocityNet to predict the camera velocity, which is used to calculate the camera pose for each image row with the constant velocity assumption. Meanwhile, they adapt DispNet [10] (referred to as DepthNet) to predict pixel-wise depth in the RS image. With the camera pose and the depth, the undistorted GS image is generated by projection.
Other than single-view approaches, Liu et al. [3] proposed a network to remove RS distortion using two consecutive frames. They propose a motion estimation network to compute the cost volumes

---

between two consecutive frames, which are used to predict the pixel displacement between RS image and GS image. Karpenko al. [11] proposed use gyroscopes for video stabilization and RS correction based on classic geometry -hatty computer vision.
It challenging to get large set of GS/RS image pairs for RS correction training. Not only the GS/RS cameras need to have identical configures (e.g., intrinsic parameters), but also they have to be co-located, which is mostly impossible real world. Existing works use synthesized images instead. Rengarajan et al. [6] generate RS images from GS image datasets. For each GS image, they generate a random 2D camera motion and construct a mapping function to get the corresponding RS image. Subsequently, they train their neural network to perform the inverse action. Zhuang et al. [7], in the same vein, use a stereo GS image dataset generate RS images. For each GS image pair, they run stereo matching to get depth map; with randomly mandomly-genertted 6D camera velocity, they synthesize an RS image under the constant camera velocity assumption. Liu al. [3] proposed two datasets for RS correction. One is from simulation; in the other dataset, the RS image is synthesized by sequentially copying row of pixels from consecutive GS images captured by a high-speed (2400 FPS) camera, which is super expensive and its recording time is short.
The TUM rolling shutter dataset [8] has been proposed benchmark visual-inertial SLAM with RS cameras, captured by the device shown in Fig. 2. The device is equipped with a pair of synchronized RS/GS cameras, an IMU, and motion caption system recording ground-truth poses. However, the constant displacement between the RS camera and the GS camera prevents us from directly using the dataset for RS correction training, which we are going to solve in the following section.
RS Images (cam1)
Pixel Corre- spondences
PWC-Net
GS Images (camo)
Triangulation
Depth
Sparse Poses
Cubic Spline Interpolation
Row-wise Poses
Projection
GS
GS Images (cam1)
Data
Figure 2:
The data capture device used in [8] and an overview of the image processing pipeline. [Red: inputs for the neural network; Blue: ground truth for the neural network.]
Methodology
3.1 Dataset Generation
Currently, the TUM rolling shutter dataset [8] is the only publicly available dataset with synchro- nized RS/GS image pairs, camera poses, and IMU data. The RS/GS images are captured at 20Hz with 1280 x 1024 resolution; as shown in Fig. 2, we mark the GS camera as camo and the RS camera cam, The time difference between two consecutive 29.4737μu. The ground truth poses are recorded by a motion capture system running at 120Hz. The IMU runs at 200Hz. Please refer [8] for more details about this dataset.
in RS image is reported as
The RS images captured by cam, and IMU data are used as inputs to our neural network. How- cannot directly use the GS images from camo ground truth because there is a constant displacement between the cameras. Hence, we recover the GS images at the location of cam, For clarity, we mark the GS images captured from camg as GS, images and the recovered GS images as GS,- As seen in Fig. 2, the idea is to recover a depth map from the stereo configuration and project points to the pose of the first image row of cam, to get the GS; image.
Recovering depth from GS, image and corresponding RS image more challenging than from a pair of GS images. The rolling shutter distortion breaks the stereo epipolar constraint; even after stereo rectification, the pixel correspondences are not necessarily located in the same row, which makes conventional stereo matching algorithms (e.g., [12]) inapplicable. Hence, abandon the

---

stereo constraint and adopt PWC-Net [13], the state-of-the-at optical flow algorithm, to find the optical flow between the GS,/RS image pairs. For robustness, we filter the correspondences by standard bi-directional matching.
In addition to the pixel correspondence, the relative camera poses also required for depth covery. This is because the rolling shutter mechanism makes the underlying camera poses no longer consistent for all pixels in the RS image. Unfortunately, the motion capture system is not efficient enough to capture a pose for every image row, even after we downsize the image reso- lution 320 x 256 in this work. Since the motion capture system runs 120Hz and it takes 1024 x 29.4737 las 30181.1 pas to read out the entire RS image, we have about 3.6 poses for all 256 rows in the RS image, which is too sparse. To solve this problem, we adopt cubic spline interpolation [14] on the sparse poses to get a smooth pose for each row in the RS image. Since lens distortion corrected during image preprocessing, we maintain a distortion lookup table to find the original scan-line when querying the pose for each pixel; that is, each transformation matrix (T) in the following equations already considers the lens distortion lookup table implicitly.
With the pixel correspondence (uas. URS) from PWC-Net and the row-wise camera pose TGSp uas from cubic spline interpolation, we get the depth dR and corresponding 3D point X by solving the following triangulation problem:
dns
uRs 1
K Rs Nns:
das
UGS
G
TGS ugs
X
1
Here, KAS and KGs, are known camera intrinsic parameters. We also considered the depth fil- ter [15] for further refinements; however, the improvement margins were insignificant, and hence
is omitted in our final implementation. Finally, we recover the GS| image (uGS, by projecting the 3D points Rs to the pose of the first row in RS frame (i.e., TGS;
T TitSsow0) as
uas
uRs
des,
UGS1
KRS
TGI
X Rs 1
(1)
uas
3.2 Network Architecture
(10x8x512)
RexNec34
(5x16x512)
PoseCanv
(3x32x256)
Poxe Conv
PoseCorv (2x64xx128 22)
(1x122xx64)
PoseConv
PoseConv (1x256xx32)
(Ix256xx6)
Convitanh Ixl
(image-bu baased-bbeee) (256x6)
Row-wwie Pose
(256x122
Concatenate
LSTMitah layer=2 (256xx))
Row- wiise Pose (refined) (256x6)
PoseConv
Conv/ReiLU 3x3 stride
stiide
Cony ConyTrans Trans 3x3
BatchNorm
ReL.U
IMU (256x6)
Figure 3: The architecture of RsPoseNet: a feature extraction network (ResNet-34) is followed by PoseConv blocks that learn row-wise poses, which is refined by IMU data using LSTM network.
As mentioned earlier, our supervised training pipeline for learning rolling shutter correction is in- spired by Zhuang et al. [7]. We outline the end-to-end pipeline in Fig. 1. Here, we adapt the DispNet [10] as our RsDepthNet (i.e., DepthNet [7]) for depth estimation. To predict row-wise poses (i.e., the TGS; in Eq. 1), we propose a novel deep visual model named RePoseNet; its de- tailed architecture illustrated in Fig. 3. First, use ResNet-34 [16] to extract high-level semantic features from a given input image. Then, we use series of PoseConv layers to exploit those high-level features and learn wwww poses. PoseConv consists of a convolution layer, a deconvo- lution (ConvTrans) layer, batch normalization [17] layer, and a ReLU [18] activation layer. Using stride = (1, 2) for the convolution layer, we combine information along each row; whereas using stride (2, 1) for the deconvolution layer, we expand the poses to infer prediction for each row. Following that, we use a convolution layer and a tanh activation layer to finalize the poses. We use tanh the final activation function as we normalize the poses to the range of [-1, 1].

---

We will show empirically in Sec. 4.3 that predicting row-wise poses purely from images is not very accurate; hence, we propose to refine the poses by using IMU data. For unified imput, we rotate the IMU data to the RS camera frame; we also interpolate the IMU data for each image row since IMU data is sparse running at a different frequency. The processed IMU data then concatenated with the predicted poses from the image and fed into the 2-layer LSTM [19] for the final row-wise poses.
3.3 Training Details
Since our goal is to enable vision algorithms (e.g.. visual SLAM) on RS cameras, we adopt the end- point error (EPE) in [7], standard geometric loss, for training and validation. The EPE measures the average Euclidean distance between the distorted and undistorted pixels. It is defined as
EPE(1)
1 7
Σ
|| d, T²) - ugs, (u)ll2- uEl
(2)
dK
u 1
M( = RRsT
(3)
Here, for each pixel in the input RS image I, we project it the corresponding GS frame in Eq. 3, which is analogous to Eq. 1 but using predicted pose T'. We compare the projection with ground truth ugs, (u) get the EPE. Note that the here is the ground-truth depth, which decouples the training of RsDepthNet and RsPoseNet.
We initialize the ResNet-34 in RsPoseNet with the pre-trained weights from ImageNet classifica- tion [20]. We empirically find that training directly using EPE does not yield optimal results, hence initialize RsPoseNet using pose loss (mean squared error of the row-wise poses) for 50 epochs with a learning rate of le - 3; then we refine RsPoseNet using the EPE for another 50 epochs with 1e learning rate. Pose loss is straightforward for training, but minimized pose loss does not guarantee minimized EPE because rotation and translation play different roles in projection. Re- fining RsPoseNet with EPE optimizes the weights in each dimension of poses minimize the EPE, which is meaningful and necessary.
4 Experimental Evaluation
By applying the data generation process described in Sec. 3.1 on the TUM rolling shutter dataset [8], we get 10 sequences of RS images with corresponding IMU data, depth maps, row-wise camera poses, and recovered GS, images. There are 9495 frames in total, which is slightly less than the total number of frames (9523) in the TUM rolling shutter dataset, because some frames at the beginning or end of a few sequences were not recorded by the motion capture system, hence they were discarded.
4.1 Data Verification
In the evaluation, quantify RS distortion by the EPE formulations defined in Eq. 2 and Eq. 3. Since EPE is evaluated with respect to the GS₁ images generated by the proposed method, we need verify the validity of GS, images. To do so, we run DSO [9] on a continuous sequence of images; being direct method for visual odometry, DSO is very sensitive to RS distortion. Fig. shows the qualitative results for a particular sequence (#02) in the TUM dataset; as the top-left block (RS) illustrates, DSO completely fails on the RS images with an EPE of 5. 5.738 pixels. The scene in the bottom-left block (GS, gt) reconstructed by DSO on our recovered GS₁ images, which clearly shows the accurate room layout; EPE for this case is zero by definition. Moreover, we run DSO 10 times and calculate the average absolute trajectory error [21] (ATE) with Sim(3) alignment. The ATE on the RS images is O. 269 meters, while it reduces to O. 0.047 meters on the GS, images. These qualitative and quantitative results validate our data generation process and justify the use of EPE as an evaluation metric. We also generate data with the constant velocity assumption as in [7]: here we verify that its EPE 0.228 pixels. The reconstructed scene the top-right (GS) cv) is not as good the one without such assumption, which is further validated by the fact that its ATE=0. 157 meters. This shows that a constant velocity assumption is sub-optimal in real-world settings. Table 1 quantitatively summarizes these experimental results.
Fig.
5

---

Table 1: EPEs and ATES of sequence 02 on RS images, GS₁ images, GS₁ images with constant velocity assumption, and GS₁ images estimated by our network.
Images
RS
GS1
GS₁ gt cv
GS1 pred
EPE (pixels)
5.738
0.228
0.937
ATE (meters)
0.269
0.047
0.157
0.064
GS1gt.cv
GS gt
Figure 4: Top-to-bottom, left-to-right: Scenes reconstructed by DSO on RS images (RS), GS| im- ages (GS) gt), GS, images with constant velocity assumption (GS) gt.cv), and GS| images corrected by our network (GS) pred).
4.2 Network Evaluation
We split the dataset follows for training and testing: testing; the remaining 8 sequences are used for training and validation, among them, we reserve the middle 10% data for validation and the rest for training. Consequently,
sequence 02 and sequence 07 for
have 7362 frames for training, 815 frames for validation, and 534 + 784 = 1318 frames for testing. Our network Nvidia Titan V GPU with 12GB memory. We mainly focus on [7] (denoted as SMARSC) for comparison with the goal of showing the con- tribution of IMU data for RS correction. The SMARSC is also trained with the EPE. Additionally, we fine-tune the sfute-of-thh-ar two- wwo-view lcarning-base based RS correction approach [3] (denoted as TwoView) on our dataset for comparison. However, this approach outputs the corrected GS image directly so that we cannot accurately calculate EPE; thus we include it for qualitative comparison.
implementation based on Keras [22] and trained
Table 2: The ratio of images whose RS distortion reduced and EPE (in pixels) on the test data.
(a) SMARSC Ours
(b) Tests on partial IMU data.
Seq.
Input
SMARSC [7]
Ours
None
Acc.
Gyr.
Full
EPE Ratio
5.738
2.599
0.937
2.327
2.939
0.977
0.937
02
90.4%
99.3%
93.1%
90.4%
100%
99.3%
EPE
6.593
2.267
0.746
1.802
1.958
0.896
0.746
07
Ratio
86.5%
95.9%
85.6%
86.1%
96.0%
95.9%
Table 2a reports the performance comparisons on our test data. The term Ratio represents the per- centage testing images whose RS distortion (measured by EPE) gets reduced. For SMARSC [7], the EPE of sequence 02 is reduced by half and it predicts RS distortion in the correct direction for more than 85% of the tests. It shows that learning RS correction from single-view images is feasible. As for the proposed method (i.e., 'Ours') ), it predicts RS distortion in the correct direction for more

---

than 95% of the tests with sub sub-pixel EPE, which is a significant improvement over SMARSC [7]. As we will discuss in Sec. 4.3, the performance improvement mainly comes from the IMU data. We present some qualitative results of RS correction by the proposed system in Fig. 5a. The predicted GS images look very similar to the ground truth images, as most of the visible RS distortions are removed. For TwoView [3], even though it predicts the correct direction of RS distortions, the cor- rected results introduce new undesired image distortions. Fig. 5b confirms that the new distortion comes from the Two View [3] itself instead of the process of fine-tuning on our dataset.
Input PS
65 o5.0g
G5 ours
GS Taloiiew
(b)
(c)
Figure 5: Qualitative Results. (a): A few samples of predicted GS images. From top bottom are: input RS images, the ground truth GS, images, the GS images corrected by our network, and the GS images corrected by Twewiew [3]. The last row contains new undesired distortion. (b): The undesired distortion also appears in off-the-shelf results [3]. (c): Rectified GS image by projection without proc-pgoc..,,ng. The projected image incomplete with many missing pixels (black dots).
Application Study We evaluate the performance of DSO on GS images recovered by our method. The reconstructed scene is shown in the bottom-right of Fig. 4; is as good as the one on ground- truth GS, images and significantly better than the one on the original RS images. We get the average ATE of 0.064 meters, which is much smaller than the ATE RS images (0. (0.269 meters) and very close to the ATE on ground-truth GS, images (0.0e (0.047 meters). From the quantitative and qualitative results, we show that the proposed RS correction network enables downstream vision algorithms (DSO in this case) to run on RS cameras without any modification. We also run DSO on the results by SMARSC [7] and by Two View [3] the network inference is able to run in real-time (~ 60 FPS our Titan V GPU), including the projection of pixel from the input RS image to the rectified GS image (i.e., Eq. 3). However, since the projection is not surjective [23] (not all GS pixel is guaranteed to be filled), the output GS image
DSO fails consistently. For computational efficiency,
7

---

is incomplete as shown in Fig. Sc. We apply interpolation to complete the output image, with Fig. 5a showing the results after interpolation. As interpolation computationally expensive (~ 20 FPS on CPU), we will explore alternatives (e.g, median filter) for higher computational efficiency in future work.
4.3 Ablation Study on IMU Data
To analyze the contribution of IMU data for RS conrection, we disable the gyroscope, or the accel- erator, or both when feeding the IMU data into the network: we re-train the network and report the results in Table 2b. Using gyroscope alone (i.e., *Gyr.') already yields good results, while integrating the accelerator data (i.e. 'Full') further reduces the EPE slightly. However, the accuracy of using accelerator data alone (i.e., 'Ace." is low. Gyroscope measures the camera motion more directly than accelerator does (angular velocity versus linear acceleration). thus gyroscope being more im- portant for RS correction is expected. Nevertheless, even when the IMU data is completely disabled (i.e., 'None'), ) it still slightly outperforms SMARSC [7]. The potential reason is that the row-wise poses are highly correlated with the constant velocity assumption so that the velocity error affects every pixel, whereas row-wise poses are predicted individually in our method.
4.4 Generalization Performance
We also test our network on the WHU RS Visual-Inertin Dataset [24], which contains RS/GS image pairs, IMU data, and ground- poses. We use the training results on our proposed dataset without fine-tuning on the WHU dataset to test the generalization performance of our RS correction network. Fig. 6 gives some qualitative results. One important observation is that when the camera rotates, our network works most of the time, with the predicted GS images well-aligned to the ground-truth GS images as illustrated the left part of the figure; however, for pure translational motion, our method is more likely to fail (e.g., right part of Fig. 6). The potential reason is that pure translational motion is very rare in our training dataset from the TUM RS dataset. For future work, we plan to increase generalization capability by augmenting the training set with more diverse data.
success cases
failure cases
GS Ingut RS
Figure 6: Qualitative results of our network on WHU RS Dataset [24]. The top row shows the input RS images; the bottom row shows the GS images corrected by our network. The edges of ground-truth GS images (red lines) are overlaid on the results for RS comparison.
Conclusions
We propose novel neural network for single-view rolling shutter correction, where we use IMU data to significantly improve rolling shutter pose prediction. Rather than synthesizing rolling shut- ter images, we generate real images to train the proposed network. Our experiments validate the improved accuracy and robustness in rolling shutter correction and downstream vision algorithms by using the proposed approach. For future work, we will extend the proposed method for diverse environments by designing our own data acquisition device. We also intend to deploy the proposed approach for based robot localization and odometry purposes while using inexpensive and ubiquitous rolling shutter cameras.
8

---

Acknowledgments
This work was supported by the US National Science Foundation Award IIS #1637875, the Univer- sity of Minnesota Doctoral Dissertation Fellowship, and the MnRI Seed Grant.
We thank Igor Slinko for sharing the weights of PWC-Net on grayscale images with us. We are also thankful Nvidia TM for their donation of GPUs to support our work.
References
[1] C. Guo, D. Kottas, R. DuToit, A. Ahmed, R. Li, and Roumeliotis. Efficient Visual-Inertial Navigation using& Rolling-Shutter Shutter Camera with Inaccurate Timestamps. In Proceedings of Robotics:
Science and Systems, Berkeley, USA, July 2014. doi: 10.15607/RSS http://RRSSS.......... 2014.X.007.
[2] B. Zhuang, L.-F. Cheong, and G. H. Lee. Dolling-Shutter-Aie Differential SfM and Image Rectification. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 948- 956, 2017. doi: 109/1CC..201..10000
[3] Liu, Z. Cui, V. Larsson, and M. Pollefeys. Deep Shutter Unrolling Network. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 5940- 5948, 2020. doi: 10::10.1100 100.1100//CV/222200002222
[4] B. Heflin, W. Scheirer, and T. E. Boult. Correcting Rolling-Shutted Distortion of CMOS Sen- sors using Facial Feature Detection. 2010 Fourth IEEE International Conference on Bio- metrics: Theory, Applications and Systems (BTAS), pages 1-6, 2010. doi: 10t.1..110//BAS.20100 5634528.
[5] P. Purkait, C. Zach, and A. Leonardis. Rolling Shutter Correction in Manhattan World. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 882-890, 2017. doi: 10. 00.100//CCV...01...00.
[6] V. Rengarajan, Balaji, and A. N. Rajagopalan. Unrolling the Shutter: CNN to Correct Motion Distortions. In 2017 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 2345-2353, 2017. doi: 40::10.1100/CCVP22017..2
[7] B. Zhuang, Q-H. Tran, P. Ji, L.-F. Cheong, and M. Chandraker. Learning Structure-And- Motion-Aue Aware Rolling Shutter Correction. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages
2019, doi: 2009///V..2019.000000
[8] D. Schubert, N. Demmel, L. v. Stumberg, V. Usenko, and D. Cremers. Rolling-Shut Shutter Mod- elling for Direct VisutalInnrttil Odometry. In 2019 IEEE/RSJ International Conference In- telligent Robots and Systems (IROS), pages 2462-2469, 2019, 1oi: 1000100109//RSS54877.000999 8968539.
[9] Engel, V. Koltun, and D. Cremers. Direct Sparse Odometry. IEEE Transactions on Patlern Analysis and Machine Intelligence, 40(3):611-66) 2018. N0f.100.109/TPAM.211...666
[10] N. Mayer, E. Ilg, P. Häusser, Fischer, D. Cremers, A. Dosovitskiy, and T. Brox. A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estima- tion. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4040 4048, 2016. 100.100.109/
[11] A. Karpenko. D. Jacobs, J. Baek, and M. Levoy. Digital Video Stabilization and Rolling Shutter Correction using Gyroscopes. CSTR, 1(2):33, 2011.
[12] H. Hirschmuller. Stereo Processing by Semiglobal Matching and Mutual Information. JEEE Transactions on Pattern Analysis and Machine Intelligence, 30(2):328-44 2008. doi: 10. 1109/PPAMI.207.111
[13] D. Sun, X. Yang. M.-Y. Liu, and J. Kautz. PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume. In 2018 IEFF/CVE Conference on Computer Vision and Pattern Recognition, pages -944,,,43, 2018. 1oi:10.1109/CPPP 10199/9...200...00000

---

[14] I. J. Schoenberg. Contributions to the Problem of Approximation of Equidistant Data by Ana- lytic Functions Part B-On the Problem of Osculatory Interpolation. A Second Class of Ana- lytic Approximation Formulae. Quarterly of Applied Mathematics, 4(2):112-144) 1946.
[15] G. Vogiatzis and C. Hernández. Video-based, Real-Time Multi View Stereo. Image and Vi- sion Computing, 29(7):434 441, 2011. ISSN 0262-8856. doiihttss/ddooorr//...00f/./.
2011.01.006.
[16] K. He, X. Zhang, S. Ren, and J. Sun. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016. N0f:10.110//CVPP.2000
[17] S. Ioffe and C. Szegedy. Batch Normalization: Accelerating Deep Network Training by Re- ducing Internal Covariate Shift. In F. Bach and D. Blei, editors, Proceedings of the 32nd In- ternational Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 448 456, Lille, France, 07-09 Jul 2015. PMLR.
[18] V. Nair and G. E. Hinton. Rectified Linear Units Improve Restricted Boltzmann Machines. Proceedings of the 27th International Conference on International Conference on Ma- chine Learning. ICML'10, page 807-814, Madison, WI, USA, 2010. Omnipress. ISBN 97816055 188655599777.
[19] S. Hochreiter and J. Schmidhuber. Long Short-Term Memory. Neural Comput., 9(8): 1735-1780) Nov. 1997. ISSN 0899-7667.
[20] A. Krizhevsky, I. Sutskever, and G. E. Hinton. ImageNet Classification with Deep Convo- lutional Neural Networks. Commun. ACM, 60(6):84-90, May 2017. ISSN 0001-0782. doi: 10.1145/306536.
[21] 7. Zhang and Scaramuzza. Tutorial on Quantitative Trajectory Evaluation for Visual(- Inertial) Odometry. In 2018 IEEF/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7244-7251, 2018.
85939411.
[22] A. Gulli and Pal. Deep Learning with Keras. Packt Publishing Ltd, 2017.
[23] M. Fenoy-Muñoz, J. L. Gámez-Merino, G. A. Munnzz-Frrández, and E. Sáez-Maestro. A Hierarchy in the Family of Real Surjective Functions. Open Mathematics, 15()):88-5501 2017.
[24] L. Cao, J. Ling, and X. Xiao. The WHU Rolling Shutter Visual-Inertial Dataset. IEEE Access, 8:50771-50779, 2020. doi: 10. A0A11/////CESS.00022.0000
10


Reviews:
[{"review_id": "l5t1STk2uEb", "reviewer": "Reviewer_kkCY", "summary": "This paper is addressing the challenges of using global shutter cameras in slam. The problem with rolling shutter(RS) cameras are that each row is taken at a different time step and there will be a distortion when the camera moves and because of that it breaks the direct sparse odometry method. The paper argues that linear interpolation of IMU poses to estimate the position of the camera for each row of the image is not enough. They predict the depth for each row of the image and then fuse it with IMU data and further process it to predict per row pose. With the corrected pose they show the quality improves quite significantly. The model is trained on TUM rolling shutter dataset where there is a global shutter and rolling shutter cameras. Correspondences between RS and GS camera is found with optical flow (using PWC-Net) and then the pose is optimized to minimize the reprojection error between RS and GS after reprojection.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper tackles a practical problem for SLAM system\n- The paper is well written and the method makes sense.\n- The method makes significant improvement quantitatively and qualitatively.\n- Comparison on different datasets and with different methods.\n- I also like the ablation studies to show the effect of using IMU vs not using IMU.\n\nWeaknesses: \nN/A\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I am not quite familiar with the literature in this area so I don’t know about potential missing comparisons. The improvement seems significant to me and the method makes sense. I'll let other expert reviewers to point out potential flaws. \n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "N/A", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {}}, {"review_id": "L1UK8ra1mb", "reviewer": "Reviewer_BmJn", "summary": "This paper performs rolling-shutter (RS) correction by estimating pixelwise depth and row-wise camera transform given a single rolling-shutter image, and this allows correcting the image to a global-shutter(GS) one. The key contributions include: a) using additional IMU information for more accurate prediction, b) leveraging a real dataset with motion information and RS-GS pairs for training. The empirical and qualitative results clearly demonstrate the benefits of the IMU sensor inputs and the system yields improvements over prior related work [7].", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The paper tackles a very relevant and basic task that can help improve any visual odometry approach. Towards this, it demonstrates clear benefits of including IMU sensor readings when inferring row-wise camera poses and this is an interesting finding which could be applicable to several methods.\n\n- The overall approach is very sensible and intuitive. In particular, the main contributions of: a) extracting pseudo-GT labels for depth training using optical flow between RS and GS image, and b) modifying the pose prediction network to integrate IMU information, both make sense are well-executed.\n\n- The ablations presented by the paper are convincing regarding the importance of the IMU information when predicting camera motion. \n\n- The qualitative results also demonstrate a clear use-case for the proposed approach in allowing better registration by visual odometry approaches.\n\n**Weaknesses**\n\n- While the empirical results and the overall system developed are encouraging, the technical contributions of the work are limited to allowing an additional input (IMU readings) in existing frameworks e.g. [7].\n\n- The presented generalization results (Figure 6) are not very impressive and I am unsure if the learned models have perhaps overfitted to the statistics of the training dataset.\n\n- While using the E2E error is an intuitive metric, training by optimizing this metric makes the comparison slightly unfair to the baselines as this method is directly trained to optimize this metric.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, I feel this paper is a well-motivated and well-executed one, and while the specific technical innovations maybe limited, the overall approach for extracting pseudo-GT for training and incorporating IMU readings for better camera prediction is a novel and well-ablated one. I think this therefore maybe a value contribution that could benefit the SLAM-based approaches.\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {"issues": "A presentation concern is that the abstract claims that the paper 'releases' a dataset, but I think this is not an accurate claim. More precisely, the paper leverages an existing dataset with off-the-shelf systems to extract additional information e.g. 'ground-truth' depth."}}, {"review_id": "JecpNZL1Mr7", "reviewer": "Reviewer_DMMe", "summary": "This paper proposes an imu-assisted, learning-based rolling shutter correction method. The key to the approach is two neural networks - one that predicts a depth map for the rolling shutter image (RsDepthNet) and one that predicts a per-row pose (RsPoseNet). The output of these networks is then used to reproject the rolling shutter pixels into an estimate of the global shutter image. The experimental results show that the method performs very well.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The paper is clear and very well written. The figures are very well made and help the reader to easily understand the proposed method. The paper contains sufficient details about the implementation and parameters used.\n- Unlike existing methods, the proposed approach makes very little assumptions on the motion of the camera. This is due to the fact that row-wise poses and pixel-wise depths are predicted with the help from neural networks. Therefore the method hypothetically will outperform most existing approaches which assume purely rotational, or other limited degrees of motion.\n- The experimental evaluation is very thorough. The approach has been compared to the state of the art (SMARSC), and a nice ablation study has been done to investigate the effect that the IMU has on the accuracy of the approach.\n- The results show that the method outperforms the state of the art (even when the IMU is not used, which is a fair comparison). The reasons for this are nicely explained and confirm the claims made in the paper.\n\n**Weaknesses**\n- The approach is very similar to [7]. The only differences are that the prediction is now conditioned on an IMU and per-row poses are predicted. This detracts (slightly) from the novelty of the paper.\n- The generalization performance seems a bit poor. This is somewhat expected as the method relies on accurately predicting the depth of the rolling shutter image. Perhaps an evaluation of the sensitivity of the method to depth / pose prediction errors will help the reader to understand this aspect better.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, this is a very useful method that seems to work well. While there is still some room for improvement in terms of generalization, this could possibly be addressed with better training data.\n\n\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "I would like the authors to comment on the sensitivity of the approach to the depth prediction as this seems to be the main weakness of the approach. ", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies an important problem since rolling-shutter correction can enhance the performance of visual-odometry and SLAM pipelines. The reviewers also appreciated the technical approach (e.g., integration of IMU information, pseudo-labels for depth), the clarity of the presentation, and the experimental results and ablation. The main weakness of the paper is that the experiments do not make a strong case for the generalization capability of the proposed approach. However, after the rebuttal, the authors added more discussion about generalization, which the reviewers and area chair found satisfactory.

Decision: Accept (Poster)