Learning Interpretable BEV Based VIO without Deep Neural Networks
Authors: Zexi Chen
Haozhe Du
Xuecheng XU
Rong Xiong
Yiyi Liao
Yue Wang
Venue: CoRL 2022 Poster (2022)
Keywords: VIO
Interpretable Learning

Abstract: Monocular visual-inertial odometry (VIO) is a critical problem in robotics and autonomous driving. Traditional methods solve this problem based on filtering or optimization. While being fully interpretable, they rely on manual interference and empirical parameter tuning. On the other hand, learning-based approaches allow for end-to-end training but require a large number of training data to learn millions of parameters. However, the non-interpretable and heavy models hinder the generalization ability. In this paper, we propose a fully differentiable, and interpretable, bird-eye-view (BEV) based VIO model for robots with local planar motion that can be trained without deep neural networks. Specifically, we first adopt Unscented Kalman Filter as a differentiable layer to predict the pitch and roll, where the covariance matrices of noise are learned to filter out the noise of the IMU raw data.  Second, the refined pitch and roll are adopted to retrieve a gravity-aligned BEV image of each frame using differentiable camera projection. Finally, a differentiable pose estimator is utilized to estimate the remaining 3 DoF poses between the BEV frames: leading to a 5 DoF pose estimation. Our method allows for learning the covariance matrices end-to-end supervised by the pose estimation loss, demonstrating superior performance to empirical baselines. Experimental results on synthetic and real-world datasets demonstrate that our simple approach is competitive with state-of-the-art methods and generalizes well on unseen scenes.

Learning Interpretable BEV Based VIO without Deep Neural Networks
Zexi Chen Haozhe Du Xuecheng Xu Rong Xiong Yiyi Liao" Yue Wang"
Zhejiang University
{chenzexi, hzdu, xuecheeegguguuuxiiiooggg yiyi 1ia0, ywang24)ezju edu. cn
Abstract: Monocular visual-inertial odometry (VIO) is critical problem in robotics and autonomous driving. Traditional methods solve this problem based filtering optimization. While being fully interpretable, they rely manual interference and empirical parameter tuning. On the other hand, learning braneing-based approaches allow for end-to-end training but require large number of training data to learn millions of parameters. However, the non-interpretable and heavy models hinder the generalization ability. In this paper, propose a fully differ- entiable, and interpretable, bird-eye-vis (BEV) based VIO model for robots with local planar motion that can be trained without deep neural networks. Specifically, we first adopt Unscented Kalman Filter as differentiable layer to predict the pitch and roll, where the covariance matrices of noise are learned to filter out the noise of the IMU raw data. Second, the refined pitch and roll are adopted to retrieve a gravity- aliine BEV image of each frame using differentiable camera projec- tion. Finally, differentiable pose estimator is utilized to estimate the remaining DoF poses between the BEV frames: leading to 5 DoF pose estimation. Our method allows for learning the covariance matrices end-to-end supervised by the pose estimation loss, demonstrating superior performance to empirical baselines. Experimental results synthetic and real-world datasets demonstrate that simple approach is competitive with state-of-th of-th--rtt methods and generalizes well unseen scenes.
Keywords: VIO, Interpretable Learning
Introduction
Visual-inertial odometry (VIO) is fundamental component of the visual-inertial simultaneous lo- calization and mapping system with many applications in robotics and autonomous driving. As a cheap and efficient solution, monocular VIO has attracted growing interest recently. However, monocular VIO is a challenging task considering the limited sensory information.
Traditional monocular VIO approaches have demonstrated promising results based on classical fea- ture detection and feature matching []][]][][[4] see Fig. (top). These methods are interpretable and generalize well, but require manual interference and empirical parameter tuning. Moreover, these methods only leverage sparse features, resulting in the degraded performance in textureless regions [5]. More recently, learning-based approaches address these limitations by learning the ego-motion in end-to-end fashion from the raw, dense monocular images, as illustrated in Fig. (middle). However, they lack interpretability and thus pose a new challenge to generalization. While many works improve the interpretabilityy by predicting map [[]]][[[1[(13)) the following pose regression network remains uninterpretable. Furthermore, existing learning-based approaches rely on a large number of training images learn millions of parameters while the ego-motion has essentially 6 DoF only. We thus ask the following question: Is
intermediate optical flow [6][]][]][9] or depth
co-corresponding author,
corresponding author.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Traditional Mathods
Non- Differentiable
Front View t
Covariance
Interpretable
Manual Tuning
Matrices
Deep Neural Network Methods
Front View tz
Differentiable
Millions
lon-lnntepreetblee
IMU
Tuning
Parameters
OURS
Interpretable
Forward Back--ropangttion
Auto Tuning
Covariance Matrices
the standard visual-inertial odometry that requires Deep learning based VIO approaches learn depth or optical flow prediction before pose regression, and are loaded with tons of heavy parameters for training, and bottom: our differential interpretable model based learning approach that solves the problem in an end to end manner, training covariance matrices without deep neural networks.
Figure 1: Comparison of VIO methods. Top:
manual tuning parameters, middle:
it possible obtain a fully interpretable model with minimal set of parameters being learned end- to-end? this paper, we focus on the BEV based VIO for DoF local planar motion with elevation fixed. This is not trivial as such motion pattern is applicable for unmanned ground vehicles and au- tonomous driving vehicles in urban, underground parking lots, tunnels and highways [14][15]. We combine the advantages of the traditional methods with bearning-basek approaches. Several former works [[]] explore this possibility using traditional filtering for interpretable fusion, but leaving the sensor model a plain regression network. Our key idea is to design fully differentiable, inter- pretable model that eliminates all black-box neural networks, yet allowing for replacing empirical parameter tuning with learning from the data, Fig. (bottom). Additionally, ploits dense appearance information instead of sparse features. As it is difficult to differentiate the DoF pose estimation that involves robust and non-linear optimization, focus the DoF image based pose estimation conditioned on the estimated pitch and roll angles.
method ex-
More formally, consider the BEV based monocular VIO as function that outputs relative pose taking as input i) two consecutive frames, and ii) IMU data. First, we utilize a Diffcrentiable Un- scented Kalman Filter (DUKF) to filter out noise from the IMU raw measurements and retrieve pitch and roll. As the covariance matrices of noise in both the motion model and the measurement model not known precisely in practice, classical UKF relies on empirical parameter tuning given spe- cific scene. In contrast, we exploit the fact that UKF differentiable and optimize the covariance matrices in end-to-end manner leveraging the training data. This requires the estimator of the remaining 3 DoF' pose to be differentiable as well. Therefore, in the second step, we estimate the re- maining DoF poses by converting monocular images to bird-eye view conditioned on the denoised pitch and roll [14]. The bird-eye view representation allows us to obtain global optimal solution using Differentiable Phase Correlation (DPC) [18] in single forward pass. Note that DPC leverages dense appearance information and does not introduce any trainable parameters. More importantly, allows for back- pack-propagating the gradient the DUKF module update the covariance matrices. The combination of DUKF and DPC provides us fully differentiable, and interpretable BEV based model for VIO, named BEVO, with the parameters of the covariance matrices as the only trainable parameters and no deep neural networks involved. We emphasize that by carefully combining the designed mechanistic model with learning approaches, the model is inherently interpretable. [19]. We summarize our contributions follows:
We present a fully differentiable and interpretable monocular VIO framework for planar moving robots, e.g., autonomous driving unmanned ground vehicles, that combines the advantages of both traditional and learning- based approaches.
We propose to use the Unscented Kalman Filter as differentiable layer and leverage the Differ- entiable Phase Correlation BEV images for pose estimation. This novel combination allows for learning the covariance matrices from data in an end-to-end manner.
Extensive experiments on KITTI, CARLA, and AeroGround demonstrate that, surprisingly, our simple approach shows competitive results compared to the state-of-the-art.

---

Front View
State
IMU
MU Gyro
Motion Modil
Motion
Messurement
Ace
Differontianle
UKF
Pitch & Roll
BEV Projection
State
IMU Ano
f1
MU t-1-b-1
Pront View
Meamummmnnt Model
Motion
Diftreentable
Extimatorr
ODOM LOSS
Aod Ii-
Cross Goy
Kalman Gath)
Messurement
State 1o-1
MU Oyrotu-1
Differnntable LKF
Diferrnniable BEV Projection
Mation
Measurement
Forward
Full Pipeline
Differentiable UKF
Bock-Propagation
Figure 2: The overall pipeline of the proposed BEVO. As demonstration, the frequency of the front image 10Hz while the frequency of IMU is 100Hz. For each frame of IMU, the pitch rate, roll rate, and 3D acceleration are fed into the differentiable UKF to learn noise filtering (elaborated in the right part of the figure). Afterward, the filtered pitch and roll aid the differentiable BEV projections which are further estimated for pose to be supervised. However, due the frequency difference, only frames with front images have back-propagated gradients (red dashed lines) for the Differentiable UKF (DUKF) training.
Fully Differentiable and Interpretable Model for VIO
We aim to estimate the odometry with the input of one monocular camera and IMU data by differentiable and interpretable model. Following the pipeline in Fig. 2, we employ a trainable UKF to integrate IMU data for pitch and roll estimation. Then the denoised pitch and roll. together with the corresponding front image are fed into the second differentiable module, for differentiable pose estimation of the remaining 3 DoF pose. To this goal, transform the image the BEV images conditioned on the denoised pitch and roll. Following the BEV projection, the (2M) pose of consecutive BEV images is estimated by the DPC. Note that the UKF is continuous forward function and is differentiable in essence. In this work, we emphasize its backward path by referring to it DUKF, distinguish it from the traditional UKF utilization which focuses only on the forward path. As a result, the only trainable parameters are the 4 numbers that make up the covariance matrices of motion and measurement in the DUKF and are trained end-to-end with the pose loss altogether.
2.1 Differentiable Unscented Kalman Filter
For the DUKF, there are two basic models in the pipeline, namely, the motion model and the mea- surement model. We follow the right-hand system through the paper.
Motion Model: Given the filtered pitch and roll in time / - 1, we have the state in time / - 1 as:
-z-1 fau-1, _,-1)²,
(1)
where a, and B are the pitch and roll angle respectively. Following the sampling theory of UKF [20]. the sampled state in t - becomes X,-1 = [1),-1), where I' is the sampling rule. Given the motion model G, we have the initial guess of pitch and roll time as:
X = G(X.-1) = [a,- Wor Br-1 + Wa., At)
(2)
P Σ WmXi o. = Σ wx - - - R1)² + +0+,
(3)
where wa, and W/ are the pitch and roll rate from the gyroscope, and The and ã; are the mean and variance the initial guess X;, respectively. At is the interval of the IMU data. Note that wi and we are the weights for each sampled state in X; based on the theory in [20], and O₂ is the covariance matrix for the noise the motion model to be trained.
Measurement Model: After updating the covariance matrix Oz which is the motion prediction, we retrieve new sampled state X, in time t:
F1 N ( 7,) A [āt, 3.]
(4)
X, = [(T.),
(5)
3

---

where 3, satisfies the normal distribution. Given the measurement model H, the initial guess of measurement
74 = -(xx) = Jāi, [e]
(6)
M. = Σ Wm Zr
(7)
Σ Σ we - M1) - M,) + Qr:
(8)
where M, and E, are the mean and variance of X, respectively, and Qe is the covariance matrix for the noise of the measurement model to be trained.
Now, considering real-world measurement is obtained:
di = aarclan(ace] /V acee?² + aec;?);
i. arclan(aec) /uce)),
Z, z, = [dr. ".]"
(9)
where di, and 3r are the measurement of pitch and roll in time with respect the raw acceleration data in each axes acci accy, and acci. Altogether, the real-world measurement state formed as Z,
By now, we come the last stage of the DUKF, update the final state 2, as follows:
$ (X;
[z.) M.)
(10)
1=0
K₁ En
Ex.,75-1
(11)
Pr = D. + (2, M,),
ox = o + K,E,K,"
(12)
zt N (pr, σ4) - 1at, 3.1"
(13)
where E/ SX.2 K₁ are the cross-covariance and Kalman Gain respectively, μ4 and og are the mean and variance of 24 which will also guide the sampling in the following time + 1.
2.2 Differentiable Bird-eye View Projection
To achieve the projection from the front view the birds-eye view with DBEV, we first define the transformation matrix from the IMU to the ground beneath the vehicle considering its pitch, roll, and elevation:
cos0:
O
sinat sin
R MU
sin,, sina,cosB: sinB:
cos
T.MU = [0 const
(14)
the rotation matrix R.MU of the IMU at time changes constantly with a and 3 which are es- timated by the DUKF. The translation vector TMM of the IMU is known and fixed since is the urban road and highway that we are considering in which the height of the IMU to the ground const, stays unchanged through the journey [14]. Transformation matrices from camera to the world [RC TCCMM are obtained with the calibration from IMU to camera and [R] [R MU)
Besides the unchanged height, one more assumption can be made for safe trip: there should be sufficient distance ahead which should be about 10m at 34km/h according to the mounting height and angle (similar [14] [15]), forming that the center bottom part of the front image should represent a plane ground, shown as the yellow dashed box in Fig. 2.
With the assumption above, consider pixels within the yellow dashed box as set of points in the 3D space, and map them from the camera plane to the ground plane.
To begin with, we generate set of even points Py on the ground plane that stands for the location that the points {u, 8) in the BEV should be projected to:
Pg P [X Y 3T
(15)
where X", Y',,, and Z' are the coordinates of points. Pa is then transformed to the camera coordinate with:
pº = Ry CAM (Po - -.CAM & 1x", Y.", 1T
(16)

---

and P. is projected to the camera plane with the focal lengths fa, fy and principal point (Co, Cy):
fx
0
X
Z
1/
fu
Cy
Y
(17)
0
Z
where (ui, v/) are the indexes on the front camera image of each corresponding point in Pg time t. Therefore, by remapping the original pixels (u, u) to (ut. v)), we retrieve the BEV projection L out of the yellow dashed box I in Fig. 2:
New (Lus vr)) I2 (F(fue, v.))),
(18)
where {u4, w/} F(fu, vz3) the remapping from BEV to front. Since DBEV is parameterized by the pitch and roll, which is related to the variance of IMU data, with the help of DUKF, the gradient can hack-propagated the variance.
2.3 Differentiable Phase Correlation
Given two images Hey and Feev with pose transfor- mations, a variation of the differentiable phase cor- relation [18] is utilized to estimate the the overall relative pose & between / and Thev. The rotation and scale are calculated by:
Figure The visual demonstration DPC.
p(E!"t) = C(((((1) ((3(1)=1))
(19)
(0t, E(p(s?")))
(20)
where 5/ is the rotation and scale part of & between π and Te 3 is the discrete Fourier Transform, is the log-polar transform Log-Polar Transform), and e the phase correlation solver. Fourier Transformation š transforms images into the Fourier frequency domain of which the magni- tude has the property of translational insensitivity, therefore the rotation and scale are decoupled with displacements and are represented in the magnitude (Fig. 3 Magnitude of FFT). Log-polar transfor- mation & transforms Cartesian coordinates into log-polar coordinates so that such rotation and scale in the magnitude of Fourier domain are remapped into displacement in the new coordinates, making solvable with the afterward phase correlation solver (Fig. 3. Phase correlation solver C outputs a heatmap indicating the displacements of the two log-polar images, which eventually stands for the rotation O, and scale 8 of the two input images 1 and hew (Fig. 3 Phase Correlation Rotation). Since assume that the height of the robot does not change drastically along the trajectory, the scale 8 is fixed as 1. To make the solver differentiable, we use expectation E(.) as the estimation of
Then hew rotated and scaled referring to <i.> with the result of Thee In the same manner, trans- lations & between Teev and 1hey calculated with p(tt) C(pper 1h2y) and t, Ex((())). With the relative rotation 8,, scale St, and translation t, between Ibey and Ihew and by consulting with the fixed Distance per Pixel (DPP), we finally arrive at the SIIM(2) pose between frame -- and I of vehicle:
S 0r, St. DPP .p].
(21)
Note that the gradient in the part can be back prok-prop all the way back to the variance of the IMU data since the phase correlation is differentiable.
2.4 Loss Design, Back-Propagation, and Training
With DUKF, DBEV and DPC, define one-step VIO function:
{0,, St, DPP 4,, az 3.} 1IO0.qQ4:-.. 4e,,,,,,,,
(22)
where VIOo.q is fully differentiable and interpretable with 4 trainable parameters in Oz and Q T, w, and a are the front images, gyroscope data, and accelerometer data. Since the BEVO fully differentiable, it is trained end to end and supervised only with the ground truth of the odometry
5

---

between frames. Such supervision is applied to optimize the covariance matrices of noise in both the motion model and the measurement model of DUKF.
Elaborately, we wish to supervise the distribution of the poses E/" and E and therefore, we calculate the Bealback-Liibberr Divergence loss KLD between (p(E) p({()) and the ground truth:
L KLD(p(E) 1g.) KLD(p(()), 1r),
(23)
where 10- and 1,- are the Gaussian Blurred one-peak distributions centered at the ground truth #" and t* respectively.
Training Details: all trainings are conducted on a GPU server with NVIDIA RTX 3090. For the initialization, we randomize all initial values of O and Q within the range of [0, 1] uniformly. Since O and Q are 2 2 diagonal matrices, we consider the learning process of the spherical covariances of them following [1][2]] I] so that the number of trainable parameters will be reduced to 4. We supervise the model only with DoF ground truth (GT) data of the same frequency as the images, i.c. the high frequency GT of the IMU is not needed. We set the BEV size to 100 x 100 as a trade-off between speed and accuracy. The inference time is 21ms and takes up about 500M B of the resources. The pseudocode for training BEVO will be given in Appendix I.
Plug into Map-Based Localization System: addition to the odometry, we also explore the pos- sibility of integrating BEVO with fully differentiable localization to achieve drift-free system. We show results demonstration of the application along with the pseudocode shown in Appendix II.
3 Experiment
We conduct two main experiments: i) odometry estimation on the KITTI [22] odometry dataset on which the experiments on ablation study, testing, generalization are conducted, and ii) heterogencous localization on CARLA [23] and the AeroGround Dataset [24].
Dataset and Corresponding Experiments:
KITTI & CUD: We evaluate BEVO on the KITTI odometry dataset using sequences 00-08 for train- ing and 09-10 for the test following the same tain//vaiidatetest split as [25][26]. In addition to the widely adopted train/validate split where 20% samples are randomly sampled from the training set for validation, we also train model in the first three quarters of each sequence of 00-08, and evaluate the performance in the last quarter. This train/validate split is more realistic even though it might present worse result. Moreover, ing BEVO with 20% data of cach sequence. Note that sequence 03 cannot be used for training or evaluation in this study since the raw data where we grab the IMU data is missing.
further investigate the generalization ability by train-
CARLA & AG: We further verify the plausibility of applying the BEVO as front-end plugin for end- to-end heterogeneous localization following the routine in [27]. Such experiments are conducted in CARLA and on the real-world AeroGround Dataset. The setups are elaborated in Appendix III.
Evaluation Matrices: We evaluate our trajectories using beat and Frel from the standard KITTI toolkit as in [26]. our case, we do not estimate the elevation due to the assumption that the vehicle will not leave the ground. However, we still record the initial elevation and keep it constant through the journey for comparison, which might degrade our performance in fret to some extent.
Baselines: For the odometry experiment, we compare to both traditional method VINS [1] and SOTA depth dependent learning-based methods DEEPVIO [5], SELEVIO [28], LI ET AL. [26], GEONET [29], and depth independent learning based method TARTANVO [30]. For the localization experiments, we compare the performance with
[1] and ORB-M [21].
3.1 Odometry Estimation
We now compare the odometry estimation performance of the proposed BEVO to the state-of-the- art methods in two training settings as introduced in Dataset: i) following the train/validate splits in [25][26] (Ours'), ) and ii) split each sequence by quarters, take the first three quarters of each sequence train and the last quarter validate (Ours2). In addition these two settings, we further train BEVO using the first 20% of the training data of each sequence in Ours to evaluate its performance on insufficient data (Ours*).

---

TABLE 1: Quantitative results of odometry estimation KITTI dataset from sequence 00 to 10. Red indicates the best performance and Blue indicates the second best.
Doeph11
TamanV
Li al.
GeoNNe
SeHNIO
VINS
ORIE-M
Ourr!
Oure
Ourr?
Sey
franses
Teal
Frad
Teal
frel
Fryl
Frat
Fert
Text
feat
Feod
Eeel
Teal
Frul
00
x541
11.62
2.45
14.21
5.93
44.58
14.89
1.24
0.45
18.83
2.49
25.20
7.73
3.26
1.44
1.35
0.53
2.01
1.02
012
4661
4.52
1.40
16.21
2,50
73.59
12.53
0.50
0.25
21.03
2.61
20.30
3.10
4.56
1.32
1122
0.88
1.79
1.55
9.08
17.91
3.70
24.52
32.47
13:12
21.90
26.01
10.62
06
1101
4.72
2.03
9.77
3.58
40.28
16.68
3.86
2.08
1.07
1.36
2.77
1.99
1101
2.71
1.66
4.50
X.41
12.85
3.50
57.05
17:30
0.91
0.49
15 99
2.42
34
51
10.85
3.58
0.58
1.35
1.15
1.79
27.10
11.45
32.65
32.
12:13
09
1591
1.28
1.12
6.0
3.11
15:21
5.24
13.02
67.06
1.95
1.15
41.47
2.41
45.52
3.10
1.22
1.05
1.29
1.12
1.32
1.31
10
1201
0.85
1,03
6.89
2.73
25.63
7,69
58.52
23.02
1.31
1.30
20.35
2.73
6.39
3.20
1.01
1.01
1.14
1.08
1.17
1.30
00:-10
5.40
16.00
4.60
33.78
22.90
21
26
05
09-10
1.12
1.88
2.90
20.42
55.37
45.04
1.88
10.91
15
Fig.4 and Table 1 show the qualitative and quantitative comparisons, respectively. Table shows that when trained in the same settings as other baselines, Ours² outperforms most of the baselines in validation results in sequence 00 to 08. However, regarding the unseen testing se- quences, our Ours beats all other baseline in both lyet and Tret except for the lyet of DEEP- VIO in sequence 10. This may be explained by the fact that VIE utilizes stereo cam- cras instead of monocular camera considered in method. The result of the testing sequence shows that our method is better at generaliza- tion compared to other learning- based meth- ods. Fig. 4 also shows that, even though pitch and roll are not directly supervised, but with the low-frequency image-level supervision in 3 DoF only, they are still accurate. This proves the effectiveness of the end to end training and that the interpretability provides inductive bias, which accounts for good generalization.
KITTI Odometry
KITTI Odometry 10
DUKF Noise Filtering: Pitch
DUKF Noise Filtering: Roll
Clarms
Figure 4: The qualitative results of Top: the evaluation on KITTI sequence 09, and 10, Bot- tom: Noise filtering of the pitch and roll angle through sequence 10. Visual results on other se- quences can be found in Appendix IV.
It is worth addressing that without employing any nonlinear optimization such as sliding window, our method surprisingly comparable with others on sequences 00-08, and is even slightly better in testing sequences 09 and 10. The result of Ours² suggests that, in contrast to learning- baaneng-bbsedd approaches reliant on a large number of training images, our method is more data-efficient thanks the fully interpretable model.
3.2 Plugin for Heterogencous Localization
In this part, we show the possibility of enabling an end-to-end localization with BEVO as the odom- etry. Elaborations on the implementation details of the fully differentiable drift-free localization can be found in the supplementary material. By combining BEVO with the heterogeneous localization (denoted as BEVO+), the whole pipeline is fully differentiable, thus allowing for more accurate pose (de-
estimation. We compare the localization BEVO+ with BEVO itself as well as the noted as VINS in the table). Results in Fig. 5 and Table prove that the BEVO is able achieve good performance even when it acts as plugin in an end-to-end trainable localization framework. It indicates promising future for upcoming end-to-end autonomous driving, e.g., wuto-wheelchair.
3.3 Ablation Study
Several experiments conducted on KITTI are designed for the ablation study to reassure the im- portant role a DUKF plays in the BEVO as well as the role trainable covariance matrices play in DUKF. First, we replace the learned covariance matrix O, by one that empirically set in [20]. The quantitative result in Table. 3 shows that the accuracy drops drastically when us-
7

---

AeroGround
CARLA
Quantitative sults of localization. Note: in CARLA, we localize the vehi- cle in different weathers against sunny satellite map. (S-sunny, F-foggy, N-night)
TABLE 2:
BEVO Locolization
Test
Train
BEVO
VINS
Dunset
Trus
Frel
Trai
Frui
frrt
Frei
The visual demonstration localizing a front cam- era's BEV of ground moving robot in drone's 's map. More Qual- itative demonstration are shown in the Appendix V.
Figure 5:
CARLA
2.55
0.30
ing manually tuned matrix, suggesting that the empirical parameter tuning leads to inaccurate pitch and roll estimation, and subsequently deteriorates the BEV generation and odometry esti- mation. Next, we restore O, with the learned matrix and replace Q, with an empirical covari- matrix as [20], and note as "w/o Q.". The quantitative result also verifies that train- able covariance matrices for noise help to improve the performance of UKF. However, even though the performances are degraded with empirical parameters, they are still competitive due to the dense matching of DPC who seeks global optima. We further study the importance of DUKF on the BEVO by eliminating the whole DUKF and feeding the raw pitch and roll (di- rectly calculated from the accelerometer) to the BEV projection. We note it as "w/o DUKF". The quantitative result shown in Table. proves that when the filter is gone, the BEVO will en- counter serious tracking failure in bumpy roads. One more ablation study is conducted by re- placing the output pitch and roll of DUKF with the ground truth, denoted as "w/ GT". The re- sult that BEVO is comparable with "w/ GT" in- dicates that by learning the covariance matri- ces, the error brought by DUKF is too small to influence the odometry than the error by DPC, showing the effectiveness of the training.
TABLE 3: Quantitative results of ablation study conducted on KITTI All of the methods are trained in sequence 00-08 following the train spit in [26] and tested in sequence 09-10.
wio O2
wio Q2
W/o DUKF
BEYO
W/ GT
Sog
1,4
Fee
Feel
7ea
f,1
09
4.2
4.73
5,00
5:11
22.05
71.58
1.29
1,12
1/6
1.04
5.06
4,87
6.25
5.02
25.64
68.54
1.14
1.03
0.96
1.01
3.4 Limitation
Though being competitive in odometry estimation for most of the ground mobile robots, BEVO, limited by the assumption of constant elevation, yet hardly applicable to robots that move with large elevation changes, e.g. drones. However, since the phase correlation in Section 2.3 is able to estimate the scale changes along with the rotation (yaw) between images, it should potentially serve way to calculate the relative elevation changes. We will address the above limitations in future works and hopefully make BEVO compatible in all scenarios.
Conclusion
We present fully differentiable, interpretable and lightweight model for monocular VIO, namely, BEVO. It comprises a UKF to denoise pitch and roll via trainable covariance matrices, a BEV pro- jection, and phase correlation estimate the SIM((2 pose on the BEV image. The differentiability allows us to train the covariance matrices end-to end-to-end, thus avoiding empirical parameter tuning. In various experiments, BEVO presents a competitive performance in accuracy and generalization. We believe our interpretable, simple approach provides an alternative perspective of traditional methods and should be considered as a baseline for future works on learning- based monocular VIO.
8

---

Acknowledgments
This work was supported in part by the National Key R&D Program of China (Grant No. No.2020YFB13133000 Zhejiang Provincial Natural Science Foundation of China (LD22E050007), and Science and Technology on Space Intelligent Control Laboratory (2021-JCIQ-LB-0000-13) We also thank reviewers who gave useful comments.
References
[1] T. Qin, P. Li, and S. Shen. Vins-mono: A robust and versatile monocular visual-inertial state estimator. IEEE Transactions on Robotics, 34(4): 2018.
[2] Y. He, J. Zhao, Y. Guo, W. He, and K. Yuan. Pl-vio: Tightly-coupled monocular visual-inertial odometry using point and line features. Sensors, 18(4):1159) 2018.
[3] E. Rublee, V. Rabaud, K. Konolige, and G. Bradski. Orb: An efficient alternative to sift or surf. In 2011 International conference on computer vision, pages 2564-2571. leee, 201
[4] M. Brossard, S. Bonnabel, and A. Barrau. Unscented kalman filter on lie groups for visual iner- tial odometry. In 2018 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 649 649-655. IEEE, 2018.
[5] L. Han, Y. Lin, G. Du, and S. Lian. Deepvio: Self-supervised deep learning of monocular visual inertial odometry using 3d geometric constraints. In 2019 IEEE/RSJ International Con- ference on Intelligent Robots and Systems (IROS), pages 1906-6913 IEEE, 2019.
[6] T. Pandey, D. Pena, Byrne, and D. Moloney. Leveraging deep learning for visual odometry using optical flow. Sensors, 21(4):1313, 2021.
[7] P. Muller and A. Savakis. Flowdometry: An optical flow and deep learning based approach to visual odometry. In 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 624 631. IEEE, 2017.
[8] C. Chuanqi, H. Xiangyang, Z. Zhenjie, and Z. Mandan. Monocular visual odometry based on optical flow and feature matching. In 2017 29th Chinese Control And Decision Conference (CCDC), pages 4554 4558. IEEE, 2017.
[9] X. Ban, H. Wang. Т. Chen, Y. Wang, and Y. Xiao. Monocular visual odometry based on depth and optical flow using deep learning. IEEE Transactions on Instrumentation and Measurement, 70:1-19, 2020.
[10] R. Wang, S. M. Pizer, and J.-M. Frahm. Recurrent neural network for (un-) supervised learning of monocular video visual odometry and depth. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 5555-5564, 2019.
[11] H. Zhan, R. Garg, C. S. Weerasekera, K. Li, H. Agarwal, and I. Reid. Unsupervised learning of monocular depth estimation and visual odometry with deep feature reconstruction. Pro- ceedings of the IEEE conference computer vision and pattern recognition, pages 340-349, 2018.
[12] M. Xiong, Z. Zhang, W. Zhong, J. Ji, Liu, and H. Xiong. Self-supervised monocular depth and visual odometry learning with scale-consistent geometric constraints. In IJCAI, pages 063-969, 2020.
[13] X. Li, Y. Hou, Q. Wu, P. Wang, and W. Li. Dvonet: unsupervised monocular depth estimation and visual odometry. In 2019 IEEE Visual Communications and Image Processing (VCIP), pages 1-4. IEEE, 2019.
[14] T. Qin, T. Chen, Y. Chen, and Q. Su. Avp-slam: Semantic visual mapping and localization for autonomous vehicles in the parking lot. In 2020 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 5939-5945. IEEE, 2020.
[15] D. Zhou, Y. Dai, and Li. Ground soaund-plane-baree based absolute scale estimation for monocular visual odometry. IEFE Transactions on Intelligent Transportation Systems, 21(2):791-822, 2019.

---

[16] R. G. Krishnan, U. Shalit, and D. Sontag. arXiv:1511.05121, 2015.
Deep
kalman filters.
arXiv preprint
[17] R. Jonschkowski, D. Rastogi, and O. Brock. Differentiable particle filters: End-to-end learning with algorithmic priors. arXiv preprint arXiv: /122, 2018.
[18] Z. Chen, X. Xu, Y. Wang, and R. Xiong. Deep phase correlation for end-to-end heterogeneous sensor measurements matching. In J. Kober, F. Ramos, and C. Tomlin, editors, Proceedings of the 2020 Conference on Robot Learning, volume 155 of Proceedings of Machine Learning Research, pages 2359-2375. PMLR, 16-18 Nov 2021.
[19] C. Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206-215, 2019.
[20] E. Wan, R. Van Der Merwe, and Haykin. The unscented kalman filter. Kalman Kaltrifilttring and neural networks, 5(2007):211-28)) 2001.
[21] C. Campos, R. Elvira, J. J. Gomez, J. M. M. Montiel, and J. D. Tardos. ORB-SLAMN An accurate open source library for visual, visual-inertial and multi-may SLAM. arXiv preprint MiXiv:2007.118% 2020.
[22] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun. Vision meets roboties: The kitti dataset. The International Journal of Robotics Research, 12:1131221-1123,, 2013.
[23] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun. Carla: An open urban driving simulator. In Conference on robot learning, pages 1-16. PMLR, 2017.
[24] ZJU--Rootics--aah Aeroground dataset, 2020.
[25] R. Mahjourian, M. Wicke, and Angelova. Unsupervised learning of depth and ego-motion from monocular video using 3d geometric constraints. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5667-5675, 2018.
[26] S. Li, X. Wang, Y. Cao, F. Xue, Z. Yan, and H. Zha. Self-supervised deep visual odometry with online adaptation. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pages 6339-6348, 2020.
[27] H. Yin, R. Chen, Y. Wang, and R. Xiong. Rall: end-to-end radar localization on lidar map using differentiable measurement model. IEEE Transactions on Intelligent Transportation Systems, 2021.
[28] Y. Almalioglu, M. Turan, A. E. Sari, M. R. U. Saputra, P. P. de Gusmão, A. Markham, and N. Trigoni. Selfvio: Self-supervised deep monocular visual-inertial odometry and depth esti- mation. arXiv preprint arXiv:1911 191.:9991..096.. 2019.
[29] Z.
Yin and J. Shi. Geonet: Unsupervised learning of dense depth, optical flow and camera pose. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1983-1992, 2018.
[30] W. Wang, Y. Hu, and S. Scherer. Tartanvo: A generalizable learning- based vo. arXiv preprint a0X1v 2020.
10


Reviews:
[{"review_id": "qHAATrXeax", "reviewer": "Reviewer_p14S", "summary": "This work introduced a novel visual-inertial odometry system, which is learning-based but interpretable without deep neural networks. Specifically, the IMU data is used by a differentiable UKF to estimate the roll and pitch, which are subsequently used to project images to bird-eye view (BEV). The BEV images are transformed to the frequency domain to estimate SIM(2) camera motion. The leaning part comes from the UKF covariance; the authors maintain the following BEV projection and motion estimation differentiable so that the odometry error could be back-propagated to the UKF covariance, which makes it learnable and the estimated roll/pitch optimal. The evaluation is mainly based on the standard KITTI dataset. The proposed method achieves great results.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n- The paper is well-written, and the methodology is clearly stated and technically sound. It is a really solid work with significant novelties and contributions. It is refreshing to review a learning-based VIO work that is not based on deep networks. \n\nWeakness\n- The unchanged height assumption for driving scenarios seems fine; however, the reviewer is slightly concerned about the sufficient distance assumption: what is the minimal distance? is it also related to the camera mounting height and angle? if there are objects in that yellow box, how will it distort the BEV images and decrease the VIO accuracy?\n- The evaluation is mainly based on the KITTI dataset, focusing on sequences 09-10 for testing. The validation would be more persuasive if the authors include more VIO testing. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "It is a really solid work with novel method.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- Line 121: ‘within the red dashed box’, do you mean ‘within the yellow dashed box’?\n- In Section 2.3, the ‘scale’ term is slightly confusing. Since the proposed method is estimating SIM(2) motion, the reviewer assumes the ‘scale’ refers to the odometry scale instead of image scaling. If so, what is the reason to fix the scale to 1 (line 150)? fixing scale to 1 makes SIM(2) to SE(2), right?\n- How do the authors handle the IMU bias terms? are they not modeled in the UKF?\n"}}, {"review_id": "fVs6Ietqrm1", "reviewer": "Reviewer_3Som", "summary": "This paper proposed an interpretable architecture for VIO, which admits differentiable training. Experimental results on two domains demonstrate that the proposed approach is effective and competitive against recent deep learning-based architectures. ", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\nThe architecture is novel, as far as I can tell. It also achieves very high performance. \n\nWeaknesses: \n\nLet me preface my comment by saying that I am not an expert on VIO, but come more from the general model interpretability side. With that said, I am quite disappointed about the lack of empirical evaluation of the model interpretability. While the model achieves comparable performance as deep models, which is refreshing to see as many interpretable models struggle to match the performance of their non-interpretable counterparts, it is not clear in what ways the proposed model is exactly interpretable. \n\nDemonstrating interpretability requires validating that people, developers and/or users, can gain concrete understanding of the model. Thus, at the very least, there should be more analysis on the model itself, besides that on model performance. Ideally, it would also show some specific aspects corresponding to the model understanding. For example, maybe the developer can identify some issues of the model so that they can re-train a better one, or that a domain expert can understand the weakness of it, so that they can choose when not to trust the model prediction. Without any of these demonstrations, I am not convinced that this model is any more interpretable than a standard neural network. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Due to the lack of experiments and analyses on the interpretablity front, I could not recommend this paper for publication. I believe that there could be two avenues forward: \n\n1. Remove all claims of interpretability, so that this paper's contribution is limited to a novel architecture for VIO that matches SoTA performance. Or better, \n\n2. Conduct experiments to validate interpretability, and position this model as an interpretable and high-performing one. ", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "None. ", "reviewer_expertise": "2: The reviewer is willing to defend the evaluation, but it is quite likely that the reviewer did not understand central parts of the paper"}, "extra_text": {}}, {"review_id": "EChgurqDk4Dt", "reviewer": "Reviewer_vD2G", "summary": "This paper proposes a visual inertial odometry system based on kalman filtering. The main idea of this work is to learn the noise filtering parameters by making the filter differentiable. While the idea of learning filters is not new (See [1]), the application to the problem of VIO from a bird-eye perspective is new to me.  The approach is evaluated against a set of baselines on the KITTI dataset, where it obtains comparable or better results.\n\n\n[1] Krishnan et al,, Deep Kalman Filters", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strength of the paper is the nice balance between end-to-end learning and interoperability. Learning only a few parameters end-to-end makes the system easy to debug and investigate. It is to be said, however, that similar ideas have been long investigated in the community (See [1],  [2] or many others), but none really took off. This could be because learning so few parameters makes the algorithm strongly reliant on algorithm priors, which might be inappropriate in several real-world applications.\nA second interesting takeaway from this work is that traditional VIO methods, like VINS, perform terribly on KITTI. However, I find this somehow difficult to believe since, in the KITTI benchmark, most monocular systems are traditional VIO methods. Is this setup somehow different? What makes it challenging for them in the proposed evaluation? \nFinally, I find the idea of passing through the Bird-eye view quite interesting. I would have expected to be a suboptimal choice given the presence of other vehicles and humans, whose upper part is generally occluded. However, this does not seem to be a problem for the proposed system, which I find quite interesting.\n\nThe main weakness of the approach is, in my opinion, the evaluation setup and results. As mentioned before, the terrible performance of traditional VIO methods is difficult to interpret. In addition, I feel that the main claim of this paper (rely more on algorithmic priors than parameters) is not sufficiently justified. It would be interesting to see results on more challenging datasets, possibly in 3D, where the assumption on the motion model would not always hold. If the paper only wishes to specialize in AV scenarios, then such choices must be made clear from the beginning (abstract/intro) and claims weakened. The limitation section mentions that the approach would be applicable for all ground robots, but there is not enough evidence to support this claim. Overall, the main question I feel should be answered is: when is desirable to enforce algorithmic priors instead of parameters? \nAnother weakness is the fact that previous work on deep state estimators (like [1],[2], and others) has not been sufficiently covered and discussed. In what does the approach differ? Why is it better? \nFinally, a (minor) limitation is that the approach is said to be lightweight, but computation time is not compared to existing methods.\n\n[1]  Deep Kalman Filters, Krishnan et al.\n[2] Differentiable Particle Filters: End-to-End Learning with Algorithmic Priors, Jonschkowski et al.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "My recommendation is based on the disparity between the claims and the provided results. In addition, the paper does not make a strong case on why this approach would be relevant in general ground robotic applications (AVs also have GPS, so why not use it?).", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "I would be happy to see a more aligned introduction and experimental section in the revision. Also, I would like to see a discussion of previous methods and a clearer answer to the question: \"When are algorithmic priors a good idea?\""}}, {"review_id": "0a6V5h_iyYs", "reviewer": "Reviewer_Me8P", "summary": "This paper proposes a novel VIO method which estimates 5 DoF poes (with the elevation fixed) from monocular camera and IMU. In particular the proposed method adopt Unscented Kalman Filter (UKF) which is natually differentiable for the state estimation. With the aid of filtered pitch and roll using UKF, the front-view images can be converted to bird eye view images which are then used for pose estimation with Differentiable Phase Correlation. The whole method is differentiable and can be backpropagated using ground-truth pose supervision to update the covariance matrices. Evaluation results show the the proposed method achieves competitive results on real-world KITTI dataset for odometry, CARLA and AeroGround datasets for map-based localization. The authors also conducted ablation studies on the learned covariance matrices.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- The paper is very well written and one can read very smoothly.\n- The authors make use of the differentiable nature of Unscented Kalman Filter (UKF) and proposes to learn the covariance matrices in a end-to-end manner to capture the motion and measurement noise distribution. This way it retains the interpretability of the pipeline and also leverage the power of data-driven approach to learn the otherwise hand-tuned covariance matrices.\n-  The estimated roll and pitch from UKF is integrated seamlessly to the bird eye view projection followed by camera pose estimation using Differentiable Phsace Correlation. This enables the differentiability of the whole piple with reasonable assumptions.\n- The results show very competitive results on KITTI for camera pose estimations. Further, the authors also show its effectiveness to apply it for map-based localization on CARLA and AeroGround datasets.\n- The authors provide the code in the supplement.\n\n## Weaknesses\n- Altough the whole method only learns 4 parameters, it is a bit confusing why it needs 2 RTX 3090 to train.\n- No runtime and memory statistics are provided.\n- The authors did not show cross dataset generalization evaluation. It would be interesting to see if the covariances learned from one dataset/sensor can transfer to other datasets.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The authors propose a quite novel approach for 5 DoF VIO which leverages differentialble UKF filter and BEV projection. The learned covarinances are shown to be very effective and I believe such an approach could have a big impact to the community.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "It would be great if the authors could address my concers in the *Strengths And Weaknesses* section.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)