Title: ON LEARNING VISUAL ODOMETRY ERRORS

Abstract: This paper fosters the idea that deep learning methods can be sided to classical visual odometry pipelines to improve their accuracy and to produce uncertainty models to their estimations. We show that the biases inherent to the visual odometry process can be faithfully learned and compensated for, and that a learning architecture associated to a probabilistic loss function can jointly estimate a full covariance matrix of the residual errors, defining a heteroscedastic error model. Experiments on autonomous driving image sequences and micro aerial vehicles camera acquisitions assess the possibility to concurrently improve visual odometry and estimate an error associated to its outputs.

1. INTRODUCTION:

Visual odometry (VO) is a well established motion estimation process in robotics (Scaramuzza & Fraundorfer, 2011) , successfully applied in a wide range of contexts such as autonomous cars or planetary exploration rovers. Seminal work resorted to stereovision: by tracking point features in images, 3D points correspondences are used to recover the motion between two stereovision acquisitions -the integration of elementary motions yielding an estimate of the robot pose over its course. Continuous work on VO led to a well established processes pipeline, composed of feature extraction, matching, motion estimation, and finally optimization. This scheme has been extended to single camera setups, in which case motions are estimated up to a scale factor, retrieved e.g. by fusing inertial information. Direct methods for VO have recently been proposed: they bypass the feature extraction process and optimize a photometric error (Engel et al., 2018) . These methods overcome the limits of sparse feature-based methods in poorly textured environments or in presence of low quality images (blurred), and they have proven to be on average more accurate.

The advent of convolutional neural networks (CNN) sprouted alternate solutions to VO, that achieve the full estimation process to deep-learning architectures in an end-to-end fashion (see e.g. (Konda & Memisevic, 2015; Li et al., 2017) , and especially (Wang et al., 2018) -note these work consider the monocular version of the problem, leaving the scale estimation untackled). In such approaches, the system has to learn the various information necessary to perform vision-based egomotion estimation, which can be a daunting task for a CNN.

The work presented here builds upon existing work that exploits a CNN to predict corrections to classic stereo VO methods (Peretroukhin & Kelly, 2017) , aiming at improving their precision. This concurs with the idea that it is more beneficial to side deep-learning based methods with classical localization estimation processes rather than delegating the full pose estimation to a CNN. Our developments consider that visual odometry estimation errors do not have zero mean, as assessed in e.g. (Dubbelman et al., 2012; Peretroukhin et al., 2014) , and provide corrections that improve the precision of VO. Furthermore, they provide a full error model for each computed motion estimation (in form of a Gaussian model), akin to (Liu et al., 2018) . This is a significant achievement, as to our knowledge no precise error models have been derived for classic VO methods.

2. PROBLEM STATEMENT AND RELATED WORK:

Consider a robot moving in a three dimensional environment. Let x i ∈ R 6 (3 translations and 3 orientations) be its pose at time i in a given reference frame. The actual motion (ground truth) between time instants i and i + 1 is represented by a homogeneous 4 × 4 transformation matrix i T i+1 .

A vision-based motion estimator uses raw image data I i ∈ R n to obtain an estimate i Ti+1 . In the VO case, the raw data I i is a pair of monocular or stereoscopic images captured at two different time instants i, i + 1 (i.e. 2 or 4 images). The error e i of VO is:

e i = i T i+1 • i T-1 i+1 (1)

The dataset to feed a learning architecture is D = {I i , e i |∀i ∈ [1, d]}, where d is the size of the dataset.

The literature provides three different approaches to leverage this type of dataset. The first one directly learns the motion estimate and associated error (Wang et al., 2018) . The two other approaches side a classic VO process with learning to either estimate (i) a motion correction to apply to i T-1 i+1 , thus improving its accuracy (Peretroukhin & Kelly, 2017) , or (ii) an error model associated to i T-1 i+1 (Liu et al., 2018) , thus allowing its fusion with any other motion or pose estimation process.

2.1 DIRECTLY LEARNING VO AND AN ERROR MODEL Wang et al. (2018) introduce an end-to-end, sequence-to-sequence probabilistic visual odometry ("ESP-VO") based on recurrent CNN. ESP-VO outputs both a motion estimate i T-1

i+1 and an associated error. The learned error model is a diagonal covariance matrix, hence not accounting for possible correlations between the different dimensions of the motions. It is unclear how the probabilistic loss is mixed to the mean squared error of the Euclidean distance between the ground truth and the estimated motions, and the makes use of a hand-tuned scaling factor to balance rotation and translation. . The article present significant results obtained with an impressive series of varied datasets, with comparisons to state of the art VO schemes. The results show that ESP-VO is a serious alternative to classic schemes, all the more since it also provides the variances associated to the estimations. Yet, they are analysed over whole trajectories, which inherit from the random walk effect of motion integration, and as such do not provide thorough statistical insights -e.g. on the satisfaction of the gaussianity of the error model.

2.2. LEARNING CORRECTIONS TO VO:

The work presenting DPC-net (Peretroukhin & Kelly, 2017 ) learns an estimate of e i , which is further applied to the VO estimate i T-1 i+1 to improve its precision. The authors introduce an innovative pose regression loss based on the SE(3) geodesic distance modelled with matrix Lie groups approach. Instead of resorting to a scalar weight to unify with a linear combination the translation and rotation errors, the proposed distance function naturally balances these two types of errors. The loss takes the following form:

EQUATION

where ξ ∈ R 6 is a vector of Lie algebra coordinates estimated by the network, g(ξ) computes the equivalent of Eq.1 in the Lie vector space, and Σ is an empirical average covariance of the estimator pre-computed over the training dataset. The paper provides statistically significant results that show DPC-net improves a classic feature-based approach, up to the precision of a dense VO approach.

In particular, it alleviates biases (e.g. due to calibration errors) and environment factors. Worth to notice though is that the authors interlace the corrections estimated at lower rate than the underlying VO process with VO, which processes all the images, using a pose-graph relaxation approach.

2.3. LEARNING AN ERROR MODEL OF VO:

Inferring an error model for VO come to learn the parameters of a predefined distribution to couple VO with uncertainty measures. Liu et al. (2018) introduces DICE (Deep Inference for Covariance Estimation), which learns the covariance matrix of a VO process as a maximum-likelihood for Gaussian distributions. However, they consider the distribution over measurement errors as a zero-mean Gaussian N (0, Σ). Such model is acceptable for unbiased estimators, which unfortunately if not the case of VO. Yet the authors show that their variance estimates are highly correlated with the VO errors, especially in case of difficult environment conditions, such as large occlusions.

3. SIMULTANEOUSLY LEARNING CORRECTIONS AND UNCERTAINTY:

To jointly estimate a correction to the VO process and a full error model after having applied the correction, we expand the network structure of Liu et al. (2018) adding a vector µ i ∈ R 6 to the output layer, which is incorporated in the negative log-likelihood loss that is derived as follows.

Given a dataset D of size d, where the observations {e 1 , . . . , e d } T of VO errors are assumed to be independently drawn from a multivariate Gaussian distribution, estimate the parameters of the Gaussian by arg max

EQUATION

This is equivalent to minimize the negative log-likelihood (NLL)

arg min

EQUATION

= arg min

EQUATION

≈ arg min

EQUATION

We split the output of the network in two different parts: the mean vector f µi (I i ) and the covariance matrix f Σi (I i ), where f (I i ) represents the full output given a pair of stereo images.

To enforce a positive definite covariance matrix we use the LDL matrix decomposition. f Σi (I i ) is reformulated as a vector

α i = [l i , d i ] T with l i ∈ R (n 2 -n) 2

and d i ∈ R n . We have then

Σ i ≈ L(l i )D(d i )L(l i ) T (7)

where l i and d i are the vectors containing the elements of the respective L and D matrices. The LDL decomposition is unique and exists as long as the diagonal of D is strictly positive. This can be enforced using the exponential function exp(d i ) on the main diagonal. Thanks to some additional properties around the computation of its log determinant the first term of Eq.6 can be simplified as sum(d i ), that is the sum of the elements of the vector d i . In the second term f Σi (I i ) -1 is replaced by the LDL product. Replacing f µi (I i ) with the mean output vector μi we finally obtain

L(I 1:d ) = arg min μ1:d ,α 1:d d i=1 sum(d i ) + (e i -μi ) T (L(l i )D(exp(d i ))L(l i ) T ) -1 (e i -μi ) (8)

Formulating the problem as in Eq.8, the loss function recalls the formulation of the Lie algebra loss in Eq.2. The covariance matrix in this case is learned in relation to the input, capturing the heteroscedastic uncertainty of each sample. The learned covariance matrix acts as in Kendall & Cipolla (2017) , weighting position and orientation errors. The main difference resides in the nature of the learned uncertainty, homoscedastic vs heteroscedastic: through back-propagation with respect to the input data, Kendall & Gal (2017) , we learn a heteroscedastic error.

Assuming that errors can be drawn from a distribution N (µ i , Σ i ), estimating µ i corresponds to predicting the maximum likelihood value that the error model can assume. This corresponds to the desired correction in our case. At the same time, we estimate a covariance matrix Σ i , returning an uncertainty measure relative to each particular input and predicted correction.

4. EXPERIMENTS:



4.1. SETUP:

We use the open-source VO implementation libviso2 (Kitt et al., 2010) . It is a feature-based approach, that uses a Kalman Filter in combination with a RANSAC scheme to produce SE(3) estimates using rectified stereoscopic pairs.

4.1.1. DATASETS:

We exploit two different datasets: the KITTI dataset (Geiger et al., 2012) and the Euroc Micro Air Vehicles dataset (Burri et al., 2016) .

KITTI provides various sequences of rectified images acquired while driving in urban areas. We tested training the network using several combinations and obtained consistent results splitting train and validation trajectories in different configurations: for all results shown here we trained using sequences 04 to 10 excluding one or two for validation purposes. The estimated motions are expressed in camera frame (z axis pointing forward), and the Tait-Bryan angles are defined wrt. this reference frame (e.g. yaw encodes rotations around the optical axis z).

Euroc contains different stereoscopic sequences recorded in different environments. We rectified the images in order to use them as input for both the VO process and the network. We use the first three sequences MH 01 to MH 03. Note that in some cases VO fails to produce a pose estimate (mainly when images have a strong motion blur): these data have simply been discarded from the dataset. Contrary to the KITTI dataset, Euroc estimations and ground truth are expressed in the vehicle body frame (x pointing forward, y leftward).

4.1.2. NETWORK STRUCTURES:

We initially compared the results produced using the architectures in DPC-net (Peretroukhin & Kelly, 2017) and DICE (Liu et al., 2018) . The first trial was to adapt the loss in Eq. 8 to DPCnet. We noticed that the mean output vector was still rather constant throughout entire trajectories, regardless of the dataset, and the same behavior was experienced using the loss in Eq. 2. Similar tests were conducted with DICE. We experienced problems in reducing the average mean error along the six dimensions, and an increase in the standard deviation. Alleging these issues a being caused by the shallow architecture of DICE, we modified its network structure, first removing the max pool layers to preserve spatial information (Handa et al., 2016) , and achieving dimension reduction by setting the stride to 2 in early layers. We also increased the number of convolutional filters to tackle the estimation of both the corrections and error model, adding 50% dropout after each layer to prevent over-fitting. We kept respective nonlinear activation function using parametric ReLu for DPC-net and leaky ReLu for DICE. For the rest of the paper, we refer to this network as Deeper-DICE (D-DICE, table 1 ).

The convolutional layers are followed by two fully connected layer, respectively composed of 2048 and 27 output units. In the six-dimensional case we need 21 values for the LDL decomposition and 6 for the mean vector. No rectification or dropout is applied to the last fully connected layer.

We trained using Adam optimizer with a learning rate of 1e-04 and halted the learning when test and train loss start diverging. All the experiments have been carried out using using an Nvidia GeForce RTX 2080 Ti with a batch size of 64. 

4.2. EVALUATION:

Most of the results presented in this section aim to analyse the improvements brought by the neural network to VO, by complementing it with SE(3) pose corrections and an error estimate.

4.2.1. LOSS COMPARISON:

Here we compare the results in terms of corrections between the loss based on the Lie algebra formulation (Equ. 2) and our full negative log-likelihood loss (NLL, Equ. 4). Both are evaluated on DPC-net. None of the two losses comes out as a clear winner, even if both improve the mean error along five out of six dimensions compared to VO alone. Note small increases in the average standard deviations using NLL loss, which are not significant compared to the improvements over the average mean error.

Of course, the advantage of using the proposed NLL loss is to jointly provide an uncertainty estimation.

4.2.2. ARCHITECTURE COMPARISON:

We noticed that DPC-net tends to output rather constant corrections throughout a whole sequence, certainly compensating biases. D-DICE behaves differently, as can be seen figure 1 , exhibiting more data-dependant corrections -which is the main point of a deep learning approach.

4.2.3. UNCERTAINTY ESTIMATION:

Here we inspect the error estimates produced by D-DICE. Since we assumed a Gaussian error model ∼ N (µ, Σ), a common way to measure its relevance is to check the fraction of samples that do not respect the following inequality:

µ i -nσ i ≤ e i ≤ µ i + nσ i (9)

where e i is a the value assumed by the error along the dimension i, and µ i , σ i respectively are the mean and standard deviation predicted for the input associated to e on the i-th dimension. The parameter n is associated to the number of considered standard deviations. We consider the three sigma interval (n = 3) that, in case of samples drawn from a normal distribution, covers 99.7% of the samples. uncertainty estimation in the 3-sigma interval, as shown in Table4. The results from considering an unbiased error yield a non-significant increase of precision, as we already sit around 99%, at the cost of being more uncertain, which is a much bigger downside.

5. CONCLUSIONS:

We presented an insight into the learning of errors in visual odometry. Relying on existing stateof-the-art techniques, we iterated on analysing what type of error and uncertainty can be learned by deep neural networks. We concentrated our efforts on approaches that can be paired with classical visual odometry pipelines, in order to ease the work done by the network and exploiting the powerful feature-based processes. We demonstrated that it is possible to assimilate the distribution over visual odometry errors to Gaussians, and proceeded to cast the error prediction to a full maximum likelihood for normal distributions case. Knowing that the errors are biased, we model such Gaussians as non-zero mean distributions, showing the beneficial aspects of this approach compared to works that rely only on the estimation of the covariance matrix. On the other hand, we pair visual odometry corrections with a more precise model, inferred thanks to the assumption of biased distributions. In future we would like to explore similar approaches, with different perception processes that are yet to associated with precise error models, e.g. iterative closest points algorithm based on LiDAR scans (Pomerleau et al., 2013) .

Under review as a conference paper at ICLR 2020 

:



Figures and Tables:

Fig.4shows D-DICE uncertainty bounds trained considering the visual odometry error ∼ N (0, Σ i ), in red, and ∼ N (µ i , Σ i ), in light blue. Our proposed loss already reaches a good precision on the

Figure 1: The plots show the different nature of the learned corrections using DPC-net and D-DICE (Euroc dataset). Each plot shows the difference between VO estimates and the corrected estimates produced by the network.

Figure 2: D-DICE. Euroc, MH 01. Uncertainty prediction in the six dimensions (translation top, rotation bottom).

Figure 4: KITTI, sequence 05. Uncertainty prediction estimating the mean of the distribution (blue) or not (red).

KITTI, sequence 05. Y-estimate evolution over time.



Table content: <table><tr><td colspan="4">Layer Kernel size Stride Number of channels</td></tr><tr><td>conv1</td><td>5x5</td><td>2</td><td>64</td></tr><tr><td>conv2</td><td>5x5</td><td>2</td><td>128</td></tr><tr><td>conv3</td><td>3x3</td><td>2</td><td>256</td></tr><tr><td>conv4</td><td>3x3</td><td>2</td><td>512</td></tr><tr><td>conv5</td><td>3x3</td><td>1</td><td>1024</td></tr></table>

shows that corrections learned with the Lie loss and extracted from the Gaussian model learned minimizing the negative log-likelihood. The results have been obtained after training without sequences 05, left for validation and 06, used for testing.

Table content: <table><tr><td/><td>µ V O</td><td>σ V O</td><td colspan="4">µ corr Lie σ corr Lie µ corr N LL σ corr N LL</td></tr><tr><td>x</td><td>0.20</td><td>0.60</td><td>0.07</td><td>0.60</td><td>-0.92</td><td>0.66</td></tr><tr><td>y</td><td>-0.25</td><td>0.58</td><td>0.07</td><td>0.58</td><td>-0.02</td><td>0.62</td></tr><tr><td>z</td><td>-0.62</td><td>0.95</td><td>-0.25</td><td>0.95</td><td>-0.12</td><td>0.98</td></tr><tr><td>roll</td><td>1.21</td><td>28.17</td><td>-0.98</td><td>28.16</td><td>-0.2</td><td>28.55</td></tr><tr><td>pitch</td><td>0.81</td><td>28.16</td><td>0.82</td><td>28.13</td><td>1.01</td><td>28.06</td></tr><tr><td>yaw</td><td colspan="2">-14.32 30.59</td><td>0.28</td><td>30.60</td><td>2.26</td><td>30.85</td></tr></table>

KITTI, sequence 05. Statistics for VO with and without corrections, applied using DPC-net trained with the two losses of in Eq.2 and 8. Units are cm for x, y, z and in mrad for the angles.

Table content: <table/>

Euroc, sequence MH 01.

Table content: <table><tr><td/><td>σ</td><td>2σ</td><td>3σ</td></tr><tr><td>x</td><td colspan="3">87.24% 98.91% 99.63%</td></tr><tr><td>y</td><td colspan="3">91.15% 99.42% 99.67%</td></tr><tr><td>z</td><td colspan="3">79.49% 96.12% 98.94%</td></tr><tr><td>roll</td><td colspan="3">75.72% 94.89% 98.69%</td></tr><tr><td>pitch</td><td colspan="3">72.97% 95.14% 98.76%</td></tr><tr><td>yaw</td><td colspan="3">70.57% 93.55% 98.69%</td></tr><tr><td>Percentages of samples that lie in the various</td><td/><td/></tr><tr><td>sigma-intervals around the mean. Mean and</td><td/><td/></tr><tr><td>standard deviations are produced by D-DICE</td><td/><td/></tr><tr><td>and correspond to Fig.2.</td><td/><td/></tr></table>

KITTI, Sequence 05.

Table content: <table><tr><td>Per-</td></tr></table>

Peer review (iclr2020):
The proposed method addresses the problem of predicting per-time-step corrections for the motion estimates produced by a visual odometry (VO) system. A training dataset is aggregated, collecting errors of the considered VO system (libviso2) as observations and the images for the respective time steps as conditional inputs. A convolutional neural network is then trained to amortize a predictive multivariate Normal distribution that should fit the observed errors. The inputs to the network are the images at the current time step. The network parameters are optimized with stochastic gradient descent, maximizing the log-likelihood of the observed error w.r.t. the predicted Gaussian model. The authors also propose modifications to an existing convolutional neural network architecture. The method is evaluated on the KITTI and EuroC data sets. The reviewer recommends rejecting the paper. In its current state it is not possible to judge the contributions due to a lack of clarity in the experiments section and because a highly relevant metric is omitted (long-term translation and rotation error). The results on uncertainty quantification are hard to judge. Major Comments: 1) The method described in the paper is justified. 2) Section 3 fails to point out the exact differences to the cited prior work. For example, the assumed LDL decomposition appears the same as in the work by Lie et al. 2018. It should be stated more clearly what the delta is to the work of Peretroukhin & Kelly 2018 that lets the CNN predict the Normal mean, and to the work of Lie et al. 2018 that has the CNN predict the Normal covariance matrix. After looking at the respective papers, it appears the two methods have been combined in a single consistent Gaussian model without further modification. 3) The summarized statistics in Table 2 are not clear. Are those the predicted statistics from the amortized conditional distribution? Or are they the empirical diagonal Gaussian statistics computed over the VO transformations which have been corrected by the mean predicted by the network? The latter is my current best guess given the highlighting. However it is not clear what this metric would show us in terms of model performance. 4) Quantifying the quality of the residual uncertainty estimates via the ratio of observed residuals that fall into the +/- 3 sigma band of a diagonal Gaussian seems like an arbitrary choice. The neural network is supposed to predict a full multivariate Gaussian distribution - why not utilize the full covariance matrix? For example, one could compute log-likelihood scores w.r.t. the predictive model. The authors of DICE evaluate their uncertainty estimates by illustrating a correlation between uncertainty and error magnitude. A similar experiment would be helpful here to judge the quality of the uncertainties. 5) The last sentence right before the conclusion is unclear. What is meant under *the results from considering an unbiased error*—was a second model that assumes a zero-mean Gaussian fit to the data, similar to DICE? 6) The paper is missing evaluations that demonstrate the overall and piece-wise reduction of drift in the visual odometry system when the predicted corrections are applied. The last two figures showing a single example from KITTI are not sufficient to judge the effectiveness of the method. For instance DPC-net, which the authors compare against, reports three metrics of long-term pose error (See: C. Evaluation Metrics of https://arxiv.org/pdf/1709.03128.pdf). It is not possible to evaluate the contribution of the method without looking at long-term trajectory estimates and how they change after applying the proposed model. Suggested improvements: 1) The paper needs an experiment detailing long-term pose error with and without the proposed method. 2) It is hard to interpret section 4.2.3. The paper would benefit from an experiment showcasing the usefulness of uncertainty.