Information-Theoretic Odometry Learning

Abstract: In this paper, we propose a unified information-theoretic framework for odometry learning, a crucial component of many robotics and vision tasks such as navigation and virtual reality where 6-DOF poses are required in real time. We formulate this problem as optimizing a variational information bottleneck objective function, which eliminates pose-irrelevant information from the latent representation. The proposed framework provides an elegant tool for performance evaluation and understanding in information-theoretic language. Specifically, we bound the generalization errors of the deep information bottleneck framework and the predictability of the latent representation. These provide not only a performance guarantee but also practical guidance for model design, sample collection, and sensor selection. Furthermore, the stochastic latent representation provides a natural uncertainty measure without the needs for extra structures or computations. Experiments on two well-known odometry datasets demonstrate the effectiveness of our method.

Reviews and author responses:

We would like to thank you for the instructive feedbacks and suggestions. We provide more clarifications and discussions on the concerns below:

Q1: What is the problem being attacked and what is the contribution of this work？

In short, we target the specific supervised learning problem for relative camera pose prediction of an image sequence, which provides a crucial part in motion tracking and localization. And we make contribution by providing a theoretically inspired framework and a more informative representation for the learning process. 

The word odometry can be a bit confusing for different communities. For supervised relative camera pose prediction, “odometry” is termed exactly for predicting the relative poses [1-3]. While in geometry methods, “odometry” is usually used in a broader sense and referred to a more complete system, in which case local back-end might also be incorporated to further optimize the initial “guess” from the front-end.  

We agree that learning-based methods can by no means replace well-designed geometry-oriented methods, either feature-based or direct ones. Nevertheless, results from the learned models still hold values in that they can shed light on some aspects that are difficult for geometry methods to capture, e.g. textless areas for feature-based methods and photometric inconsistency for direct methods. As a recent example, D3VO [4] incorporates learned depth and relative camera motion results from an unsupervised model into DSO [5] by using them as initial values with learned uncertainties for the back-end. If equipped with pose labels, supervised models have shown to exhibit superior performance than unsupervised counterparts [6] and hold the potential to provide better initialization. 

If we view odometry as a full system which can be built from many different perspectives, then there remain many crucial research problems as mentioned in the comments, each of which of course deserves lots of research efforts. Nevertheless, a full discussion of all problems in this field is outside the scope of this paper. In this paper,  we focus on a specific topic, i.e. how to improve initial relative camera pose prediction using learning models and make contribution by providing (1) an information-theoretic framework that addresses a crucial problem for learning-based methods, i.e. (how to learn a more informative representation for the final goal, (2) insights on how information bottleneck achieves this in Theorem 1-2 and Corollary 1-2, (3) insights on how to choose and integrate multiple sensors from the information-theoretic perspective, and especially (4) the connection with geometry methods in the sense of information bottleneck in Section 4.4. We are not targeting odometry from the full systematic view but only focus on the potential learning and representation part of this system, which we believe better fits the scope of ICLR.  


We would like to thank you for the constructive comments and suggestions. We provide more details on the concerns below:

Q1: (1) Derivation for Equation 13.  (2) Minor point 1: Duplicate Theorem and Corollary numbers. (3) Derivation for Corollary 1.

(1) WLOG, we first take the condition on \xi out, then the problem reduces to deriving the bound for I(o || s). From Equation (1) in page 2 in [1], we have I(X||Y) <= E[D_KL(p(y|x) || q(y))]. Let s and o be Y and X respectively, we have I(o || s) <= E[D_KL(p(s|o) || q(s))]. We then further condition both parts on \xi, thus leading to I(o || s | \xi) <= E[D_KL(p(s|o, \xi) || q(s|\xi)]. 

(2) The numbers in the main text and appendix do duplicate. We are sorry for the inconvenience and misunderstanding caused by this issue. Theorem 3 and Theorem 4 in the appendix refer to Theorem 1 and Theorem 2 in the main text respectively. Lemma 2 in the appendix refers to Lemma 1 in the main text. And Corollary 3 and Corollary 4 in the appendix correspond to Corollary 1 and Corollary 2 respectively.

(3)By Equation (44), we have I(X||S) <= log|S|. We get Corollary 1 by substituting this inequality into Theorem 1 and replacing I(X||S) with log|S|. 

Q2: Practical usage of the generalization bounds. 

This is a great question how the generalization bounds can guide practices. In practice the involved mutual information is hard to calculate exactly due to the large variable space, leading to difficulties in evaluating exact values of the bound as well. Nevertheless, we can still gain insights from the theoretical results w.r.t. some practices: (1) By Theorem 1, minimizing the information bottleneck objective helps constraining the expected generalization error; (2) By Corollary 2, a small growing rate of the latent dimension d compared with n/log(n) is also beneficial for constraining the expected generalization error. Though the bound might be loose in some cases, depending on the information loss, bottleneck effect, network loss landscape, and sample size, we provided a tightness analysis compared with an existing related work by Vera et al. [2], where our results provide a tighter bound.

Q3: Theoretical contributions w.r.t. (Zhang, 2018) and (Xu & Raginsky, 2017).

(Zhang, 2018) and (Xu & Raginsky, 2017) obtained the generalization bound w.r.t. the mutual information I(X||\Theta) between input data X and the learning parameters \Theta for general learning algorithms and neural networks. However, what information bottleneck methods regularize is the mutual information I(X||S) between X and the latent representation S. We establish the connection between I(X||\Theta) and I(X||S) so that previous results can be extended into the information bottleneck framework, which to the best of our knowledge is novel. More discussions on our theoretical contributions are provided in Section 4.1 in page 5 and the remark on more discussions on Lemma 1 and Theorem 1 in page 14.

Q4: The structure of the paper. 

Thanks for the suggestions. We will add more descriptions in the paper accordingly.

Typos in Section 4.4.

Thanks for the suggestions. We will revise the paper accordingly.

[1] Poole, B., Ozair, S., Oord, A. V. D., Alemi, A. A., & Tucker, G. “On variational bounds of mutual information”. ICML. 2019
[2]Matias Vera, Pablo Piantanida, and Leonardo Rey Vega. “The role of the information bottleneck in　representation learning.” In 2018 IEEE International Symposium on Information Theory (ISIT), 2018.


We would like to thank you for the valuable comments and suggestions. We provide more discussions on the concerns below:

Q1: Comparison with other methods. 

Currently for supervised odometry learning methods, there is no consensus on the performance measure and even data split. We followed the experimental setup of Chen el al. [1] and thus compared with the methods re-implemented in Chen el al. [1]. Different methods target this problem from different aspects. For instance, Xue et al. [2] proposed a memory and refinement module based on the deepvo backbone to account for drift errors, which is compatible with our method, where we aim at providing a better backbone. It is an important future work to incorporate all proposed improvements and achieve the performance limit, which is non-trivial since the source codes and results of most supervised works are not released.


Q2: The effect of the potential bias introduced by using estimated pose rather than the ground-truth pose during inference.

This is a very good point. We will conduct an extra experiment to study this effect by comparing results from both using the estimated pose and ground-true pose during inference. 

Q3: The difference between original results and the results of re-implementation. 

We follow the experimental setting in Chen et al. [1]. And our reimplementation of DeepVO/ VIO, SoftVO/VIO, HardVO/VIO perform better on KITTI compared with Chen’s (re-)implementation, while worse on EuRoC. Since the authors did not release either the source code or results, it is difficult to provide a comparable trajectory with their results. We will prepare trajectory visualization of our own results as a reference.

Typo 1 in page 5.

Thanks for pointing this out. We will revise the paper accordingly.

Typo 2 in page 12.

The Markov chain o->s->\xi can be regarded as an assumption in IB and is what IB tries to achieve, while \xi->o->s is naturally implied since s is extracted from o, which means o contains all the information of s and thus \xi and s are independent given o. We kindly refer to footnote 3 in page 10 in the original paper of information bottleneck [3] for more discussions. 

[1] Chen, Changhao, et al. "Selective sensor fusion for neural visual-inertial odometry." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019. 
[2] Xue, Fei, et al. "Beyond tracking: Selecting memory and refining poses for deep visual odometry." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019 
[3] Tishby, N., Pereira, F. C., & Bialek, W.  “The information bottleneck method.” arXiv preprint physics/0004057. 2000


Q2: “Odometry is challenging due to the difficulties to model the complexity and diversity of real-world scenarios from a limited number of on-board sensors” -> What exactly does that mean? E.g. is it that most sensors have peculiar noise distribution not well-modeled by Gaussian, and yet oversimple models are often used?

The challenges lie in the underlying assumptions of each method tailored for each sub-problem, including point correspondence modeling, noise modeling, measurement integration, scales for monocular methods, etc. We would add more references to refer readers to more complete discussions of this field.

Q3: Can odometry be accurately described as a “time prediction” problem? Would it not be better understood as interpolation, or maybe reconstruction / regularization / smoothing? How does the proposal follow a strict filter-based approach? And drift error?

In this work, we follow the terminology used in learning-based odometry methods and focus on providing initial relative pose estimates from an image sequence, without including a backend which is crucial for the overall drift error. If we consider a full system that incorporates a backend, either filtering or optimization, then the back-end can be interpreted as interpolation, smoothing, etc. We can regard the RNN as an intrinsic filtering counterpart in the sequence prediction problem, though a real back-end would definitely benefit the final results.

Q4: Integrating a motion model

We agree that a well-defined motion model for a certain scenario can benefit real-world applications, though it remains non-trivial to apply such models for each scenario. For geometry-based methods, since the front-end is usually formulated as an optimization problem by minimizing the reprojection error or the photometric error, motion models could provide initial values at this stage. For deep learning-based methods, the results of motion models can also be integrated into the network in multiple ways, such as serving as context information or as a prediction prior s.t. we only need to predict a residual for it.

Q5: Parameter tuning

It is a great question and we will tone down our description. Parameter tuning is annoying for both geometry and learning methods, though in different senses. For geometry methods, we need to determine the method to use, selection criteria of keyframes, densities of landmarks to check, etc., while for learning methods, what matters are network design, training scheme, loss weights, learning rate adjustment, etc. 

Deep learning based methods do require non-trivial tuning for those (hyper) parameters. This is also a problem of deep learning methods. Frankly speaking, parameter tuning in deep learning can be even more annoying in practice than in geometry methods, due to the long training time and computation capability required, which takes days for a GPU to finish training even on a relatively small dataset like KITTI. While within the same amount of computation source and time, we can do many more trials for geometry methods. Pretrained flownet features are commonly used to relieve this training burden, which however limits the capability of the network as a negative result. Moreover, it seems that training tricks matter a lot for odometry learning methods to really perform, from which we also suffered a lot when trying to reproduce results of current supervised methods which mostly are not open-sourced.

Collecting ground-truth labels in new environments presents another challenge for supervised learning methods. Unsupervised methods can relieve this problem though at a sacrificed accuracy. Training in simulated environments also holds potential for this problem, while the effectiveness of current domain adaptation techniques used in odometry need to be verified towards this direction.

[1] Wang, Sen, et al. "Deepvo: Towards end-to-end visual odometry with deep recurrent convolutional neural networks." 2017 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2017 
[2] Xue, Fei, et al. "Beyond tracking: Selecting memory and refining poses for deep visual odometry." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019 
[3]  Chen, Changhao, et al. "Selective sensor fusion for neural visual-inertial odometry." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019. 
[4] Yang, Nan, et al. "D3VO: Deep Depth, Deep Pose and Deep Uncertainty for Monocular Visual Odometry." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020. 
[5] Engel, Jakob, Vladlen Koltun, and Daniel Cremers. "Direct sparse odometry." IEEE transactions on pattern analysis and machine intelligence 40.3 (2017): 611-625. 
[6] Chen, Changhao, et al. "A survey on deep learning for localization and mapping: Towards the age of spatial machine intelligence." arXiv preprint arXiv:2006.12567 (2020).


Summary

The paper addresses the odometry problem i.e. estimating the relative pose of a camera or device as it moves through the environment. The authors propose an information-theoretic model for visual and visual inertial odometry. The method is based on the hypothesis that features contain pose-irrelevant information that leads to overfitting which is alleviated through the using of an information-bottleneck (IB) objective. The information-theoretic formulation allows them to use existing results to provide guarantees related to generalization. The experiments on KITTI and EuRoC indicate that the IB objective does help to improve performance

Strengths
- The paper provides a principled information-theoretic formulation for learning-based VO. The observation that the features learned by a typical VO model might contain pose-irrelevant information that hurts generalization is sensible and does happen in practice and thus the proposed approach is very well motivated.


- The generalization bounds for the problem are theoretically interesting as overfitting is quite a significant issue with learning-based VO models. 


- The empirical results in terms of effectiveness of the IB objective are strong. Specifically, the results support the effectiveness of the proposed IB-based loss with respect to the generalization ability of the trained model and the proposed method outperforms all the existing methods in terms of accuracy.


- The intrinsic uncertainty obtained from the model seems to be well calibrated and can be used to infer anomalies such as missing or noisy data and are roughly predictive of the translation and rotation error.

Concerns:
- The derivations are not that clear and seem to skip steps making them a bit difficult to follow. As an example, how is Equation 13 obtained? According to my derivation the bound for $I(o_{1:T} || s_{1:T}|\xi_{1:T})$ should be $E_{o^m_{1:T},\xi_{1:T}} \left[D_{KL}\left[p(s_{1:t} | o^m_{1:T}, \xi) || q(s_{1:t} | o^m_{1:T}, \xi_{1:T}) \right]\right]$. Also, how does Corollary 1 follow from Theorem 3? It would be good if the authors could put all steps in the derivations. 


- A lot of emphasis is put on the theoretical guarantee of the generalization ability, however, it is not mentioned how tight this bound is or what practical implications it has. It would be good if the authors could demonstrate how this bound can be used to in practice.


- A lot of the contributions in the paper seem to be based on (Zhang, 2018) and (Xu & Raginsky, 2017). Although I recognize that applying these to the VO domain is not trivial, it would be good if the authors could better highlight what their contributions are. 


- Overall, the structure of the paper is a bit difficult to follow. It would be better if the key results relating to the odometry problem being addressed are presented in the paper with a description of how these apply to the problem. The existing results can be put in the appendix. 

Minor points

In Section 4.3 the authors introduce Theorem 2 but then only refer to Theorem 4. Is this a duplicate?

There are some grammar errors that need ot be fixed throughout the paper. For example, “Besides,” is used incorrectly in a number of places where the authors probably mean “In addition,” e.g. section 4.4


Pros:
+ This paper proposes a novel solution for learning odometry by using an information bottleneck constraint 
+ This paper is well-written with clear derivation.

Concerns: 
The main issue of this paper is the insufficient experiment to validate the advantages of the method.
- The main weakness is the lack of the comparison with the state-of-the-art methods. The comparing methods, especially visual odometry methods are few. The experiment of visual odometry only compares with DeepVO [1]. However, to  the best of my knowledge, the visual odometry methods including non-learning methods (e.g. ORB-SLAM2 [2]), unsupervised methods (e.g. Bian et al. [3], Li et al. [4]), and the more recent supervised methods (e.g. Xue et al. [5], Xue et al. [6]) are not compared with the proposed method. Besides, the plots of odometry trajectory are preferred to present in the paper for demonstrating the qualitative comparison. One of the reasons is to visualize the drift of the odometry. The ground-truth pose \xi_t serves as the input of the deterministic function f^p in the training phase. But the estimated pose is used in the test phase instead. This would somehow cause the drift due to the biased estimation of the previous pose. However, the RMSE of the entire trajectory is insufficient to examine the drift.
- The comparing state-of-the-art methods are re-implemented and share the same network architecture. I am curious about the difference between the original results and the results of re-implementation. Thus, the trajectory plots would be a kind of evidence for demonstrating the re-implementation. Besides, I would like to see the quantitative results of both RMSE and absolute error, which is provided by Chen et al. [7]. The absolute error could be a reference to the original results of Chen et al. [7].

Minor: typos:
- \xi = g(X, \Theta) is "an" one-to-one function instead of "a" on page 5.
- The formulation of information bottleneck implies that "o \to s \to \xi" forms a Markov chain, instead of "\xi \to o \to s" on page 12 (Appendix).

Reference: 
[1] Wang, Sen, et al. "Deepvo: Towards end-to-end visual odometry with deep recurrent convolutional neural networks." 2017 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2017.
[2] Mur-Artal, Raul, and Juan D. Tardós. "Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras." IEEE Transactions on Robotics 33.5 (2017): 1255-1262.
[3] Bian, Jiawang, et al. "Unsupervised scale-consistent depth and ego-motion learning from monocular video." Advances in neural information processing systems. 2019.
[4] Li, Shunkai, et al. "Sequential adversarial learning for self-supervised deep visual odometry." Proceedings of the IEEE International Conference on Computer Vision. 2019.
[5] Xue, Fei, et al. "Local supports global: Deep camera relocalization with sequence enhancement." Proceedings of the IEEE International Conference on Computer Vision. 2019.
[6] Xue, Fei, et al. "Beyond tracking: Selecting memory and refining poses for deep visual odometry." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019.
[7] Chen, Changhao, et al. "Selective sensor fusion for neural visual-inertial odometry." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019.

The authors seem to have great knowledge of interesting information-theoretical tools that can be used to analyze the performance of pattern-recognition methods, and guide the design of model fitting procedures. There are however very important details regarding the application being studied that are not well-discussed which are fundamental to establish the relevance of the paper. The main contributions of the paper are also not well defined or demonstrated experimentally, making it mostly a paper with apparently interesting theoretical foundation that lacks clear and compelling experimental evidence or application appeal. The recommendation is unfortunately not to publish.

The first issue that can be pointed out regards what is even the problem being attacked. From the start the authors present the problem as one of odometry, what means fusing and integrating the readings from sensors such as IMU, wheel counters or images and provide relative pose measurements that can be accumulated over time. Odometry has no motion model or global localization sensors. A motion model would imply tracking or localization, and the ability to make pose predictions instead of strictly relative pose predictions implies (global) localization.

Naturally these different tasks and available information all relate to each other in different ways, and it gets more fuzzy when we get closer to practical application. Being able to perform tracking and obtain better pose estimates may be a good thing in an application, and why would any engineer stick to pure odometry if it were possible to employ a better solution?

A scientific paper must make great efforts to elucidate these differences, though, and it would appear the authors have not been careful enough here. The confusion might be shared with other recent authors bringing deep-learning into geometric vision and mobile robotics. All these authors must be urged to more carefully look at what is being claimed, and at what exactly different models can do, and what exactly are the benefits and costs of the proposed methods to potential applications.

The following two sentences from the start of the paper may be a good illustration of the issues being raised here:
 1. "Odometry is challenging due to the difficulties to model the complexity and diversity of real-world scenarios from a limited number of on-board sensors."

What exactly does that mean? For example, is it that most sensors have peculiar noise distributions now well-modeled by Gaussians, and yet oversimple models are often used?

Regarding 6DOF odometry, this is from the start a pattern recognition problem with complex geometry. From its very nature the problem involves challenges such as how to even represent poses and deal with the non-euclidean spaces involved, and then how to deal with the ill-posedness of unavailable information, the integration of measurements, scale invariance of monocular vision, etc.

Are the authors referring to these classing great challenges of the problem, or are they focusing on the even greater challenge of taking into account time-variant systems and dynamical environments? If the former is the case, these greater challenges might better go unspoken of. Just focus on the more fundamental issues first. It's understandable that the proposal might eventually contribute to the greater challenges. It's important to convey the message of where the current contribution lies first, though.


 2. "Furthermore, since odometry is essentially a time-series prediction problem, how to properly handle time dependency and environment dynamics presents further challenges."

Apart from the previous note regarding the focus of the proposal and dynamic environments, etc, this sentence demonstrates some confusion regarding the understanding of the nature of the odometry problem. Can odometry be accurately described as a "time prediction" problem? Would it not be better understood as interpolation, or maybe reconstruction / regularization / smoothing?

Another important detail not mentioned in the paper is how the proposal follows a strict filter-based approach, while in general odometry might be performed with regression applied to a longer sequence of observation, as done in OKVIS. Would it be fair to say that
the latent state learned by the proposed technique can somehow encode some of the information that is retained in the alternative method by not following a strict filtering scheme. This is the kind of question that would be interesting to see investigated here, and the authors do not seem very aware of that.

Even more relevant is the issue that odometry in principle merely integrates physical measurements, without a motion model. In practice one might argue that there's some modeling regarding the connection between inertia and accelerations. There should be no exploitation of biases such as a mean height, for instance, or upright constraint, or the fact that a car moves mostly straight-ahead and following the dynamics of Ackermann steering, or whatever. These are all constraints that can be taken into account in a tracking system, and they might greatly benefit the tracker's performance in an application. It's very interesting research. Such a tracking system should not be directly compared to a stricter odometry technique, though, since there are fundamental differences in what kind of data can be taken into account, and what is being modelled --- sensor noise, not motion.

By approaching the problem as one of pose prediction in general, it might be possible to go even further. The PoseNet paper cited performs precisely what we might perhaps understand as a complementary problem to odometry. Given a sensor reading, come up with a global pose estimate. This is excellent, and certainly relevant to many applications. It is concerning, though, that anyone may be using techniques capable of that, and then compare it to odometry systems. In especial because odometry is more or less fated to contain drift error that will accumulate in time, and a global localization technique will therefore eventually seem infinitely superior.

Any occasion where learning techniques are employed for strict odometry, or even tracking, it is paramount to verify that there is no way the model is picking up on clues in the environment to improve on its pose predictions be performing global localization, and not relative pose estimations. And this may be very challenging. Take the Euroc challenge for instance. The occurrence of high z-axis acceleration may serve as a clue for especially high height values to be predicted. It's just bias that is naturally in the data. This sort of bias must be guaranteed to be removed, either through constraints in the model itself, of for instance through data augmentation (much less desirable). The author cite PoseNet as an inspiration, and do not seem to worry that there may be characteristics in their system that fundamentally go against this being a research on odometry, a term utilized all the way from the title of the paper.


Apart from these issues regarding the nature of the research, and its motivation, there is another related concern with what was achieved. Taking again another paper sentence to motivate the discussion:

"Their hard-coded systems require extensive parameter tuning under different environments"

And deep learning requires no extensive parameter tuning?

One great thing about learning methods is precisely that they can be tuned to specific applications to improve their performance. The authors seem to be proposing something different than that, though, that a model would be trained that will exhibit great performance with no tuning. This would maybe replace the geometry processing core from alternative methods, with rotation encoded somehow in the latent state, and then different sensors would require just training part of the complete model. Is this really what is being sought? Is it interesting? And how do we test that this precise goal has been reached, opposite to a system that requires extensive parameter tuning under different environments?

Methods such as OKVIS require things like camera calibration and inertia measurements. How does the proposed method differ from that? Is the research somehow related to the possibility of a self-calibrating system?

These are all important questions that should be discussed in the experiments, although in the end the authors seem only to care more about how they compare to other learning based methods in different benchmarks, and then with the issue of detecting failures. This is indeed a very interesting thing, even with practical appeal. If this is the main point of the research, though, it should be given much more attention, from the start, and then more comparisons with alternative methods should be made with attention given to that specific issue, etc.

And is this what the authors mean by dealing with time variance? Fault detection? It certainly makes sense, and it's interesting. It would be nice to make it clear this is what is being considered, from the beginning of the paper.

The authors should be advised to try to make a more clear case regarding what is the application they have in mind, and how their method compares to others, and prepare a paper that attacks a more specific problem, concisely. Good papers have specific issues that are addressed, discussed from start to finish, preferably on top of solid theory. The authors seem perfectly capable of producing this with some more discussion and work put in the research.


Meta-review: This paper performs visual odometry using variational information bottleneck. It assumes video and pose observations and aims to find a latent state that is maximally predictive of the pose observations, while minimizing the mutual information between the image observations and the latent state. It approximates this cost using variational inference. The paper also makes use of deterministic+stochastic latent transition models, as in Hafner et al 2019, 2020. The paper contributes generalization bounds that are based on recent work by Xu and Raginsky 2017 and Zhang et al 2018. This is useful to include, but this contribution is not the main focus of the work in my opinion, and it is not clear how tight the bounds are, especially considering that the original cost function is approximated. 


Pros:

The idea of using the deterministic+stochastic transition models of Hafner et al and related works for visual odometry is very interesting and promising avenue for research. 

The fact that the experiments are done on some of the major camera and IMU datasets is great.



Needs fixing:

Major: The paper mentions "Extensive ablation studies were conducted to examine the effects of (1) the deterministic component, (2) sample size and (3) extra sensors.". These are great, but I would also have expected to see a range of variations in terms of the weight gamma, including a value of zero. The appendix is vague in this regard and says " and
perform a non-intensive and small-range grid searching." The utility of the information bottleneck idea depends heavily on the weight gamma, and I would have liked more results confirming that the method does well under a range of choices for gamma.

Major: The rotation results produced by this method on EuroC should be improved to be more competitive with okvis. As it stands, it is unclear why the method does not perform as well and the explanations offered in the paper are speculative.  
 
Medium: The paper also mentions in the appendix: "Though 3D von Mises-Fisher distribution and 4D-Bingham distribution can be
arguably more appropriate to model Euler angles and quaternions respectively, it is non-trivial to evaluate and use them for training in practice." So, the paper represents rotations using Euler angles. The authors are encouraged to look at https://www.gilitschenski.org/igor/publication/202004-iclr-deep_orientation_uncertainty_learning/

Minor: MSCKF was originally coined by Mourikis and Roumeliotis https://ieeexplore.ieee.org/document/4209642 and even though the term is used by follow-up works,  it is worth adding the reference.

Finally, I would disagree with one of the reviewers that this paper needs to compare with ORBSLAM2. I think comparing with OKVIS and an MSCKF variant is sufficient for "classic" SLAM and odometry methods.  

I think the paper needs one more iteration to fix these issues, even though it is very promising work.

    

Decision: withdrawn-rejected-submissions