Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper proposes a novel VIO method which estimates 5 DoF poes (with the elevation fixed) from monocular camera and IMU. In particular the proposed method adopt Unscented Kalman Filter (UKF) which is natually differentiable for the state estimation. With the aid of filtered pitch and roll using UKF, the front-view images can be converted to bird eye view images which are then used for pose estimation with Differentiable Phase Correlation. The whole method is differentiable and can be backpropagated using ground-truth pose supervision to update the covariance matrices. Evaluation results show the the proposed method achieves competitive results on real-world KITTI dataset for odometry, CARLA and AeroGround datasets for map-based localization. The authors also conducted ablation studies on the learned covariance matrices.
strengths_and_weaknesses: ## Strengths
- The paper is very well written and one can read very smoothly.
- The authors make use of the differentiable nature of Unscented Kalman Filter (UKF) and proposes to learn the covariance matrices in a end-to-end manner to capture the motion and measurement noise distribution. This way it retains the interpretability of the pipeline and also leverage the power of data-driven approach to learn the otherwise hand-tuned covariance matrices.
-  The estimated roll and pitch from UKF is integrated seamlessly to the bird eye view projection followed by camera pose estimation using Differentiable Phsace Correlation. This enables the differentiability of the whole piple with reasonable assumptions.
- The results show very competitive results on KITTI for camera pose estimations. Further, the authors also show its effectiveness to apply it for map-based localization on CARLA and AeroGround datasets.
- The authors provide the code in the supplement.

## Weaknesses
- Altough the whole method only learns 4 parameters, it is a bit confusing why it needs 2 RTX 3090 to train.
- No runtime and memory statistics are provided.
- The authors did not show cross dataset generalization evaluation. It would be interesting to see if the covariances learned from one dataset/sensor can transfer to other datasets.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The authors propose a quite novel approach for 5 DoF VIO which leverages differentialble UKF filter and BEV projection. The learned covarinances are shown to be very effective and I believe such an approach could have a big impact to the community.
issues: It would be great if the authors could address my concers in the *Strengths And Weaknesses* section.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a visual inertial odometry system based on kalman filtering. The main idea of this work is to learn the noise filtering parameters by making the filter differentiable. While the idea of learning filters is not new (See [1]), the application to the problem of VIO from a bird-eye perspective is new to me.  The approach is evaluated against a set of baselines on the KITTI dataset, where it obtains comparable or better results.


[1] Krishnan et al,, Deep Kalman Filters
strengths_and_weaknesses: The main strength of the paper is the nice balance between end-to-end learning and interoperability. Learning only a few parameters end-to-end makes the system easy to debug and investigate. It is to be said, however, that similar ideas have been long investigated in the community (See [1],  [2] or many others), but none really took off. This could be because learning so few parameters makes the algorithm strongly reliant on algorithm priors, which might be inappropriate in several real-world applications.
A second interesting takeaway from this work is that traditional VIO methods, like VINS, perform terribly on KITTI. However, I find this somehow difficult to believe since, in the KITTI benchmark, most monocular systems are traditional VIO methods. Is this setup somehow different? What makes it challenging for them in the proposed evaluation? 
Finally, I find the idea of passing through the Bird-eye view quite interesting. I would have expected to be a suboptimal choice given the presence of other vehicles and humans, whose upper part is generally occluded. However, this does not seem to be a problem for the proposed system, which I find quite interesting.

The main weakness of the approach is, in my opinion, the evaluation setup and results. As mentioned before, the terrible performance of traditional VIO methods is difficult to interpret. In addition, I feel that the main claim of this paper (rely more on algorithmic priors than parameters) is not sufficiently justified. It would be interesting to see results on more challenging datasets, possibly in 3D, where the assumption on the motion model would not always hold. If the paper only wishes to specialize in AV scenarios, then such choices must be made clear from the beginning (abstract/intro) and claims weakened. The limitation section mentions that the approach would be applicable for all ground robots, but there is not enough evidence to support this claim. Overall, the main question I feel should be answered is: when is desirable to enforce algorithmic priors instead of parameters? 
Another weakness is the fact that previous work on deep state estimators (like [1],[2], and others) has not been sufficiently covered and discussed. In what does the approach differ? Why is it better? 
Finally, a (minor) limitation is that the approach is said to be lightweight, but computation time is not compared to existing methods.

[1]  Deep Kalman Filters, Krishnan et al.
[2] Differentiable Particle Filters: End-to-End Learning with Algorithmic Priors, Jonschkowski et al.

robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: My recommendation is based on the disparity between the claims and the provided results. In addition, the paper does not make a strong case on why this approach would be relevant in general ground robotic applications (AVs also have GPS, so why not use it?).
issues: I would be happy to see a more aligned introduction and experimental section in the revision. Also, I would like to see a discussion of previous methods and a clearer answer to the question: "When are algorithmic priors a good idea?"
review_title: null

final_score: rating: Strong Reject: I recommend rejecting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper proposed an interpretable architecture for VIO, which admits differentiable training. Experimental results on two domains demonstrate that the proposed approach is effective and competitive against recent deep learning-based architectures. 
strengths_and_weaknesses: Strengths: 

The architecture is novel, as far as I can tell. It also achieves very high performance. 

Weaknesses: 

Let me preface my comment by saying that I am not an expert on VIO, but come more from the general model interpretability side. With that said, I am quite disappointed about the lack of empirical evaluation of the model interpretability. While the model achieves comparable performance as deep models, which is refreshing to see as many interpretable models struggle to match the performance of their non-interpretable counterparts, it is not clear in what ways the proposed model is exactly interpretable. 

Demonstrating interpretability requires validating that people, developers and/or users, can gain concrete understanding of the model. Thus, at the very least, there should be more analysis on the model itself, besides that on model performance. Ideally, it would also show some specific aspects corresponding to the model understanding. For example, maybe the developer can identify some issues of the model so that they can re-train a better one, or that a domain expert can understand the weakness of it, so that they can choose when not to trust the model prediction. Without any of these demonstrations, I am not convinced that this model is any more interpretable than a standard neural network. 
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Due to the lack of experiments and analyses on the interpretablity front, I could not recommend this paper for publication. I believe that there could be two avenues forward: 

1. Remove all claims of interpretability, so that this paper's contribution is limited to a novel architecture for VIO that matches SoTA performance. Or better, 

2. Conduct experiments to validate interpretability, and position this model as an interpretable and high-performing one. 
issues: None.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This work introduced a novel visual-inertial odometry system, which is learning-based but interpretable without deep neural networks. Specifically, the IMU data is used by a differentiable UKF to estimate the roll and pitch, which are subsequently used to project images to bird-eye view (BEV). The BEV images are transformed to the frequency domain to estimate SIM(2) camera motion. The leaning part comes from the UKF covariance; the authors maintain the following BEV projection and motion estimation differentiable so that the odometry error could be back-propagated to the UKF covariance, which makes it learnable and the estimated roll/pitch optimal. The evaluation is mainly based on the standard KITTI dataset. The proposed method achieves great results.
strengths_and_weaknesses: Strength
- The paper is well-written, and the methodology is clearly stated and technically sound. It is a really solid work with significant novelties and contributions. It is refreshing to review a learning-based VIO work that is not based on deep networks. 

Weakness
- The unchanged height assumption for driving scenarios seems fine; however, the reviewer is slightly concerned about the sufficient distance assumption: what is the minimal distance? is it also related to the camera mounting height and angle? if there are objects in that yellow box, how will it distort the BEV images and decrease the VIO accuracy?
- The evaluation is mainly based on the KITTI dataset, focusing on sequences 09-10 for testing. The validation would be more persuasive if the authors include more VIO testing. 

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: It is a really solid work with novel method.
issues: - Line 121: ‘within the red dashed box’, do you mean ‘within the yellow dashed box’?
- In Section 2.3, the ‘scale’ term is slightly confusing. Since the proposed method is estimating SIM(2) motion, the reviewer assumes the ‘scale’ refers to the odometry scale instead of image scaling. If so, what is the reason to fix the scale to 1 (line 150)? fixing scale to 1 makes SIM(2) to SE(2), right?
- How do the authors handle the IMU bias terms? are they not modeled in the UKF?
review_title: null

Meta-review: metareview: The paper proposes a differentiable approach for monocular VIO estimation based on BEV, without relying on deep neural networks. The reviewers find the paper well written and the idea of using BEV to be interesting. This paper received highly mixed reviews. The major concerns raised by the reviewers include empirical evaluations of the model interpretability, justification for relying on algorithmic priors than parameters, results on more challenging datasets, positioning this work with respect to existing work on deep state estimators, and clarifications regarding the claims made, among others. Most of the concerns raised by the reviewers have been thoroughly addressed in the rebuttal. I thank the authors for the engaging discussions during the rebuttal. Some minor concerns still exist. Nevertheless, I agree with the reviewers that the paper is an interesting contribution.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)