Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a data-driven estimator for the odometry estimation of a legged robot from IMU data. In particular, the authors propose to learn a 1-D CNN to predict the displacement vector from a history of IMU states. They demonstrate the benefit of their learning-based odometry estimation by combining it with odometry from leg kinematics in an EKF estimator and in a factor graph estimator with a visual blackout. The proposed method improves relative pose error by 37 % in challenging scenarios when compared to traditional kinematic-inertial estimators.

strengths_and_weaknesses: Comments:
1. The proposed method is quite simple and elegant and seems to substantially improve the results compared to the state of the art. Using a learned prediction model to predict the displacement vector from buffered IMU data is a good place to leverage data-driven priors
2. Section 6.2, 6.3 and 7.1 do a great job of extensively evaluating the proposed method with comparisons to all the relevant non-vision baselines. in particular, the real-world evaluation is quite convincing.    
3. Although I understand that the scope of the paper is limited to improving odometry with missing vision/lidar input, it is unclear to me how practical is a setup with completely missing vision. For example, it would be interesting to see the effective improvements when a learning-based Visual Odometry method [A] is also adding estimates to the factor graph from very noisy lidar/perception inputs. I suspect the gains would not be as much in that case. 
4. It would be interesting to see ablations that plot the pose error as the history length of IMU changes, and also see how the performance changes with the loss function used in [5] instead of the huber loss function described in Section 4.1 (eqn 7). 

[A] Wang, Sen, et al. "Deepvo: Towards end-to-end visual odometry with deep recurrent convolutional neural networks." 2017 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2017.

summary_of_recommendation: Although, I have some general concerns related to the specific setup of completely missing vision sensors, and the effective benefits of the proposed method with learning-based visual odometry which can potentially handle very noise perceptual inputs, the proposed method will still contribute to improved odometry since the contributions of the paper are complementary to my concern. That setup might reduce the performance gains of the proposed method, but my concern is more to understand what problems practically occur in the field of odometry estimation. I think the paper is quite strong in empirical demonstration of its results. I am inclined to accept the paper.     
issues: One modification that would help in understanding the different components of the proposed method would be to include ablations that plot the pose error as the history length of IMU changes and also see how the performance changes with the loss function used in [5] instead of the Huber loss function described in Section 4.1 (eqn 7). Although this is not critical for the acceptance of the paper, this should definitely be included in the final version of the paper.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes a method to estimate the position and orientation of a legged robot by combining the IMU data, the robot kinematics and visual observations. A focus of the paper is on a drift-free inertial odometry that uses a recurrent neural network. Another is on the application of the factor graph in order to exploit observations of landmarks that might be intermittently available. The method is effective particularly in environments where firm contacts between the robot feet and the ground are not available and the vision guidance is hardly expected due to bad optical conditions.

strengths_and_weaknesses: It seems reasonable to apply the neural network to the inertial odometry that might be able to lower the effect of sensor drift as well as the accumulation of numerical errors, so that a more robust estimation than a naive integral of the acceleration is expected with it. On the other hand, it is suspected that the method reduces the accuracy if the sufficient number of training data under sufficient variety of situations are not provided. The results in Table 1 pretty matches this prediction, and thus, is not surprising for the reviewer.

The technique around the factor graph is not clearly explained. Although the reviewer understands that the original technique by the authors is the use of learned transitions of the robot states (magenta lines in Figure 3), he/she does not understand why it connects every other nodes. It would be strongly suggested that the authors carefully explain how the graph is constructed and how several hypotheses of transitions between nodes including that based on the learned process are combined.

Regarding the results in Figure 7 (with a view of the supplement movie), the reviewer can hardly justify the claim that FG-VIKD is better than FG-VIK since the closeness of the latest positions of each trajectory to the ground truth seems to be frequently switched.

summary_of_recommendation: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
issues: To clearly explain the technique around the factor graph.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper introduces a neural network for state estimation of a legged robot. The NN outputs positional displacements and their corresponding uncertainty based on raw IMU data. These displacements are then integrated into prior methods for state estimation with multiple sensors to output a more robust pose (with reduced sensor drift). Experiments were conducted for challenging terrains (like slippery or soft surfaces) and in a real-world mine environment.
strengths_and_weaknesses: Integrating a learned approach for state estimation is a powerful idea that might improve the handling of challenging situations. However, the experimental results show that the proposed IKD methods perform worse than the default TSIF method. Nonetheless, the learned method improves the corresponding approach without learning, however only with very similar training and test data distribution. We think that the paper does not put enough emphasis on the limitations of learning: a possible distribution mismatch and the (probably constrained) generalization capability compared to a model-based approach.
summary_of_recommendation: The paper has an interesting idea, however the evaluation and discussion should be extended to support the claims (e.g. “our approach greatly reduces the drift of ...” in the abstract).
issues: - We conclude from the experimental results that the proposed method performs worse than related work: For example, in comparison between the proposed EKF-IKD and the default TSIF state estimator, the proposed method performs 40% worse on average (RPE) in the lab experiments (Table 1). The abstract states a reduction by 37% in comparison to a traditional kinematic-inertial estimator, but doesn’t that include TSIF? Apart from the drift of the z-axis, the TSIF also shows qualitatively better results in Fig. 5 and 6. Why are the best results of the factor graph-based methods highlighted, although they are just different methods for solving the same task? We strongly suggest adding a baseline for the mine experiment.
- While it is not inherently bad that the results are not better than related work, we think that the paper should acknowledge and might discuss reasons for that. 
- Again, we think that the discussion does not emphasize the obvious disadvantages of the learned approach. We suggest adding experiments for out-of-distribution terrains (like surfaces with different softness) to show the robustness of the proposed work. In particular, the robustness is a key motivation of your work (and arguably very important for real-world usage of your state estimator). In robot learning, we find it very important to get a feeling about the limitations of the approach. Why doesn’t the learned approach fix the drift in the z-direction completly?
- Why doesn’t the NN get the joint states q as input? We think that the NN could detect (and then correct) challenging situations in particular based on a mismatch between IMU and joint data.
- We suggest clearing up Section 3. From our perspective, there is no real “problem statement” or task in there. What should be minimized or calculated? The biases b are not used furthermore (as they are learned). Between which points of time are the position increments d_i defined?
- Fig. 7 (left side) should be cleared up, e.g. by combining all three trajectories into one figure. As no start point is mentioned, it might make sense to shift and rotate the trajectories by a constant offset first.
- The title of the tables should be above.
- To make room for additional discussion: Fig. 5 and 6 could be made smaller and combined into one, and Section 2 and 5 could be shortened.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a learned state estimator that can be fused with kinematics and IMU measurements as well as factor graphs to estimate robot state over long experiments, without significant drift. Authors present experiments in a mine where the camera image clearly shows a lot of noise that would make estimation hard using visual features. In such cases, the learned estimator can augment visual estimation reduce the robot state estimation drift.
strengths_and_weaknesses: Strengths:
1. The paper is well-written and the use of learned models especially in the presence of large visual noise is well motivated. 
2. The robot experiments are well-conducted and run over long episodes, increasing the confidence in the performance of the estimator.
3. The paper presents ways of using the proposed approach with both extended kalman filters that take IMU and IK data, as well as factor graphs that can take visual data as input. This makes the method versatile and easy to adapt to new problems.

Weaknesses:
1. The proposed approach requires re-learning of the estimator on each terrain. This is a major shortcoming of the approach as the major advantage of a learned estimator would be generalization to unseen terrains, where the parameters of baseline filters cannot be hand-designed. There is no discussion of this in the paper.
2. The proposed approach only estimates the x,y,z position of the robot, not the orientation. This is also a major shortcoming which is not discussed. I would suspect that in situations such as slipping, the orientation estimate goes inaccurate, but no data is provided.
3. While the paper compares their approach against Pronto which uses IMU and IK information, they do not compare against any learning-based baselines. Since the proposed approach learns an estimator on the ground profile it is tested on, but the baselines do not, the comparison is not fair. Some learning based baselines that might make experiments more compelling are [a], [b], [c]. A simple baseline would even be to learn the dynamics model used in the EKF.
[a] Liu, Wenxin, et al. "TLIO: Tight learned inertial odometry." IEEE Robotics and Automation Letters 5.4 (2020): 5653-5660.
[b] Kim, Youngji, et al. "Unsupervised Balanced Covariance Learning for Visual-Inertial Sensor Fusion." IEEE Robotics and Automation Letters 6.2 (2021): 819-826.
[c] Teng, Sangli, Mark Wilfried Mueller, and Koushil Sreenath. "Legged Robot State Estimation in Slippery Environments Using Invariant Extended Kalman Filter with Velocity Update." arXiv preprint arXiv:2104.04238 (2021).

Detailed comments:
1. The introduction should clarify that the proposed estimator only applies to 3D positions and not orientations.
2. Eq 1 and Eq 2 are mixing discrete and continuous time conventions, and variables k, t
3. Why do authors use MSE to start and switch to GML loss? What was the motivation behind this choice?
4. Is each dimension considered independent is Eq 7? Is this a reasonable assumption, especially in situations like slipping? Can authors motivate this choice?
5. Why do authors ignore covariance estimates when predicting state in Eq 8? Could covariances be useful in fusing the estimates from filter and NN better than simply adding the displacements?
6. Section 5.2 is not very well described. What happens between visual events? Is the past estimate of state held constant or is the EKF run on proprioceptive data?
7. In Section 6.1, is the data of 2h16m per terrain or across all terrains? Is the same model fit to all terrains or one per terrain?
8. What is the frame of reference for experiments in Figure 5, 6, etc. Is y-movement forward?
9. The main advantage of the proposed approach is in keeping the z-direction drift. However, in the experiments the z remains mostly fixed (except on ramp, where TSIF outperforms the proposed approach). The NN could just be learning to keep the z fixed. Moreover, the performance does not look better in x and y directions.  More discussion on poor performance in x on soft ground would be useful.
10. How different is the mine from lab experiments? Though the video is impressive, overall the generalization to the mine terrain from lab is unconvincing. Why do authors use a vision-based approach here? How does EKF-IKD perform in this setting? This section needs to be motivated better and the choice of controller, approach, etc be described in more details.

summary_of_recommendation: The paper presents an incremental approach to estimation which involves a trained network predicts displacements in 3D positions based on IMU history. Similar ideas are present in literature on 6-dof pose estimates, like [1]. The paper could benefit from learning based baselines, recommended in the main review.
[1]  Silva do Monte Lima, João Paulo, Hideaki Uchiyama, and Rin-ichiro Taniguchi. "End-to-end learning framework for imu-based 6-dof odometry." Sensors 19.17 (2019): 3777.
issues: Main issues:
1. No learning-based baselines. Since approach trains a model on each terrain separately, it gives the approach significant advantage over baselines, making comparisons unfair.
2. only a 3D pose estimate, not 6-dof estimates of state presented in the paper.
review_title: null

Meta-review: The authors propose a learning-based proprioceptive state estimator for legged robots using IMU data.  Comparison of the proposed approach to EKF and a factor-graph based estimators is presented.  As pointed out by the reviewers, the authors should discuss why they choose to ignore using the kinematic information (joint encoders) as input to the neural network.  Comparison of the method with other learning-based estimators should be presented. Failures of the proposed method should be discussed, particularly inference on out-of-distribution terrains (relearning on each terrain may not be feasible). 

Having gone through the author response to reviewer comments and discussion, I am recommending an accept with poster.  I commend the authors on carefully addressing all the reviewer comments.
Decision: Accept (Poster)