Legolas: Deep Leg-Inertial Odometry
Authors: Justin Wasserman
Ananye Agarwal
Rishabh Jangir
Girish Chowdhary
Deepak Pathak
Abhinav Gupta
Venue: CoRL 2024 (2024)
Keywords: State and Odometry Estimation
Quadruped robots
Sim-to-Real
TL;DR: Legolas learns to predict odometry from simulation and succesfully deploys to multiple real-world quadruped platforms.

Abstract: Estimating odometry, where an accumulating position and rotation is tracked, has critical applications in many areas of robotics as a form of state estimation such as in SLAM, navigation, and controls. During deployment of a legged robot, a vision system's tracking can easily get lost. Instead, using only the onboard leg and inertial sensor for odometry is a promising alternative. Previous methods in estimating leg-inertial odometry require analytical modeling or collecting high-quality real-world trajectories to train a model. Analytical modeling is specific to each robot, requires manual fine-tuning, and doesn't always capture real-world phenomena such as slippage. Previous work learning legged odometry still relies on collecting real-world data, this has been shown to not perform well out of distribution. In this work, we show that it is possible to estimate the odometry of a legged robot without any analytical modeling or real-world data collection. In this paper, we present Legolas, the first method that accurately estimates odometry in a purely data-driven fashion for quadruped robots. We deploy our method on two real-world quadruped robots in both indoor and outdoor environments. In the indoor scenes, our proposed method accomplishes a relative pose error that is 73% less than an analytical filtering-based approach and 87.5% less than a real-world behavioral cloning approach.
More results are available at: learned-odom.github.io

Legolas: Deep Leg-Inertial Odometry
Justin Wasserman 1.2 Ananye Agarwal', Rishabh Jangir', Girish Chowdhary' Deepak Pathak Abhinay Gupta" University of Illinois at Urbana-Champane Carnegie Mellon University
Figure 1:
Robot state estimation by Legolas across diverse environments and robotics plat- forms. Legolas operates at high frequency to predict small incremental motions of robot by only utilizing leg and IMU sensing. We introduce the first accurate, purely data-driven method for predicting odometry.
Abstract: Estimating odometry, where an accumulating position and rotation is tracked, has critical applications in many areas robotics as form of state es- timation such as in SLAM, navigation, and controls. During deployment of a legged robot, vision system's tracking can easily get lost. Instead, using only the onboard leg and inertial sensor for odometry is promising alternative. Previous methods in estimating leg-inertial odometry require analytical modeling or col- lecting high-quality real-world trajectories to train model. Analytical modeling is specific to each robot, requires manual fine-tuning, and doesn't 't always capture real-world phenomena such as slippage. Previous work learning legged odome- try still relies on collecting real-world data, this has been shown to not perform well of distribution. In this work, we show that it is possible to estimate the odometry of a legged robot without any analytical modeling real-world data collection. In this paper, we present Legolas, the first method that accurately esti- mates odometry in purely data-driven fashion for quadruped robots. We deploy our method on two real-world quadruped robots in both indoor and outdoor en- vironments. In the indoor scenes, our proposed method accomplishes relative pose error that is 73% less than an analytical filtering- approach and 87.5% less than a real-world behavioral cloning approach. More results are available at: coirned-odom.githhb...
Keywords: State and Odometry Estimation, Quadruped robots, Sim-to-Real
Eqqul Advising
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Introduction
Many robotics tasks require tracking the state of the robot, one of these states that is commonly tracked is the pose of the robot. Doing so with odometry requires estimating and accumulating small incremental ego-centric displacements of the robot at a high frequency. Within robotics, these estimates are used in many downstream tasks. An example application is within SLAM, where odometry estimates are used motion estimates improve mapping
[1]. It is also common in controls to use odometry estimates as source of feedback [2, 3]. Odometry has applications across various fields where fast tracking is required. Some notable examples include VR [4], pedestrian motion estimation [5], and autonomous car navigation [6].
typical deployment robot will often incorporate visual components into its odometry [7, 8, 9, 10J. These components are useful for creating a map to reduce drift and to assist when other sensors fail. However, in many cases, a quadruped will need act in an environment where it is required to perform quick movements such going up and down stairs or dodging obstacles. These quick motions cause the visual component to lose tracking and cause the odometry system to fail. During these failure modes we seek to estimate odometry throughjus, the proprioceptive and inertial sensing that are ubiquitously found onboard quadruped platforms. Fig. 1 and the accompanying videos show examples of our robots estimating odometry across many environments.
Previous methods for solving leg-inertial odometry can be split into analytical and learned meth- ods. Analytical methods for estimating odometry on legged robot often require direct sensing or estimating the foot contact [1 11 12, 13, 14]. However, foot contact sensors are not always available on robots, and estimating the foot contacts can be challenging due to modeling the robot's interac- tions with the environment such as slippage, and deformable surfaces. Due to the analytical models only being correct for a studied robot, immediate transfer to a different robot configuration may not be possible. On the learning-based side, recent methods have shown that it possible estimate odometry by training model from real-world data [15, 16, 17]. However, collecting a diverse and large dataset of trajectories with high-quality labels is challenging. Challenges include making the training environments diverse, limitations of devices that can be used to obtain the target pose, and time to collect the dataset. These systems have also previously relied on analytical models and utilized the prediction in their update step rather than only using the learned estimate.
In this work, we introduce Legolas, or Legged odometry learned as simulated as purely data- driven solution for solving leg-inertial odometry. In contrast to analytical filtering methods and recently developed learned odometry prediction models, our proposed work does not require analyt- ical modeling such as sensing modeling, kinematic modeling, or foot contact estimation.
This work builds off the recent success robot locomotion and navigation [18, 19, 20, 21J in sim-to- real transfer. These works demonstrated that complex plamning. mapping. and analytical modeling not necessary for deploying their methods to real-world platforms. We extend this line of work by showing that predicting the robot's odometry can be successfully learned from simulation first step toward sim-to-real transfer of explicit state estimation. Through simulation, months' worth of trajectory data across diverse scenes are collected in single day. After training, direct sim- to-real transfer of odometry prediction on the Unitree Go2 and DEEP Robotics Lite3 quadrupeds demonstrated. After training individual models for each of the quadrupeds, no additional manual tuning is required for deployment. This is contrast to many current filtering- and factor graph methods for predicting odometry that suffer from the "Curse of Manual Fine-Tuning" [22]. Furthermore, Legolas can predict odometry at high frequency previous filtering and learned baselines while being competitive with a visual odometry method in indoor scenes.
accuracy much higher than
2 Related Works
Filtering- Based State Estimators: An Extended Kalman Filter (EKF) is class of filtering- based methods that use non-linear analytical modeling of the robots' kinematics and sensors to estimate the

---

state of the robot [23]. EKF methods [24, 25] for tracking the state of legged robots are promising as the roll and pitch orientations are fully observable and drift-free on quadrupeds [26]. However, the models to build the EKF often assume that at least one foot is in contact with the ground at all times [12]. Additionally, for quadruped models, it often assumed that there no or little slippage on the floor [12, 27J. In contrast to these filtering- based methods, Legolas makes no assumption about analytical modeling, meaning that Legolas can be used train odometry model on any robot configuration. Furthermore, Legolas makes no assumptions about the environment as Legolas is trained in simulation, it collects data across many environment variations and configurations. Furthermore, filtering-basee based methods will often require or model the foot contacts of the robot [28]. As foot sensing is not straightforward to model [29], susceptible to damage through impact [30], and not all pedal robots have this sensor, Legolas does not require the use of this sensor.
Learning-Based Inertial Odometry: There is broad range of literature that is interested in learn- ing odometry through just an IMU [16, 31, 32] as IMUs are cheap and prevalent. These methods have been applied to applications outside of robotics such as consumer phones [4] and drones [33]. Work such as Cioffi et al. [33] apply this idea to robotics and have shown that it is possible to use only sensing from an IMU collected from real-world trajectories, in conjunction with model-based system to predict accurate odometry estimates. However, Buchanan el al. [34] has shown that these predictions degrade when the robot's behavior out of distribution from the training dataset. In contrast to these previous methods, as Legolas is trained in simulation, months worth of trajectories of the robots moving in diverse environments such as over stones, stairs, and inclines are collected single day without the need for analytical models. This allows us to train a robust odometry predic- tion model that is less likely to go out of distribution during deployment. Furthermore, in this work, we will show that predictions from just an IMU lead to degraded predictions, and demonstrate supe- rior performance by adding additional observations that are readily available on quadruped robots.
Robot Sim-To-Re Ro-Real Transfer: Sim-to-real transfer has been notably of interest recently due the promising scale of simulators, safety, and the difficulty in collecting real-world data on physical systems [35, 36, 37]. RMA [19] used a two-phase learning procedure to adapt the locomotion policy to unknown environment parameters. Agarwal et al. [18] extended the two-phase framework by first training with low-resolution scandots. Then in the second phase, supervised training from the first phase policy performed with depth images in the observation. Both this method and RMA were demonstrated to transfer to the real world on difficult locomotion tasks such as climbing stairs and moving through slippery environments without fine-tuning. Hoeller et al. learns state representation over a sequence of images and the trajectory of the robot in simulation to perform obstacle avoidance while reaching a goal. This method was shown to even dodge dynamic obstacles after transfer to the real world. Zhuang et al. [38] demonstrated successful transfer of highly precise and reactive parkour skills. Adding onto this line of work, Legolas shows a successful sim-to-real transfer of state estimation from simulation across multiple robot embodiments.
3 Legolas
The goal is to train a pose difference prediction model P that predicts small incremental motions of the robot at each time step. This model takes as input vector of size R containing previous actions and command velocities as well as IMU and proprioceptive sensing which are common on quadruped platforms. The IMU data corresponds the gyroscope (IR3) and roll and pitch estimates (R2 The command velocity is in the x, y, and angular directions 3). The proprioceptive sensing corresponds to the joint angle and velocity of each joint of size RR22 Finally, the previous actions are given as desired joint angles, this is of size 12 P outputs a prediction and variance vector of shape R 18 on how much the robot has translated (R3) and rotated as represented by a 6D rotation, Rs [39]. This representation was chosen as it was found to lead to the lowest error on the validation dataset as discussed in See. C. A variance is also predicted for each element of the translation and rotation component Rº. An additional head introduced to predict confidence (R-) whether the robot is moving to perform zero- velocity updates [40]. The connection between data collection and training is visualized in Fig. 2a.
3

---

A relative pose prediction error is utilized in the proposed loss that compares the predicted and ground truth incremental motions of the robot. A key insight into this loss as visualized in Fig. 2b, that this error should not be propagated to future predictions. This formulation allows for the creation of independent data points in the collected dataset.
In this section, we will first provide details dataset collection in Sec. 3.1. After this, will discuss the loss used to train P in Sec. 3.2 and finally training details in Sec. 3.3.
Data Collection
Training
Prediction Errors
OL..... Oj
P
π
Q
C2
(a) Overview of data collection and training Legolas.
(b) Relative prediction errors.
Figure 2: Legolas overview. (a) 13.7M data points containing a history the robot's observations and ground truth pose are collected from simulation. P takes in history of observations and outputs the change in the pose of the robot. In conjunction with the ground truth pose difference, an error is calculated. (b) The key idea behind the proposed error is to calculate the magnitude of the difference between the predicted and ground truth pose changes.
3.1 Odometry Dataset Collection
In the simulator Issac Gym [35], a trajectory is collected by using a robust locomotion policy â for n steps in order to ensure diverse dataset collection. The choice of this policy, whether it is learned or an analytical approach, for example, will be compatible with Legolas. Legolas only assumes that a given policy 7, that takes in the current observations and returns action a, is available both in the simulator and the real world. The simulation environment contains many scenes that a robot would encounter such tripping hazards, uneven surfaces, and stairs. Collected trajectories contain both the current observation Og and the ground truth pose of the robot. O contains the proprioceptive angle and velocity, IMU gyroscope and roll and pitch estimates, the previous commanded joint angles from â. and commanded velocity. While creating this dataset, thousands of robots are simulated to create trajectories in parallel. Data collection for both the Go2 and the Lite3 are the same. Details and visualizations dataset collection can be found in Sec. B.
3.2 Displacement Error and Training Loss
The error is calculated by comparing trajectories represented as a ks length sequence of SE(3) ma- trices where Q E 4x1x1 is the ground truth trajectory and P E Rxxx1x is the predicted trajectory. At a given timestep i, the ground truth pose is Q, E SE(3) and the predicted pose is P. E S3). The incremental ground truth and predicted motion from time step to is Q Q. and P. P, respectively. The pose difference, D E SE(3) between these incremental motions, defined in (1).
D = (Q
(I)
To formulate the proposed loss as independent predictions in the training procedure, the difference
in the predicted trajectory (AP) is estimated directly from P as shown in (2). AP is composed of AP, E IR3, the incremental motion the (ar, v) directions as well as a rotation component
APR E 1R.. (+)) is then converted into a translational and 6D vector, denoted as AQ.
D = AQ - AP
(2)
Training with just D as the error, i.e. e = ||D112 leads accumulation of drift while the robot is stationary. To counteract this masking head is introduced into the network. The purpose of the masking head is to learn to manage zero-velocity updates [40]. In practice the masking head performs binary classification on whether the robot is stationary, outputting m. 0 when stationary and m. = I when in movement. Therefore with the masking head, the final error is defined in (3).
m, |1 + (1 -m.) [[(Q)]]
(3)

---

The enror can be thought of as just 2 when the robot is predicted to be stationary. This error minimized when the robot is standing still and penalized when the robot is actually moving. The emoor otherwise equivalent to c e= || when the robot is predicted to be moving.
Training started with MSE loss shown in (4) and then switched Gaussian Maximum Likelihood Loss shown in (5) after training for sufficient number of samples.
N
LASE =
1
(4)
N
1=1
N
exp(-2 |le 1|s
N
1
LGML = N - log
log det (2,) 0,)+
+ 2 |le (e,13,)) + C'st
C'st
(5)
det
N
2
Where is the еπтог for given data point in a mini-batch of size N, and Σn is the covariance matrix predicted by P to represent uncertainty. Using the Gaussian Maximum Likelihood Loss improves the motion prediction and facilitates predicting variance. The variance allows for more interpretability of the output Legolas and facilitates downstream integration with SLAM systems. However, starting training with LMSE found improve P and reduce training instability. Further details regarding these losses can be found in Sec. C.
3.3 Training Procedure
n After the trajectories are collected as specified in Sec. 3.2, each trajectory is split into H+1 parts to form an offline dataset to train P. Where n is the length the trajectory and H is the num- ber of previous observation steps that will be given to the model. Fach data point contains the current observations of the robot as well as a history of the last H observations before it, (O--H,O:HHH1-- O;-1, O,}- These observations provide history for the model to improve the prediction of the odometry. Finally, each data point also contains the relative pose difference over the ground truth trajectory for the last step. Q
After collecting the ground truth observations, noise added to each sensor. For the Go2 dataset, 7M data points are collected, equivalent to 160 days of data collection. This took approximately 48 hours collect. During training, for each data point, P takes in the observations {0,-H ... 0t) and predicts incremental motion AP and covariance matrix Σ. During training and deployment, H = 50. This is equivalent to taking the last one second of observation history and predicting the incremental motion over the last 0.02 seconds. Consistent with prior work that estimates odome- try [15, 17], a ID ResNet architecture [41] is adopted for P. Further details can be found in Sec. B.
Experiments
4.1 Experimental Setup
Ground truth trajectories are collected on real-world quadruped robots. Ground truth information is collected with motion capture for indoor scenes and RTK-GPS for outdoor scenes. When neither is available due to their limitations, SLAM with a LIDAR [42] used to obtain the ground truth pose. Legolas deployed onto the Unitree Go2 quadruped and the DEEP Robotics Lite3 quadruped at 50 Hz match the simulation training speed. the following experiments, more focus is given the Go2 as baselines are more readily available for this platform.
We benchmark using sitering-bbsed, behavioral cloning, and visual baselines and variations: EKF [26]: Analytical Alter-based approach, stemming from the work Cerberus2.0 [25]. The kinematic modeling in the EKF is modified to match the Go2 rather than the Gol. Manual fine- tuning on its parameters performed [43].
BC: Trained by collecting trajectories the real world. These trajectories are then processed into dataset in the same manner performed in Sec. 3.3 create dataset of 90,000 data points.
Training also follows the same as Legolas with the same loss in Sec. 3.2.
5

---

Indoors
Outdoors
#
Method
RPE
ATE. 4
ATE,
ATE.
ATE, 4
1
EKF [26. 25]
0.38
0.116
0.062
0.23
0.07
2
BC
0.81
0.175
0,099
0.20
0.12
3
Ours w/ Gol
0.12
0.022
0.014
0.08
0.04
4
Ours
0.11
0.026
0.015
0.04
0.03
5 VINS-Fusion [7]
0.09
0.022
0.011
Table 1: Legolas prior method real-world deployment. In the indoor setting, Legolas achieves performance at a level comparable to a visual baseline and outperforms relevant leg-inertial baselines. However, in the outdoor setting, where the visual baseline fails to deploy, Legolas still surpasses the other leg-inertial baseline.
VINS-Fusion [7]: Visual-inertial odometry baseline. Additional hardware and manual fine-tuning was required for suitable deployment [44, 43]. See Sec. for more details. Has access stereo images, whereas Legolas and the other basclines do not.
Ours w/ Gol: Legolas, but instead of collecting training dataset in the simulator with the Go2, Gol is simulated and its respective स is utilized instead.
4.2
Metrics
The RPE and ATE metrics [45] are utilized as they are standard visual odometry and SLAM [46]. The RPE metric is more relevant odometry as this metric an average error of the odometry models across predictions. Both metries are utilized be consistent with relevant literature [15, 33].
RPE: Relative Pose Error is "local" error that ignores drift between Q and P. it is defined in (6).
RPE
Eynse
(6)
Where A corresponds to the index of the next pose in the ground truth trajectory that is some distance away from Q. In the following experiments, distance of I meter is utilized, therefore this metric reported as RPE@1m.
ATE: Absolute Trajectory Error is "global" error that penalizes drift and mismatches between the ground-truth Q and predicted trajectory P of lengths n. In this work, the root mean squared error of the ATE, which is defined in (7) is reported:
Σ ((OOSPP)))))
Eymse ==
Where t() retrieves the translation component of the matrix, S is transformation that aligns Q and P. and N is the number of steps in given trajectory. S is necessary the trajectories are not guaranteed to be immediately aligned. In this work ATE is utilized in two manners, the first is when a transformation that aligns the first frame of both trajectories. The second is when S is found using Umeyama's 's method [47] to find the best transformation align all points across both Q and P. These metries are referred as ATE, and ATE, respectively. In addition the above, the error normalized by the path length so the metric is not dominated by long trajectories.
4.3 Real- World Quadruped Odometry
Legolas and baselines are deployed in the real world across 11 indoor and 5 outdoor scenes. These scenarios cover a range of locomotion tasks on flat and hilly terrains across diverse trajectory varia- tions. In Tab. the metrics achieved by each method on both indoor and outdoor scenes are reported. In the outdoor scenes, the @PE 1m metric is forgone as GPS does not accurately report the heading.
Legolas achieves near vision-based performance indoor scenes (indoors rows 4 and 5). While Legolas (row 4) does fall behind VINS-Fusion for indoor scenes, methods with equal

---

ground -rrun
Ours
EKF
VINS-Pusion
you 150
150
30m
(s) 30
(a) Indoors trajectory tracking across all methods.
(b) Outdoors trajectory.
Figure 3: Indoor and outdoor trajectories collected Go2 quadruped. Across indoor and outdoor scenes, Legolas demonstrates successful deployment. (a) Legolas and
most closely track the ground-truth. EKF captures relevant shape but does not track the ground-truth. (b) Legolas produces accurate odometry estimation even in large environments. More full rollouts be found in Sec.
sensing, Legolas exceeds the baselines. When compared the EKF-based approach (indoors row 1), Legolas achieves 73% lower RPE@1m. Furthermore, when compared the real-world behav- ioral cloning approach, Legolas achieves an 87% lower RPE@1m.
In Fig. 3a, the predicted trajectory across methods is compared. Both VINS-Fusion and Legolas closely track the ground-truth trajectory. EKF does capture relevant shape but fails to closely track the ground truth over time. We find that this common error mode stems from foot contact estimation being required. However, this estimate comes directly from the foot contact sensor which is prone to noise and degradation [30]. We find that BC can successfully reproduce trajectories it was trained but fails to generalize novel trajectories. This is due to the low amount of data used to train the model when compared to Legolas. This phenomenon has been found in other works as well [34].
Legolas improves over baselines in outdoors dataset split (outdoors rows 1, 2, 4) The out- doors dataset split was notably challenging as there were variations in weather conditions and terrain. Despite this, in Fig. 3b Legolas tracks the robot in slippery environment, where the ground is cov- ered in rain. Across all baselines in the outdoor dataset split, Legolas improves
the baselines.
MNNS-Fusion was tuned for indoor environments and immediate outdoor deployment failed. More fine-tuning didn't improve performance due to brightness changes, using a downward-facinn cam- era, and quick motions. These failure modes are described in-depth along with an example of how Legolas can be integrated with VIO method to improve robustness can be found in Sec.
Legolas works across embodiment (indoors rows 3 and 4). While trained with different physical robot and policy than the one used in deployment, Legolas is capable of accurate odometry predic- tion indoor scenes. Notably, when trained with the Gol and deployed on the Go2, a decrease in ATE, from 0.024 to 0.019 and ATE, from 0.014 to 0.013 (indoor rows and 4) occurs. However, the more relevant metric for odometry, RPE@ 1m, increases from 0.11 to 0.12.
4.4 Simulator Odometry Prediction
In this experiment, validation dataset is collected in simulation to perform ablations on the obser- vations given to Legolas. This dataset consists trajectories collected over 90m of walking over a rugged ground environment. Results are showcased in Tab. 2.
Using the full sensor suite and actions history is critical for effective odometry prediction. Notably, only using the IMU for odometry prediction, yields RPE@1m of 0.32 and ATE, of 0.244 (row 1). Just adding the last actions performed by the robot to the observation space improves

---

Observation Space Modalities
Metries
#
Proprioceptic
Last Action
IMU
RPE +
ATE,
ATEg
x
x
0.32
0.244
0.094
2
x
V
0.40
0.243
0.111
3
x
V
V
0.27
0.186
0.053
4
V
X
x
0.34
0.255
0.111
5
V
X
V
0.36
0.247
0.109
0.24
0.132
0.087
0.05
0.051
0.016
Table 2: Efffec of imputs on Legolas. Thre of the inputs to the Legolas system are Proprioceptic (angle and velocity), the Last Action performed, and IMU. Removing any of the inputs affects the best-performing variation (row 7) of Legolas. When compared to the complete system, the Last Action most crucial removing this yields an increase ATE, of 484% (row 5), followed by Proprioceptic with 364% larger error (row 3), and then IMU with an increase of 258% (row 6). Predicting with just the IMU (row 1) as done in previous methods leads to a 478% increase of ATE.-
this result to a RPE@1m of 0.27 and an ATE. of 08186, 15.6% and a 23.7% decrease respectively (rows and 3). Finally, adding the proprioceptic sensing to the observation space yields a RPE@1m of 0.05 and an ATE, of 0.051, an incredibly large 81.4% and 72.5% decrease respectively (rows 3 and 7). Rollouts of Legolas trained with the full sensing suite and only the IMU visualized in Sec. E. The improved odometry predictions clearly show Legolas is capable of closely estimating the ground truth trajectory while only using the IMU leads to degraded behavior.
4.5 Real-World World Lite3 Deployment
Legolas, when trained for the Lite3, is able to deploy onto real-world hardware as demonstrated in Fig. 4. As visualized Fig. 4b, Legolas tracks the ground-truth trajectory of the robot, showcas- ing that Legolas is able deploy across multiple robots. A visualization of the physical robot with ground truth and predicted trajectories in the scene shown in Fig. 4a.
5m
Estimaate
Ground Truth
Predicted and ground truth trajectory imposed on a map.
(a)
(b) Lite3 deployment with predicted and ground truth trajectories visualized.
Figure 4: Legolas deploys across robotics platforms. After collecting a new dataset of Lite3 roll- outs in simulation and training new model, Legolas successfully deploys to the new platform. The reconstructed trajectory is visualized from the top-down Fig. 4a and in the real-world in Fig. 4b.
Conclusion
In this work, we introduced Legolas, leg-inertial model capable of achieving odometry prediction at a level near visual odometry without explicit modeling of the robot or the environment. Legolas was trained by first collecting a dataset a simulator to take advantage of its ability to parallelize and produce diverse dataset. After this, Legolas is trained across multiple quadruped embodiments with successful real-world transfer for both. Through real-world experiments in both indoor and outdoor scenarios, demonstrated Legolas's capabilities of estimating odometry at accuracy higher than both filtering- based and other baseingg-bseed baselines.

---

References
[1] F. Fraundorfer and D. Scaramuzza. Visual odometry: Part ii: Matching, robustness, optimiza- tion, and applications. IEEE Robotics & Aulomation Magazine, 2012.
[2] D. Falanga, P. Foehn, P. Lu, and D. Scaramuzza. Pampe: Perception-aware model predictive control for quadrotors. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1-8. IEEE, 2018.
[3] D. Q. Mayne, J. B. Rawlings, C. V. Rao, and P. O. Scokaert. Constrained model predictive control: Stability and optimality. Automatica, 36(6):789-8 781-814,, 2000.
[4] S. Herath, H. Yan, and Y. Furukawa. Ronin: Robust neural inertial navigation in the wild: Benchmark, evaluations, & new methods. In 2020 IEEE international conference robotics and automation (ICRA), pages 3146-3152. IEEE, 2020.
[5] C. Chen, Zhao, C. X. Lu, W. Wang, A. Markham, and N. Trigoni. Deep-learning-aas pedestrian inertial navigation: Methods, data set, and on-device inference. IEEE Internet of Things Journal, 7(5):4431-4411 2020.
[6] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32(11))1231-1237, 2013.
[7] Qin, Li, and S. Shen. Vins-mono: A robust and versatile monocular visual-inertial state estimator. IEEE Transactions on Robotics, 34(4): 2018
[8] D. Nistér, O. Naroditsky, and J. Bergen. Visual odometry for ground vehicle applications. Journal of Field Robotics, 23(1):3-20, 2006.
[9] D. Scaramuzza and F. Fraundorfer. Visual odometry [tutorial]. IEEE robotics & automation magazine, 18(4):80-92, 2011.
[10] T. Shan, B. Englot, C. Ratti, and R. Danicla. Lvi-sam: Tightly-cuuledd lidar-visual- risuallinertial odometry via smoothing and mapping. In IEEE International Conference on Roboties and Automation (ICRA), pages 5692 1992-5698. IEEE, 2021.
[11] D. Youm, H. Oh, S. Choi, H. Kim, and J. Hwangbo. Legged robot state estimation with invariant extended kalman filter using neural measurement network. arXiv preprint arXiv:2402.00366, 2024.
[12] M. Bloesch, C. Gehring, P. Fankhauser, M. Hutter, M. A. Hoepflinger, and R. Siegwart. State estimation for legged robots on unstable and slippery terrain. In IROS, 2013.
[13] D. Wisth, M. Camurri, and M. Fallon. Vilens: Visual, inertial, lidar, and leg odometry for all-terrain legged robots. IEEE Transactions on Robotics, 39(1)3309-326, 2022.
[14] D. Wisth, M. Camurri, and M. Fallon. Robust legged robot state estimation using factor graph optimization, IEEE Robotics and Automation Letters, 4(4):4507 4514,, 2019.
[15] W. Liu, D. Caruso, E. llg, J. Dong, A. Mourikis, K. Daniilidis, V. Kumar, and J. Engel. Thio: Tight learned inertial odometry. IEEE Robotics and Automation Letters, 2020.
[16] C. Chen, X. Lu, A. Markham, and N. Trigoni. Ionet: Learning to cure the curse of drift in inertial odometry. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[17] R. Buchanan, M. Camurri, F. Dellaert, and M. Fallon. Learning inertial odometry for dynamic legged robot state estimation. In CoRL, 2022.
[18] A. Agarwal, A. Kumar, J. Malik, and D. Pathak. Legged locomotion in challenging terrains using egocentric vision. In CoRL, 2023.

---

[19] A. Kumar, Z. Fu, D. Pathak, and J. Malik. Rma: Rapid motor adaptation for legged robots. arXiv preprint (eXiv:2107.040000 2021.
[20] D. Hoeller, L. Wellhausen, Farshidian, and M. Hutter. Learning a state representation and navigation in cluttered and dynamic environments. RAL. 2021.
[21] X. Cheng. K. Shi, A. Agarwal, and D. Pathak. Extreme parkour with legged robots. arXiv preprint 2rXiv:2399.14444 2023.
[22] C. Cadena, L. Carlone, H. Carrillo, Latif, D. Scaramuzza, J. Neira, I. Reid, and J. J. Leonard. Past, present, and future of simultancous localization and mapping: Toward the robust-perception age. IEEE Trans. Robotics, 2016.
[23] G. Welch, G. Bishop, et al. An introduction to the kalman filter. 1995.
[24] V. Dhédin, H. Li, S. Khorshidi, L.. Mack, A. K. C. Ravi, A. Meduri, P. Shah, F. Grimminger, L. Righetti, M. Khadiv, et al. Visual-inertial and leg odometry fusion for dynamic locomotion.
In ICRA, 2023.
[25] Yang, Z. Zhang, Fu, and Z. Manchester. Cerberus: Low-drift iisualineeriia-leg odometry for agile locomotion. 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 4193-4199, IEEE, 2023.
[26] M. Bloesch, M. Hutter, M. A. Hoepflinger, S. Leutenegger, C. Gehring, C. D. Remy, and R. Siegwart. State estimation for legged robots-consistent fusion of leg kinematics and imu. Robotics, 17:17-24, 2013.
[27] N. Rotella, M. Bloesch, L. Righetti, and S. Schaal. State estimation for humanoid robot. In IROS, 2014.
[28] M. Maravgakis, D.-F. Argiropoulos, S. Piperakis, and P. Trahanias. Probabilistic contact state estimation for legged robots using inertial information. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 2266-12269, IEEE, 2023.
[29] G. Bledt, P. M. Wensing, S. Ingersoll, and S. Kim. Contact model fusion for event-based locomotion in unstructured terrains. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 4399-4406. IEEE, 2018.
[30] Hwangbo, D. Bellicoso, Fankhauser, and M. Hutter. Probabilistic foot contact esti- mation by fusing information from dynamics and differential/forward kinematics. In 2016 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 3872- 3878. IEEF, 2016.
[31] C. Chen, P. Zhao, C. Lu, W. Wang, A. Markham, and N. Trigoni. Oxiod: The dataset for deep inertial odometry. arXiv preprint arXiv: 09099,7499,... 2018.
[32] H. Yan, Q Shan, and Y. Furukawa. Ridi: Robust imu double integration. In Proceedings of the European conference on computer vision (ECCV), pages 621-636, 2018.
[33] G. Cioffi, Bauersfeld, Kaufmann, and D. Scaramuzza. Learned inertial odometry for autonomous drone racing. IEEE Robotics and Automation Letters, 8(5):2684-269) 2023.
[34] R. Buchanan, V. Agrawal, M. Camurri, F. Dellaert, and M. Fallon. Deep imu bias inference for robust visual-inertial odometry with factor graphs. IEEE Robotics and Automation Letters, 8(1):41-43 2022.
[35] V. Makoviychuk, L. Wawrzyniak, Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2108. 10470, 2021.
10

---

[36] M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, Liu, V. Koltun,
J. Malik, et al. Habitat: A platform for embodied ai research. In Proceedings of the IEEF/CVF international conference on computer vision, pages 9339-9347, 2019.
[37] S. James, Z. Ma. D. R. Arrojo, and A. J. Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2): 302-0000,,,,, 2020.
[38] Z. Zhuang, Z. Fu, J. Wang, C. Atkeson, S. Schwertfeger, C. Finn, and H. Zhao. Robot parkour learning. arXiv preprini arXiv:2309.0560 2023.
[39] Y. Zhou, C. Barnes, J. Lu, J. Yang, and H. Li. On the continuity rotation representations in neural networks. Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, 2019.
[40] E. Foxlin. Pedestrian tracking with shoe-mounted inertial sensors. IEEE Computer graphics and applications, 25(6):38 46, 2005.
[41] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. CVPR, 2016.
[42] Macenski and I. Jambrecic. Slam toolbox: Slam for the dynamic world. Journal of Open Source Software, 2021.
[43] O. J. Woodman. An introduction to inertial navigation. Technical report, University of Cam bridge, Computer Laboratory, 2007.
[44] P. Furgale, J. Rehder, and R. Siegwart. Unified temporal and spatial calibration for multi-sensor systems. In IROS. IEEE, 2013.
[45] Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers. benchmark for the evalua- tion of rgb-d slam systems. In IROS, 2012.
[46] Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers. A benchmark for the evalu- ation of rgb-d slam systems. 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pages 573-580, 2012.
[47] S. Umeyama. Least-squares estimation of transformation parameters between two point pat- terns. IEEE Transactions on Pattern Analysis & Machine Intelligence, 1991.
[48] M. Labbé and F. Michaud. Rtab-map as open-source lidar and visual simultancous local- ization and mapping library for large-scale and long-term online operation. JFR, 2019.
[49] E. Pennestri, V. Rossi, Salvini, and Valentini. Review and comparison of dry friction force models. Nonlinear dynamics, 83:1785-188 2016.
[50] N. Rudin, D. Hoeller, P. Reist, and M. Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. Conference on Robot Learning, pages 91-100. PMLR, 2022.
[51] C. Wang, D. Gao, K. Xu, J. Geng, Y. Hu, Y. Qiu, B. Li, F. Yang, B. Moon, A. Pandey, et al. Py- pose: A library for robot learning with physics-based optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22024-22034, 2023.
[52] G. Wenliang. imu utils. http://tttw..oooeeeeaaaaaaannnna 2018

---

Appendix Legolas: Deep Leg-Inertial Odometry
In this Appendix, we include additional details about the following:
A Limitations of the proposed method.
Further details regarding the training of Legolas such as improvements the chosen archi- tecture and details of data collection.
Elaborations the loss used to train Legolas.
D Explanation of the steps taken to improve deployment of the baseline methods and details of the deployment Legolas.
E Additional trajectories visualizations on the validation dataset and in real-world deploy- ments.
Limitations
We provide a comprehensive overview potential failure modes and limitations, along with strate- gies for future improvements. (1) Legolas locomotion-speciff Through testing, it has been found that Legolas's odometry prediction becomes poor when the locomotion policy used during deployment is different than the one used in training. A potential solution to this could be by col- lecting a diverse set of walking gaits in simulation and using this as larger, more diverse source to train Legolas. (2) Legolas has no immediate solution for removing accumulating drift. Integrating the predictions of Legolas into a SLAM system as an odometry estimate, which is very common across modern SLAM systems [48, 42] could fix this. (3) Our proposed method constrained by the fidelity and capabilities of the simulator. These limitations include simple foot-contact physics, limited sensing models resulting in poorer sim-to-real transfer, and lack of dynamic obstacles in current simulation framework.
B Further Training and Data Collection Details of Legolas
Training and data collection details are provided to improve the reproducibility of Legolas.
B.1 Training Details
During deployment, it was found that the default ID ResNet architecture [41] had too many pa- rameters, causing slower than desired model prediction on real-world hardware. Fine-tume Fine-tuning of the model, by reducing the number of residual layers from to decreased the number of parameters in the network from 4.7M to 1.5M allowing for successful deployment. As shown Tab. 3, this led to substantial change in the relevant metrics.
# Model Parameter Sized
RPE@11 +
ATE. 1
4.7M
0.045
0.017
2
1.5M (Ours)
0.050
0.016
Table 3: Effect of Model Size on Validation Performance. While decreasing the size of the model reduced model performance in RPE@1m, ATE, improved slightly when using the smaller model. Due to the changes not substantially changing the relevant metrics, but boosting on-board speed, the smaller model was utilized for deployment.
B.2
Data Collection
Simulator Ground-Truth Data: The ground truth sensor data generated from the simulator Isaac-Gym with the real-time physies engine PPhysX.' PhysX simulates the robot's configuration by integrating forces, velocities, and constraints based on the robot's properties (such mass, and kine- matic configuration) and environmental factors (such friction, gravity, and other applied forces).
12

---

During each simulation step, PhysX updates the robot's state, including its position, orientation, and velocities, and accurately resolves any interactions with the environment. This detailed simulation data is then made available within Isaac-Gym, allowing seamless access to the ground-truth sensor data for use in various applications. Slippage is handled through PhysX through the coulomb friction model [49].
We utilize the noise model from Rudin et al. [50] where the noise is sampled from a uniform distribution and then added to the observations. Tab. 4 includes the relevant minimum and maximum values of this distribution.
Additive Noise:
Observation
Min and Max Values of Unif. Distribution
IMU Gyro
+.22 rad/s
IMU Pitch and Roll
0.05 rad
Command Velocity X/Y
+.00 m/s
Command Velocity Angular
10.0 rad/s
Joint Angle
00.01 rad
Joint Velocity
1.5 rad/s
Policy Action
+.00 rad
Table 4: Sensor Noise. Additive noise sampled over a uniform distribution from the minimum and maximum values given. These ranges were found to be more noisy than their real-world coun- terparts.
Simulator Dataset Collection: The only difference in Legolas for training the pose prediction model P for the Go2 and Lite3 occurs during dataset collection, the "urdf" file that describes the robot's kinematics is changed for the respective robots.
Figure 5: Snapshot of robots collecting trajectories in simulation as performed in Sec. 3.1.
To collect given traeectory, robot follows its policy á with random velocity command with a target linear velocity uniformly sampled from [-2.0, 2.0] A target linear velocity between [-0.15, 0.15] m is instead set to 0 in order to add standing and rotating in-place behaviors into our dataset. A target angular velocity in yaw is then uniformly sampled from [-T, T] red If angular velocity chosen between [-0.05, 0.05] rad the angular velocity is set forward behaviors in dataset this
create more walking common motion in the real world. This large range of linear and angular velocities covers the full range of motions possible with म. A visualization of the training environment can be found in Fig. 5.
C Training Loss Details
During training and deployment, different rotation representations were utilized. Using 6D rep- resentation [39] during training improved performance on relevant metries on the validation split demonstrated in Tab. 5. Testing was performed against yaw-pitch-roll representation and combined translational representation. The combined translation representation is a logarithm map
13

---

# Rotation Representation
RPE +
ATEg +
1 Yaw-Pitch- Sww-Pitch-Roll
0.233
0.023
2
(3)) Loss
0.085
0.022
3 6D [39] (Ours)
0.050
0.016
Table 6D rotation improves validation metrics. Predicting changes in the 6D rotation produced better odometry prediction (RPE@ 1m decreased by 78.5%) when compared to predicting the changes yaw, pitch, and roll.
of SE(3) with velocity vector and axis-angle representation for rotation [51]. During training, APn is 6D representation, and the ground-truth incremental motion of the robot is converted into a 6D representation. Otherwise, during deployment and data collection, a matrix representation is utilized.
D Robotics Experiment Details
The average trajectory length for the indoor and outdoor datasets are 24m and 325m respectively. The indoor dataset consists of scenes an office building with flat, carpeted and stone floors as well as stairs trajectory. The outdoor dataset consists of urban scenes on concrete, dirt, and grass.
Legolas is deployed real-world hardware with a frequency of 50 Hz to match the frequency used in the simulator. However, with the optmizations provided in Sec. B.I, the model is capable of being run at up to 600 Hz on the Jetson Orin Nano.
During deployment of Legolas, the incremental motion of the robot is predicted at every time step. To estimate the trajectory of the robot, these predictions are accumulated. After converting the output of the models to an SE)33 matrix at some arbitrary step i, S,, the state at the current step, can be computed as S = -
D.1 Visual-Inertial Odometry Deployment Details
(a) 0.0 seconds
(b) 0.25 seconds
(c) 0.5 scconds
(d) 0.75 seconds
(c) 1.0 seconds
Figure 6: Quick motions and large changes in rotation occuring over one second. These quick motions cause degraded image captures on the quadruped affecting the deployment of the visual baseline. Changes in pitch (Fig. 2a.66,6c,,6,66 and roll (Fig. 6a and Fig. 6е) are notable.
VINS-Fusion found successful deployment when carefully tuned using tools including iimuuutils' [43, 52] and 'Kalibr' [44]. imu_tools utilized to retrieve relevant IMU statisties used by both the EKF and VINS-Fusee baselines such the gyroscope's and accelerometer's bias and noise. Kalibr
used find the extrinsic matrix of the camera frame with respect the IMU frame and to find the timing offiet between the camera and the IMU. With these steps, more manual fine-tuning of the parameters was required for deployment.
However, direct deployment to outdoor environments the previously tuned and measured param- eters failed. We found that quick motions and rotations such the one visualized in Fig. 6 became exaggerated on quadruped robot. Furthermore, interactions between the sun and the onboard cam- caused degraded depth prediction and tracking. Using polarized film with metal shield around the camera reduced this error mode, but still didn't lead to consistent results as shown in Fig. 7,
14

---

(a) Stereo camera without shield (b) Stereo camera with shield and (c) Stereo camera with shield and and full cloud coverage.
full cloud coverage.
no cloud coverage.
Figure A camera shield necessary for VINS-Fusiee deployment. Without the camera shield, the stereo camera equipped on the robot fails to capture the scene due to interference. However, we still find issues with outdoor deployment of VINS-Fusie if the environment sunny even when equipped with shielding.
D.2 Improving VIO with Legolas
VIO:
+IO+
Legolas:
Figure 8: Legolas improves VTO: RTAB-Map visual baseline loses tracking while the robot is moving. However, with Legolas integration, RTAB-Map is able to continue mapping.
As Legolas predicts both an odometry estimate as well as a covariance, these estimates can be used in downstream SLAM applications. RTAB-Map [48] is utilized this experiment as allows for easy integration of odometry estimates from outside sources. In Fig. 8, adding Legolas to RTAB-Map allows for the method continue mapping even when visual tracking fails. The complementary video for this experiment is available at: https:/learned-odom wwt.mm.ggttubiiiiieiio
D.3 Details Training Test Dataset
A visualization of the collected indoors dataset is visualized in Fig. 9. Ground truth was collected with motion capture and LiDAR-SLAN system.
15

---

63
45-
30
3.6
35.
34
90
3.
33
25.
.44
2
o8
_0
6
.05
2
(a) Forward motion, ground truth collected with motion capture.
10
15
20
2
3
5
6
(b) Forward motion, ground truth collected with -IDAR-SLAM system.
5.
5,5
53
4.
5.
5,00
45
6.
4.
35
35
19
23
25
03 8
.00
6 o8 y
(c) Backwards motion, ground truth collected with motion capture.
Figure Collected Indoor Test Dataset: Units meters. The indoor test dataset consists of 10 collected trajectories. 6 trajectories were collected with the Go2 moving forward. Of the col- lected forward trajectories, obtained their ground truth from a motion capture system and from LiDAR-SIAV system. Four trajectories were collected with the Go2 moving backward with ground truth obtained from a motion capture system.
16

---

E
Additional Trajectory Rollouts
Additional validation rollouts: Fig. 11 demonstrates the superior trajectory reconstruction of using the full sensing suite rather than just the IMU. The use of the full sensing suite allows Legolas to directly predict the incremental motions of the robot rather than rely on analytical models of the robot.
E.1 Real-World Variance Predictions
Predected Deplacement Vasiance
Prodiceed Diplocerent
18
13
(a) Building Scene:
The robot walks through hallways and carpeted office arcas. The predicted magnitude of the displacements and variances are shown. Notably the Y variance and displacements are much smaller than the forward-facine X direction as the displacements from swaying are much smaller.
we pt Ardiited recaaomeent vunuer
Preddimed treonnereerr
10.00
12.5
32.5
(b) Stairs Scene: the robot goes stairs. The predicted magnitude in the forward-facing X and upwards Z direction cyclical with matching variance.
Figure 10: Predicted variance and displacements. Across a building scene Fig. 10a and going up stairs Fig. 10b, results for the predicted magnitude in displacements and variances shown.
A qualitative analysis of the robot's predicted variance across two scenes is examined. A companion video showcasing these rollouts, well as the confidence intervals, can be found at https://leareed- odom.github.i www.dex..////ixxi//// htt.eeeeimmmmmmmeeeecc
Building Scene Fig. 10a: this scenario, the displacement and variance are much smaller than the displacement. This matches what expected as the robot is moving forward, and this would be dominated by an X displacement with smaller Y displacement from swaying. After completing the first hallway, the robot turns sharply and causes dip in displacements, which matched with dip in variance the robot is moving slower. The lower variances when the robot is moving slower can also be observed at the start of the plot as the robot starts to move.
Stairs Scene Fig. 10b: While going up the stairs the X and displacements are very cyclical with their peaks being matched with peaks in variance occurring wherever the displacement changes quickly. When the robot reaches the new floor, variances for X and Z decrease there is the robot's locomotion is more stable.
Mean relative variance increases with task difficulty. As visualized in Fig. 10, the variance magnitude changes with the robot's 'ss speed; slower movement narrows the range of reasonable pre- dictions. Differences in error should also occur with varied robot locomotions. Tab. 6 utilizes mean variance magnitudes divided by prediction magnitudes across scenarios. This metric represents the certainty of the network in its prediction. During stable locomotion (Building and the last 2 seconds of Stairs), this ratio is lower meaning that the network is more certain of its prediction. During the
17

---

Environment
| X Variance
Y
Variance
Z Variance
|X] Prediction
Prediction
Z Prediction
Building
0.0054
0.0032
0.0011
Stairs (Full)
0.0213
0.0184
0.0061
Stairs (Last 2 Seconds)
0.0039
0.0029
0.0008
Table 6: Mean relative variance increases with the difficulty of the task. The mean ratio of the variance magnitude over the prediction magnitude lower for simple walking tasks (Building and last 2 seconds of Stairs). This ratio increases when the robot is climbing up stairs in the full Stairs environment.
full stairs rollout, where quick motions occur, the ratio increases as predicting changes in motion on stairs is more challenging.
Additional real-world rollouts: Supplementing the rollout given in Fig. 3a, additional rollouts are visualized. Fig. 12 demonstrates the robot moving in straight line in an indoor scene. In this scenario Legolas tracks the straight line while VINS-Fusion suffers from losing visual tracking due to reflections on the floor. Fig. 13 demonstrates the robot moving in double circular pattern an indoor environment. this scenario, Legolas able to track the ground-truth. The EKF baseline capable of reproducing similar shape up until the 1st completed loop where its trajectory reconstruction degrades.
Ours
IMU Only
-2
-4
mm
-10
[m]
x [mm]
Estimate
Ground Truth
Ours
IMU Only
5.0
2.5
2.5
0.0
0.0
10
15
20
10
15
20
[m)
xm]
Estimate
Ground Truth
Figure 11: Top-Down visualization of rollouts of Legolas trained with different sensing modal- ities. Legolas demonstrates improved odometry prediction through its use of the full sensing suite on the robot. Previous work has attempted to estimate displacements through learning with only the IMU sensor, this produces inferior predictions.
18

---

25
ground-truth
Ours
20
VINS-Fusiio
(m)
15
BC
10
y (m)
-2
-8
-10
150
100
yaw (deg)
50
-50
-100
-50
40
(s)
Figure 12: Straight line rollout. this scenario the robot moves a straight line for approximately 23 meters. Legolas tracks the trajectory. while VINS- VINS-Fusion suffers from losing visual tracking around (s) 37 and the estimated position of the robot jumps.
19

---

ground-truth
4
Ours
EKF
3
VINS-Fusion
(m)
2
BC
2
1
0
150
100
yaw (deg)
50
0
-50
-100
-150
20
40
60
80
100
(s)
Figure 13: Circular rollout. this scenario the robot moves in a circle two times, both VINS- Fusion and Legolas are capable of tracking the ground-truth trajectory. EKF tracks closely until the first loop is completed. BC is able track the heading, but poorly track and y positions.
20


Reviews:
[{"review_id": "YbW30P2UgY", "reviewer": "Reviewer_hmFN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents a learning-based method to estimate legged odometry using simulated IMU and proprioceptive data. Using different rollouts of a policy in simulation, a network is trained to predict incremental pose estimates, which are concatenated to produce a SE(3) pose. Results show significant improvements to other baselines that rely on exteroceptive data as well.", "summary_of_recommendation": "I think the paper is good and interesting, so I’m happy to accept it. But I strongly encourage the authors to address the previous comments, because these are weak aspects that people working in state estimation will very likely criticize otherwise, since they omit or rebrand some known ideas in the field.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "- Please be more explicit about the loss definition for the incremental pose change. Currently it is not clear how the actual Euclidean error is obtained if the pose difference is defined on SE(3). I had to check the appendix to understand that the translation and rotation are decoupled and a special parametrization for rotations was used.\n- Further, I wonder if using a different error definition (i.e metric) that couples rotation and translation (e.g. the logarithm map of SE(3)) could change the results, given that’s the standard procedure to define factors in SLAM or odometry estimation with factor graphs.\n- Given that the method predicts the covariance of the estimate, it would be interesting to see the confidence bounds of the predictions, as this information can be used in downstream methods (e.g. SLAM). This could provide further insights on the consistency of the estimate.\n- The “masking head” term introduced is doing nothing else than “zero velocity updates” in standard legged state estimation literature. I recommend making this connection because otherwise it’s an unnecessary rebranding of a widely studied concept. The $m$ variable introduced is also similar to the covariance increases during flying phases of the legs used in other papers, such as Bloesch (2012), which acts like a soft switch.\n- Please explain how the IMU and other data is simulated. Which analytical models and noises did you use? How far are they from the nominal specifications of the real IMU and sensors used?\n- Evaluation: Using the Umeyama method for odometry evaluation is not right and it is expected to produce lower errors. Umeyama makes sense when you evaluate trajectories estimated by a SLAM system, which introduces loop closures that change the reference frame of the trajectories. In this setting of odometry estimation, aligning against the initial pose is the only case that makes sense.\n- Similarly, the ATE is not really a good metric to evaluate odometry, as it is expected to drift, and the numbers will get worse over time. The RPE is a better metric, because it effectively characterizes error per distance traveled, which encodes the amount of drift expected for the method.\n- The real world deployment example on Lite3 is not really useful. Fig 4 doesn’t show a scale, so it’s difficult to understand how much the estimate drifted. Running a standard odometry system on a different robot platform is also expected to work. The question always is how much is the actual drift.\n- Lastly, I wonder how much effort was made to make the VIO methods work. None of the trajectories look particularly challenging (and let’s note that not much information about the sequences was shown in the paper itself), so I’m a bit hesitant. Maybe testing against OpenVINS or explaining the failure cases in a better way could be a useful insight for the readers.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ODnW4YM6qm", "reviewer": "Reviewer_KiPa", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper continues prior work to learn odometry estimates from IMU and leg encoders for quadruped robots. The innovation on top of prior work is to train on simulated data.", "summary_of_recommendation": "The rebuttal significantly improved the accuracy of the experimental results. The mix of the experiments is now a bit chaotic with different camera placements, environments, etc. I think there is now sufficient evidence that this works in general.  I am however still not able to conclusively answer the questions \"How useful is this compared to VIO with a correctly placed camera?\" and \"If, as the authors suggest, this needs to be combined with a visual SLAM system, does it really yield improvements over other odometry methods?\"; I therefore raise my score to weak reject.", "technical_quality": 2, "clarity_of_presentation": 2, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "see review above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dKpkG3EeTf", "reviewer": "Reviewer_8eXK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces a sim-to-real odometry prediction system for legged robots using IMU+leg odometry+velocity commands.", "summary_of_recommendation": "I lean towards accept and ask that the authors provide the additional explanations that I have requested.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1. The inputs and outputs of the neural network must be much more clearly explained.\n\n2. The poor performance of the EKF method should be more clearly explained", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received mixed initial reviews with the following summary of strengths and weaknesses.

Strengths:
- The approach is simple and straightforward which could make it easily adaptable to other robot platforms.
- The paper is well written and easy to follow.

Weaknesses:
- Novelty is limited, as a similar approach has been proposed before trained on real-world data.
- Further clarifications of the method are needed.
- The method should be better related to the previous literature in legged robot odometry estimation.
- The experimental results are weak and evaluation needs to be improved.
- The results for the VIO baseline in the experiments are questionable.

The author response and discussion has partially addressed the concerns of the reviewers. After some debate, the reviewers and AC concluded that the reasons for accept outweigh reasons for rejecting the paper and it can be recommended for acceptance. 

However, the additional covariance evaluation did not address the questions/concerns by the reviewers appropriately (it seems there is a misunderstanding) and does not provide sufficient evidence if the variance follows the prediction error. This part must either be fixed entirely following the suggestions by the reviewers or removed from the paper. If it is fixed and included, it is suggested to move this experiment to the supplementary material to also keep within the page limit.

The revised paper violates the page limit by over 1 page. Please write some parts more compactly and move some parts (e.g. the covariance evaluation) to the supplementary material. If the paper gets accepted, the final paper must not violate the page limit.
Decision: Accept