SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 3: Trajectory errors for Drunkard's test scenes for all difficulty levels. Note that COLMAP is an offline method and is only shown for reference. Our odometry method mostly outperforms the compared online odometry methods. Best results are highlighted as first, second, and third.
We chose scenes 0, 4 and 5 as the test ones with 3.816, 4.545 and 1.655 frames respectively. The rest of the scenes were used for training. The trajectories estimated by COLMAP and EDaM are compared against the ground truth after Sim(3) alignment, as they are up-to-scale. DROID-SLAM and our Drunkard's are aligned to the ground truth with a SE(3) transformation, as the RGB-D input allows them to estimate the real scale. As using Sim(3) against SE(3) is not totally fair since the former applies a scale correction, we also test EDaM using the ground truth depth maps so we can apply SE(3) alignment. The reported metrics are: Relative Position Error (RPE) for translation and rotation, that measures the local accuracy of the estimated trajectory against the ground truth between consecutive frames, and the Absolute Trajectory Error (ATE) for translation that computes the global consistency between both trajectories (see [74] for details).
For each sequence, the percentage of registered frames over the total is shown, a metric in which COLMAP shows poor performance. As a consequence, its trajectory metrics are influenced positively as it excludes frames that are challenging to track and probably would have increased the error. This is beneficial in particular for the ATE, as it takes into account the global consistency rather than frame-to-frame errors like the RPE, and happens earlier in higher deforming scenes. Also note that DROID-SLAM is very GPU-memory demanding, in part because of the final global bundle adjustment and it is not able work with long sequences like Scene 4. However, our Drunkard's Odometry and EDaM are more robust, partly due to tracking only between adjacent frames.
Table 3 shows our results. Note that our Drunkard's Odometry practically always outperforms DROID-SLAM and EDaM (with and without ground truth depth) in RPE and ATE, also in rigid scenes, even if our model is trained exclusively in non-rigid scenes and does not use loop closure or full bundle adjustment. The gap is significantly larger at higher deformation levels, for which the Drunkard's Odometry errors increase much less. This demonstrates that our method is able to generalize at predicting surface deformations. Only COLMAP is able to outperform our Drunkard's Odometry in ATE. Note, in any case, that COLMAP has a much lower recall, as it only estimates camera motion for a substantially lower percentage of frames. If we focus on RPE, a more fair metric in this case, our Drunkard's Odometry is on par to COLMAP at lower deformation levels, and clearly outperforms it at higher ones.
Validation in real endoscopies. We used the Hamlyn dataset [30], that contains intracorporeal endoscopic RGB scenes with weak textures, deformations and reflections. Specifically, we chose scenes 1 and 17 (see Figure 5), which are significant exploratory ones. Most of Hamlyn's videos have very small camera motions, being of no interest for benchmarking odometry methods. We slightly cropped the images to remove black pixels at the borders. Depth data was taken from the public tracking test data of EDaM [9] which was estimated by a single-view dense depth network trained in a self-supervised manner in all Hamlyn scenes except for the test ones. Note that this depth does not have the same quality as the real ground truth one from the Drunkard's Dataset.
(a) Scene 1
Figure 5: Sample color and depth frames and APTE results for the Hamlyn test videos.
(b) Scene 17
Scene #Frames Method	APTE↓
1	1058	EDaM [9]	0.0044
DROID-SLAM [34]	0.0019
Drunkard's trained w/o deform.	0.0055
Drunkard's trained w/ deform.	0.0045
17	1306	EDaM [9]	0.0236
DROID-SLAM [34]	0.0338
Drunkard's trained w/o deform.	0.0309
Drunkard's trained w/ deform.	0.0214
(c) Tracking Results
Since the Hamlyn dataset lacks ground truth camera poses, we propose a novel ground truth-free trajectory metric to measure the quality of the estimated odometries. The key idea is to generate loopy videos by duplicating and reversing a given image sequence and concatenating it to the end of the original sequence. The generated "palindrome video" is twice as long as the original sequence and any tracking method should ideally loop back to its starting position. We then simply measure the discrepancy between start and end pose and accumulate it over various loop lengths. We denote the new metric as the Absolute Palindrome Trajectory Error (APTE) and define it as
$$
\mathrm {A P T E} = \frac {\sum_ {k = 1} ^ {N} \mathrm {A P T E} _ {k}}{N} \quad \text {w i t h} \quad \mathrm {A P T E} _ {k} = \left\| \left(\prod_ {j = 1} ^ {k} \mathbf {T} _ {c j} ^ {\text {b a c k}}\right) \left(\prod_ {j = 1} ^ {k} \mathbf {T} _ {c j}\right) \right\| _ {2}, \tag {3}
$$
where  $N$  is the total number of frames and  $\mathrm{APTE}_k$  the translation Root Mean Squared Error (RMSE) of between the first and last pose of the sub-trajectory loop  $k$ . This last pose of the loop  $k$  is the result of sequentially applying to the starting camera pose—the identity  $\mathcal{I} \in \mathrm{SE}(3)$ —the first  $k$  estimated relative camera poses  $\mathbf{T}_c$  for the scene plus the first  $k$  estimated relative camera poses for the same scene but ran backwards  $\mathbf{T}_c^{back}$ . Then, the  $\mathrm{APTE}_k$  is the module of the translation of this last estimated pose. We note that the APTE error is trivially minimized if all transformations are zero or close to the origin, but these cases are easy to check visualizing the trajectory.
Table 5c shows our APTE metric in the Hamlyn test scenes for the baselines that are able to track the whole scene (EDaM and DROID-SLAM) and two versions of our Drunkard's Odometry, one trained only in rigid scenes of the Drunkard's Dataset, i.e. scenes from difficulty level 0, and other in deformable scenes, specifically from level 1. SD-DefSLAM [26] was also tested here and breaks after a few frames, far from the full lenght of the trajectory, even having originally been built and tested in Hamlyn. In consequence, it is not possible to compute the APTE since we need the full trajectory run forward and backward. Before computing the APTE, all the trajectories are scaled with the one given by Sim(3) alignment, having as reference the trajectory estimated by EDaM, since it showed a good qualitative tracking performance in Endo-Depth-and-Motion [9].
Figure 6 helps to visualize how the  $\mathrm{APTE}_k$  values evolve as the length of the loops grows. Notice that a longer loop does not always mean higher  $\mathrm{APTE}_k$ . This is because there can be some specific problematic frames with a bad pose estimation that induce a drift accumulation in the following ones getting worse  $\mathrm{APTE}_k$  measures. However, this can be reversed if the tracking recovers partially during the way back in the loop. The single APTE values shown in the main paper are the average of all the  $\mathrm{APTE}_k$  measures.
Figure 7 collects the estimated 3D camera trajectories in the Hamlyn test scenes for all the baselines. It is noteworthy how DROID-SLAM is incorrectly producing sharp and chaotic trajectories around the origin, particularly evident in Scene 1 and partially in Scene 17. This can be further observed in Figure 6b, where the plot of DROID-SLAM becomes similar to the others in the second half of the video. The other methods produce smoother and plausible estimates. The qualitative evaluation of visualizing the trajectories and the quantitative by the APTE complement each other. The former helps in recognizing when a method is producing entirely incorrect camera poses, while the latter provides an objective measure for comparing methods when the trajectories already appear good and plausible.
One clear conclusion we can extract is that our model performs better if it is trained in non-rigid scenes rather than rigid. Again, this shows its capacity to learn deformation patterns and retrieve
(a)  $\mathrm{APTE}_k$  plots in Hamlyn's Scene 1.
(b)  $\mathrm{APTE}_k$  plots in Hamlyn's Scene 17.
Figure 6: Camera tracking performance on the Hamlyn dataset.  $\mathrm{APTE}_k$  values along the  $N$  different  $k$ -frames-length-loops in Hamlyn's scenes 1 (a) and 17 (b) for EDaM [9], DROID-SLAM [34] and Drunkard's Odometry (ours) with and without having been trained in deformable scenes of the Drunkard's Dataset (i.e. trained in level 1 and 0, respectively). The training on deforming scene substantially improves the performance of our method.
Figure 7: Estimated 3D trajectories in Hamlyn test scenes for different methods. See animations with Adobe Reader, KDE Okular or YouTube.
a more stable camera pose under real-world non-rigid challenging scenes, and even trained in a different domain. Besides, DROID-SLAM's superiority of APTE in Scene 1 results invalidated by observing how it completely fails to estimate the trajectory in Figure 7. This validates the robustness of the Drunkard's Odometry pipeline in addressing the domain gap problem. Another interesting point to comment on is that the advantage of our Drunkard's Odometry over EDaM may not be clear enough. Their performances are similar in Scene 1, while Drunkard's is better for Scene 17. The evaluation in our dataset shows that our Drunkard's Odometry is significantly better than EDaM. This indeed shows the motivation and benefit for our Drunkard's Dataset: looking at Table 3 we can confidently claim that our Drunkard's Odometry is better than EDaM for camera trajectory estimation. The apparent disagreement with the Hamlyn evaluation may arise from the fact that deformations are not graduated in Hamlyn, and qualitatively they are very small, which may benefit rigid methods.
Limitations. Our Drunkard's Dataset's most clear limitation is that it is synthetic. However, as we argued, we believe that the difficulties for acquiring high-quality data with ground truth annotations in the target application domains motivate their use. Notice that it is not possible to record true ground truth optical flow in real-world sequences, as we do not have access to the exact pixel motion information. It is precisely this optical flow availability which unlocks the use of powerful flow-based models trained in synthetic deformable data which generalize well to real non-rigid scenes. Despite the real indoor images of the Drunkard's Dataset may resemble real deformable scenes in texture and
shape, such as bouncy castles, moving fabrics or canvas, they are far from the medical application environment. Still, we think that generating medical data with realistic deformations, fluids or textures in large scale is out of reach. As a proof, such data does not exist yet and motivates the use of the Drunkard's data. Our Drunkard's Odometry has all the limitations inherent to a frame-to-frame tracking method. Drift accumulates very quickly, and even if our sequences are loopy we do not either detect loop closures or correct our trajectories based on them. However, the SLAM literature shows that SLAM methods (e.g., [75]) can be built on top of odometry ones (e.g., [76]).
Estimating camera motion in deformable scenes is a challenging research problem relatively underexplored in the literature, and for which a lack of clear benchmarks slows down research progress. In this work, we created the Drunkard's Dataset, a large-scale simulated dataset with perfect ground truth and a wide variety of scenes and deformation levels to train and validate deep neural models. In addition, we propose the Drunkard's Odometry method for deformable scenes to validate our dataset. The method minimizes a scene flow loss, but as its main contribution, intrinsically decomposes the estimated twist flow into two components: The majority of motion is aimed to be explained by a rigid-body camera motion, and all remaining motion is explained by scene deformations. In contrast to most existing works our method does not require a static scene part for estimating a reference coordinate frame which is crucial in fully deforming scenarios like endoscopy. To also assess odometry estimates in the absence of ground truth data, we further define a novel ground truth-free metric for trajectory evaluation that measures the cyclic consistency of a tracking algorithm. Both the dataset and source code for our baseline method are publicly available. Our experimental results validate our dataset, illustrates its challenges, and also shows that our Drunkard's Odometry is able to outperform relevant baselines.
Acknowledgements. This work was supported by the EU Comission (EU-H2020 EndoMapper GA 863146), the Spanish Government (PID2021-127685NB-I00 and TED2021-131150BI00), the Aragon Government (DGA-T45 17R/FSE), and a research grant from FIFA.
[1] A. Badias, I. Alfaro, D. Gonzalez, F. Chinesta, and E. Cueto, "Morph-dslam: Model order reduction for physics-based deformable slam," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 11, pp. 7764-7777, 2021. 1
[2] T. Li, S. Liu, T. Bolkart, J. Liu, H. Li, and Y. Zhao, "Topologically consistent multi-view face inference using volumetric sampling," in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3824-3834, 2021. 1, 2, 3
[3] N. Qian, J. Wang, F. Mueller, F. Bernard, V. Golyanik, and C. Theobalt, "Html: A parametric hand texture model for 3d hand reconstruction and personalization," in Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16, pp. 54–71, Springer, 2020. 1
[4] J. Wang, S. Tan, X. Zhen, S. Xu, F. Zheng, Z. He, and L. Shao, “Deep 3d human pose estimation: A review,” Computer Vision and Image Understanding, vol. 210, p. 103225, 2021. 1
[5] C. Li and G. H. Lee, "Coarse-to-fine animal pose and shape estimation," Advances in Neural Information Processing Systems, vol. 34, pp. 11757-11768, 2021. 1
[6] Z. Su, W. Wan, T. Yu, L. Liu, L. Fang, W. Wang, and Y. Liu, "Mulaycap: Multi-layer human performance capture using a monocular video camera," IEEE Transactions on Visualization and Computer Graphics, vol. 28, no. 4, pp. 1862-1879, 2020. 1
[7] J. Lamarca, S. Parashar, A. Bartoli, and J. Montiel, "Defslam: Tracking and mapping of deforming scenes from monocular sequences," IEEE Transactions on robotics, vol. 37, no. 1, pp. 291-303, 2020. 1, 4
[8] A. Sengupta and A. Bartoli, “Colonoscopic 3d reconstruction by tubular non-rigid structure-from-motion,” International Journal of Computer Assisted Radiology and Surgery, vol. 16, no. 7, pp. 1237–1241, 2021. 1, 4
[9] D. Recasens, J. Lamarca, J. M. Fácil, J. Montiel, and J. Civera, "Endo-depth-and-motion: reconstruction and tracking in endoscopic videos using depth networks and photometric constraints," IEEE Robotics and Automation Letters, vol. 6, no. 4, pp. 7225-7232, 2021. 1, 4, 6, 7, 8, 9
[10] S. Shao, Z. Pei, W. Chen, W. Zhu, X. Wu, D. Sun, and B. Zhang, "Self-supervised monocular depth and ego-motion estimation in endoscopy: Appearance flow to the rescue," Medical image analysis, vol. 77, p. 102338, 2022. 1, 4