The Drunkard’s Odometry: Estimating Camera Motion in Deforming Scenes
Authors: David Recasens
Martin R. Oswald
Marc Pollefeys
Javier Civera
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: odometry
non-rigid dataset
RGB-D tracking
endoscopy
TL;DR: The Drunkard's Dataset provides synthetic data for camera motion estimation in deformable scenes, allowing for quantitative evaluation of baselines and the development of the Drunkard's Odometry, a novel odometry method.

Abstract: Estimating camera motion in deformable scenes poses a complex and open research challenge. Most existing non-rigid structure from motion techniques assume to observe also static scene parts besides deforming scene parts in order to establish an anchoring reference. However, this assumption does not hold true in certain relevant application cases such as endoscopies. Deformable odometry and SLAM pipelines, which tackle the most challenging scenario of exploratory trajectories, suffer from a lack of robustness and proper quantitative evaluation methodologies. To tackle this issue with a common benchmark, we introduce the Drunkard's Dataset, a challenging collection of synthetic data targeting visual navigation and reconstruction in deformable environments. This dataset is the first large set of exploratory camera trajectories with ground truth inside 3D scenes where every surface exhibits non-rigid deformations over time. Simulations in realistic 3D buildings lets us obtain a vast amount of data and ground truth labels, including camera poses, RGB images and depth, optical flow and normal maps at high resolution and quality. We further present a novel deformable odometry method, dubbed the Drunkard’s Odometry, which decomposes optical flow estimates into rigid-body camera motion and non-rigid scene deformations. In order to validate our data, our work contains an evaluation of several baselines as well as a novel tracking error metric which does not require ground truth data. Dataset and code: https://davidrecasens.github.io/TheDrunkard'sOdometry/

The Drunkard's Odometry: Estimating Camera Motion in Deforming Scenes
David Recasens University of Zaragoza
Martin R. Oswald ETH Zurich, University of Amsterdam
Mare Pollefeys ETH Zurich, Microsoft
Javier Civera University of Zaragoza
Abstract
Estimating camera motion in deformable scenes poses a complex and open research challenge. Most existing non-rigid structure from motion techniques assume observe also static scene parts besides deforming scene parts order to establish anchoring reference. However, this assumption does not hold true in certain relevant application cases such as endoscopies. Deformable odometry and SLAM pipelines, which tackle the most challenging scenario of exploratory trajectories, suffer from a lack of robustness and proper quantitative evaluation methodolo- gies. To tackle this issue with common benchmark, introduce the Drunkard's Dataset, a challenging collection of synthetic data targeting visual navigation and reconstruction in deformable environments. This dataset is the first large of exploratory camera trajectories with ground truth inside 3D scenes where every surface exhibits non-rigid deformations over time. Simulations realistic 3D build- ings lets us obtain vast amount of data and ground truth labels, including camera poses, RGB images and depth, optical flow and normal maps at high resolution and quality. We further present novel deformable odometry method, dubbed the Drunkard's Odometry, which decomposes optical flow estimates into rigid-body camera motion and non-rigid scene deformations. In order to validate our data, our work contains an evaluation several baselines as well as a novel tracking етror metric which does not require ground truth data. Dataset and code: ht https: / //avvirreeaeen github. io/ Dunkark sOdometry/
Introduction
Deformable scenes are among the most challenging cases for visual navigation and multi-view
reconstruction. They may also be among the ones with the most relevant potential applications, ranging from the reconstruction of deforming objects [1], faces [2], hands [3], human bodies [4] (or animals' [5]), clothing [6]) or the body interior for medical applications [7, 8, 9, 10]. Among non-rigid scenes, medical ones stand out
all potential applications for mapping and navigation as very different from the rest and are the target of our work. In certain medical procedures, such as endoscopies, a camera navigates inside the human body, performing exploratory trajectories that extend far beyond its field of view. For the rest of the applications mentioned, the camera remains nearly or fully stationary and most views have high degree overlap. The field of Non- Rigid Structure from Motion (NRSIM) has experienced significant progress in the last decades, e.g. [11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24J. However, most of them address small-scale reconstructions and are of limited use in medical applications. For the few exceptions that cover SLAM [25] deformable scenes (c.g., [7, 8, 10, 26]), there are no clear benchmarks nor datasets to support and track progress in the field.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

camera view
3rd person view
Level O
Level
Level 2
Level 3
Figure 1: Sample scene of the Drunkard's Dataset. The dataset provides various levels of scene deformation. Top row: Sample frames from Scene 0 over all difficulty levels 0 3. Bottom row: External views showing the ground truth camera trajectory in green and the camera frame in purple. With increasing deformation level the camera motion more abrupt. See animations with Adobe Reader, KDE Okular or YouTube.
Our main contribution in this work motivated by this need for benchmarking exploratory camera motion estimations in deforming scenes. Publicly available datasets imaging deforming scenes do not cover exploratory trajectories, and the ones that cover them do not have ground truth geometric annotations. Our proposal is synthetic dataset that we denote as the "Drunkard's Dataset", containing a set of high-resolution RGB images, ground truth depth, optical flow, normal maps and camera pose trajectories in synthetic but realite-c deforming scenes. In order to generate sufficient amount and variety of data, we imported the real-world scanned indoor 3D models of the Habitat-Matterporr 3D dataset [27], added dynamic deformations, and generated trajectories within them. Figure shows several data samples with the camera trajectory that advances over time. As the original 3D models along scenes. To make it a scalable benchmark dataset, every scene was recorded under four different levels of reconstruction difficulty, increasing the deformation and camera trajectory perturbations (observe them in Figure 1). The Drunkard's Dataset is unique in its kind, providing large-scale data in deformable scenes, which will enable both, benchmarking non-rigid navigation and reconstruction methods, as well as sufficient data to evaluate the potential benefits of deep learning. Please visit our project website for further details and access to the dataset and source code.
sal-w scans, camera trajectory and depth are in consistent metric scale
Scientific progress is in many occasions based on well-established and public benchmarks. This has been the specific case in computer vision research in the last decades, up to the point that the existence of some benchmarks is highly correlated with scientific progress in the field. Data repositories are also essential nowadays not only benchmark different methods, but also to train deep neural networks. However, collecting large amounts of data with non-rigid content is challenging due to difficulties in annotation, as argued by Li al. [2]. Indeed, some very popular datasets are synthetic [28, 29].
Capturing large-scale data in sufficiently large non-rigid spaces, in order to benchmark odome- try/SLAM methods, is even more challenging. Having ground truth annotations, in particular in the medical domain, hugely increases the challenge. The most popular medical datasets [30, 31, 32, 33] small, lack geometric ground truth, both. As result, comparisons between methods are very often inaccurate, inconclusive or questionable. This motivates our work.
The second contribution of this paper is the Drunkard's Odometry odometry method for camera motion estimation from RGB-D sequences in deformable scenes. Our method is inspired by the pose estimation of DROID-SLAM [34] and by the 3D scene flow prediction of RAFT-3D [35]. Our novelty compared to both that our architecture models potential scene deformations.
(see Figure 2), flow-based

---

Depth 2 Depth
RGB 2
Feature extraction and matching
RGB
Posc and 3D
Iterative block
Pose and 3D flow initialization
Figure 2: Overview of the Drunkard's Odometry pipeline.
As third contribution, in order to validate methods also in non- non-annotated data, we present a novel metric, the Absolute Palindrome Trajectory Error (APTE). Our novel metric is based running odometry methods forwards and backwards a sequence of images, and comparing the pose errors between the first and last frames through different loop lengths. This metric may be useful for validating methods in realistic setups, but is limited to relative errors between two poses, and hence it is not as informative metrics using ground truth labels. For this reason, we believe that APTE in real sequences should not be trusted on its own, but as a complement to a larger and more informative set of metrics in our simulated Drunkard's Dataset.
2
Related Work
Structure from Motion, visual odome- try and visual SLAM methods for rigid scenes commonly evaluated in well established set of publicly avail-
Explo- ratory
Dataset
Sim/Real
Frames
Domain
Aguiar al. [36]
6K
bodies Human bodics
FAUST [37]
Real
9K
able datasets [42, 43, 44, 45, 46, 47, 48], with ground truth geometric an- notations and under the same metrics, which facilitates comparisons between them and progress in the field. Large- scale annotations for geometric ground truth are quite challenging in real sce- narios, and still contain small errors due the accuracy of the equipment (GPS
NRSMM Challenge [32]
Real
Objects
DeforminTThngg<D [38]
Sim
122K
Humans & animals
ToFu [2]
Sim
20K
Faces
EndoMapper
Real
4M
Colomoscopies
Hamlyn [39, 40,
Real
93K
Laparoscopies
Drunkard's (ours)
Sim
416K
Indoor scenes
Table 1: Overview of existing non-rigid dataset specifica- tions compared our Drunkard's dataset.
or motion capture systems) used. An accurate approximation of the depth and camera pose could be recorded with sensors, but could not obtain such a good quality of optical flow and normal maps. In real-world applications that involve significant deformations, such as endoscopies, these sensors cannot even equipped. While we could attempt simulate a small-scale scenario using moving blankets bouncy castles, it would fall short in terms of capturing a wide range of textures, defor- mations, and camera trajectories. Virtual environments, on the other hand, offer superior solution, providing rich and diverse representation of the complex conditions that are found in real-world settings. Synthetic datasets solve these issues at the price of the sim-to-real gap, and are common for benchmarking methods navigation and reconstruction similar reasons, synthetic datasets are widely used other computer vision tasks, such as stereo and flow [53, 28, 54, 29, 55, 56], depth [57, 58], object recognition and segmentation [59], object pose estimation and tracking [60], and scene segmentation, understanding and reasoning [61, 62, 63]. For additional insights and references the use of synthetic data in deep learning, the reader is referred to the excellent survey by Nikolenko [64].
rigid scenes [49, 50, 51, 52, 27]. For
Note that, in most of the datasets cited, in particular those for faces and humans bodies, the camera is almost stationary. For the few datasets where the camera moves sufficiently, exploring new areas, the covered region is in any case reduced, deformations are small or there is absence of geometric

---

ground truth (see Table 1). This poses difficulties for benchmarking methods for odometry and SLAM targeting non-rigid environments, such as [7. 9]. Odometry and SLAM methods for deformable scenes are scarce in the literature, being [65, 7, 8, 9, 26, 10, 66, 67] the most representative ones. However, they are all based on feature matching direct tracking, which make them unstable in challenging sequences. Differently, our Drunkard's s Odometry is based on scene flow, which makes it significantly more robust.
3
The Drunkard's Dataset
Depilt
Optical
Fal
Noemai
Figure 3: Sample images of the Drunkard's Dataset with their corresponding depth, optical flow and normal maps. Non-rigid deformations are simulated by smooth deformations of all scene parts.
The Drunkard's Dataset is publicly available 19 different deforming indoor scenes, where each one has been recorded four times, one for each difficulty level. For each level, we generated over 100K frames and recorded camera poses, RGB images, depths, optical flow and normal maps at 1920 x 1920, being the camera poses and the depth consistent real-world metric scale throughout different scenes. Find sample images Figure 3.
19 different camera trajectory recordings of
We used Blender [68] to render the deformations of the 19 real-world scanned indoor 3D models of the Habitat-Matterpor 3D dataset [27]. We manually designed camera trajectories such that every room in each building is visited once and the camera trajectory returns at the starting point, except for scenes 4, and 14, in which the camera traverses the building three times, but in each loop visiting the rooms in different order. The Blender files are publicly available, along with the scripts we used for generating them, so that anyone can render in in-house versions of the Drunkard's Dataset scenes modifying parameters of deformations, pose trajectory, resolution or camera type among others.
As Table details, level 0 stands for zero deformation and camera noise, resulting in rigid scene and smooth camera motion, well suited for rigid SIM/SLAM methods. The following levels have an increasing degree of deformation and trajectory noise. Having four levels of difficulty allows benchmark both rigid and non-rigid methods in a graduated manner.
Diffitlty level
Deformatious
Trajectory noise
Table Overview of the four difficulty levels. Defor- mation and camera trajectory noises increases with higher levels. At level 0 there is neither deformation nor camera noise, i.e. it represents rigid scene and smooth motion.
Lovel
Level
Level
Medium High
Medium
Level
High

---

The Drunkard's Odometry
Given pair of RGB-D images (I, Z. I, E R@xxx3 and Z₁ Ruwh, i (1,2), our Drunkard's Odometry estimates dense scene flow T SE(3) between them, and the relative camera motion T E SE(3) iteratively (k: E (1...., M) stands for the iterative block step). T contains pixel-wise rigid-body transformations that ideally, i.e. in absence of noise, maps every 3D point P, π (u,, (g) -corresponding pixel j I1. back-projected from the pixel with image coordinates uj E S4 (St, is the image domain for I1) and its sensor (ground truth) depth 2j = [1,]]] to its ground truth equivalent 3D point Py- - (ūy 2g.) back-projecto from the true associated pixel with image coordinates u,' E S2 (522 the image domain for 12) and sensor depth E Zg أ -- (u,=) =) stands for the inverse projection model. As the estimated scene flow T might be affected by noise, we will denote as P, = T [u,] P3 the transformation of P3 to the local frame of 12, that in general will not coincide the true corresponding point p,. -r T [uy] is composed of the rigid transformation coming from the camera motion Th and the coming from the non-rigid surface deformations T [u,] SE(3), that T and if is deforming Pl T [u,]P;
[uy] T [uy].
If the scene is rigid P T.P.
lientire block
Few g
Fostuee medder
somaned
A rorkkal
raps
Coenest cocsder
Cersiiaka Inarace
Tebll
Eav()
BGB
Floer apáika
Invane Sipth
IrieTic Akyli
Enteeiings coniiianc
Cyelate llock
Hrkkee
Prie twoll
Porc aetwark
Figure 4: Drunkard's Odometry architecture in detail.
Our iterative pose estimation model, shown in detail in Figure 4, is based the foundations of the 3D flow estimation architecture of RAFT-3D [35], which does not have pose estimation capabilities all. Firstly, pose regression network encodes both color images I and I2 and outputs a initial pre-estimate for the camera motion Thee This rigid transformation is used for initialization, so that for any u, pixel of camera Thr [u,] The Additionally, we encode both images I and I2 in two feature maps, which we use to build a 4D correlation pyramid between the features of all pixel pairs
four different scales, each scale halving the resolution of the previous one. From T and Z, at each iteration k; we can obtain dense 2D pixel correspondences U1 -> 11y
=
Ey), T (uyl)
(1)
where (P, T [u]) stands for the pinhole camera projection model. At the beginning of each iteration,
these correspondences are used to sample the correlation features from the fixed 4D correlation
pyramid and one of the inputs of the update block. The estimated optical flow O I mxxxx
coming from these correspondences is also an input for the update block, being O [u,] u u,
The update block also takes input inverse depth residual obtained from the difference between
the estimated inverse depth map A E Ruxh and the sensor inverse depth Ag E Rwxh A₂ [u]]
1/2, [uy], Vul. E Ω2. Values for A are interpolated from the grid defined by N2.y A uu e pl. e3 (00 1) acts selector of the third dimension of the 3D point.
5

---

A pair of context features that extracts semantic and contextual information from image I are also pre-rendered before entering the iterative process using context encoder network and given to the update block. One context feature map is kept fixed during all iterations and another is used as initialization of the hidden state of the General Recurrent Unit (GRU) [69], which at the heart of the update block (see the network fine-grained details in the supplementary material).
At each iteration the estimates for T and T are mapped the Lie algebra with logarithm map to result in twist field t" logsr(3) (T*) and t logss(3))
(TH) before being given to the update block. The update operator outputs a set of updates for the optical flow O*, for the twist camera pose E and for the inverse depth Ač. the hidden state of the recurrent network, mask and set of rigid-motion embeddings and confidence maps. These last two maps, next to the updated estimate of A and O$ and Z are used by the least-squares optimizer block to update T (see details in [35] for this update). Scene flow is updated as T = T exPsE(3) (t*).
Internally the network works at 1/8th of the original resolution, and the estimated mask at each iteration of the update block is used to perform a convex upsampling to the original resolution of T and intermediate updated 2D flow O E R=xxx2 by the update block.
The supervision comes from comparing the estimated optical flow O* obtained from T and Z, with Eq. and the intermediate pre-estimated O with the ground truth optical flow O E R=xAx2 the inverse depth error between A₂ and A2, the relative camera pose error of Te and the initial guess pre-estimated by the pose network To against Te. The total loss results in
M
L =
Lre pose
+
Σ k=
C M-k (Chow + Cippth + Spisse
with klow = En; (1 O11 + wi 0 - Ö111) being the optical flow loss term, Lapp = W12 En,, || A A [1,] Ag[u,]] 1I the inverse depth loss term, and f wys w3|llogsE(3) (T.T= [| and Cpose wa ||logse(3) (T 1)11 the relative camera pose loss terms. wy stands for the relative weight of the 1th loss term and y weights each loop.
Experiments
In this section, we show the evaluation results of our Drunkard's Odometry against several relevant baselines in two non-rigid datasets: our synthetic Drunkard's ' and the real Hamlyn data [39, 30, 40, 41].
Drunkard's Setup. We trained our Drunkard's 's Odometry in all scenes of the Drunkard's Dataset except the test ones, with around 90 - 10% ratio for training and testing. respectively, of the difficulty level with an input resolution of 320 x 320, batch size of 12, learning rate of 10 Adam optimizer [70], weight decay of 10 10-5, 12 iterations of the iterative block during training and test, hyperparameters w11 = 0.2, w² = 100, wz = 200 and u² 6, and during 10 epochs (~~ 3 days) on a single RTX Nvidia Titan. An ablation study available in the Supplementary Material was performed to obtain the hyperparameter values. Everything was trained from scratch except for the pose network encoder which was pre-trained on ImageNet [71].
Drunkard's Benchmark. We put to fight our Drunkard's 's Odometry in the Drunkard's Dataset benchmark against the gold-standard SfM pipeline COLMAP [72, 73], that searches for matches across all images in an offline manner using only the RGB channels of our images; the robust and accurate DROID-SLAM [34] that uses combination of local-online and global-offling bundle adjustment refinements, trained also in virtual environments buildings with optical flow and with a well-proved generalization capability between datasets, evaluated using the same RGB-D images as Drunkard's Odometry; and the frame-to- to-fame tracking designed for non non-rigid endoscopic scenes Mado-Depth-anp -nt-Motion (EDaM) [9] that here uses the RGB plus the single-view view estimated depth maps by network trained self-supervisely on monocular images of KITTI [43]. The non-rigid SfM SD-DefSLAM [26] was tested but fails in the beginning of the sequences. Complex non-rigid optimization methods such as this one are unstable and tend to fail in difficult sequences with complicated 3D surfaces and abrupt camera trajectories like the ones in our Drunkard's 's Dataset.

---

Level D
Level 1
Level 2
Level 3
Align-
frames [9:]
RPE
RPE ATE
frames
RPE
RPE ATE
ftames
RPE
RPE ATE
frames [%]1
RPE
RPE ATE
Scene Method
ment
[em]4
['14
[m]]]
[8]1
[em]] 1'14 [m]]
[em]]
1"1
[m]
[em]]
I'1L
[m]
COLMAP [72]
Sim(3)
42
0.36
0.10
0.11
42
0.85
0.18
0.12
42
1.89
0.35
0.81
25
3.64
D.51
1.14
DROID-SLAM [4]
SE(3)
100
0.77
0.28
2.38
100
0.42
1.79
100
2.53
1.26
100
EDaM [9]
Sim(3)
100
1.83
1.21
1.49
100
1.82
1.37
1.83
100
2.05
1.72
1.95
100
2.66
2.27
2.01
EDaM w/ GT depth
SE(3)
100
2.05
1.13
2.01
100
2.28
1.28
1.78
100
2.89
1.17
2.24
100
3.57
2.15
2.42
Drunkard's Odometry
SE(3)
100
0.34
0.10
0.67
100
0.59
0.16
1.08
100
1.14
0.28
1.35
100
1.2
0.48
COLMAP [72]
(im(()
32
0.38
0.083
0.10
32
1.2
0.15
0.24
32
3.12
0.30
0.90
23
8.59
0.71 2.31
DROID-SLAM [34]
SE(33
EDDM [9]
(im
100
5.55
2.16
4.85
100
5.27
2.27
4.81
100
5.39
2.56
4.79
100
5 88
2.96 4.90
EDaM w/ depth
SE(3)
100
6.01
2.02
7.58
100
2.14
7.26
100
6.31
2.40
7.94
30
6.81
2.90
2.60
Drunkard's Odometry
SE(3)
100
0.60
0.14
1.21
100
0.83
0.18
1.39
100
1.43
0.28
2:46
100
2.26
0.46
4.66
COLMAP 172]
Sim(3)
100
0.40
0.08
0.20
8U
1.12
0.16
0.53
100
3.58
0.356
1.38
31
4.95
0.46
2.45
DROID SLAM [34]
SE(3)
100
0.56
0.21
1.25
100
1.52
0.39
1.56
100
3,16
0.67
2.43
100
4.69
1.02 2.70
EDaM [9]
Sim(3)
100
3.05
1,98
2.82
100
3.13
2.11
2.73
100
3.57
2.46
2.99
100
4.12
2.98 2.86
EDaM depth SE(3)
100
4.63
1.95
4.00
2.07
100
5.21
4.19
100
Drunkard's ' Odometryy SE((3
100
0.45
0.13
0.47
100
0.74
0.18
0.70
100
1.44
0.29
1.24
100
2.40
0.49 2.45
Table 3: Trajectory errors for Drunkard's test scenes for all difficulty levels. Note that COLMAP an offline method and is only shown for reference. Our odometry method mostly outperforms the compared online odometry methods. Best results are highlighted as first. second, and third.
We chose scenes and 5 as the test ones with 3.816, 4.545 and 1.655 frames respectively. The rest of the scenes were used for training. The trajectories estimated by COLMAP and EDaM are compared against the ground truth after Sim(3) alignment, as they are up-to-scale. DROID-SLAM and our Drunkard's s are aligned to the ground truth with SE(3) transformation, as the RGB-D input allows them to estimate the real scale. As using Sim(3) against SE(3) is not totally fair since the former applies a scale correction, we also test EDaM using the ground truth depth maps so we can apply SE(3) alignment. The reported metrics are: Relative Position Error (RPE) for translation and rotation, that measures the local accuracy of the estimated trajectory against the ground truth between consecutive frames, and the Absolute Trajectory Error (ATE) for translation that computes the global consistency between both trajectories (see [74] for details).
For each sequence, the percentage of registered frames over the total is shown, a metric in which COLMAP shows poor performance. As a consequence, its trajectory metrics are influenced positively as it excludes frames that are challenging to track and probably would have increased the error. This is beneficial in particular for the ATE, as it takes into account the global consistency rather than frame-to-free frame errors like the RPE, and happens earlier higher deforming scenes. Also note that BROID-SLAM is very GPU-memory demanding, in part because of the final global bundle adjustment and it is not able work with long sequences like Scene However, our Drunkard's Odometry and EDaM are more robust, partly due to tracking only between adjacent frames.
Table shows our results. Note that our Drunkard's Odometry practically always outperforms DROID-SLAM and EDaM (with and without ground truth depth) in RPE and ATE, also in rigid scenes, even if our model is trained exclusively in non-rigid scenes and does not use loop closure or full bundle adjustment. The gap is significantly larger at higher deformation levels, for which the Drunkard's Odometry errors increase much less. This demonstrates that our method is able to generalize at predicting surface deformations. Only COLMAP is able to outperform our Drunkard's Odometry in ATE Note, in any case, that COLMAP has a much lower recall, as it only estimates camera motion for substantially lower percentage of frames. If we focus on RPF, more fair metric in this case, our Drunkard's Odometry is on par to COLMAP at lower deformation levels, and clearly outperforms it at higher ones.
Validation in real endoscopies. We used the Hamlyn dataset [30], that contains intracorporeal endoscopic RGB scenes with weak textures, deformations and reflections. Specifically, we chose scenes and 17 (see Figure 5). which are significant exploratory ones. Most of Hamlyn's 's videos have very small camera motions, being of no interest for benchmarking odometry methods. We slightly cropped the images to remove black pixels at the borders. Depth data was taken from the public tracking test data of EDaM [9] which was estimated by a single-view dense depth network trained in a self-supe manner in all Hamlyn scenes except for the test ones. Note that this depth does not have the same quality the real ground truth one from the Drunkard's 's Dataset.
7

---

Scene #Frames Method
APTE,
EDaM
0.0044
DROID-SLAM [:4]
0.0019
1
1058
Drunkard's trained w/o deform.
0.0055
Drunkard's trained w/ deform.
0.0045
0,0236
DROID-STAM [14]
0.0338
17
Drunkard's x trained wlo deform.
0.0309
Drunkard's 's trained w/ deform.
0.0214
(a) Scene
(b) Scene 17
(c) Tracking Results
Figure 5: Sample color and depth frames and APTE results for the Hamlyn test videos.
Since the Hamlyn dataset lacks ground truth camera poses, we propose a novel ground truth-fres trajectory metric to measure the quality of the estimated odometries. The key idea is to generate loopy videos by duplicating and reversing given image sequence and concatenating to the end of the original sequence. The generated "palindrome video" is twice as long as the original sequence and any tracking method should ideally loop back its starting position. We then simply measure the discrepancy between start and end pose and accumulate it over various loop lengths. We denote the new metric as the Absolute Palindrome Trajectory Error (APTE) and define it as
APTE = ΣN APTER N
|(1i
k
k (IT
with APTER
T j=1
back
Tej
(3)
where N the total number of frames and APTF& the translation Root Mean Squared Error (RMSH) of between the first and last pose of the sub-trajectory loop k. This last pose of the loop l is the result of sequentially applying to the starting camera pose -the identity I E SE(3)- the first I estimated relative camera poses To for the scene plus the first & estimated relative camera poses for the same scene but ran backwards T back Then, the APTE& the module of the translation of this last estimated pose. We note that the APTE error is trivially minimized all transformations are zero or close to the origin, but these cases are easy check visualizing the trajectory.
Table 5c shows our APTE metric in the Hamlyn test scenes for the basclines that are able to track the whole scene (EDaM and PROID-SLAM. and two versions of our Drunkard's Odometry, one trained only in rigid scenes of the Drunkard's Dataset, i.e. scenes from difficulty level 0, and other in deformable scenes, specifically from level 1. SD-DefSLAM [26] was also tested here and breaks after a few frames, far from the full lenght of the trajectory, even having originally been built and tested in Hamlyn. In consequence, it is not possible compute the APTE since we need the full trajectory run forward and backward. Before computing the A ATE,, all the trajectories are scaled with the given by Sim(3) alignment, having as reference the trajectory estimated by EDaM, since it showed good qualitative tracking performance in Endo- Endo-Depth-aee Depth-and ant-ootion [9].
Figure 6 helps to visualize how the APTEş values evolve as the length of the loops grows. Notice that a longer loop does not always mean higher APTE,, This is because there can be some specific problematic frames with bad pose estimation that induce a drift accumulation in the following ones getting worse APTE& measures. However, this can be reversed if the tracking recovers partially during the way back in the loop. The single APTE values shown in the main paper are the average of all the A APTE, measures.
Figure 7 collects the estimated 3D camera trajectories in the Hamlyn test scenes for all the baselines. is noteworthy how PROID-SLAM, is incorrectly producing sharp and chaotic trajectories around the origin, particularly evident in Scene I and partially in Scene 17. This can be further observed in Figure 6b, where the plot of DROID-SLAM becomes similar to the others in the second half of the video. The other methods produce smoother and plausible estimates. The qualitative evaluation of visualizing the trajectories and the quantitative by the APTE complement each other. The former helps in recognizing when method is producing entirely incorrect camera poses, while the latter provides an objective measure for comparing methods when the trajectories already appear good and plausible.
One clear conclusion we can extract that our model performs better if it trained in non-rigid scenes rather than rigid. Again, this shows its capacity to learn deformation patterns and retrieve
8

---

0.053
DRGID-SLA Durs ni deformat on Cu' nilo defomaaionn
IDaM
0.07
EDoM DROID SLAM
Oura wr deformation
Ours wit deformation
0.04
0.010
0.005
0.01
0.000
0.00
200
400
6:00
800
1200
200
400
6.00
800
1000
12000
Loop length number frammas
Loop langth k.tn number of frammss
(a) APTE; plots in Hamlyn's Scene 1.
(b) APTE, plots in Hamlyn's 's Scene 17.
Figure 6: Camera tracking performance on the Hamlyn dataset. APTER values along the N different d-frames-length-loope in Hamlyn's 's scenes 1 (a) and 17 (b) for EDaM [9], DROID- SLAM [34] and Drunkard's s Odometry (ours) with and without having been trained in deformable scenes of the Drunkard's Dataset (i.e. trained in level and respectively). The training deforming scene substantially improves the performance of our method.
Scenee
Score
Figure 7: Estimated 3D trajectories in Hamlyn test scenes for different methods. See animations with Adobe Reader, KDE Okular or YouTube.
more stable camera pose under real-world non-rigid challenging scenes, and even trained in different domain. Besides, DROID-SLAM's superiority of APTE in Scene results invalidated by observing how it completely fails to estimate the trajectory in Figure This validates the robustness of the Drunkard's 's Odometry pipeline in addressing the domain gap problem. Another interesting point to comment on is that the advantage of our Drunkard's Odometry over EDaM may not be clear enough. Their performances are similar in Scene 1, while Drunkard's is better for Scene 17. The evaluation in our dataset shows that our Drunkard's Odometry is significantly better than EDaM. This indeed shows the motivation and benefit for our Drunkard's Dataset: looking at Table 3 we can confidently claim that our Drunkard's 's Odometry is better than EDaM for camera trajectory estimation. The apparent disagreement with the Hamlyn evaluation may arise from the fact that deformations not graduated in Hamlyn, and qualitatively they are very small, which may benefit rigid methods.
Limitations. Our Drunkard's ' Dataset's 's most clear limitation is that it is synthetic. However, we argued, we believe that the difficulties for acquiring high-qualit, data with ground truth annotations in the target application domains motivates their use. Notice that it is not possible to record true ground truth optical flow in real-world sequences, as we do not have access to the exact pixel motion information. It is precisely this optical flow availability which unlocks the use of powerful flow-based models trained synthetic deformable data which generalize well to real non-rigid scenes. Despite the real indoor images of the Drunkard's Dataset may resemble real deformable scenes in texture and

---

shape, such as bouney castles, moving fabrics or canvas, they are far from the medical application environment. Still, we think that generating medical data with realistic deformations, fluids textures in large scale is out of reach. As proof, such data does not exist yet and motivates the use of the Drunkard's data. Our Drunkard's Odometry has all the limitations inherent to a frame-to-frame tracking method. Drift accumulates very quickly, and even if our sequences are loopy we do not either detect loop closures or correct our trajectories based on them. However, the SLAM literature shows that SLAM methods (e.g.. (75]) can be built on top of odometry ones (e.g.. [76]).
6 Conclusions
Estimating camera motion in deformable scenes is challenging research problem relatively under- explored in the literature, and for which a lack of clear benchmarks slows down research progress. In this work, we created the Drunkard's is Dataset, a large-scale simulated dataset with perfect ground truth and a wide variety of scenes and deformation levels to train and validate deep neural models. In addition, we propose the Drunkard's Odometry method for deformable scenes validate our dataset. The method minimizes a scene flow loss, but as its main contribution, intrinsically decomposes the estimated twist flow into two components: The majority of motion is aimed to be explained by rigid-body camera motion, and all remaining motion is explained by scene deformations. In contrast to most existing works our method does not require a static scene part for estimating a reference coordinate frame which is crucial in fully deforming scenarios like endoscopy. To also assess odometry estimates the absence of ground truth data, we further define a novel ground truth-free metric for trajectory evaluation that measures the cyclic consistency of a tracking algorithm. Both the dataset and source code for our baseline method are publicly available. Our experimental results validate our dataset, illustrates its challenges, and also shows that our Drunkard's Odometry is able to outperform relevant baselines.
Acknowledgements. This work was supported by the EU Comission (EU-H2020 EndoMapper GA 863 863146), the Spanish Government (PID202I-12768SNBB -127685NB-1000 and 1ED2021-13115081000 the Aragon Government (DGA-T45 17R/FSE). and a research grant from FIFA.
References
[1] A. Badias, I. Alfaro, D. Gonzalez, F. Chinesta, and E. Cueto, "Morph-dslam: Model order reduction for chysics-based deformable slam," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 11, pp. 7764-7777, 2021.
[2] T. Li, S. Liu, T. Bolkart, J. Liu, H. and and Y. Zhao, "Topologically consistent multi-view face inference using volumetric sampling." in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3824-3834, 2021. 2,
[3] N. Qian, J. Wang. F. Mueller, F. Bernard, V. Golyanik, and C. Theobalt, "Html: parametric hand texture model for 3d hand reconstruction and personalization," in Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pp. 54-71, Springer, 2020. 1
[4] J. Wang, S. Tan, X. Zhen, S. Xu, F. Zheng, Z. He. and L. Shao, "Deep 3d human pose estimation: A review." Computer Vision and Image Understanding, vol. 210, p. 103225, 2021. 1
[5] C. Li and G. H. Lee, "Coarse- to- fone animal pose and shape estimation." Advances in Neural Information Processing Systems, vol. 34, pp. 1757-11766 2021.
[6] Z. Su, W. Wan, T. Yu, L. Liu, L. Fang, W. Wang, and Y. Liu, "Mulaycap: Multi-layer human performance capture using a monocular video camera," IEEE Transactions on Visualization and Computer Graphics, vol. 28, no. 4, pp. 1862-1879, 2020.
[7] J. Lamarca, S. Parashar, A. Bartoli, and J. Montiel, "Defslam: Tracking and mapping of deforming scenes from monocular sequences," IEEE Transactions robotics, vol. 37, no. 1, pp. 291-303, 2020. 1,4
[8] A. Sengupta and A. Bartoli, "Colonoscopic 3d reconstruction by tubular non-rigid stuucure-frommmmotionn" International Journal of Computer Assisted Radiology and Surgery, vol. 16, no. 7, pp. 1237-1241, 2021. 1,
[9] D. Recasens, J. Lamarca, J. M. Fácil, J. Montiel, and J. Civera, "Endo-depth-an -ndttooond-oo reconstruction and tracking in endoscopic videos using depth networks and photometric constraints," IEEE Robotics and Automation Letters, vol. no. 4, pp. 7225-7232, 2021. 1, 4, 6. 7, 9
[10] S. Shao, Z. Pei. W. Chen, W. Zhu, X. Wu. D. Sun, and B. Zhang, "Self supervied monocular depth and ego-motion estimation in endoscopy: Appearance flow to the rescue," Medical image analysis, vol. 77, p. 102338, 2022. 1,4
10

---

[11] V. Rabaud and S. Belongie, "Re-thinking non-rigid structure from motion," in 2008 IEEE Conference on Computer Vision and Puttern Recognition, pp. 1-8, IEEE, 2008. 1
[12] P. E. Gotardo and A. M. Martinez, "Non-rigid structure from motion with complementary rank-3 spaces," in CVPR 2011, pp. 3065 -0722, IEEE, 2011. 1
[13] M. Lee, J. Cho, C.-H. Choi, and S. Oh. "Procrustean normal distribution for non-rigid structure from motion," in Proceedings of the IEEE Conference on computer vision and pattern recognition, pp. 1280-
1287, 2013. 1
[14] R. Garg, A. Roussos, and L. Agapito, "Dense variational reconstruction of non-rigid surfaces from monocular vidco." in Proceedings of the IEEE Conference on computer vision and pattern recognition,
pp. 1272-1279, 2013. 1
[15] Y. Dai, H. Li, and M. He, "A simple prior-free methad for non-rigid fructure-from-mootie factorization," International Journal of Computer Vision, vol. 107. pp. 101-122, 2014.
[16] A. Agudo, L. Agapito, B. Calvo, and J. M. Montiel, "Good vibrations: A modal analysis approach for sequential non-rigid structure from motion," " Proceedings of the IEEE Conference on computer vision and pattrrn recognition, pp. 1558-1565, 2014. 1
[17] C. Kong and S. Lucey, "Deep non-rigid structure from motion," in Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 1558-1567, 2019. 1
[18] A. Bozic, M. Zollhofer, C. Theobalt, and M. Nießner, "Deepdeform: Learning non-rigid rgb-d reconstruc- tion with semi- supervise data," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7002-7012. 2020.
[19] G. Yang, D. Sun, V. Jampani, D. Vlasic, F. Cole, H. Chang, D. Ramanan, W. T. Freeman, and C. Liu, "Lasr: Learning articulated shape reconstruction from a monocular video." in Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 15980- 15989, 2021.
[20] E. Tretschk. A. Tewari, V. Golyanik, M. Zollhöfer, C. Lassner, and C. Theobalt, "Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video." in Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 1 12959-12970. 2021. 1
[21] A. Pumarola, E. Corona, G. Pons-Moll, and F. Moreno Moreno-Noguer, "D-nerf: Neural radiance fields for dynamic scenes," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
pp. 10318-10327, 2021. 1
[22] G. Potje, R. Martins, F. Chamone, and E. Nascimento, "Extracting deformation-awan local features by learning to deform," Advances in Neural Information Processing Systems, vol. 34, pp. 10759-10771, 2021.
[23] S. Parashar, D. Pizarro, and A. Bartoli, "Robust isometric non-rigid structure- mrommmmooomm IEEE Transac- tions on Pattern Analysis and Machine Intelligence, vol. 44, no. 10, pp. 6409-6423, 2021.
[24] C.-Y. Weng, B. Curless, P. P. Srinivasan, J. T. Barron, and I. Kemellaacher-Shiizermaa "Humannerf: Free-viewpoint rendering of moving people from monocular video," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16210-16222 2022.
[25] C. Cadena, L. Carlone, H. Carrillo, Y. Latif, D. Scaramuzza, J. Neira. I. Reid, and J. Leonard, "Past, present, and future of simultaneous localization and mapping: Toward the robust-pereeption age." IEEE Transactions on robotics, vol. 32, no. 6, pp. 1309-1332, 2016.
[26] J. J. Gómez- Rodríguez, J. Lamarca, J. Morlana, J. D. Tardós, and J. M. Montiel, "Sd-defslam: Semi-direct monocular slam for deformable and intracorporeal scenes," in 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 5170-5177, IEEE, 2021. 1,4, 6, 8
[27] S. K. Ramakrishnan, A. Gokaslan, F. Wijmans, O. Maksymets, A. Clegg, J. Turner, E. Undersander, W. Galuba, A. Westhury, A. X. Chang, al., "Habitat-matterport 3d dataset (hm3dy: 1000 large-scale 3d environments for embodied ai," arXiv preprint
htt::2109.082.31 2021. 2, 3, 4
[28] D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black, "A naturalistic open source movie for optical flow evaluation," in Computer Vision-ECCV 2012: 12th European Conference Computer Vision, Florence, Itly, October 7-13, 2012, Proceedings. Part VI 12, pp. 1-625, Springer, 2012. 2,3
[29] N. Mayer, F. 1lg, P. Hausser, P. Fischer, D. Cremers, A. Dosovitskiy, and T. Brox, "A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation." in Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4040-4044 2016. 2,
[30] D. Stoyanov, G. P. Mylonas, F. Deligianni, A. Darzi, and G.-Z. Yang, "Soft-tissue motion tracking and structure estimation for robotic assisted mis procedures," in MICCAI (2), pp. 139-146, 146, 2005. 2, 3, 6, 7
[31] D. Stoyanov, "Stereoscopic scene flow for robotic assisted minimally invasive surgery," in Medical Image Computing and Compuder-Assiste Intervention Intervention-MCCAll MICCAI 2012: 15th International Conference, Nice, France, October 1-5, 2012, Proceedings, Part I 15, pp. 479 486, Springer, 2012. 2
11

---

[32] S. H. N. Jensen. M. E. B. Doest, H. Aanæs, and A. Del Bue. "A benchmark and evaluation of non-rigid structure from motion," " International Journal of Computer Vision, vol. 129, no. 4, pp. 882-899, 2021. 2,
[33] P. Azagra, C. Sostres, A. Ferrandez, L. Riazuclo, C. Tomasini, O. L. Barbed, J. Morlana, D. Recasens, V. Batlle, J. Gómez- Rodríguez, et al., "Endomapper datasct of complete calibrated endoscopy procedures," arXiv preprint arXiv:2204. 14240, 2022. 2,3
[34] Z. Teed and J. Deng. "Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras," Advances in Neural Information Processing Systems, vol. 34, 2021. 2, 6, 7, 8,
[35] Z. Teed and J. Deng, "Raft-3d: Scene flow using rigid-motion embeddings." in Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 8375-8384, 2021. 5,6
[36] E. De Aguiar, C. Stoll, C. Theobalt, N. Ahmed, H.-P. Seidel, and S. Thrun, "Performance capture from sparse multi-view video," in ACM SIGGRAPHI 2008 papers. pp. 1-10, 2008. 3
[37] F. Bogo, J. Romero, M. Loper, and J. Black, "Faust: Dataset and evaluation for 3d mesh registration," in Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 3794-3801, 2014. 3
[38] Y. Li, H. Takehara, T. Taketomi, B. Zheng, and M. Nießner, "Adcomplete: Non-rigid motion estimation beyond the observable surface," " in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12706-12716, 2021. 3
[39] P. Mountney, D. Stoyanov, and G.-Z. Yang. dinees-iimensionll tissue deformation recovery and tracking," IEEE Signal Processing Magazine, vol. 27, no. 4, pp. 14 24, 2010. 3, 6
[40] D. Stoyanov, M. V. Scarzanella, P. Pratt, and G.-Z. Yang. "Real-time stereo reconstruction in robotically assisted minimally invasive surgery," International Conference on Medical Image Computing and Computer-Assisted Intervention, 2010. 3, 6
[41] P. Pratt, D. Stoyanov, M. Visentini-Scarzanele Scarzanella,, and G.ZZ. Yang. "Dynamic guidance for robotic surgery using image-constrained biomechanical models." in International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 77-85, 2010. 3,
[42] J. Sturm, N. Engelhard, F, Endres, W. Burgard, and D. Cremers, "A benchmark for the evaluation of rgb-d slam systems," " in 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 573-580, IEEE, 2012. 3
[43] A. Geiger, Lenz, C. Stiller, and R. Urtasun, "Vision meets roboties: The kitti dataset," The International Journal of Robotics Research, vol. 32, no. 11. pp. 1231-1237, 2013. 3,6
[44] K. Wilson and N. Snavely, "Robust global translations with dsffm, in Computer Vision-ECCY 2014: 13th European Conference, Zurich, Switzerland, September 6- 2014, Proceedings, Part III 13, pp. 61-75, Springer, 2014.
[45] M. Burri, J. Nikolic, P. Gohl, T. Schneider, J. Rehder, S. Omari, M. W. Achtelik, and R. Siegwart, "The euroc micro aerial vehicle datasets," " The International Journal of Robotics Research, vol. 35, no. 10,
pp. 1157-1163 16,, 2016. 3
[46] A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner, "Scannet: Richly annotaeed 3d reconstructions of indoor scenes," in Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 5828-5839, 2017.
[47] D. Schubert, T. Goll, N. Demmel, V. Usenko, J. Stückler, and D. Cremers, "The tum vi benehmark for evaluating visual- inertial odometry." in 2018 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 1680-1664 1687, JEEE, 2018. 3
[48] T. Schops, T. Sattler, and M. Pollefeys. "Bad slam: Bundle adjusted direct rgb-d slam." in Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 134-144, 2019. 3
[49] A. Handa, T. Whelan, J. McDonald, and A. J. Davison, "A benchmark for rgb-d visual odometry, 3d reconstruction and slam," " in 2014 IEEE international conference on Robotics and automation (ICRA).
pp. 1524-1531, IEEE, 2014.
[50] J. Straub, T. Whelan, L. Ma, Y. Chen, E. Wijmans, S. Green, J. J. Engel, R. Mur-Artal, C. Ren, Verma, et al., "The replica dataset: A digital replica of indoor spaces," arXiv preprint arXiv: 1906.05797, 2019.
[51] W. Wang, D. Zhu, X. Wang, Y. Hu, Y. Qiu, C. Wang, Y. Hu, A. Kapoor, and S. Scherer, "Tartanair: A dataset push the limits of visual slam," in 2020 IEFF/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 4909-4916, IEEE, 2020. 3
[52] K. Minoda, F. Schilling, V. Wüest, D. Florcano, and T. Yairi, "Viode: A simulated datasct to address the challenges of visual-inertial odometry in dynamic environments," IEEE Robotics and Automation Letters, vol. 6, no. 2, pp. 1343-1350, 2021. 3
[53] S. Baker, D. Scharstein, J. Lewis, S. Roth, M. J. Black, and R. Szeliski, "A database and evaluation methodology for optical flow," International journal computer vision, vol. 92, pp. 1-31, 2011.
12

---

[54] A. Dosovitskiy, P. Fischer, E. 1lg, P. Hausser. C. Hazirbas. V. Golkov, P. Van Der Smagt, D. Cremers. and T. Brox. "Flownet: Learning optical flow with convolutional networks," in Proceedings of the IEEE international conference on computer vision, pp. 2758-2766, 2015.
[55] E. Ilg, T. Saikia, M. Keuper, and T. Brox, "Occlusions, motion and depth boundaries with a generic network for disparity, optical flow scene flow estimation," in Proceedings of the European conference
computer vision (ECCV), pp. 614-630, 2018. 3
[56] L. Mehl, J. Schmalfuss, A. Jahedi, Y. Nalivayko, and A. Bruhn, "Spring: A high-resolution high-detail dataset and benchmark for scene flow, optical flow and stereo," arXiv preprint arXiv:2303.01943 2023. 3
[57] F. M. Carlucci, P. Russo, and B. Caputo. "A deep representation for depth images from synthetic data." in 2017 TEEE international conference on robotics and automation (ICRA), pp. 1362-1369, TEFE, 2017.
[58] B. Planche, Z. Wu, K. Ma, S. Sun, S. Kluckner, O. Lehmann, T. Chen, A. Hutter, S. Zakharov, H. Kosch, et al., "Depthsynth: Real-time realistic synthctic data generation from cad models for 2.5 d recognition," in 2017 International Conference on 3D Vision (3DV), pp. 1-10, IEEE, 2017. 3
[59] A. X. Chang. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song. H. Su, et al., "Shapenet: An information-rich 3d model repository." arXiv preprint arXiv:1512.03012
2015. 3
[60] A. Gaidon, Q. Wang, Y. Cabon, and E. Vig, "Virtual worlds proxy for multi-object tracking analysis." in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4340-4349, 2016.
[61] J. McCormac, A. Handa, S. Leutenegger, and A. J. Davison, "Scenenet rgh-d: Can 5m synthetic images beat generic imagenet pre-training on indoor segmentation?," in Proceedings of the IEFF International Conference on Computer Vision, pp. 2678-2678802 2687, 3017.
[62] J. Johnson, B. Hariharan, L. Van Der Maaten, L. Fei-Fei, C. Lawrence Zitnick, and R. Girshick, "Clevr: A diagnostic dataset for compositional language and elementary visual reasoning." in Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2901-2910, 2017.
[63] M. Roberts, J. Ramapuram, A. Ranjan, A. Kumar, M. A. Bautista, N. Paczan, R. Webb, and J. M. Susskind, "Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding." in Proceedings of the IEEF/CVF international conference on computer vision, pp. 10912-10922, 2021.
[64] S. L. Nikolenko, Synthetic data for deep learning, vol. 174. Springer, 2021. 3
[65] Song. J. Wang. L. Zhao, S. Huang, and G. Dissanayake, "Mis-slam: Real-time large-scale dense deformable slam system in minimal invasive surgery based on heterogeneous computing." " IEEE Robotics and Automation Letters, vol. 3, no. 4. pp. 4068 4075, 2018. 4
[66] J. Lamarca, J. G. Rodriguez, D. Tardos, and J. M. Montiel, "Direct and sparse deformable tracking," " IEEE Robotics and Automation Letters, vol. 7, no. 4, pp. 11450-11457, 2022.
[67] J. J. G. Rodríguez, J. M. Montiel, and J. D. Tardós, "Tracking monocular camera pose and deformation for slam inside the human body." in 2022 IEFP/RS] International Conference on Intelligent Robots and Systems (IROS). pp. 5278-5285, IEEE, 2022. 4
[68] B. O. Community, Blender a 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018. 4
[69] K. Cho, B. Van Merrienboer, D. Bahdanau, and Y. Bengio, "On the properties of neural machine translation: Encoder-decoder approaches," arXiv preprint arXiv:/409 /259, 2014.
[70] I. Loshchilov and F. Hutter, "Decoupled weight decay regularization," arXiv preprint 1rXiv:1711.05101,
2017. 6
[71] J. Deng, W. Dong, R. Socher, I..J. Li, K. Li, and I.. Fei-Fei, "Imagenet: A large-scale hicrarchical image database," 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255, leee, 2009.
[72] J. L. Schönberger and J.-M. Frahm, "Structure-from-mottoe revisited," in Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 6.7
[73] J. L.. Schönberger, F., Zheng, M. Pollefeys, and J.-M. Frahm, "Pixelwise view selection for unstructured multi-view stereo," in European Conference on Computer Vision (ECCV), 2016. 6
[74] D. Prokhorov, D. Zhukov, O. Barinova, K. Anton, and A. Vorontsova, "Measuring robustness of visual slam," in 2019 16th International Conference on Machine Vision Applications (MVA), pp. 1-6, IEEE, 2019.
[75] X. Gao. R. Wang, N. Demmel, and D. Cremers. "Ldso: Direct sparse odometry with loop closure." in 2018 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 2198 2204, IEEE, 2018. 10
[76] J. Engel, V. Koltun, and D. Cremers, "Direct sparse odometry," IEEE transactions on pattern analysis and machine intelligence, vol. 40, no. 3, pp. 611 611-625, 2017. 10
13


Reviews:
[{"review_id": "y4rbqB60S8", "reviewer": "Reviewer_etHK", "summary": "", "questions": "", "limitations": "The limitations readers may have are well discussed.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The paper proposed useful synthetic dataset for odometry in deformable scenes, which is quite underexplored problem. \n- The dataset is generated in the multiple levels of noise and deformations, which makes dataset be a strong benchmark. \n- They proposed robust scene flow-based baseline which works robustly in deformable scene scenarios.\n- Quantitative results are well-organized and relevant analyses are given.\n- They proposed a novel metric which can be used as supplementary metric where the groudtruth camera poses are inaccessible.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "kiS1bSnSBY", "reviewer": "Reviewer_yRQV", "summary": "", "questions": "", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper addresses the problem of non-rigid deformations in the camera and the unreliable of current trajectory methods in such situations.  It proposes the first large-scale camera trajectory in deformable environments. The proposed evaluation method is reasonable. \nThe proposed odometry method achieves promising results on synthetic and real datasets. The paper is well written. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "21zCo22X3G", "reviewer": "Reviewer_8ABf", "summary": "", "questions": "", "limitations": "The authors have adequately addressed the limitations of their work. The checklist provided at the end of the paper is helpful in this regard. ", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "- Motivation: The paper effectively highlights the motivation behind the research, addressing a significant challenge in medical applications like endoscopy due to the lack of large-scale datasets for deforming scenes. This sets a strong foundation for the study's importance and relevance.\n\n- Scale of dataset: The scale of the proposed dataset stands out, covering a wide range of scenes with varying deformities (based on levels) and providing comprehensive annotations due to the synthetic nature, including camera poses, RGB images, depth, optical flow, and normal maps. This richness enhances the dataset's value and utility for researchers in the field. Although a good thing of annotations about this dataset is also a point of concern - extensive evaluation (on top of what is already presented) on other tasks would be useful (expanded more in the improvement section below).\n\n- Result on real endoscopy example: The paper demonstrates performance on a real endoscopy example, offering practical insights into its applicability in real-world scenarios. While not perfect, this validation contributes to the credibility of the proposed approach. However, more comments are provided in the next section.\n\n- Reasonable conclusion:  The conclusion effectively emphasizes the model's better performance when trained on deformable scenes compared to rigid scenes, when the evaluation task is a deformable scene (Hamlyn dataset) as demonstrated in Table 3 and Figure 4. This highlights the method's effectiveness and its potential benefits for relevant applications.\n\nOverall, the paper showcases a compelling approach to tackle the camera motion estimation challenge in deformable scenes, providing a diverse and valuable dataset along with a novel data-driven method. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "AHCYSJyq5W", "reviewer": "Reviewer_s3zk", "summary": "", "questions": "", "limitations": "I think the limitations have been adequately discussed. In particular, I do not think using synthetic data is a limitation but it is defininately ok to include it.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "- Interesting dataset that has a decent size with a novel setting.\n\n- The proposed method achieves impressive results, especially it is transferable to medical applications.\n\n- The proposed metric makes sense especially in absense of ground truth trajectory.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper got positive ratings from all reviewers. A common concern among the reviewers is about the motivation and practical application of the dataset, specifically regarding camera pose estimation in deforming scenes. While endoscopies in medical applications could serve as a direct application, it is not clear to come up with other compelling applications.

In their rebuttal, the authors address these concerns, which successfully convince the reviewers. Apart from these concerns, the paper is in good shape overall. It includes a large-scale dataset and demonstrates its advantages through various experiments, including real endoscopies.

The AC concludes that the paper meets the NeurIPS D&B bar. The authors are advised to carefully review the feedback provided by the reviewers and enhance the paper's quality for the camera-ready version.
Decision: Accept (Poster)