Deep Patch Visual Odometry
Authors: Zachary Teed
Lahav Lipson
Jia Deng
Venue: NeurIPS 2023 poster (2023)
Keywords: SLAM
Simultaneous Localization and Mapping
Visual Odometry
Structure from motion
SfM
TL;DR: An architecture for state-of-the-art visual SLAM at real-time framerates.

Abstract: We propose Deep Patch Visual Odometry (DPVO), a new deep learning system for monocular Visual Odometry (VO). DPVO uses a novel recurrent network architecture designed for tracking image patches across time. Recent approaches to VO have significantly improved the state-of-the-art accuracy by using deep networks to predict dense flow between video frames. However, using dense flow incurs a large computational cost, making these previous methods impractical for many use cases. Despite this, it has been assumed that dense flow is important as it provides additional redundancy against incorrect matches. DPVO disproves this assumption, showing that it is possible to get the best accuracy and efficiency by exploiting the advantages of sparse patch-based matching over dense flow. DPVO introduces a novel recurrent update operator for patch based correspondence coupled with differentiable bundle adjustment. On Standard benchmarks, DPVO outperforms all prior work, including the learning-based state-of-the-art VO-system (DROID) using a third of the memory while running 3x faster on average. Code is available at https://github.com/princeton-vl/DPVO

Deep Patch Visual Odometry
Zachary Teed" Princeton University zteed0princeton.. edu
Lahay Lipson'
Jia Deng Princeton University
Princeton University
Dlipson@princeton.cem edu
iadeng@princeton edu
Abstract
We propose Deep Patch Visual Odometry (DPVO), deep learning system for monocular Visual Odometry (VO). DPVO uses a novel recurrent network architecture designed for tracking image patches across time. Recent approaches to VO have significantly improved the state-of-the- networks predict dense flow between video frames. However, using dense flow incurs large computational cost, making these previous methods impractical for many cases. Despite this, it has been assumed that dense flow is important as it provides additional redundancy against incorrect matches. DPVO disproves this assumption, showing that it possible to get the best accuracy and efficiency by exploiting the advantages of sparse patch-based matching over dense flow. DPVO introduces a novel recurrent update operator for patch based correspondence coupled with differentiable bundle adjustment. On Standard benchmarks, DPVO outperforms all prior work, including the learning-baste based state-of-the (DROID) using third of the memory while running 3x faster on average. Code is available htttt:::/// // gith.... com/ wwt.cooonnettt.eeeeeeeeeeeeeccce pwrinctton-vl////
accuracy by using deep
VO-system
Introduction
Visual Odometry (VO) the task of estimating robot's position and orientation from visual measurements. In this work, we focus on most challenging casc-monocular VO-where the only input is a monocular video stream. The goal of the system is to estimate the 6-DOF pose of the camera at every frame while simultaneously building map of the environment.
VO closely related to Simultaneous Localization and Mapping (SLAM). Like VO, SLAM systems aim to estimate camera pose and map the environment but also incorporate techniques for global corrections-suce loop closure and relocalization (3. SL.AM systems typically include VO frontend which tracks incoming frames and performs local optimization.
Prior work typically treats VO optimization problem solving for a 3D model of the scene which best explains the visual measurements (3). Indirect approaches first detect and match keypoints between frames, then solve for poses and 3D points which minimize the reprojection distance (26 4
20). Direct approaches, on the other hand, operate directly on pixel intensities, attempting to solve for poses and depths which align the images (13: The main issue with prior systems, both direct and indirect, is the lack of robustness. Failure cases are too frequent for many important applications such autonomous vehicles. These failure typically stem from moving objects, lost feature tracks, and poor convergence.
43) have been introduced to address the robustness issue. The main advantage of deep learning is better features as well as differentiable optimization layers guided by neural networks, DROID- (37). which includes VO frontend, uses neural networks to estimate dense flow fields which are subsequently used to optimize depth
Several deep learning approaches (37 45
39
Equal Contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

and camera pose. DROID- has significantly improved the state-of-tie th-the-art in visual SLAM and VO. However, it comes with large computational cost: in VO mode (without the backend), averages 40FPS on an RTX-3090 using 8. 8.7GB GPU memory, which can be impractical for resource- constrained devices. this work, we aim to significantly improve the efficiency of deep VO, and therefore SLAM, without sacrificing its accuracy.
We propose Deep Patch Visual Odometry (DPVO), new deep learning system for monocular VO. Our method achieves the accuracy and robustness of deep learning-based approaches while running 1.5-8.9x faster using 57%-29% the memory. It has lower average error than all prior work on common benchmarks. On an RTX-3090, it averages 60FPS using only 4.9GB of memory, with a minimum frame rate of 48FPS. can run at 120FPS on the EuRoC dataset (2) using 2.5GB while still outperforming prior work on average. The frame rate of DPVO is relatively-constan and does not significantly depend on the degree of camera motion, as opposed to prior methods including DROID-SLAM which slow down during fast camera motion (12 37 27). We train our entire system end-to-end on synthetic data but demonstrate strong generalization on real video.
in practice,
This leap in efficiency is achieved by the combination of 1) a deep feature-based patch representation for keypoints which encodes their local context, and novel recurrent architecture designed to track sparse collection of these patches through time-alternating patch trajectory updates with a differentiable bundle adjustment layer to allow for end-to-end learning of reliable feature matching. This efficiency boost enables the design of DPVO to allocate resources towards components which improve accuracy, which are described in Sec.
Existing approaches such as DROID-SLAM estimate camera poses by predicting dense flow 24 23 37) Designing efficient VO system with the same-or-better accuracy poses a significant challenge, since using dense matches in SLAM provides additional redundancy against incorrect matches. However, DPVO still achieves robustness by introducing components which improve matching accuracy, using resources which would otherwise be spent estimating dense flow. We observe, surprisingly, that patch-based correspondence improves both efficiency and robustness over dense flow. In classical approaches to VO. patch-based matching (12) has been shown to improve accuracy over keypoint- based methods (27). However, has remained unclear how to leverage this idea using deep networks without underperforming dense- flow approaches.
To summarize, contribute Deep Patch Visual Odometry, new method for tracking camera motion from video. DPVO uses a novel recurrent network designed for sparse watch-based correspondence. DPVO outperforms all prior work across several evaluation datasets, while running 1.5-8.9x faster than the previous state-of-the-art and using only 57-29% as much memory.
2 Related Work
Visual Odometry (VO) systems aim to estimate robot state (position and orientation) from a video. Overtime, a VO system will accumulate drift, and modern SLAM methods incorporate techniques to identify previously mapped landmarks to correct drift (i.e loop closure). VO can be considered a subproblem of SLAM with global optimization and loop closures disabled (3)
Many different modalities of VO have been explored by past work, including risual-inertia odometry (VIO) [4) and stereo VO (42 Here, we focus on the monocular case, where the only input a monocular video stream. Early works approached the problem using filtering and maximum- likelihood methods (6 25). Modern methods almost universally perform Maximum a Posteriori (MAP) estimation over factor graphs with Gaussian noise; in which case, the MAP estimate can be found by solving a non-linear least-squares optimization problem (8). This problem has lead to the development of many librarics for optimizing non-linear least-squares problems (I [7
Among VO systems, our method borrows many core ideas of Direct Sparse Odometry (DSO) 12) classical system based on least-squares optimization. Namely, we adopt a similar patch representation and reproject patches between frames to construct the objective function. Unlike DSO, the residuals are not based on intensity differences but instead predicted by a neural network which can pass information between patches and across patch lifetimes. Outlier rejection is automatically handled by the network, making our system more robust than classical systems like DSO and ORB-SLAM 12 26). One important component of classical systems is the careful selection of

---

which image regions to use. We find, surprisingly that our system works well on a small number (64 per frame) of randomly sampled image patches.
With regards to deep SLAM systems, our method is closely related to DROID-SLAM (7 but uses different underlying representation and different network architecture. DROID-SLAM is end-to- end deep SLAM system which shows good performance compared to both classical and deep baselines. Like our method, it works by iterating between motion updates and bundle adjustment. However, it estimates dense motion fields between selected pairs of frames which has a high computational cost and large memory footprint. While it capable of 40FPS inference on average, its speed varies depending on the amount of motion in the video. Our method selects sparse patches from the video stream, with relatively constant runtime per frame and 1.5-8.9x faster inference than DROID-SLAM,
Prior works such as BA-Net (35) and (21) also have embedded bundle adjustment layers in end-to-end differentiable network architectures. However, BA-Neet does not use patch-based correspondence. (21) and (11) have proposed neural networks which sit atop COLMAP (31) and perform subpixel- level refinement; these approaches are not, however, able perform 3D reconstruction on their own and are subject to failure cases in the underlying SIM system.
3 Approach
Given an input video, we represent scene as a collection of camera poses T SE and a set of square image patches P extracted from the video frames. Using d to represent inverse depth and (x,y) to represent pixel coordinates, we represent each patch as the 4 x p² homogeneous array
Preliminaries:
P& =
xy
where p is the width of the patch. We assume a constant depth for the full patch, meaning that forms fronto-parallel plane in the frame from which it was extracted. Letting denote the index of the source frame of the patch, i.c. the frame from which which patch Ps was extracted, we can reproject the patch onto another frame j
Pky KT,T." -CT,T.-KK-T -Pp
taking K to be the calibration matrix
fz
0
Cx
fy
Cy
K
0
0
1
0
0
The pixel coordinates x'= (x',y") =x==x',v
can be recovered by dividing by the third element, For the rest of the paper, we use the shorthand Pk, = wiy(T, (T, P&)) to denote the reprojection of patch k onto frame j in terms of pixel coordinates.
Patch Graph: We use a bipartite patch graph to represent the relations between patches and video frames. Edges in the graph connect patches with frames. We show an example in Fig. [ By default, the graph is constructed by adding an edge between each patch and every frame within distance r from the index of the source frame of the patch. The reprojections of patch in all of its connected frames in the patch graph form the trajectory of the patch. Note that a trajectory is a set of reprojections of single patch into multiple frames; it is not set of original square patches. The graph is dynamic; as new video frames are received, new frames and patches are added while old ones are removed. We provide an example trajectory in Fig. 3
Approach Overview: At a high level, our approach works similarly to a classical system: it samples set of patches for each video frame, estimates the 2D motion (optical flow) of each patch against each of its connected frames in patch graph, and solves for depth and camera poses that are consistent with the 2D motions. But our approach differs from a classical system that these steps are done through recurrent neural network and differentiable optimization layer. Our approach can be understood as sparsified variant of DROID-SLAM VO mode; instead of estimating dense optical flow, estimate flow for sparse set of patches.
3

---

Patch Fesland
Cortaed Fasturn Valching Fasturs Pyseeed
(a) Feature and patch extraction
(b) The patch graph
Figure 1: Constructing the patch graph. (a) Residual networks extract 1) a context feature map at
1 image-resolution and 2) 2-level pyramid of matching features at 1/2 and is resolution. Many p x patches are cropped from this feature map at random pixel coordinates using bilinear sampling. (b) Multiple patches are extracted from each frame (e.g. blue) and are connected to nearby frames (green and purple). In the resulting patch graph, edges connect patches with frames.
Feature Extraction: To estimate the optical flow of the patches, we need to extract per-pixel features and use them for computing visual similarities. We use a pair of residual networks. One network extracts matching features while the other extracts context features. The first layer each network is x 7 convolution with stride 2 followed by two residual blocks at 1/2 resolution (dimension 32) and 2 residual blocks at 1/4 resolution (dimension 64), such that the final feature map is one-quarter the input resolution. The architectures of the matching and context networks are identical with the exception that the matching network uses instance normalization and the context network uses no normalization. We construct two-level feature pyramid by applying average pooling to the matching features with a 4 x filter with stride 4.
We create patches by randomly sampling 2D locations, which we find to work well despite its simplicity. We associate each patch with a per-pixel feature map of the same size, cropped with bilinear interpolation from both the full matching and full context feature maps. We provide example in Fig. [a The patch features are used to compute visual similarities, but unlike DROID-SIL -LAM, we compute them on the fly instead of precomputing them as correlation volumes.
Patch Extraction:
3.1 Update Operator
The core of our approach is the update operator, which is a recurrent network that iteratively refines the depth of each patch and the camera pose of each video frame. At each iteration, it uses the patch features to propose revisions to the optical flow of the patches, and updates depth and camera poses through differentiable bundle adjustment layer. Because the patches and spatially separated, the recurrent network includes special designs that facilitate exchange of information between patches. We provide schematic overview of the operator in Fig. 2 The operator operates on the patch graph and maintains a hidden state for each edge. Its first three components (Correlation, Temporal Convolutions, Softmax-Aggregae information across edges, the Transition block produces an update to each hidden state, and the final two components (Factor-Head + Bundle-Adjustment Adjustment) produce an update to the camera poses and patch depths. When new edge is added, its hidden state is initialized with zeros.
sparsely located
produce and aggregate
Correlation:
For each edge (k. in the patch graph, compute correlation features (visual sim- ilarities) to assess the visual alignment given by current estimates of depth and poses and to propose revisions. We first use Eqn. 2 to reproject patch i from frame into frame j: P't Wiy (T, Px). Given patch features g E R and frame features E IR XWxD xD for each pixel (u, v) in patch k. we compute its correlation C E I2xxx777 with grid of pixels centered at its reprojection in frame j. using the inner product:
C Cavas= = (gues f(Pk) (u, +
where we take to be a x integer grid centered at indexed by a and 99, and f(-) denotes bilinear sampling. We compute these correlation features for both levels in the pyramid and concatenate the results. This operation is implemented as an optimized CUDA layer which leverages the regular grid structure of the interpolation step. This implementation is identical to the alternative correlation

---

furres paache
dx
À 8.
Facker
Σ A
(T, P}
pasch graph
Figure 2: Schematic of the update operator. Correlation features are extracted from edges in the patch graph and injected into the hidden state alongside context features. We apply ID convolution, message passing and a transition block. The factor head produces trajectory revisions which are used by the bundle adjustment layer to update the camera poses and the depth of patches. Each "+" operation a residual connection followed by layer normalization.
implementation used by RAFT (66) and is equivalent to indexing correlation volumes due to the linearity of the inner product and interpolation.
ID Temporal Convolution: DPVO operates on real-tims video, where neighboring frames tend be highly correlated. To leverage this correlation, we apply a IDDconvolution in the temporal dimension to each patch trajectory. Since trajectories vary in length and keyframes are actively added and removed, it is not straightforward to implement convolution edge (k, j) we index the features of its temporally-adjacen neighbors at (k. - 1) and (k: + 1), concatenate, then apply linear projection. The temporal convolution allows the network to propagate information along each patch trajectory and model appearance changes of the patch through time.
batched operation. Instead, for each
SoftMax Aggregation: Even though the patches are sparsely located, their motion and appearance can be still correlated they may belong to the same object. We leverage this correlation through global message passing layers that propagate information between edges in the patch graph. This operation has appeared before in the context of graph neural networks (30). Given edge e and its neighbors (e) we define the channel aggregation function
Σ a(x) M(r) xEN(c)
Σ #EN(C)
(x))
where di and 6 are linear layers and a linear layer followed by sigmoid activation. We perform two instantiations of soft aggregation: (1) patch aggregation where edges are neighbors if they connect to the same patch (2) frame aggregation where edges are neighbors they connect both the same destination frame and different patches from the same source frame. Note that this graph-based aggregation is unique to our sparse patch-based representation, because for dense flow the same effect easily achieved with convolution in prior approaches (37). Following the softmax- aggtrax-aggregation, we include a transition block (shown in Fig. 2 to produce an update to the hidden state for each edge in the patch graph. Our transition block is simply two gated residual units with Layer Normalization and ReLU non non-linearities. We find that layer normalization helps avoid exploding values for recurrent network modules.
Transition Block:
The final learned layer in our update operator is the factor head, which proposes 2D revisions to the current patch trajectories and associated confidence weights. This layer consists of 2 MI.Ps with hidden unit each. For each edge (k, j) in the patch graph, the first MI.P predicts the trajectory update ōØz E 2 a 2D flow vector indicating how the reprojection of the patch center should be updated in 2D; the second MLP predicts the confidence weight Σxj E IR2 which is bounded to (0, 1) using sigmoid activation.
Factor Head:
Differentiable Bundle Adjustment: Given the proposed 2D revisions to trajectories, we need to solve for the updates depth and camera poses to realize the proposed revisions. This is achieved through a differentiable bundle adjustment (BA) layer, which operates globally on the patch graph and outputs updates to depth and camera poses. The predicted factors (6, Σ) are used to define an
5

---

Figure 3: A subset of the patch trajectories predicted by our method. Patches extracted from the green keyframe are tracked through subsequent frames. When a new keyframe is added (blue), additional patches extracted and tracked. Our method produces confidence values which weight their respective contribution to the bundle adjustment.
optimization objective:
Σ (R,))EE
(T, Pi.) - [P/r + وes J Eius
where ||-1|/ is the Mahalanobis distance and P/r denotes the center of Pky. We apply two Gauss-Newton iterations to the linearized objective, optimizing the camera poses as well as the inverse depth component of the patch while keeping the pixel coordinates constant. This optimization seeks to refine the camera poses and depth such that the induced trajectory updates agree with the predicted trajectory updates. Like DROID-SLAM (37), we use the Schur complement trick for efficient decomposition and hackpropagate gradients through the Gauss-Newton iterations.
3.2 Training and Supervision
DPVO implemented using PyTorch. We perform supervised training of our network the TartanAir dataset. On each training sequence, we precompute optical flow magnitude between all pairs of frames using ground truth poses and depth. During training, we sample trajectories where frame-to to-nett--frae optical flow magnitude is between 16px and 72px. This ensures that training instances are generally difficult but not impossible.
We apply supervision to poses and induced optical flow (i.e. trajectory updates), supervising each intermediate output of the update operator and detach the poses and patches from the gradient tape prior to each update.
Pose Supervision: We scale the predicted trajectory to match the groundtruth using the Umeyama alignment algorithm (40). Then for every pair poses (i,j), we supervise on the error
Σ
Logse(3) |G, (T,
(1,5) 6
where G is the ground truth and T are the predicted poses.
Flow Supervision:
We additionally supervise on the distance between the induced optical flow and the ground truth optical flow between each patch and the frames within two timestamps of its source frame. Each patch induces a x p flow field. We take the minimum of all p x errors.
The final loss is the weighted combination
10 0.1 f flow-
Training Details: We train for total of 240k iterations on a single RTX-3090 GPU with a batch size of 1. Training takes 3.5 days. We use the AdamW optimizer and start with an initial learning rate of 8e-5 which is decayed linearly during training. We apply standard augmentation techniques such as resizing and color jitter.
We train on sequences of length 15. The first frames are used for initialization while the next frames are added one at time. We unroll 18 iterations of the update operator during training. For the first 1000 training steps, we poses with the ground truth and only ask the network to estimate the depth of the patches. Afterwards, the network is required to estimate both poses and depth.

---

1.0
Ours
0.0
0.0
0.2
0.4
0.6
0.8
1.0
ATE [m]
(a) Sparse reconstruction example
(b) Ours vs DROID-SLAN
Figure 4: Results on the TartanAir (44) validation split. Our method gets an AUC of 0.80 compared to 0.71 for DROID-SLAM while running 4x faster.
Frame Queue
Visualization
Patch Extraction
Edge Addition
Update Op BA
Keyframe
Figure 5: Overview of the VO System. The visualization and frame loading are performed in separate threads.
3.3 VO System
We implement the logic of the full VO system primarily in Python with bottleneck operations such as bundle adjustment and visualization implemented in C++ and CUDA.
Overview: In Fig. we show overview of the VO system. The visualization and frame loading performed in separate threads.
Initialization:: We use frames for initialization. We add new patches and frames until 8 frames are accumulated and then run iterations of our update operator. There needs be some camera motion for initialization; hence, we only accumulate frames with an average flow magnitude of at least 8 pixels from the prior frame.
When new frame is added we extract features and patches. The pose of the new frame initialized using constant velocity motion model. The depth of the patch initialized as the median depth of all the patches extracted from the previous frames.
Expansion:
We connect each patch to every frame within distance from the frame index where the patch was extracted. This means that when a new patch is added, we add edges between that patch and the previous keyframes. When last keyframes with the new frame. This strategy means that each patch is connected to no more than (2r 1) frames, bounding the worst-case latency.
frame is added, add edges between each patch extracted in the
Following the addition of edges we run one iteration of the update operator followed by two bundle adjustment iterations. We fix the poses of all but the last 10 keyframes. The inverse depths of all patches are free parameters. The patches are removed from optimization once they fall outside the optimization window.
Optimization:
The most recent frames are always taken to be keyframes. After each update, we compute the optical flow magnitude between keyframe / - 5 and - 3. If this is less than 64px, remove the keyframe at 4. When keyframe removed, we store the relative pose between its neighbors such that the full pose trajectory can be recovered for evaluation.
Keyframing:

---

000
MB
001
ME
002
ME
ME
003
ME
005
ME
007
ME
M11
000
M11
002
5111
084
M11
M11
005
MIl
037
AME
ORB SLAME E
13.61
16.95
20.57
16.00
22.27
9.25
21.61
7.74
15.44
2,92
12.51
8.15
2.59
21.91
11.70
25.88
14.38
COLMAP 83
15.20
5.38
10.36
3.93
2.62
14.78
7.00
18.47
12.26
13:45
12.45
20.95
26.97
16.79
7,01
7.97
12.50
DSO 13
9.55
3.84
12.30
8.37
9.27
2.34
8.15
5.43
9.92
0.35
7.56
3,45
12.58
8.42
7.50
7.32
DROID SLAM* 37
0.17
0.06
0.36
0.57
1.14
0.13
1.13
0.16
0
0.05
0.04
0
0.01
0.68
0.50
0,88
0.35
DROID-V
0.32
0.15
0.24
1.27
1.04
0.14
1.32
0.77
0.32
0.13
0.06
0.09
1.50
0.69
0.59
0.97
0.58
Our: (Defaul))
0.16
0.11
0.11
0.66
0.31
0.14
0.30
0.13
0.21
0.84
0.04
0.08
0.58
0.17
0.11
0.15
0.21
Our (Foet)
0.35
0.13
0.27
0.71
0.47
0.16
0.30
0.13
0.34
0.05
0.06
0.07
0.51
0.41
0.09
0.14
0.78
Table I: Results on the Tir monocular test split from the ECCV 2020 SLAM competition.
Results are reported as ATE with scale alignment. For our method, we report the median of 5 runs. Methods marked with (*) use global optimization loop closure.
360
desk
desk2
floor
plant
room
rpy
toddy
xyz
|
avg
ORB [LA-S4)
0.017
0.210
x
0.034
0.009
D50[13]
0.173
0.567
0.916
0.080
0.121
0.379
0.058
0.036
0.172
0.718
0.728
0.068
0.167
0.767
0.031
DROID VO[32]
0.161
0.028
0.099
0.033
0.028
0.327
0.028
0.169
0.013
0.098
Ours (Default)
0.135
0.038
0.048
0.040
0.036
0.394
0.034
0.064
0.012
0.089
Ours (Fast)
0.169
0.029
0.064
0,047
0.047
0.396
0.034
0.074
0.012
0.097
Table 2: Results (ATE) on the freiburg1 set of TUM-RGBD (3)
This evaluates monocular visual
odometry (No method uses stereo or sensor depth), and is identical to the evaluation setting in DROID-SLAM (37). For each sequence. we report the median result across 5 independent trials. Missing values (i.e. x) indicate that the method completely failed and refused to output a trajectory. Only learning based methods are robust enough to not fail on any sequence, and out of them Ours (Default) performs best on average.
4 Experiments
We evaluate DPVO on the TartanAir (44) TUM-RGBD (33), EuRoC (2) and ICL-NUIM (19) benchmarks. On each dataset, we run five trials each with a different set of patches and report the median results obtained. Example reconstructions in Fig. 4a and in the Appendix, respectively. We benchmark two configuration settings of DPVO: Ours (Default) uses 96 patches per image and 10 frame optimization window and Ours (Fast) uses 48 patches and a 7 frame optimization window. Our frame rates are practically near constant, since method uses constant number of FLOPS per-frame, unlike prior methods which slow down and make more keyframes during fast motion (37) 12). We also benchmark two versions of DROID-SLAME the full version and version where loop closure and global bundle adjustment is disabled (DROID-VO) VO). DROID-VO is more comparable to our method as we only perform local optimization. Across all benchmarks, DPVO achieves the lowest average error among all previous VO systems, and even outperforms full SLAM systems on some benchmarks. We focus on average error as it reflects the expected performance of our system in the wild.
the TartanAir and ETH3D datasets are shown
We use the same 32-sequence validation split as DROID- and report aggregated results in Fig. [4b and compare with DROID- and ORB-SLAM3. We run method 3 times on each sequence and aggregate the results. In the [0, 1mm error window, we get an AUC of 0.80 compared to 0.71 for DROID-SLAM.
TartanAir Validation Split:
TartanAir Test Split: Tab. D reports results on the test-split used the ECCV 2020 SLAM competition compared to state-of-the-art methods. Classical methods such as DSO and ORB-SLAM fail on more than 80% of the sequences, hence we use COLMAP as a classical bascline as was used in the two winning solutions of the ECCV SLAM competition. We attain an error 40% lower than DROID-SLAM and 64% lower than DROID- VO. COLMAP takes 2 days to complete the 16 sequences and typically produces broken reconstructions which lead to large errors in evaluation.
EuRoC MAV (2 In Tab. 3 we benchmark on the EuRoC MAV dataset and compare to other visual odometry methods including SVO (15), DSO 12) and the visual odometry version of DROID- SLAM (36). Video from the EuRoC benchmark recorded at 20FPS. Like DROID-SLAM, we skip every other frame, doubling the effective frame rate of the system. Our default system outperforms prior work on the majority of the EuRoC sequences. The average error 43% lower than DROID- VO (7). Even our 120-FPS system outperforms DROID-VO most video.
8

---

M1103
Avg.
TartanVO 33
0.639
0.325
0.550
1.153
1.021
0.447
0.389
0.622
0.433
0.749
1.52
0.680
SVO hB
0,100
0.120
0.410
0.430
0.300
0.070
0.210
0,110
0.110
1,080
0.294
DSO P
0.046
0.946
0.172
3.810
0.110
0.089
0.107
0.903
0.044
0.132
1.152
0.601
DROID VO 37
0.163
0.121
0.242
0.399
0.270
0.103
0.165
0.158
0.102
0.115
0.204
0.186
Ours (Default)
0.087
0.055
0.158
0.137
0.114
0.050
0.140
0.086
0.057
0.049
0.211
0.105
Ours (Fast)
0.101
0.067
0.177
0.181
0.123
0.053
0.158
0.095
0.095
0.063
0.310
0.129
Table 3: Monocular SLAM on the EuRoC datasets, ATE[m] compared to other visual odometry methods. For our method, we report the median of 5 runs.
QRA
SFT-
Superpoink
3.4
3.A
Grad
9.2
All Softigg
Random
0.5
3.2
0.5
0.5
0.5
10
0.2
0.2
0.6
5.5
0.8
10
[m)
A72 (m]
(a) Point vs Patch fcatures
(b) Update operator components
(c) Selecting patch centroids
Figure 6: Ablation experiments. (a) We show the importance of using patches over point features.
(b) Removal of different components of the update operator degrades accuracy. (c) Randomly
selecting patch centroids works better than 2D keypoint locations produced via SIFT (22). ORB (29) Superpoint (9), or pixels with high image gradient.
TUM-RGBD (33): Tab. 2 we benchmark on TUM-RGBD (3 and compare to DSO (2). the visual odometry version of DROID-SLAM (36), and ORB-SLAME (27). We evaluate only visual-only monocular methods, identical to the TUM-RGED evaluation in (37). This benchmark
evaluates motion tracking in indoor environment with erratic camera motion and significant motion blur. Video from the TUM-RGBD benchmark recorded at 30FPS.
Classical approaches such ORB-SLAM (27 and DSO 12) perform well on some sequences but exhibit frequent catastrophic failures. Similar DROID-VO, our system is also robust to such failures but our average етror is 9% lower. This indicates that our sparse, patch-based approach performs better in-the-wild compared to dense flow.
ICL-NUIM (9): In Tab, 4 we evaluate on the ICL-NUIM (19) SLAM benchmark and compare to other visual odometry and SLAM methods including SVO (5), DSO (22 and DROID-SLAM (37). Video from the ICL-NUIM benchmark is recorded at 30FPS. ICL-NUIM is synthetic dataset for evaluating SLAM performance indoor environments with repetitive or monochrome textures (e.g. blank white walls). Our system outperforms prior work on the majority of the ICI-NUIM sequences.
8
GPU Memory (GB)
DROID-VO
(11 FPS)
DPVO (Default)
DPVO (Fast)
(40 FPS)-1
(60 FPS)
(48 FPS)
(120 FPS)-1
(98 FPS)
Average Runtime (FPS 1)
Worst-Case Runtime (FPS-1)
DROID-VO (Default) (Fast)
(a) Average/worst-cass runtime
(b) GPU memory
Figure 7: Efficiency on EuRoC. (a) Compared DROID- VO (the front-end of PROID-SLAM,, the Default and Fast variants of DPVO run 1.5x (60FPS) and 3x (120FPS) faster on average. DPVO's frame-rate is relatively stable, remaining above 48FPS and 98FPS for 95% of frames. In contrast, DROID-VO drops to 1 11PPS, an 8.9x difference. (b) DPVO (Default) and DPVO (Fast) use 57% and 29% much memory as DROID-VE VO, respectively. (c) We can trade-off speed for performance by increasing the number of patches tracked. However, performance quickly saturates beyond 96.

---

Living
Living Room
Living
Living
Office
Office
Office
Office
Room
Room
Room
Room 0
Room
Room
Room
Avg
ORB-SI AMM-
27
0,461
0.172
0.806
N/A
0.589
0.813
1,000
N/A
N/A
DROID-SLAM
27
0.008
0.027
0.039
0.012
0.065
0.025
0.858
0,481
0.189
DROID-VO BY
0.010
0.123
0.072
0.032
0.095
0.041
0.842
0,504
0.215
SVO
0.07
0.09
0.34
0.28
0,08
0.136
DSO (22
0.01
0.02
0.06
0.03
0.21
0.83
0.36
0.64
0.270
DSO (Realtime)
12
0.02
0.03
0.33
0.29
0.64
0.23
0.46
0.258
(Default)
0.635
Ours (Fast)
0.008
0.007
0.021
0.010
0.071
0.015
0.018
0,593
0.093
Table 4: Results on the ICL-NUIM SLAM benchmark. Results are reported ATE with scale alignment. Methods marked with (*) use global optimization / loop closure. SVO and DSO results are from (16). ORB-SLAM2 results are from (IO) (only partial results were provided). Despite our best efforts, we were unsuccessful in running monocular ORB-SLAM3: The official software would not produce any keyframes while running-
The average error of our faster model is 51% lower than DROID-SLAM (37) and 32% lower than SVO (15). Our faster and default systems perform similarly.
Efficiency: In Fig. we compare the efficiency of DPVO and DROID-VO. Our 60-FPS variant is 1.5x faster on average and 4.3x faster in the worst case (5th percentile), using 57% the GPU memory of DROID-VO. The 120-FPS DPVO variant is 3x faster average and 8.9x faster in the worst-case, using less than a third of the memory.
4.1 Ablations
We perform ablation experiments on the TartanAir validation split and show results in Fig. C We use the same parameter settings in all experiments with augmentation disabled. We run each ablation experiment three times on the validation split and aggregate the results.
Point Patch Features: In Fig. (a demonstrate the importance of the patches simply using point features (i.e 1x1 patches). The patch features encode local context which is lacking with point features. The additional information stored in the correlation features allows for more precise tracking.
Update Operator: Fig. 6b) we test the effect of removing various components from the update operator. Both removing ID-Convolution and Softmax- Aggreegtion degrade performance on the validation set.
Patch Selection: In Tab. 6c we compare different methods for selecting 2D patch centroids. Surprisingly, our method performs best when randomly selecting keypoint centroids.
5 Conclusion
DPVO is new deep visual odometry system built using a sparse patch representation. It is accurate and efficient, capable of running 60-120 FPS with minimal memory requirements. DPVO out- performs all prior work (classical or learned) on EuRoC, TUM-RGBD, the TartanAir ECCV 2020 SLAM competition, and ICL-NUIM.
Acknowledgments: This work was partially supported by the Princcton University Jacobus Fellow- ship, NSF, Qualcomm, and Amazon.
References
[1] Sameer Agarwal, Keir Mierle, and The Ceres Solver Team. Ceres Solver, 2022.
[2] Michael Burri, Janosch Nikolic, Pascal Gohl, Thomas Schneider, Joern Rehder, Sammy Omari, Markus W Achtelik, and Roland Siegwart. The
micro aerial vehicle datasets. The International Journal of Roboties Research. 35(10):1557-116 2016.
Cesar Cadena, Luca Carlone, Henry Carrillo, Yasir Latif, Davide Scaramuzza, José Neira, Ian Reid, and John J Leonard. Past, present, and future of simultancous localization and mapping: Toward the pouust-erceetion age. IEEE Tranxactions on robotics, (206):139-11322 2016.
[4] Carlos Campos, Richard Elvira, Juan J Gómez Rodríguez, José MM Montiel, and Juan Tardós. Orb- slam3: An accurate open-source library for visual, visual- inertiall and multimap slam. IEEE Transactions Robotics, 1874 1890, 2021.
10

---

[5] Christopher B Choy, Jun Young Gwak, Silvio Savarese, and Manmohan Chandraker. Universal correspon- dence network. Advances neural information processing systems, 29, 2016.
[6] Andrew Davison. Real-time simultancous localisation and mapping with single camera. In Computer Vision, IEEE International Conference on, volume 3, pages 1403-1403. IEEE Computer Society, 2003. Frank Dellaert. Factor graphs and gtsam: A hands-on introduction. Technical report, Georgia Institute of Technology. 2012.
[7]
[8] Frank Dellaert, Michacl Kaess, et al. Factor graphs for robot perception. Foundations and Trends in Robotics, 6(1-2)11-13, 2017.
[9] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superpoint: Self-supervised interest point detection and description. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 224-236, 2018.
[10] ZhiMin Duan, YingWen Chen, HuJie Yu, BoWen Hu, and Chen Chen. Rgb-fusion: Monocular 3d
reconstruction with learned depth prediction. Displays. 70: 10:102100, 2021.
[1I] Mihai Dusmanu, Johannes L Schönberger, and Mare Pollefeys. Multi-view optimization of local feature geometry. In European Conference on Computer Vision, pages 770-686. Springer, 2020.
[12] Jakob Engel. Vladlen Koltun, and Daniel Cremers. Direct sparse odometry. IEÉE transactions on pattern analysis and machine intelligence, (063):611-625, 2017.
[13] Jakob Engel, Thomas Schöps, and Daniel Cremers. Lsd-slam: Large-scale direct monocular slam. In European conference on computer vision, pages 834-849. Springer, 2014,
[14] Christian Forster, Luca Carlone, Frank Dellaert, and Davide Scaramuzza. Imu preintegration on manifold
for efficient visual-inertial maximum-a-posterion
estimation. Georgia Institute of Technology, 2015.
[15] Christian Forster, Matia Pizzoli, and Davide Scaramuzza. Svo: Fast semi-dired monocular visual odometry. In 2014 IEEE international conference on robotics and outomation (ICRA), pages 15-22. IEFE, 2014.
[16] Christian Forster. Zichao Zhang, Michael Gassner, Manuel Werlberger. and Davide Scaramuzza. Svo: Semidirect visual odometry for monocular and multicamera systems. IEEE Transactions on Robotics, 33(2):249-2665 2016.
[17] Giorgio Grisetti, Rainer Kümmerle, Hauke Strasdat, and Kurt Konolige. g2o: A general framework for (hyper) graph optimization. In Proceedings of the IEEE international conference on roboticy and automation (ICRA). Shanghai, China, pages 9-13, 2011.
[18] Michael Grupp. evo: Python package for the evaluation of odometry and slam. https://github..em com/ 2017.
Michae1Grupp/era
[19] A. Handaa, I. Whelan, J.B. McDonald, and A.J. Davison. A benchmark for RGB-D visual odometry, 3D reconstruction and SLAM. In IEEE Inil. Conf. on Robotics and Automution, ICRA, Hong Kong. China.
May 2014.
[20] Stefan Leutenegger, Paul Furgale, Vincent Rabaud, Margarita Chli, Kurt Konolige, and Roland Siegwart. Keyframe-based visual-inertial slam using nonlinear optimization. Proceedings of Robotis Science and Systems (RSS) 2013, 2013.
[21] Philipp Lindenberger, Paul-Edouard Sarlin, Viktor Larsson, and Mare Pollefeys. Pixel-perfect structure- from-motion with featuremetric refinement. In Proceedings of the IEEE/CVF International Conference on
Computer Vision, pages 5987-5997, 2021.
[22] David G Lowe. Object recognition from local scale-invariant features. In Proceedings of the seventh IEEE international conference on computer vision, volume 2. pages 1150-1157. Ieee, 1999.
[23] Zhixiang Min and Enrique Dunn. Voldor+ slam: For the times when feature-based or direct methods are not good enough. In 202/ IEFF International Conference on Robotics and Automation (ICRA), pages 1381 13813-13819 19. IEEF, 2021.
[24] Zhixiang Min, Yiding Yang, and Enrique Dunn. Voldor: Visual odometry from log-logistic dense optical
flow residuals. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages
2020.
[25] Anastasios Mourikis, Stergios I Roumeliotis, et al. A multi-state constraint kalman filter for vision-aided inertial navigation. In ICRA, volume 2, page 6, 2007.
[26] Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos. Orb-slam: versatile and accurate monocular slam system. IEFF transactions robotics, 31(5):1 147-1163, 2015.
[27] Raul Mur-Artal and Juan D Tardós. Orb-slam2: An open-source slam system for monocular, stereo, and
rgb-d cameras. IEEE transactions on robotics, 33(5): 12555:2255 2017.
[28] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high perfformae deep
learning library. Advances in neural information processing systems, 32, 2019.
[29] Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski. Orb: An efficient alternative to sift or surf. In 2011 International conference computer vision, pages 2564-2571 leee, 2011.
[30]
Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4938-4947,, 2020.
[31] Johannes L Schonberger and Man-hichael Frahm. Strueture- -romre--foomoae motion revisited. In Proceedings of the
IEEE conference on computer vision and pattern recognition, pages 4104-4113 2016.
[32] Thomas Schöps, Torsten Sattler, and Mare Pollefeys. BAD SLAM: Bundle adjusted direct RGB-D SLAM.
In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
11

---

[33] J. Sturm, N. Engelhard. F. Endres, W. Burgard, and D. Cremers. A benchmark for the evaluation of rgb-d slam systems. In Proc. the International Conference Intelligent Robot Systems (IROS), Oct. 2012. [34] Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. Loftr. Detector free local feature matching with transformers. In Proceedings of the IEEE/CVF conference computer vision and pattern
recognition, pages 8922-8931, 2021.
[35] Chengzhou Tang and Ping Tan. Ba-net: Dense bundle adjustment network. arXiv preprint
1106. 04807.447...20 2018. [36] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow, In European
conference on computer vision, pages 402-419. Springer, 2020.
[37] Zachary Teed and Jia Deng. Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras.
Advances in Neural Information Processing Systems, 34:1655 14:16558-BB -65566 2021.
[38] 7achary Teed and Jia Deng. Raft-3d: Scene flow using rigid-motion embeddings. In Proceedings of the Computer Vision and Puttern Recognition, pages 8375-8384, 2021.
IEEE/CVF Conference
[39] Prune Truong. Martin Danclljan, Luc Van Gool, and Radu Timofte. Learning accurate dense correspon- dences and when to trust them. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5714-5724 2021.
[40] Shinji Umeyama. Least-squares estimation of transformation parameters hetween point patterns. IEEE Transactions on Puttern Analysis & Machine Intelligence. (3(4):3776 --07, 1991.
[41] Lukas Von Stumberg. Vladyslav Usenko, and Daniel Cremers. Direct sparse visual-inertial odometry using dynamic marginalization. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 25 10-25117 IEEE, 2018. 8.
[42] Rui Wang. Martin Schworer, and Daniel Cremers. Stereo dso: Large-scale direct sparse visual odometry with sterco cameras. In Proceedings of the IEEE International Conference on Computer Vision, pages
3903 3911. 2017.
[43]
Wenshan Wang. Yaoyu Hu, and Schastian Scherer. Tartanvo: A generalizahle learning-based vo. arXiv preprint vrXiv:2011000355 2020.
[44] Wenshan Wang, Delong Zhu, Xiangwei Wang, Yaoyu Hu, Yuheng Qiu, Chen Wang, Yafei Hu, Ashish Kapoor, and Scbastian Scherer. Tartanair: dataset to push the limits of visual slam. In 2020 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4909-4916. IEEE, 2020.
[45] Huangying Zhan, Chamara Saroj Weerasekera, Jia- Wang Bian, and Ian Reid. Visual odometry revisited: What should be learnt? In 2020 IEEE international conference on robotics and automation (ICRA), pages
4203 4210. JEEE, 2020.
12

---

A
Stable Runtime
The runtime of our approach relatively constant compared prior VO/SLAM systems. Our default configuration averages 60FPS, and rarely drops below 50FPS. In Fig. A we show the runtime distribution on EuRoC. The runtime stability DPVO results from the comparatively simple keyframing mechanism.
Discussion on Keyframing: Most VO/SLAM systems 12: 27 37),
including DROID-SLAM,
contain protocol for deciding when produce new keyframes from video stream. Keyframing mechanisms serve to discard repetitious frames early-on, saving on computation and potentially improving performance by limiting redundancy. Many VO/SLAM systems require low keyframing frequency in order to run at real-time framerates. If there is significant motion between subsequent frames, keyframes created drastically. DROID-SLAM for instance, slows from 40FPS to 11FPS during sufficiently large increases in the number of keyframes being created.
high frequency which
such methods to slow down
Unlike previous works, DPVO treats all incoming frames as keyframes and only removes redundant frames later using motion computed from the dtready-estimate pose. Although this design decision sub-optimal (in terms speed) during very slow or still camera movement (e.g. EuRoC), our approach is both simpler and ensures our frame-rate is approximately constant matter the degree of camera motion.
10
20
25
30
35
Latency (ms)
Figure A: Runtime distribution on EuRoC. Our system averages 60FPS with each new frame taking
1-17mm to process. The distribution is very centralized and rarely drops below 50FPS.
B ATE Error Metric
We compare results using the ATE (average trajectory error) metric, which is standard for VO and SLAM (16) 37 27 (1).. The ATE metric between the predicted trajectory T and the ground -tuthh trajectory T is computed using
E
T myz
f Tryzll2
(9)
after aligning the two trajectories using a similarity transformation, to account for the scale and SE(3) gauge freedoms. This metric is computed using the EVO library (18).
C Additional Training Details
DPVO is trained entirely TartanAir (44) synthetic dataset. This is the same synthetic training
data used by previous VO systems (43, 37) TartanAir provides large number of training scenes of both indoor and outdoor environments with varied lighting and weather conditions. The dataset provides depth and camera pose annotations, enabling one to generate optical flow by re-projecting
https://theairraab wtw.:aaa..ogggaataai--eaarr
13

---

Reconstruction the ETH-3D (32 dataset.
(b) Reconstruction from video demo
Figure B: Sparse reconstructions produced by our model.
the depth using the camera poses. All dataset parameters are identical to those used in DROID- SLAM (37). This includes random cropping, color-jitter, random greyscale, random color inversion, as well as re-scaling the training scenes in order to improve training stability.
D Visualization
Reconstructions are visualized interactively using separate visualization thread. Our visualizer implemented using the Pangolin library directly reads from PyTorch tensors avoiding all unnccessary memory copies from CPU to GPU. This means that the visualizer has very little overhead- only slowing the full system down by approximately 10%. In Fig. Ba we show a reconstruction on the ETH-3D (32) dataset, and in Fig. Bb we show the reconstruction produced during video demo.
E Bundle Adjustment Layer
Our bundle adjustment layer is functionally identical to the BA layer in DROID-SLAM (37 except that DROID-SLAM predicts the damping factor using its update operator, while we found this not to be necessary and hard-code it to 10 instead. We refer the reader to Appendix C in the DROID-SLAS SLAM paper for the analytical gradients of the Gauss-Newton update iterations. Since DROID-SLAN uses dense flow, we implement our own sparse version as an optimized CUDA layer.
F
Additional Figures
Correlation Operation: In Fig. Ca we show high-level visualization of the correlation operation which performed the start of the update operator. Patch k's pa matching-feature are each projected into the two-level feature pyramid of frame j. A x 7 grid of D 128 dimension feature vectors are bilinearly sampled from the pyramid around each point reprojection and dot producted with the matching features. This operation results in sets of p p 7 7 dot products, one for each resolution (1/4 and 1/8th of the original image size). This operation is implemented as an optimized CUDA layer.
we show high-level view of the rest of the update operator. The correlation and context features are fed into additional learn-able layers, including ID
Update Operator: Following Fig. Ca in Fig. Cb
14

---

Matching Feature Pyramid for frame
Features for patch
Features for edge (k,
(from frame
Bry CT Pa)
AP
AT
6,Σ
1) 1D Convolution
BA
h k.j
2] Message Passing
(hidden state)
3) Transition Block
h t++ k.
Features for edge (k.))
(a) Correlation Step
(b) Update Operator
The update operator, including the correlation step. (a) For each edge (k:,j) in the patch graph, patch & is reprojected into a frame j using equation (2). Frame j's matching features are then cropped using the reprojected patch. The correlated matching features (blue, green) and context features (grey) form the edge features. Each dot in the sampled matching features (green) represents 7 7 neighborhood of points, however this is omitted for clarity. (b) Given the features for edge (k,j), several learnable layers (yellow) are used to predict an update Ó(k.)) E IR² to the flow, confidence weight E(k,)) E IR? and an update to the hidden state hy.. The predicted factors are used in the bundle adjustment layer to produce an update to the camera poses and patch depths. The factor head omitted here for clarity.
Figure C:
Framerate (FPS)
60.0
30.1
20.2
8.0
EURoC Error (ATE)
DPVD Default Configuration
0.16
0.14
0.12
0.08
280
Number of Patches per Frame
Figure D: Accuracy on the EuRoC dataset as a function of the number of patches. We can trade-off speed for performance by increasing the number of patches tracked. However, performance quickly saturates beyond 96.
Convolution Layers, Softmax- Aggreeattonn block (i.e. Message Passing), and Transition Block. This results in an updated hidden state for each edge. Finally, the updated hidden state is used to produce 2D flow revisions and confidence weights which guide the solution the bundle adjustment. The bundle-adjustment layer ultimately produces an update to the depths and camera poses.
Confidence Weights: Our approach learns to reject outliers by predicting confidence weight associated to each predicted 2D flow update, which DROID-SLAM (37) does as well. We show visualization of the weights for small number of edges in Fig. E These weights are not supervised directly, rather they are learned by supervising on the pose output of the differentiable bundle- adjustment layer.
Number of Patches: In Fig. D we show the accuracy and runtime of DPVO on EuRoC while varying the number patches. Results are accumulated across independent trials. The accuracy quickly saturates after 96 patches.
G Network Architecture
Update Operator: In Fig. we show the architecture of the update operator, excluding non-learnable layers such as the correlation layer and the bundle adjustment layer. Since the ID-Convolutions and the Softmax-Aggregatiio layers operate on neighoring edges, they are partially implemented using PyTorc (28) scattering operations.
15

---

Figure E: Factor confidence weights. For each edge in the patch graph, the factor-head of our update operator predicts confidence weight w E R bounded to (0, 1) and 2D flow update. Cold colors (blue) represent high confidence edges while hot colors (red) represent low confidence edges.
Feature Extractor: In Fig. G show a visualization of the architecture of the feature extractor. The same network architecture is used for extracting context features and for extracting matching features, except the former uses no normalization and output dimension D 384 while the latter uses instance normalization and D 128. We use instance normalization for the matching features since they should be calculated independently for each input image in batch, which other flow-based networks (36) 38 (37) do as well. Our feature extractor is similar to the architecture used in DROID-SLAM (37), half size. Consequently, the output resolution 1/4 of the image resolution, as opposed to 1/8th. Since DPVO only tracks a sparse collection of patches instead of predicting dense flow, we can afford to use higher spatial-rese restualir features without significant memory overhead.
Correlation
Net2
SayazSioc
Scctnan ARE
Softmas ASE
Context
Yeatures
pabcc
dest. Seane
State
Layachicm
20 Flow
Linear
Layections
Confidance Weight
Sati Rigeaid
Sineer
Ginear
Lineaz
New #idden State
Linwarr
Misce- Lensoua
Tesporal cnev
Esansition Bleck
Multiplicaation
Fassing)
Figure F: Architecture of the Update Operator, excluding non-learnable layers such as the correlatio layer and the bundle-adjustment layer.
16

---

2D-Conv 7x7 (32) Stride
Residual- Besic BBock Stride
Residunl-Bloce (64) Staide
2D-Conv 1xl1
Figure G: Architecture of the feature extractors. D 128 for the setchinn-featuury extractor and 384 for the context-feature extractor.
17

---

H Predicted Trajectories on EuRoC
Figure H: Trajectory predictions the EuRoC test dataset.
18

---

I Predicted Trajectories on TartanAir
1
Figure Trajectory predictions on the TartanAir test dataset.
19


Reviews:
[{"review_id": "cDGuicwke9", "reviewer": "Reviewer_Uwmu", "summary": "This paper proposes a deep learning based visual odometry (VO) system which estimates the 6DoF poses of each frame and outputs a sparse reconstruction of the scene from an input monocular video sequence. The pipeline extracts deep features from the sampled image patches and updates deep optical flow of the patches with a recurrent module, then perform deep BA with patch correspondences within the local optimization windows on the fly. The reported statistics show that this system is able to run at 60FPS (Default) and 120FPS (Fast), respectively, while maintaining the camera pose estimation accuracy. Results show that the proposed method outperforms DROID-VO and ORBSLAM in terms of accuracy on TARTAN AIR, EuRoC MAV, and ICL-NUIM datasets.", "questions": "1. Are there any insights for the reasons of randomly sampled patches outperforming other feature based patch selection mechanisms (ORB, SIFT, and Superpoint)? Although the empirical results in this paper favor randomly sampled patches, it is still counter-intuitive to imagine most patches are randomly sampled on regions with repetitive patterns or occlusion. Are there any additional experiments to demonstrate the robustness of random patch selection?\n\n2. As mentioned in Weaknesses, why ORB-SLAM or D3VO are not included when comparing the results on the EuRoC MAV dataset? Are these methods not considered visual odometry systems?\n\n3. Both DROID-SLAM and DVPO are trained from synthetic datasets, are there any suggestions on the reasons for the worse performance of DROIDSLAM than DPVO on the ICL-NUIM dataset, even with global bundle adjustment?\n\n4. Are the running parameters (window size, patch size, number of update iterations) of the system tuned for different datasets, or are they fixed? Does patch size matter (not including the 1x1 point feature)?\n\n5. Are there any comparison between the system and classical methods in terms of efficiency, with different input frame sizes?", "limitations": "The model can only be trained on synthetic datasets with ground truth optical flow. Although the system shows the ability of generalization over the real data, its performance and robustness may be limited in the real scenarios.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. One of the main contribution of this work is to alleviate the large computational cost from the existing deep dense VO method (DROID-SLAM) while maintaining its pose estimation accuracy.\n\n2. The paper shows the contribution of 1D Temporal Convolution and SoftMax Aggregation on neighboring frames to the pose estimation, which are novel.\n\n3. The learned network is trained purely on synthetic data and it shows competency to perform well on the real datasets, including TUM-RGBD and EuRoC MAV.\n\n4. The paper is clearly written and easy to follow.", "weaknesses": "1. The novelty is limited, the iterative update mechanism and differential bundle adjustments are already used in DROID-SLAM, even with the introduction of 1D Temporal Convolution and SoftMax Aggregation modules in the update operator.\n\n2. Another major contribution of the proposed method is to track image patches instead of using dense correspondence on the entire image. However, it is not theoretically sound to support the claim that the patch-based approach improves the accuracy of pose estimation results over the dense flow based approach when the patches are selected randomly rather than feature based methods.\n\n3. I suspect that the model overfits the TartanAir dataset. That’s why there is a big performance gap between the results on synthetic and real datasets for the classic methods. When it comes to real datasets, the results are mixed.\n\n4. The selection of baseline methods is inconsistent and questionable in Table 1-4, the ORB-SLAM version used is different, and authors did not compare the results to ORB-SLAM or D3VO on the EuRoC MAV dataset, while other VO papers do include them as baseline methods. Also, there are better results can be found for running ORB-SLAM on TUM-RGBD fr1 and TartanAir sequences (reported in https://paperswithcode.com/paper/droidslam-deep-visual-slam-for-monocular/review/).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**The novelty is limited, the iterative update mechanism and differential bundle adjustments are already used in DROID-SLAM, even with the introduction of 1D Temporal Convolution and SoftMax Aggregation modules in the update operator.**\n\nWe do not claim the idea of an iterative update mechanism nor the differentiable bundle adjustment to be contributions of our method. Please see above.\n\n**It is not theoretically sound to support the claim that the patch-based approach improves the accuracy of pose estimation results over the dense flow based approach when the patches are selected randomly rather than feature based methods.**\n\nOur empirical results support the claim that DPVO, which uses randomly selected patches, outperforms DROID-VO, which uses dense flow (see tables [1,2,3,4]). This result may be counter-intuitive, but it is not a weakness of our approach. \n\nWe believe one possible theoretical reason is that random patches reduce redundant computation, and allow us to effectively re-allocate the computation and network capacity to learn better features for matching. We will revise our paper to clarify this point. \n\n**When it comes to real datasets, the results are mixed.**\n\nOn real datasets, we outperform other methods in terms of *average error”, which is a standard way to compare performance.  The results are “mixed” only in the sense that we are not No. 1 on every single test sequence, but requiring superiority on every single test example would be an unusually strict standard. \n\n**The selection of baseline methods is inconsistent and questionable in Table 1-4, the ORB-SLAM version used is different.**\n\nORB-SLAM has multiple versions, and on each dataset we use the version that produces the best results. \n\nThe appearance of inconsistency, as mentioned in the caption of table 4, is because  ORB-SLAM3 would not work on TUM-RGBD despite our best efforts; The v1.0 implementation (the latest) fails to produce any poses for the keyframes.\n\n**authors did not compare the results to ORB-SLAM or D3VO on the EuRoC MAV dataset.**\n\nWe omitted ORB-SLAM on the EuRoC table because we follow standard practice of VO evaluation [e.g. TartanVO ‘21] which focuses on comparing  strictly VO methods, i.e., those that do not perform global optimization, as opposed to SLAM methods that include global optimization. Global optimization can substantially improve results on input sequences with loops (such as those on EuRoC). As a result, VO methods are typically not expected to outperform SLAM methods and are not evaluated with SLAM methods as baselines. \n\nWe omitted D3VO as it would  not be a fair comparison - D3VO performs unsupervised training on 5 of our 11 test sequences , and then evaluates on the remaining ones which contain the same scenes. \n\nThis is a non-standard setting and gives D3VO an unfair advantage because unsupervised training on the same test scenes means that a method can essentially perform offline 3D reconstruction in advance for the same scenes and memorize the solutions. In offline reconstruction, a system can access all frames, including those from the future, whereas a VO system cannot use future frames. In contrast, our method is trained on synthetic data only, and is evaluated on each test sequence zero-shot, on the fly, without prior exposure to the same scenes.  \n\nWith the above caveats, in Table A in the rebuttal PDF, we nonetheless report results of D3VO and ORB-SLAM3 on EuRoC. We observe that D3VO outperforms DPVO on a subset of the evaluation sequences. This is expected because D3VO performs test-time-training on the remaining sequences that contain the same scenes, effectively allowing offline 3D reconstruction of the same scenes in advance. In contrast, DPVO is evaluated on all the EuRoC test sequences zero-shot, trained only on synthetic data. ORB-SLAM3 outperforms DPVO on all sequences (except it catastrophically fails on V202), but this is to be expected because it uses global optimization.\n\n**There are better results can be found for running ORB-SLAM on TUM-RGBD fr1 and TartanAir sequences (reported in https://paperswithcode.com/paper/droid-slam-deep-visual-slam-for-monocular/review/).**\n\nIt is not clear that there are better results. ORB-SLAM has three versions (ORB-SLAM1, ORB-SLAM2, & ORB-SLAM3), and the cited URL reports results from ORB-SLAM1. We reported ORB-SLAM3 results because it is newer and has fewer catastrophic failures (0/16 versus 2/8) than ORB-SLAM1. Considering  the catastrophic failures, it is not clear which version is better. \n\nRegardless, we include both versions in Table G in the rebuttal PDF and show that this choice has no impact on our conclusions. Our approach has better results than both versions of ORB-SLAM on TUM-RGBD and TartanAir. \n\nSpecifically, we outperform or match ORB-SLAM 1 & 3 on all TartanAir sequences.\n\nBoth DPVO and DROID-SLAM report identical results for running ORB-SLAM3 on TUM-RGBD:  [X 0.017 0.210 X 0.034 X X X 0.009]\n\nDROID-SLAM also reports ORB-SLAM2 results on TUM-RGBD, which are even worse than ORB-SLAM3 (it fails on more sequences): X 0.071 X 0.023 X X X X 0.010\n\n**Response to Q1**\n\nYes, we provide a discussion of this exact question in section G of the supplement. Tldr; we suspect that random patches have the most uniform coverage out of the 5 tested methods, which may help robustness.\n\n**Response to Q2**\n\nAddressed above.\n\n**Response to Q4**\n\nThe running parameters are fixed across all datasets in the default and fast variants, respectively.\n\n**Limitation: The model can only be trained on synthetic datasets with ground truth optical flow.**\n\nThis is not true, our method is not limited to synthetic datasets; it can be trained on any video dataset with poses and depth (e.g. Scannet, SUN3D, MannequinChallenge, among many others). \n\nAs stated on L85-86 of the supplement, we only train on a synthetic dataset to fairly compare with DROID-SLAM. \n\n<ran out of space>"}, {"review_id": "gsg3RjrDYM", "reviewer": "Reviewer_5bXL", "summary": "In this paper, the author propose a new method to solve monocular visual odometry (VO) in and end-to-end fashion. In contrast to previous work - DROID-SLAM, which utilizes dense optical to build correspondences across frames, the proposed method leverages sparse patches to avoid redundancy of dense pixels and therefore works faster. Experiments on public datasets including TartanAir, TUM-RGBD, EuRoC and ICL-NUIM demonstrate its higher efficiency and competitive accuracy.", "questions": "1.\tL141: random sampling. It is not very clear that random sampling of patches works better. Intuitively, with the predicted camera motion, sampling according to the predicted pose could give more correct matches between patches. Even in the classic VO/SLAM systems like ORB-SLAM and DSO, they sample keypoints/pixels from regions with rich textures because these regions could be better observed in following frames. The comparisons of random sampling with other strategies could solidate the claim.\n\n2.\tIn the Differential Bundle Adjustment module, the cameras poses and inverse depths are optimized with pixel coordinates fixed. In practice, the matches between patches across frames have outliers, impairing the performance. Basically, the BA and outlier removing are jointly updated in classic systems to guarantee the accuracy. It seems that this can be achieved in the recurrent module and as this step is very important especially to monocular VO/SLAM systems, it would be great to show how the module ‘corrects’ wrong matches.", "limitations": "NA", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tGood motivation. Dense correspondences provided by optical flow estimation are not necessary for visual odometry because basically several good matches are enough to find a good pose (more matches may increase the robustness). Therefore, I agree with the author of using sparse patches to achieve higher time and memory efficiency.\n2.\tExtensive results. The proposed method is evaluated on TartanAir, TUM-RGBD, EuRoC and ICL-NUIM datasets to show its efficiency and accuracy.\n3.\tThe paper is well-written and easy to read.", "weaknesses": "1.\tLimited novelty. The key idea of the paper is relatively straightforward. If we review the classic monocular visual VO/SLAM systems, we can find most of them are based on sparse keypoints (ORB-SLAM) or semi-dense pixels (DSO) to avoid too much computation. From my point of view, the major contribution of the paper comes from the engineering part. The recurrent module designed to update the trajectories of patches is kind of novel as it automatically updates the covisibility graph constructed by observed patches and past frames. However, the overall novelty is still limited.\n2.\tRelated works. Another limitation is the discussion of previous works. In addition to DROID-SLAM, there are lots of excellent learning-based VO/SLAM systems proposed in the past a few years such as BeyondTracking [r1], NICE-SLAM [r2], iMAP [r3], and Li et al [r4] to name a few. These works have achieved SOTA performance. However, they are not discussed and compared neither. It would be better to give a discussion of these works in the paper.\n\nr1: Xue et al., Beyond Tracking: Selecting Memory and Refining Poses for Deep Visual Odometry.CVPR 2019\n\nr2: Zhu et al., NICE-SLAM: Neural Implicit Scalable Encoding for SLAM. CVPR 2022\n\nr3: Sucar et al., iMAP: Implicit mapping and positioning in real-time. ICCV 2021\n\nr4: Li et al., DENSE RGB SLAM WITH NEURAL IMPLICIT MAPS. ICLR 2023.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Limited novelty. The key idea of the paper is relatively straightforward. If we review the classic monocular visual VO/SLAM systems, we can find most of them are based on sparse keypoints (ORB-SLAM) or semi-dense pixels (DSO) to avoid too much computation. The recurrent module designed to update the trajectories of patches is kind of novel as it automatically updates the covisibility graph constructed by observed patches and past frames. However, the overall novelty is still limited.**\n\nRegarding the novelty of our approach, please see our message to everyone in which we highlight why using sparse matches with the framework of DROID-SLAM is novel and challenging.\n\n**From my point of view, the major contribution of the paper comes from the engineering part.**\n\nWe believe a well-engineered system is still a valuable contribution to NeurIPS, especially given our exceptionally fast runtime, low cost, and performance on-par or better than DROID.\n\n\n**Related works. Another limitation is the discussion of previous works. In addition to DROID-SLAM, there are lots of excellent learning-based VO/SLAM systems proposed in the past a few years such as BeyondTracking [r1], NICE-SLAM [r2], iMAP [r3], and Li et al [r4] to name a few. These works have achieved SOTA performance. However, they are not discussed and compared neither. It would be better to give a discussion of these works in the paper.**\n\n**r1: Xue et al., Beyond Tracking: Selecting Memory and Refining Poses for Deep Visual Odometry. CVPR 2019**\n\n**r2: Zhu et al., NICE-SLAM: Neural Implicit Scalable Encoding for SLAM. CVPR 2022**\n\n**r3: Sucar et al., iMAP: Implicit mapping and positioning in real-time. ICCV 2021**\n\n**r4: Li et al., DENSE RGB SLAM WITH NEURAL IMPLICIT MAPS. ICLR 2023.**\n\n\nWe did not compare to r1, r2, r3, or r4 for several reasons:\n1) r2 and r3 assume RGB-D input, whereas DPVO is RGB-only.\n2) r1 trains on TUM-RGBD sequences that share the same scenes with the test sequences, whereas DPVO is evaluated on TUM-RGBD zero-shot without any fine-tuning. \n3) r2, r3 and r4 use global optimization and are not typically considered VO methods.  Following standard practice [e.g., TartanVO ‘21], we focused our comparisons on strictly VO methods, i.e., those that do not perform any global optimization: Global optimization can substantially improve results on input sequences with loops, but at the expense of speed. \n\nRegardless, we’ve included comparisons of DPVO against r1, r2, r3 and r4 in tables B,C and D in the rebuttal PDF.\n\n*Comparison to r2, r3, r4 on TUM-RGBD*: \n\nWe compare to r2, r3 and r4 in Table B on the popular [fr1/desk fr2/xyz fr3/office] split of the TUM-RGBD dataset.  These sequences have been used for comparisons by multiple papers including iMAP and NICE-SLAM. These results show that our approach can outperform others on some sequences, even when they use additional depth sensor input, or use global optimization.\n\nIn Table B, we observe that DPVO outperforms all methods on fr2/xyz, and underperforms Li et al. on fr1/desk. DPVO underperforms others on fr3/office, which is expected because the entire fr3/desk sequence is a loop, which favors methods with global optimization. DPVO with default settings runs nearly 3x faster than all the three, and our model runs more than 8x faster in its “fast” configuration.\n\nr3 (iMAP) and r2 (NICE-SLAM) assume an RGB-D input sequence, while r4 (Li et al.) and our approach do not require depth, only RGB input.\n\nLi et al. and NICE-SLAM do not globally optimize all poses, but still use the global keyframe list and/or the global 3D map to optimize the keyframe poses in the optimization window. \n\nDPVO also has near-constant memory usage in all cases, as opposed to iMAP, NICE-SLAM, and Li et al. whose memory grows proportionally to the sequence length in an unbounded scene. Unlike these other approaches, DPVO’s memory is always bounded; The RGB sequence could go on forever and DPVO would never run out of memory because it never touches old keyframes or the old 3D map.\n\n\n\n\n*Comparison to r4 on EuROC*: \n\n\n\n\n\n\nWe again compare our VO system to r4 (Li et al.) on the EuRoC test dataset in Table C in the rebuttal PDF. Li et al. outperforms DPVO on 2 / 6 of the sequences, catastrophically fails on another 2 / 6 of the remaining sequences, and underperforms DPVO on the remaining 2 / 6.\n\n*Comparison to r1 on TUM-RGBD (separate split)*:\n\nWe compare to r1 (Xue et al.) in table D, on a separate split of TUM-RGBD used for evaluation by Xue et al, since they did not report results on the sequences used by r2-r4. DPVO outperforms Xue et al. on three of the sequences, while they outperform us on the remaining sequences. \n\nIt is important to note that Xue et al. is trained on sequences that share the same scenes with the test sequences, whereas our approach is trained only on synthetic data and is tested on TUM-RGBD zero-shot.\n\n\n**Response to Q1**\n\nThese comparisons were already provided in figure 5c in the main paper, where we show evidence that random sampling does indeed work better than many other approaches. One of these approaches is sampling areas with high image gradients, which is analogous to sampling from pixels with rich textures. \n\nNote that selecting patch centroids randomly is the simplest possible approach (no keypoint detector required). We do not claim that it is the optimal choice, but the fact that such a simple scheme works well is a strength, not a weakness. \n\n**Response to Q2**\n\nOur method does indeed reject outliers in the recurrent module, by predicting a confidence weight associated with each predicted match. These confidences are then treated as constants in the BA step. This enables DPVO to exclude unlikely matches from the optimization entirely, without needing to correct them.\n\nWe follow DROID-SLAM in this regard, and don’t claim this as a contribution of our method. We show an example of the predicted per-match confidence in Fig E of the supplement. \n\n"}, {"review_id": "gI4Jv6VaMw", "reviewer": "Reviewer_5ejm", "summary": "The author propose a learning-based approach for visual odometry. It is essentially a sparse version of DROID-SLAM, without the loop closure capability. Instead of predicting the dense flow field relating the different frames as in DROID-SLAM, DVPO operates on sparse image patches, which are extracted at random from each frame. In more details, a random set of patches are extracted from each video frame, a patch graph is built. A recurrent neural network predicts the residual motion of each patch in each other frame. These residual motions are then used in a differentiable BA layer which minimizes the reprojection error to optimize for the sparse depth of each patches and the camera poses. This process is repeated iteratively until convergence. The paper proposes a new architecture to predict the residual motion, adapted to patches. ", "questions": "1) see weakness\n\n2) A better figure for the patch graph would be help the understanding, showing what are the nodes, what are the edges, where the updates are and so on. Fig.1 is not very informative. It would significantly help to read sec. 3.1. \nI also find using the same index notation for the patches and for the images very confusing. ex L.159 (k, i) refers to an edge between patch k and image I. It would make it easier if the patches had a different notations like bold k. \n\n3) some implementation details: how large are the patches? \n\n4) in the ablation study, it is shown that using a patch is better than point feature. According to the formulation for estimating the motion of the patch, using a patch effectively increases the search window of the correlation in the other image. Does this difference still hold if the point feature is correlated to a larger window in the other image to estimate its motion? \n", "limitations": "yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1) Thorough and convincing evaluation: The proposed approach leads to a convincing improvement in terms of speed and accuracy, compared to DROID-SLAM. The approach is evaluated on many different datasets. \n\n2) The ablation study also pin-points the essential contributions", "weaknesses": "In DROID-SLAM, predicting the dense flow allowed to optimize for full dense depths along with the camera poses with accuracy, providing advantages over both direct and indirect approaches. However, the proposed approach DVPO is based on jointly estimating the motion of sparse patches and refining camera poses and sparse depth (of the patches).\nIn that sense, it is not obvious what advantages are brought by coupling the matches estimation with the pose/depth prediction. Classical indirect approaches first predict sparse matches, triangulate 3d point to get the depth (thus providing initialization for the depth of ‘patches’), and then track further frame by performing PnP with multiple local and global BA. The paper compares to ORD-SLAM and outperforms it. However, there are now many sparse matching approaches that significantly outperform hand crafter detectors like ORB. How would DVPO compare to using a ‘classical’ indirect approach with matching performed by a deep-learning based state-of-the-art approach like SuperPoint-SuperGlue? \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**In DROID-SLAM, predicting the dense flow allowed to optimize for full dense depths along with the camera poses with accuracy, providing advantages over both direct and indirect approaches. However, the proposed approach DVPO is based on jointly estimating the motion of sparse patches and refining camera poses and sparse depth (of the patches). In that sense, it is not obvious what advantages are brought by coupling the matches estimation with the pose/depth prediction.**\n\nThe advantages of DPVO over other methods come from a combination of (1) sparse matches, (2) optimizing pose and depth (bundle adjustment) on sparse matches, (3) coupled iterations of sparse matches and bundle adjustment, (4) end-to-end differentiability and training. \nIt is true that (1)  + (2) are also used by many existing methods, particularly indirect methods. Thus it may appear that DPVO has lost its advantages by using sparse matches. However, the full combination of (1) to (4) is unique to DPVO. In other words, the advantages of DPVO over indirect methods come from integrating (3) + (4) with (1) + (2). \n\n**Classical indirect approaches first predict sparse matches, triangulate 3d point to get the depth (thus providing initialization for the depth of ‘patches’), and then track further frame by performing PnP with multiple local and global BA. The paper compares to ORD-SLAM and outperforms it. However, there are now many sparse matching approaches that significantly outperform hand crafter detectors like ORB. How would DVPO compare to using a ‘classical’ indirect approach with matching performed by a deep-learning based state-of-the-art approach like SuperPoint-SuperGlue?**\n\nIn table E in the rebuttal PDF, we show that DPVO achieves 62% lower error on TartanAir than an approach based on COLMAP with SuperGlue+Superpoint matching, while running 80x faster. This superglue-based result is also reported in the DROID-SLAM paper. We will add this comparison to our next revision. \n\nThe comparison is on the TartanAir test set, following the evaluation in the ECCV 2020 SLAM competition and in [DROID-SLAM]. The score is computed using normalized relative pose error for all possible sequences of length {5, 10, 15, ..., 40} meters.\n\nThe Superglue + Superpoint approach runs 40x slower than real-time, as is expected since this approach is designed for offline structure-from-motion, not visual odometry.\n\n**A better figure for the patch graph would be help the understanding, showing what are the nodes, what are the edges, where the updates are and so on. Fig.1 is not very informative. It would significantly help to read sec. 3.1. I also find using the same index notation for the patches and for the images very confusing. ex L.159 (k, i) refers to an edge between patch k and image I. It would make it easier if the patches had a different notations like bold k.**\n\nIn Figure L in the rebuttal PDF, we’ve included a new figure illustrating a single edge in the patch-graph connecting two nodes (a frame and a patch), and a hypothetical update that could be made to that edge. Thank you for the suggestions, we will include them in our revision.\n\n**Some implementation details: how large are the patches?**\n\nThey are 3 x 3 at ¼ the image resolution, so they effectively cover 12 x 12 pixels in the original image.\n\n**In the ablation study, it is shown that using a patch is better than point feature. According to the formulation for estimating the motion of the patch, using a patch effectively increases the search window of the correlation in the other image. Does this difference still hold if the point feature is correlated to a larger window in the other image to estimate its motion?**\n\nThis is an empirical question on the combination of two hyperparameters: patch size and correlation radius. While we found the patch size 3x3 to work better than 1x1 while keeping other hyperparameters constant (as is common practice for hyperparameter ablations), we did not exhaustively test all combinations so as to avoid overfitting to the validation set. \n\nDue to an initial misunderstanding of this question (we mistook it to be a conceptual question rather than empirical), we did not start the necessary experiments early enough to be able to report this specific ablation by the rebuttal deadline, but we will post the result as soon as we are able. \n\n"}, {"review_id": "Zkw1VmppnL", "reviewer": "Reviewer_kzhF", "summary": "This paper proposes a deep learning-based method for monocular visual odometry, which equips two leading-edge advantages: (i) a deep feature-based patch representation for keypoints encoding local context, and (ii) a novel recurrent architecture designed for patches along with a differentiable bundle adjustment layer.  The proposed DPVO outperforms the state-of-the-art across several common benchmarks while running 1.5-8.9x faster than the DROIOD-SLAM system with less memory leveraged.", "questions": "See Weakness part", "limitations": "The paper claims that the current learning-based VO systems are impractical for resource-constrained devices. However, the proposed method still cannot be applied to mobile GPU-free devices.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The proposed monocular visual odometry performs accurately in estimating the camera pose in several datasets. And the proposed system shows a strong generalization that is trained on the synthetic TarTanVO dataset but validated on real datasets.", "weaknesses": "1. The paper makes claims that the proposed method outperforms all prior work in several evaluation datasets, but only several works are listed which is not enough, can authors give more comparisons with SOTA? For example, the DytanVO[1], RAM-VO[2], DF-VO[3]\n\n[1]https://arxiv.org/pdf/2209.08430v4.pdf\n\n[2] https://arxiv.org/pdf/2107.02974v1.pdf\n\n[3] https://arxiv.org/pdf/2103.00933v1.pdf\n\n2. The results on various datasets are measured by ATE(Absolute Trajectory Error), but some other evaluation criteria are also essential, for example, the relative pose error (RPE), the average translational error, and the rotational error. Can the author provide more comparison information about them?\n\n3. The tracking of patches is achieved by designing an RNN but fails to provide a clear explanation. Why not a CNN?\n\n4. Punctuations are necessary at the end of each equation. Please carefully check it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**The paper makes claims that the proposed method outperforms all prior work in several evaluation datasets, but only several works are listed which is not enough, can authors give more comparisons with SOTA? For example, the DytanVO[1], RAM-VO[2], DF-VO[3]**\n\nOur claim was intended to mean that we outperform all prior work that is known to us and has reported results on the 4 datasets, which are standard benchmarks for VO/SLAM. We will clarify this point in our revision. \n\nWe include an additional Table F in the rebuttal PDF of results comparing DPVO against DF-VO [Zhan et al. ‘20]. We observe that DPVO outperforms DF-VO on average. \n\nUnfortunately, we do not yet have direct comparisons with DytanVO and RAM-VO. DytanVO and RAM-VO reported no results on any of the 4 datasets we use.  To use their open-source code requires us to develop nontrivial dataloading code for each dataset. We are in the process of doing so, but have not been able to complete it before the deadline. \n\n\n**The results on various datasets are measured by ATE(Absolute Trajectory Error), but some other evaluation criteria are also essential, for example, the relative pose error (RPE), the average translational error, and the rotational error. Can the author provide more comparison information about them?**\n\nTable F in the rebuttal PDF includes comparisons using relative pose error (RPE). It shows that our method is superior also in terms of RPE.\n\nWe primarily used the ATE metric for evaluation in the main paper since this enabled us to easily compare DROID-VO/SLAM and ORB-SLAM on more datasets, such as TartanAir, TUM-RGBD, and EuRoC.\n\nWe agree that more metrics is always better, however we evaluated using ATE as it is the primary metric used in several prior works [DROID-SLAM, ORB-SLAM3], which allowed us to compare to prior works on a large number of datasets without the need to reproduce the results ourselves. It is worth pointing out that ATE is popular because the final camera positions (which ATE evaluates) are effectively a function of the accumulated error in both translation magnitude and direction (which is related to the predicted rotation), i.e. it’s extremely unlikely to have very low ATE but high rotation error.\n\n**The tracking of patches is achieved by designing an RNN but fails to provide a clear explanation. Why not a CNN?**\n\nThe update operator is recurrent (i.e. weight-tied) since it continually improves its own predictions based on its current state through repeated applications. A CNN would not be a natural choice. Some explanations are already provided on L147-L158 + Figure 2 in the main paper and L53-L58 + Figure C in the supplement, but we will revise and clarify.  As mentioned in L171-L177, one component of DPVO’s update operator is a 1D convolution across the temporal dimension of each patch trajectory. Therefore, our update operator is an RNN with a CNN layer inside it.  \n"}]

Author rebuttal:
**Message to all reviewers regarding the novelty of DPVO:**

Our novelty is sparse matches *integrated with* neural iterative updates and differentiable bundle adjustment, which has not been done before and is nontrivial to design. While sparse matches have been used in prior work, they were not integrated with neural iterative updates and differentiable bundle adjustment. 

Our method can be understood as a sparsification of DROID-SLAM, a state-of-the-art SLAM approach, to improve efficiency, but such sparsification is nontrivial. DROID-SLAM does iterations through recurrent 2D convolutions on 2D feature maps, producing a dense flow field between existing and incoming video frames. Naive sparsification schemes would encounter the following difficulties: 

-  Simply sparsifying the feature maps and applying 2D convolutions does not work because we would be doing 2D convolutions on a feature map that is 95% empty

-  Simply replacing K x K convolution filters with 1x1s means losing the ability to use neighboring spatial context when making predictions - a fundamental benefit of 2D convolutions. 

-  Using 95-99% fewer keypoint matches means there is less redundancy against incorrect matches. DROID-SLAM can afford to make many more mistakes since it produces a dense flow field rather than sparse matches.

Our solutions to these above challenges are:

-  Incorporating contextual information using patch-based correlation, and message passing between patches.

-  Adding 1D convolutions along the temporal dimension of a trajectory to pass around additional contextual information. 

-  Offset the potential accuracy loss from the less redundancy  by using the significant memory savings to use a higher spatial resolution (¼ instead of ⅛) and use a larger optimization window.


Another contribution of this paper is a VO system that is open-source,  runs at 60-120 FPS, uses little GPU memory, and is as accurate or better than the current state-of-the-art open-source VO system (which is 2x-3x slower, 2x as expensive). 
We believe a well-engineered open-source VO system is valuable and of interest to the research community.

Meta-review: This submission received mixed reviews (WA, BA, BA, R). There was a consensus among the reviewers that the technical contribution is limited, even after rebuttal. However, the contribution is sufficiently interesting for most of the reviewers to propose to accept the submission. After checking the contribution (replacing dense matching by sparse matching in "neural visual odometer" because sparse matching is more reliable and accurate) carefully, I agree it is interesting, and I propose to accept this submission.
Decision: Accept (poster)