SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Overview: In Fig. 5 we show an overview of the VO system. The visualization and frame loading are performed in separate threads.
Initialization: We use 8 frames for initialization. We add new patches and frames until 8 frames are accumulated and then run 12 iterations of our update operator. There needs to be some camera motion for initialization; hence, we only accumulate frames with an average flow magnitude of at least 8 pixels from the prior frame.
Expansion: When a new frame is added we extract features and patches. The pose of the new frame is initialized using a constant velocity motion model. The depth of the patch is initialized as the median depth of all the patches extracted from the previous 3 frames.
We connect each patch to every frame within distance  $r$  from the frame index where the patch was extracted. This means that when a new patch is added, we add edges between that patch and the previous  $r$  keyframes. When a new frame is added, we add edges between each patch extracted in the last  $r$  keyframes with the new frame. This strategy means that each patch is connected to no more than  $(2r - 1)$  frames, bounding the worst-case latency.
**Optimization:** Following the addition of edges we run one iteration of the update operator followed by two bundle adjustment iterations. We fix the poses of all but the last 10 keyframes. The inverse depths of all patches are free parameters. The patches are removed from optimization once they fall outside the optimization window.
Keyframing: The most recent 3 frames are always taken to be keyframes. After each update, we compute the optical flow magnitude between keyframe  $t - 5$  and  $t - 3$ . If this is less than 64px, we remove the keyframe at  $t - 4$ . When a keyframe is removed, we store the relative pose between its neighbors such that the full pose trajectory can be recovered for evaluation.
ME000	ME001	ME002	ME003	ME004	ME005	ME006	ME007	MH000	MH001	MH002	MH003	MH004	MH005	MH006	MH007	Avg
ORB-SLAM3* (4)	13.61	16.86	20.57	16.00	22.27	9.28	21.61	7.74	15.44	2.92	13.51	8.18	2.59	21.91	11.70	25.88	14.38
COLMAP* (31)	15.20	5.58	10.86	3.93	2.62	14.78	7.00	18.47	12.26	13.45	13.45	20.95	24.97	16.79	7.01	7.97	12.50
DSO (12)	9.65	3.84	12.20	8.17	9.27	2.94	8.15	5.43	9.92	0.35	7.96	3.46	-	12.58	8.42	7.50	7.32
DROID-SLAM* (37)	0.17	0.06	0.36	0.87	1.14	0.13	1.13	0.06	0.08	0.05	0.04	0.02	0.01	0.68	0.30	0.07	0.33
DROID-VO	0.22	0.15	0.24	1.27	1.04	0.14	1.32	0.77	0.32	0.13	0.08	0.09	1.52	0.69	0.39	0.97	0.58
Ours (Default)	0.16	0.11	0.11	0.66	0.31	0.14	0.30	0.13	0.21	0.04	0.04	0.08	0.58	0.17	0.11	0.15	0.21
Ours (Fast)	0.35	0.13	0.27	0.71	0.47	0.16	0.30	0.13	0.34	0.05	0.06	0.07	0.81	0.41	0.09	0.14	0.28
Table 1: Results on the TartanAir monocular test split from the ECCV 2020 SLAM competition. Results are reported as ATE with scale alignment. For our method, we report the median of 5 runs. Methods marked with (*) use global optimization / loop closure.
360	desk	desk2	floor	plant	room	rpy	teddy	xyz	avg
ORB-SLAM3[4]	x	0.017	0.210	x	0.034	x	x	x	0.009	-
DSO[13]	0.173	0.567	0.916	0.080	0.121	0.379	0.058	x	0.036	-
DSO-Realtime[13]	0.172	0.718	0.728	0.068	0.167	0.767	x	x	0.031	-
DROID-VO[32]	0.161	0.028	0.099	0.033	0.028	0.327	0.028	0.169	0.013	0.098
Ours (Default)	0.135	0.038	0.048	0.040	0.036	0.394	0.034	0.064	0.012	0.089
Ours (Fast)	0.169	0.029	0.064	0.047	0.047	0.396	0.034	0.074	0.012	0.097
Table 2: Results (ATE) on the freiburg1 set of TUM-RGBD (33). This evaluates monocular visual odometry (No method uses stereo or sensor depth), and is identical to the evaluation setting in DROID-SLAM (37). For each sequence, we report the median result across 5 independent trials. Missing values (i.e. x) indicate that the method completely failed and refused to output a trajectory. Only learning based methods are robust enough to not fail on any sequence, and out of them Ours (Default) performs best on average.
We evaluate DPVO on the TartanAir (44), TUM-RGBD (33), EuRoC (2) and ICL-NUIM (19) benchmarks. On each dataset, we run five trials each with a different set of patches and report the median results obtained. Example reconstructions on the TartanAir and ETH3D datasets are shown in Fig. 4a and in the Appendix, respectively. We benchmark two configuration settings of DPVO: Ours (Default) uses 96 patches per image and a 10 frame optimization window and Ours (Fast) uses 48 patches and a 7 frame optimization window. Our frame rates are practically near constant, since our method uses a constant number of FLOPS per-frame, unlike prior methods which slow down and make more keyframes during fast motion (37; 12). We also benchmark two versions of DROID-SLAM: the full version and a version where loop closure and global bundle adjustment is disabled (DROID-VO). DROID-VO is more comparable to our method as we only perform local optimization. Across all benchmarks, DPVO achieves the lowest average error among all previous VO systems, and even outperforms full SLAM systems on some benchmarks. We focus on average error as it reflects the expected performance of our system in the wild.
TartanAir Validation Split: We use the same 32-sequence validation split as DROID-SLAM and report aggregated results in Fig. 4b and compare with DROID-SLAM and ORB-SLAM3. We run our method 3 times on each sequence and aggregate the results. In the  $[0,1]\mathrm{m}$  error window, we get an AUC of 0.80 compared to 0.71 for DROID-SLAM.
TartanAir Test Split: Tab. 1 reports results on the test-split used in the ECCV 2020 SLAM competition compared to state-of-the-art methods. Classical methods such as DSO and ORB-SLAM fail on more than  $80\%$  of the sequences, hence we use COLMAP as a classical baseline as it was used in the two winning solutions of the ECCV SLAM competition. We attain an error  $40\%$  lower than DROID-SLAM and  $64\%$  lower than DROID-VO. COLMAP takes 2 days to complete the 16 sequences and typically produces broken reconstructions which lead to large errors in evaluation.
EuRoC MAV (2) In Tab. 3 we benchmark on the EuRoC MAV (2) dataset and compare to other visual odometry methods including SVO (15), DSO (12) and the visual odometry version of DROID-SLAM (36). Video from the EuRoC benchmark is recorded at 20FPS. Like DROID-SLAM, we skip every other frame, doubling the effective frame rate of the system. Our default system outperforms prior work on the majority of the EuRoC sequences. The average error is  $43\%$  lower than DROID-VO (37). Even our 120-FPS system outperforms DROID-VO on most video.
MH01	MH02	MH03	MH04	MH05	V101	V102	V103	V201	V202	V203	Avg
TartanVO (43)	0.639	0.325	0.550	1.153	1.021	0.447	0.389	0.622	0.433	0.749	1.152	0.680
SVO (15)	0.100	0.120	0.410	0.430	0.300	0.070	0.210	-	0.110	0.110	1.080	0.294
DSO (12)	0.046	0.046	0.172	3.810	0.110	0.089	0.107	0.903	0.044	0.132	1.152	0.601
DROID-VO (37)	0.163	0.121	0.242	0.399	0.270	0.103	0.165	0.158	0.102	0.115	0.204	0.186
Ours (Default)	0.087	0.055	0.158	0.137	0.114	0.050	0.140	0.086	0.057	0.049	0.211	0.105
Ours (Fast)	0.101	0.067	0.177	0.181	0.123	0.053	0.158	0.095	0.095	0.063	0.310	0.129
Table 3: Monocular SLAM on the EuRoC datasets, ATE[m] compared to other visual odometry methods. For our method, we report the median of 5 runs.
(a) Point vs Patch features
(b) Update operator components
(c) Selecting patch centroids
Figure 6: Ablation experiments. (a) We show the importance of using patches over point features. (b) Removal of different components of the update operator degrades accuracy. (c) Randomly selecting patch centroids works better than 2D keypoint locations produced via SIFT (22), ORB (29), Superpoint (9), or pixels with high image gradient.
TUM-RGBD (33): In Tab. 2 we benchmark on TUM-RGBD (33) and compare to DSO (12), the visual odometry version of DROID-SLAM (36), and ORB-SLAM3 (27). We evaluate only visual-only monocular methods, identical to the TUM-RGBD evaluation in (37). This benchmark evaluates motion tracking in an indoor environment with erratic camera motion and significant motion blur. Video from the TUM-RGBD benchmark is recorded at 30FPS.
Classical approaches such as ORB-SLAM (27) and DSO (12) perform well on some sequences but exhibit frequent catastrophic failures. Similar to DROID-VO, our system is also robust to such failures but our average error is  $9\%$  lower. This indicates that our sparse, patch-based approach performs better in-the-wild compared to dense flow.
ICL-NUIM (19): In Tab. 4 we evaluate on the ICL-NUIM (19) SLAM benchmark and compare to other visual odometry and SLAM methods including SVO (15), DSO (12) and DROID-SLAM (37). Video from the ICL-NUIM benchmark is recorded at 30FPS. ICL-NUIM is a synthetic dataset for evaluating SLAM performance in indoor environments with repetitive or monochrome textures (e.g. blank white walls). Our system outperforms prior work on the majority of the ICL-NUIM sequences.
(a) Average/worst-case runtime
(b) GPU memory
Figure 7: Efficiency on EuRoC. (a) Compared to DROID-VO (the front-end of DROID-SLAM), the Default and Fast variants of DPVO run 1.5x (60FPS) and 3x (120FPS) faster on average. DPVO's frame-rate is relatively stable, remaining above 48FPS and 98FPS for  $95\%$  of frames. In contrast, DROID-VO drops to 11FPS, an  $8.9\mathrm{x}$  difference. (b) DPVO (Default) and DPVO (Fast) use  $57\%$  and  $29\%$  as much memory as DROID-VO, respectively. (c) We can trade-off speed for performance by increasing the number of patches tracked. However, performance quickly saturates beyond 96.
Living Room 0	Living Room 1	Living Room 2	Living Room 3	Office Room 0	Office Room 1	Office Room 2	Office Room 3	Avg
ORB-SLAM2* (27)	0.461	0.172	0.806	N/A	0.589	0.813	1.000	N/A	N/A
DROID-SLAM* (37)	0.008	0.027	0.039	0.012	0.065	0.025	0.858	0.481	0.189
DROID-VO (37)	0.010	0.123	0.072	0.032	0.095	0.041	0.842	0.504	0.215
SVO (15)	0.02	0.07	0.09	0.07	0.34	0.28	0.14	0.08	0.136
DSO (12)	0.01	0.02	0.06	0.03	0.21	0.83	0.36	0.64	0.270
DSO (Realtime) (12)	0.02	0.03	0.33	0.06	0.29	0.64	0.23	0.46	0.258
Ours (Default)	0.006	0.006	0.023	0.010	0.067	0.012	0.017	0.635	0.097
Ours (Fast)	0.008	0.007	0.021	0.010	0.071	0.015	0.018	0.593	0.093
Table 4: Results on the ICL-NUIM SLAM benchmark. Results are reported as ATE with scale alignment. Methods marked with (*) use global optimization / loop closure. SVO and DSO results are from (16). ORB-SLAM2 results are from (10) (only partial results were provided). Despite our best efforts, we were unsuccessful in running monocular ORB-SLAM3: The official software would not produce any keyframes while running.
The average error of our faster model is  $51\%$  lower than DROID-SLAM (37) and  $32\%$  lower than SVO (15). Our faster and default systems perform similarly.
Efficiency: In Fig. 7 we compare the efficiency of DPVO and DROID-VO. Our 60-FPS variant is  $1.5\mathrm{x}$  faster on average and  $4.3\mathrm{x}$  faster in the worst case (5th percentile), using  $57\%$  the GPU memory of DROID-VO. The 120-FPS DPVO variant is  $3\mathrm{x}$  faster on average and  $8.9\mathrm{x}$  faster in the worst-case, using less than a third of the memory.
4.1 Ablations
We perform ablation experiments on the TartanAir validation split and show results in Fig. C. We use the same parameter settings in all experiments with augmentation disabled. We run each ablation experiment three times on the validation split and aggregate the results.
Point vs Patch Features: In Fig. 6a, we demonstrate the importance of the patches over simply using point features (i.e. 1x1 patches). The patch features encode local context which is lacking with point features. The additional information stored in the correlation features allows for more precise tracking.
Update Operator: In Fig. 6b, we test the effect of removing various components from the update operator. Both removing 1D-Convolution and Softmax-Aggregation degrade performance on the validation set.
Patch Selection: In Tab. 6c, we compare different methods for selecting 2D patch centroids. Surprisingly, our method performs best when randomly selecting keypoint centroids.
DPVO is a new deep visual odometry system built using a sparse patch representation. It is accurate and efficient, capable of running at 60-120 FPS with minimal memory requirements. DPVO outperforms all prior work (classical or learned) on EuRoC, TUM-RGBD, the TartanAir ECCV 2020 SLAM competition, and ICL-NUIM.
Acknowledgments: This work was partially supported by the Princeton University Jacobus Fellowship, NSF, Qualcomm, and Amazon.
[1] Sameer Agarwal, Keir Mierle, and The Ceres Solver Team. Ceres Solver, 3 2022.
[2] Michael Burri, Janosch Nikolic, Pascal Gohl, Thomas Schneider, Joern Rehder, Sammy Omari, Markus W Achtelik, and Roland Siegwart. The euroc micro aerial vehicle datasets. The International Journal of Robotics Research, 35(10):1157-1163, 2016.
[3] Cesar Cadena, Luca Carlone, Henry Carrillo, Yasir Latif, Davide Scaramuzza, José Neira, Ian Reid, and John J Leonard. Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age. IEEE Transactions on robotics, 32(6):1309-1332, 2016.
[4] Carlos Campos, Richard Elvira, Juan J Gomez Rodriguez, Jose MM Montiel, and Juan D Tardos. Orb-slam3: An accurate open-source library for visual, visual-inertial, and multimap slam. IEEE Transactions on Robotics, 37(6):1874-1890, 2021.
[33] J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers. A benchmark for the evaluation of rgb-d slam systems. In Proc. of the International Conference on Intelligent Robot Systems (IROS), Oct. 2012.
[34] Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. Loftr: Detector-free local feature matching with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8922-8931, 2021.
[35] Chengzhou Tang and Ping Tan. Ba-net: Dense bundle adjustment network. arXiv preprint arXiv:1806.04807, 2018.
[36] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In European conference on computer vision, pages 402-419. Springer, 2020.
[37] Zachary Teed and Jia Deng. Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras. Advances in Neural Information Processing Systems, 34:16558-16569, 2021.
[38] Zachary Teed and Jia Deng. Raft-3d: Scene flow using rigid-motion embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8375–8384, 2021.
[39] Prune Truong, Martin Danelljan, Luc Van Gool, and Radu Timofte. Learning accurate dense correspondences and when to trust them. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5714-5724, 2021.
[40] Shinji Umeyama. Least-squares estimation of transformation parameters between two point patterns. IEEE Transactions on Pattern Analysis & Machine Intelligence, 13(04):376-380, 1991.
[41] Lukas Von Stumberg, Vladyslav Usenko, and Daniel Cremers. Direct sparse visual-inertial odometry using dynamic marginalization. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 2510-2517. IEEE, 2018.
[42] Rui Wang, Martin Schworer, and Daniel Cremers. Stereo dso: Large-scale direct sparse visual odometry with stereo cameras. In Proceedings of the IEEE International Conference on Computer Vision, pages 3903-3911, 2017.
[43] Wenshan Wang, Yaoyu Hu, and Sebastian Scherer. Tartanvo: A generalizable learning-based vo. arXiv preprint arXiv:2011.00359, 2020.
[44] Wenshan Wang, Delong Zhu, Xiangwei Wang, Yaoyu Hu, Yuheng Qiu, Chen Wang, Yafei Hu, Ashish Kapoor, and Sebastian Scherer. Tartanair: A dataset to push the limits of visual slam. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4909-4916. IEEE, 2020.
[45] Huangying Zhan, Chamara Saroj Weerasekera, Jia-Wang Bian, and Ian Reid. Visual odometry revisited: What should be learnt? In 2020 IEEE international conference on robotics and automation (ICRA), pages 4203-4210. IEEE, 2020.
A Stable Runtime
The runtime of our approach is relatively constant compared to prior VO/SLAM systems. Our default configuration averages 60FPS, and rarely drops below 50FPS. In Fig. A, we show the runtime distribution on EuRoC. The runtime stability of DPVO results from the comparatively simple keyframing mechanism.
Discussion on Keyframing: Most VO/SLAM systems (12; 27; 37), including DROID-SLAM, contain a protocol for deciding when to produce new keyframes from a video stream. Keyframing mechanisms serve to discard repetitious frames early-on, saving on computation and potentially improving performance by limiting redundancy. Many VO/SLAM systems require a low keyframing frequency in order to run at real-time framereates. If there is significant motion between subsequent frames, keyframes are created at a high frequency which can cause such methods to slow down drastically. DROID-SLAM, for instance, slows from 40FPS to 11FPS during sufficiently large increases in the number of keyframes being created.
Unlike previous works, DPVO treats all incoming frames as keyframes and only removes redundant frames later using motion computed from the already-estimated pose. Although this design decision is sub-optimal (in terms of speed) during very slow or still camera movement (e.g. EuRoC), our approach is both simpler and ensures our frame-rate is approximately constant no matter the degree of camera motion.
Figure A: Runtime distribution on EuRoC. Our system averages 60FPS with each new frame taking  $\sim 17$  ms to process. The distribution is very centralized and rarely drops below 50FPS.
B ATE Error Metric
We compare results using the ATE (average trajectory error) metric, which is standard for VO and SLAM (16; 37; 27; 43; 12). The ATE metric between the predicted trajectory  $\tilde{\mathbf{T}}$  and the ground-truth trajectory  $\hat{\mathbf{T}}$  is computed using
$$
E (\tilde {\mathbf {T}}, \hat {\mathbf {T}}) := \sum_ {t = 1} ^ {N} \left\| \tilde {\mathbf {T}} _ {x y z} - \hat {\mathbf {T}} _ {x y z} \right\| _ {2} \tag {9}
$$
after aligning the two trajectories using a similarity transformation, to account for the scale and SE(3) gauge freedoms. This metric is computed using the EVO library (18).
C Additional Training Details
DPVO is trained entirely on TartanAir  $(44)^2$ , a synthetic dataset. This is the same synthetic training data used by previous VO systems (43; 37). TartanAir provides a large number of training scenes of both indoor and outdoor environments with varied lighting and weather conditions. The dataset provides depth and camera pose annotations, enabling one to generate optical flow by re-projecting
$^{2}$ https://theairlab.org/tartanair-dataset/
(a) Reconstruction on the ETH-3D (32) dataset.
Figure B: Sparse reconstructions produced by our model.
(b) Reconstruction from our video demo.
the depth using the camera poses. All dataset parameters are identical to those used in DROID-SLAM (37). This includes random cropping, color-jitter, random greyscale, random color inversion, as well as re-scaling the training scenes in order to improve training stability.
D Visualization
Reconstructions are visualized interactively using a separate visualization thread. Our visualizer is implemented using the Pangolin library<sup>3</sup>. It directly reads from PyTorch tensors avoiding all unnecessary memory copies from CPU to GPU. This means that the visualizer has very little overhead—only slowing the full system down by approximately  $10\%$ . In Fig. Ba, we show a reconstruction on the ETH-3D (32) dataset, and in Fig. Bb we show the reconstruction produced during our video demo.
E Bundle Adjustment Layer
Our bundle adjustment layer is functionally identical to the BA layer in DROID-SLAM (37), except that DROID-SLAM predicts the damping factor using its update operator, while we found this not to be necessary and hard-code it to  $10^{-4}$  instead. We refer the reader to Appendix C in the DROID-SLAM paper for the analytical gradients of the Gauss-Newton update iterations. Since DROID-SLAM uses dense flow, we implement our own sparse version as an optimized CUDA layer.
F Additional Figures
Correlation Operation: In Fig. Ca, we show a high-level visualization of the correlation operation which is performed at the start of the update operator. Patch  $k$ 's  $P^2$  matching-features are each projected into the two-level feature pyramid of frame  $j$ . A  $7 \times 7$  grid of  $D = 128$  dimension feature vectors are bilinearly sampled from the pyramid around each point reprojection and dot produced with the matching features. This operation results in 2 sets of  $p \times p \times 7 \times 7$  dot products, one for each resolution (1/4 and 1/8th of the original image size). This operation is implemented as an optimized CUDA layer.
Update Operator: Following Fig. Ca, in Fig. Cb we show a high-level view of the rest of the update operator. The correlation and context features are fed into additional learn-able layers, including 1D
<sup>3</sup>https://github.com/stevenlovegrove/Pangolin
(a) Correlation Step
(b) Update Operator
Figure C: The update operator, including the correlation step. (a) For each edge  $(k,j)$  in the patch graph, patch  $k$  is reprojected into a frame  $j$  using equation (2). Frame  $j$ 's matching features are then cropped using the reprojected patch. The correlated matching features (blue, green) and context features (grey) form the edge features. Each dot in the sampled matching features (green) represents a  $7 \times 7$  neighborhood of points, however this is omitted for clarity. (b) Given the features for edge  $(k,j)$ , several learnable layers (yellow) are used to predict an update  $\delta_{(k,j)} \in \mathbb{R}^2$  to the flow, a confidence weight  $\Sigma_{(k,j)} \in \mathbb{R}^2$ , and an update to the hidden state  $h_{k,j}^{t}$ . The predicted factors are used in the bundle adjustment layer to produce an update to the camera poses and patch depths. The factor head is omitted here for clarity.
Figure D: Accuracy on the EuRoC dataset as a function of the number of patches. We can trade-off speed for performance by increasing the number of patches tracked. However, performance quickly saturates beyond 96.
Convolution Layers, a Softmax-Aggregation block (i.e. Message Passing), and a Transition Block. This results in an updated hidden state for each edge. Finally, the updated hidden state is used to produce 2D flow revisions and confidence weights which guide the solution to the bundle adjustment. The bundle-adjustment layer ultimately produces an update to the depths and camera poses.
Confidence Weights: Our approach learns to reject outliers by predicting a confidence weight associated to each predicted 2D flow update, which DROID-SLAM (37) does as well. We show a visualization of the weights for a small number of edges in Fig. E. These weights are not supervised directly, rather they are learned by supervising on the pose output of the differentiable bundle-adjustment layer.
Number of Patches: In Fig. D, we show the accuracy and runtime of DPVO on EuRoC while varying the number of patches. Results are accumulated across 5 independent trials. The accuracy quickly saturates after 96 patches.
G Network Architecture
Update Operator: In Fig. F, we show the architecture of the update operator, excluding non-learnable layers such as the correlation layer and the bundle adjustment layer. Since the 1D-Convolutions and the Softmax-Aggregation layers operate on neighboring edges, they are partially implemented using PyTorch (28) scattering operations.
Figure E: Factor confidence weights. For each edge in the patch graph, the factor-head of our update operator predicts a confidence weight  $w \in \mathbb{R}^2$  bounded to (0, 1) and a 2D flow update. Cold colors (blue) represent high confidence edges while hot colors (red) represent low confidence edges.