SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zachary Teed*
zteed@princeton.edu
Lahav Lipson*
llipson@princeton.edu
Jia Deng
jiadeng@princeton.edu
We propose Deep Patch Visual Odometry (DPVO), a new deep learning system for monocular Visual Odometry (VO). DPVO uses a novel recurrent network architecture designed for tracking image patches across time. Recent approaches to VO have significantly improved the state-of-the-art accuracy by using deep networks to predict dense flow between video frames. However, using dense flow incurs a large computational cost, making these previous methods impractical for many use cases. Despite this, it has been assumed that dense flow is important as it provides additional redundancy against incorrect matches. DPVO disproves this assumption, showing that it is possible to get the best accuracy and efficiency by exploiting the advantages of sparse patch-based matching over dense flow. DPVO introduces a novel recurrent update operator for patch based correspondence coupled with differentiable bundle adjustment. On Standard benchmarks, DPVO outperforms all prior work, including the learning-based state-of-the-art VO-system (DROID) using a third of the memory while running 3x faster on average. Code is available at https://github.com/princeton-vl/DPVO
Visual Odometry (VO) is the task of estimating a robot's position and orientation from visual measurements. In this work, we focus on most challenging case—monocular VO—where the only input is a monocular video stream. The goal of the system is to estimate the 6-DOF pose of the camera at every frame while simultaneously building a map of the environment.
VO is closely related to Simultaneous Localization and Mapping (SLAM). Like VO, SLAM systems aim to estimate camera pose and map the environment but also incorporate techniques for global corrections—such as loop closure and relocalization (3). SLAM systems typically include a VO frontend which tracks incoming frames and performs local optimization.
Prior work typically treats VO as an optimization problem solving for a 3D model of the scene which best explains the visual measurements (3). Indirect approaches first detect and match keypoints between frames, then solve for poses and 3D points which minimize the reprojection distance (26; 4; 20). Direct approaches, on the other hand, operate directly on pixel intensities, attempting to solve for poses and depths which align the images (13; 15; 12). The main issue with prior systems, both direct and indirect, is the lack of robustness. Failure cases are too frequent for many important applications such as autonomous vehicles. These failure cases typically stem from moving objects, lost feature tracks, and poor convergence.
Several deep learning approaches (37; 45; 34; 39; 5; 43) have been introduced to address the robustness issue. The main advantage of deep learning is better features as well as differentiable optimization layers guided by neural networks. DROID-SLAM (37), which includes a VO frontend, uses neural networks to estimate dense flow fields which are subsequently used to optimize depth
and camera pose. DROID-SLAM has significantly improved the state-of-the-art in visual SLAM and VO. However, it comes with a large computational cost: in VO mode (without the backend), it averages 40FPS on an RTX-3090 using 8.7GB GPU memory, which can be impractical for resource-constrained devices. In this work, we aim to significantly improve the efficiency of deep VO, and therefore SLAM, without sacrificing its accuracy.
We propose Deep Patch Visual Odometry (DPVO), a new deep learning system for monocular VO. Our method achieves the accuracy and robustness of deep learning-based approaches while running 1.5-8.9x faster using  $57\% -29\%$  the memory. It has lower average error than all prior work on common benchmarks. On an RTX-3090, it averages 60FPS using only 4.9GB of memory, with a minimum frame rate of 48FPS. It can run at 120FPS on the EuRoC dataset (2) using 2.5GB while still outperforming prior work on average. The frame rate of DPVO is relatively-constant in practice, and does not significantly depend on the degree of camera motion, as opposed to prior methods including DROID-SLAM which slow down during fast camera motion (12; 37; 27). We train our entire system end-to-end on synthetic data but demonstrate strong generalization on real video.
This leap in efficiency is achieved by the combination of 1) a deep feature-based patch representation for keypoints which encodes their local context, and 2) a novel recurrent architecture designed to track a sparse collection of these patches through time—alternating patch trajectory updates with a differentiable bundle adjustment layer to allow for end-to-end learning of reliable feature matching. This efficiency boost enables the design of DPVO to allocate resources towards components which improve accuracy, which are described in Sec. 3.
Existing approaches such as DROID-SLAM estimate camera poses by predicting dense flow (24; 23; 37). Designing an efficient VO system with the same-or-better accuracy poses a significant challenge, since using dense matches in SLAM provides additional redundancy against incorrect matches. However, DPVO still achieves robustness by introducing components which improve matching accuracy, using resources which would otherwise be spent estimating dense flow. We observe, surprisingly, that patch-based correspondence improves both efficiency and robustness over dense flow. In classical approaches to VO, patch-based matching (12) has been shown to improve accuracy over keypoint-based methods (27). However, it has remained unclear how to leverage this idea using deep networks without underperforming dense-flow approaches.
To summarize, we contribute Deep Patch Visual Odometry, a new method for tracking camera motion from video. DPVO uses a novel recurrent network designed for sparse patch-based correspondence. DPVO outperforms all prior work across several evaluation datasets, while running 1.5-8.9x faster than the previous state-of-the-art and using only  $57 - 29\%$  as much memory.
Visual Odometry (VO) systems aim to estimate robot state (position and orientation) from a video. Overtime, a VO system will accumulate drift, and modern SLAM methods incorporate techniques to identify previously mapped landmarks to correct drift (i.e loop closure). VO can be considered a subproblem of SLAM with global optimization and loop closures disabled (3).
Many different modalities of VO have been explored by past work, including visual-inertial odometry (VIO) (41; 14) and stereo VO (42; 13). Here, we focus on the monocular case, where the only input is a monocular video stream. Early works approached the problem using filtering and maximum-likelihood methods (6; 25). Modern methods almost universally perform Maximum a Posteriori (MAP) estimation over factor graphs with Gaussian noise; in which case, the MAP estimate can be found by solving a non-linear least-squares optimization problem (8). This problem has lead to the development of many libraries for optimizing non-linear least-squares problems (1; 17; 7).
Among VO systems, our method borrows many core ideas of Direct Sparse Odometry (DSO) (12), a classical system based on least-squares optimization. Namely, we adopt a similar patch representation and reproject patches between frames to construct the objective function. Unlike DSO, the residuals are not based on intensity differences but instead predicted by a neural network which can pass information between patches and across patch lifetimes. Outlier rejection is automatically handled by the network, making our system more robust than classical systems like DSO and ORB-SLAM (12; 26). One important component of classical systems is the careful selection of
which image regions to use. We find, surprisingly, that our system works well on a small number (64 per frame) of randomly sampled image patches.
With regards to deep SLAM systems, our method is closely related to DROID-SLAM (37) but uses a different underlying representation and different network architecture. DROID-SLAM is an end-to-end deep SLAM system which shows good performance compared to both classical and deep baselines. Like our method, it works by iterating between motion updates and bundle adjustment. However, it estimates dense motion fields between selected pairs of frames which has a high computational cost and large memory footprint. While it is capable of 40FPS inference on average, its speed varies depending on the amount of motion in the video. Our method selects sparse patches from the video stream, with a relatively constant runtime per frame and  $1.5 - 8.9\mathrm{x}$  faster inference than DROID-SLAM.
Prior works such as BA-Net (35) and (21) also have embedded bundle adjustment layers in end-to-end differentiable network architectures. However, BA-Net does not use patch-based correspondence. (21) and (11) have proposed neural networks which sit atop COLMAP (31) and perform subpixel-level refinement; these approaches are not, however, able to perform 3D reconstruction on their own and are subject to failure cases in the underlying SfM system.
Preliminaries: Given an input video, we represent a scene as a collection of camera poses  $\mathbf{T} \in \mathbb{SE}(3)^N$  and a set of square image patches  $\mathbf{P}$  extracted from the video frames. Using  $\mathbf{d}$  to represent inverse depth and  $(\mathbf{x}, \mathbf{y})$  to represent pixel coordinates, we represent each patch as the  $4 \times p^2$  homogeneous array
$$
\mathbf {P} _ {k} = \left( \begin{array}{c} \mathbf {x} \\ \mathbf {y} \\ \mathbf {1} \\ \mathbf {d} \end{array} \right) \quad \mathbf {x}, \mathbf {y}, \mathbf {d} \in \mathbb {R} ^ {1 \times p ^ {2}} \tag {1}
$$
where  $p$  is the width of the patch. We assume a constant depth for the full patch, meaning that it forms a fronto-parallel plane in the frame from which it was extracted. Letting  $i$  denote the index of the source frame of the patch, i.e. the frame from which patch  $\mathbf{P}_k$  was extracted, we can reproject the patch onto another frame  $j$
$$
\mathbf {P} _ {k j} ^ {\prime} \sim \mathbf {K T} _ {j} \mathbf {T} _ {i} ^ {- 1} \mathbf {K} ^ {- 1} \mathbf {P} _ {k}. \tag {2}
$$
taking  $\mathbf{K}$  to be the  $4\times 4$  calibration matrix
$$
\mathbf {K} = \left( \begin{array}{c c c c} f _ {x} & 0 & c _ {x} & 0 \\ 0 & f _ {y} & c _ {y} & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \end{array} \right) \tag {3}
$$
The pixel coordinates  $\mathbf{x}' = (x', y')$  can be recovered by dividing by the third element. For the rest of the paper, we use the shorthand  $\mathbf{P}_{kj}' = \omega_{ij}(\mathbf{T}, \mathbf{P}_k)$  to denote the reprojection of patch  $k$  onto frame  $j$  in terms of pixel coordinates.
Patch Graph: We use a bipartite patch graph to represent the relations between patches and video frames. Edges in the graph connect patches with frames. We show an example in Fig. 1. By default, the graph is constructed by adding an edge between each patch and every frame within distance  $r$  from the index of the source frame of the patch. The reprojections of a patch in all of its connected frames in the patch graph form the trajectory of the patch. Note that a trajectory is a set of reprojections of a single patch into multiple frames; it is not a set of original square patches. The graph is dynamic; as new video frames are received, new frames and patches are added while old ones are removed. We provide an example trajectory in Fig. 3.
Approach Overview: At a high level, our approach works similarly to a classical system: it samples a set of patches for each video frame, estimates the 2D motion (optical flow) of each patch against each of its connected frames in patch graph, and solves for depth and camera poses that are consistent with the 2D motions. But our approach differs from a classical system in that these steps are done through a recurrent neural network and a differentiable optimization layer. Our approach can be understood as a sparsified variant of DROID-SLAM in VO mode; instead of estimating dense optical flow, we estimate flow for a sparse set of patches.
(a) Feature and patch extraction
(b) The patch graph
Figure 1: Constructing the patch graph. (a) Residual networks extract 1) a context feature map at  $\frac{1}{4}$  image-resolution and 2) a 2-level pyramid of matching features at  $\frac{1}{4}$  and  $\frac{1}{16}$  resolution. Many  $p \times p$  patches are cropped from this feature map at random pixel coordinates using bilinear sampling. (b) Multiple patches are extracted from each frame (e.g. blue) and are connected to nearby frames (green and purple). In the resulting patch graph, edges connect patches with frames.
Feature Extraction: To estimate the optical flow of the patches, we need to extract per-pixel features and use them for computing visual similarities. We use a pair of residual networks. One network extracts matching features while the other extracts context features. The first layer of each network is a  $7 \times 7$  convolution with stride 2 followed by two residual blocks at  $1/2$  resolution (dimension 32) and 2 residual blocks at  $1/4$  resolution (dimension 64), such that the final feature map is one-quarter the input resolution. The architectures of the matching and context networks are identical with the exception that the matching network uses instance normalization and the context network uses no normalization. We construct a two-level feature pyramid by applying average pooling to the matching features with a  $4 \times 4$  filter with stride 4.
Patch Extraction: We create patches by randomly sampling 2D locations, which we find to work well despite its simplicity. We associate each patch with a per-pixel feature map of the same size, cropped with bilinear interpolation from both the full matching and full context feature maps. We provide an example in Fig. 1a. The patch features are used to compute visual similarities, but unlike DROID-SLAM, we compute them on the fly instead of precomputing them as correlation volumes.
3.1 Update Operator
The core of our approach is the update operator, which is a recurrent network that iteratively refines the depth of each patch and the camera pose of each video frame. At each iteration, it uses the patch features to propose revisions to the optical flow of the patches, and updates depth and camera poses through a differentiable bundle adjustment layer. Because the patches are sparsely located and spatially separated, the recurrent network includes special designs that facilitate exchange of information between patches. We provide a schematic overview of the operator in Fig. 2. The operator operates on the patch graph and maintains a hidden state for each edge. Its first three components (Correlation, Temporal Convolutions, Softmax-Aggregation) produce and aggregate information across edges, the Transition block produces an update to each hidden state, and the final two components (Factor-Head + Bundle-Adjustment) produce an update to the camera poses and patch depths. When a new edge is added, its hidden state is initialized with zeros.
Correlation: For each edge  $(k,j)$  in the patch graph, we compute correlation features (visual similarities) to assess the visual alignment given by current estimates of depth and poses and to propose revisions. We first use Eqn. 2 to reproject patch  $k$  from frame  $i$  into frame  $j$ :  $\mathbf{P}_{kj}^{\prime} = \omega_{ij}(\mathbf{T},\mathbf{P}_k)$ . Given patch features  $\mathbf{g}\in \mathbb{R}^{p\times p\times D}$  and frame features  $\mathbf{f}\in \mathbb{R}^{H\times W\times D}$ , for each pixel  $(u,v)$  in patch  $k$ , we compute its correlation  $\mathbf{C}\in \mathbb{R}^{p\times p\times 7\times 7}$  with a grid of pixels centered at its reprojection in frame  $j$ , using the inner product:
$$
\mathbf {C} _ {u v \alpha \beta} = \left\langle \mathbf {g} _ {u v}, \mathbf {f} \left(\mathbf {P} _ {k j} ^ {\prime} (u, v) + \Delta_ {\alpha \beta}\right) \right\rangle \tag {4}
$$
where we take  $\Delta$  to be a  $7\times 7$  integer grid centered at 0 indexed by  $\alpha$  and  $\beta$ , and  $\mathbf{f}(\cdot)$  denotes bilinear sampling. We compute these correlation features for both levels in the pyramid and concatenate the results. This operation is implemented as an optimized CUDA layer which leverages the regular grid structure of the interpolation step. This implementation is identical to the alternative correlation
Figure 2: Schematic of the update operator. Correlation features are extracted from edges in the patch graph and injected into the hidden state alongside context features. We apply 1D convolution, message passing and a transition block. The factor head produces trajectory revisions which are used by the bundle adjustment layer to update the camera poses and the depth of patches. Each "+" operation is a residual connection followed by layer normalization.
implementation used by RAFT (36) and is equivalent to indexing correlation volumes due to the linearity of the inner product and interpolation.
1D Temporal Convolution: DPVO operates on real-time video, where neighboring frames tend to be highly correlated. To leverage this correlation, we apply a 1D-convolution in the temporal dimension to each patch trajectory. Since trajectories vary in length and keyframes are actively added and removed, it is not straightforward to implement convolution as a batched operation. Instead, for each edge  $(k,j)$  we index the features of its temporally-adjacent neighbors at  $(k,j - 1)$  and  $(k,j + 1)$ , concatenate, then apply a linear projection. The temporal convolution allows the network to propagate information along each patch trajectory and model appearance changes of the patch through time.
SoftMax Aggregation: Even though the patches are sparsely located, their motion and appearance can be still be correlated as they may belong to the same object. We leverage this correlation through global message passing layers that propagate information between edges in the patch graph. This operation has appeared before in the context of graph neural networks (30). Given edge  $e$  and its neighbors  $N(e)$  we define the channel-wise aggregation function
$$
\psi \left(\left[ \sum_ {x \in N (e)} \sigma (x) \cdot \phi (x) \right] / \sum_ {x \in N (e)} \sigma (x)\right) \tag {5}
$$
where  $\psi$  and  $\phi$  are linear layers and  $\sigma$  is a linear layer followed by a sigmoid activation. We perform two instantiations of soft aggregation: (1) patch aggregation where edges are neighbors if they connect to the same patch (2) frame aggregation where edges are neighbors if they connect to both the same destination frame and different patches from the same source frame. Note that this graph-based aggregation is unique to our sparse patch-based representation, because for dense flow the same effect is easily achieved with convolution in prior approaches (37).
Transition Block: Following the softmax-aggregation, we include a transition block (shown in Fig. 2) to produce an update to the hidden state for each edge in the patch graph. Our transition block is simply two gated residual units with Layer Normalization and ReLU non-linearities. We find that layer normalization helps avoid exploding values for recurrent network modules.
Factor Head: The final learned layer in our update operator is the factor head, which proposes 2D revisions to the current patch trajectories and associated confidence weights. This layer consists of 2 MLPs with one hidden unit each. For each edge  $(k,j)$  in the patch graph, the first MLP predicts the trajectory update  $\delta_{kj} \in \mathbb{R}^2$ : a 2D flow vector indicating how the reprojection of the patch center should be updated in 2D; the second MLP predicts the confidence weight  $\Sigma_{kj} \in \mathbb{R}^2$  which is bounded to  $(0,1)$  using a sigmoid activation.
Differentiable Bundle Adjustment: Given the proposed 2D revisions to trajectories, we need to solve for the updates to depth and camera poses to realize the proposed revisions. This is achieved through a differentiable bundle adjustment (BA) layer, which operates globally on the patch graph and outputs updates to depth and camera poses. The predicted factors  $(\delta, \Sigma)$  are used to define an
Figure 3: A subset of the patch trajectories predicted by our method. Patches extracted from the green keyframe are tracked through subsequent frames. When a new keyframe is added (blue), additional patches are extracted and tracked. Our method produces confidence values which weight their respective contribution to the bundle adjustment.
optimization objective:
$$
\sum_ {(k, j) \in \mathcal {E}} \left\| \hat {\omega} _ {i j} (\mathbf {T}, \mathbf {P} _ {k}) - \left[ \hat {\mathbf {P}} _ {k j} ^ {\prime} + \delta_ {k j} \right] \right\| _ {\Sigma_ {k j}} ^ {2} \tag {6}
$$
where  $\|\cdot\|_{\Sigma}$  is the Mahalanobis distance and  $\hat{\mathbf{P}}_{kj}'$  denotes the center of  $\mathbf{P}_{kj}'$ . We apply two Gauss-Newton iterations to the linearized objective, optimizing the camera poses as well as the inverse depth component of the patch while keeping the pixel coordinates constant. This optimization seeks to refine the camera poses and depth such that the induced trajectory updates agree with the predicted trajectory updates. Like DROID-SLAM (37), we use the Schur complement trick for efficient decomposition and backpropagate gradients through the Gauss-Newton iterations.
3.2 Training and Supervision
DPVO is implemented using PyTorch. We perform supervised training of our network on the TartanAir dataset. On each training sequence, we precompute optical flow magnitude between all pairs of frames using ground truth poses and depth. During training, we sample trajectories where frame-to-frame optical flow magnitude is between 16px and 72px. This ensures that training instances are generally difficult but not impossible.
We apply supervision to poses and induced optical flow (i.e. trajectory updates), supervising each intermediate output of the update operator and detach the poses and patches from the gradient tape prior to each update.
Pose Supervision: We scale the predicted trajectory to match the groundtruth using the Umeyama alignment algorithm (40). Then for every pair of poses  $(i,j)$ , we supervise on the error
$$
\sum_ {(i, j) i \neq j} \| \operatorname {L o g} _ {S E (3)} \left[ \left(\mathbf {G} _ {i} ^ {- 1} \mathbf {G} _ {j}\right) ^ {- 1} \left(\mathbf {T} _ {i} ^ {- 1} \mathbf {T} _ {j}\right) \right] \| \tag {7}
$$
where  $\mathbf{G}$  is the ground truth and  $\mathbf{T}$  are the predicted poses.
Flow Supervision: We additionally supervise on the distance between the induced optical flow and the ground truth optical flow between each patch and the frames within two timestamps of its source frame. Each patch induces a  $p \times p$  flow field. We take the minimum of all  $p \times p$  errors.
The final loss is the weighted combination
$$
\mathcal {L} = 1 0 \mathcal {L} _ {\text {p o s e}} + 0. 1 \mathcal {L} _ {\text {f l o w}}. \tag {8}
$$
Training Details: We train for a total of 240k iterations on a single RTX-3090 GPU with a batch size of 1. Training takes 3.5 days. We use the AdamW optimizer and start with an initial learning rate of 8e-5 which is decayed linearly during training. We apply standard augmentation techniques such as resizing and color jitter.
We train on sequences of length 15. The first 8 frames are used for initialization while the next 7 frames are added one at a time. We unroll 18 iterations of the update operator during training. For the first 1000 training steps, we fix poses with the ground truth and only ask the network to estimate the depth of the patches. Afterwards, the network is required to estimate both poses and depth.
(a) Sparse reconstruction example
(b) Ours vs DROID-SLAM
Figure 4: Results on the TartanAir (44) validation split. Our method gets an AUC of 0.80 compared to 0.71 for DROID-SLAM while running 4x faster.
Figure 5: Overview of the VO System. The visualization and frame loading are performed in separate threads.
3.3 VO System
We implement the logic of the full VO system primarily in Python with bottleneck operations such as bundle adjustment and visualization implemented in C++ and CUDA.