SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Cameron Smith Yilun Du Ayush Tewari Vincent Sitzmann
{camsmith, yilundu, ayusht, sitzmann}@mit.edu
MITCSAIL
cameronosmith.github.io/flowcam
Reconstruction of 3D neural fields from posed images has emerged as a promising method for self-supervised representation learning. The key challenge preventing the deployment of these 3D scene learners on large-scale video data is their dependence on precise camera poses from structure-from-motion, which is prohibitively expensive to run at scale. We propose a method that jointly reconstructs camera poses and 3D neural scene representations online and in a single forward pass. We estimate poses by first lifting frame-to-frame optical flow to 3D scene flow via differentiable rendering, preserving locality and shift-equivariance of the image processing backbone.  $SE(3)$  camera pose estimation is then performed via a weighted least-squares fit to the scene flow field. This formulation enables us to jointly supervise pose estimation and a generalizable neural scene representation via re-rendering the input video, and thus, train end-to-end and fully self-supervised on real-world video datasets. We demonstrate that our method performs robustly on diverse, real-world video, notably on sequences traditionally challenging to optimization-based pose estimation techniques.
Recent learning-based 3D reconstruction techniques show promise in estimating the underlying 3D appearance and geometry from just a few posed image observations, in a single feed-forward pass [1-19]. These techniques offer an exciting new perspective on computer vision: Instead of making predictions only on pixels, computer vision models might operate directly on the corresponding 3D scenes. This would be a significant step towards a generalist computer vision model, applicable to any task involving interaction with the physical world.
A core challenge to the generality of these methods is that they cannot be trained from just video, but instead require knowledge of per-frame camera poses. Existing methods thus rely on curated datasets that obtain camera poses via Structure-from-Motion, but this is prohibitively expensive to compute at scale. Lifting this dataset prerequisite would unlock orders of magnitude more training data, making large-scale 3D representation learning tractable. Meanwhile, odometry and SLAM methods offer online camera pose estimation, but may fail to track sequences with dominant camera rotation or with sparse visual features, and do not reconstruct dense 3D scene representations. While recent efforts leveraging differentiable rendering have demonstrated impressive results at joint reconstruction of camera poses and 3D scenes, they still require minutes or hours per scene. Further, these optimization-based methods cannot leverage learned priors for camera pose estimation, leaving significant progress in computer vision of the last decade untapped. While prior work has demonstrated self-supervised learning of joint depth and camera pose prediction [20, 21], these models are constrained to tight video distributions, such as self-driving video, and do not infer a full 3D representation, only depth.
We present a method for jointly training feed-forward generalizable 3D neural scene representation and camera trajectory estimation, self-supervised only by re-rendering losses on video frames, completely without ground-truth camera poses or depth maps. We propose to leverage single-image neural scene representations and differentiable rendering to lift frame-to-frame optical flow to 3D scene flow. We then estimate SE(3) camera poses via a robust, weighted least-squares solver on the scene flow field. Regressed poses are used to re-construct the underlying 3D scene from video frames in a feed-forward pass, where weights are shared with the neural scene representation leveraged in camera pose estimation.
We validate the efficacy of our model for feed-forward novel view synthesis and online camera pose estimation on the real-world RealEstate10K and KITTI datasets, as well as the challenging CO3D dataset. We further demonstrate results on in-the-wild scenes in Ego4D and Walking Tours streamed from YouTube. We demonstrate generalization of camera pose estimation to out-of-distribution scenes and achieve robust performance on trajectories on which a state-of-the-art SLAM approach, ORB-SLAM3 [22], struggles.
To summarize, the contributions of our work include:
- We present a new formulation of camera pose estimation as a weighted least-squares fit of an SE(3) pose to a 3D scene flow field obtained via differentiable rendering.
- We combine our camera pose estimator with a multi-frame 3D reconstruction model, unlocking end-to-end, self-supervised training of camera pose estimation and 3D reconstruction.
- We demonstrate that our method performs robustly across diverse real-world video datasets, including indoor, self-driving, and object-centric scenes.
Generalizable Neural Scene Representations. Recent progress in neural fields [23-25] and differentiable rendering [10, 26-31] have enabled novel approaches to 3D reconstruction from few or single images [1-19], but require camera poses both at training and test time. An exception is recently proposed RUST [32], which can be trained for novel view synthesis without access to camera poses, but does not reconstruct 3D scenes explicitly and does not yield explicit control over camera poses. We propose a method that is similarly trained self-supervised on real video, but yields explicit camera poses and 3D scenes in the form of radiance fields. We outperform RUST on novel view synthesis and demonstrate strong out-of-distribution generalization by virtue of 3D structure.
SLAM and Structure-from-Motion (SfM). SfM methods [33-35], and in particular, COLMAP [34], are considered the de-facto standard approach to obtaining accurate geometry and camera poses from video. Recent progress on differentiable rendering has enabled joint estimation of radiance fields and camera poses via gradient descent [36-40], enabling subsequent high-quality novel view synthesis. Both approaches require offline per-scene optimization. In contrast, SLAM methods usually run online [22, 41, 42], but are notoriously unreliable on rotation-heavy trajectories or scenes with sparse visual features. Prior work proposes differentiable SLAM to learn priors over camera poses and geometry [43, 44], but requires ground-truth camera poses for training. Recent work has also explored how differentiable rendering may be directly combined with SLAM [45-48], usually using a conventional SLAM algorithm as a backbone and focusing on the single-scene overfitting case. We propose a fully self-supervised method to train generalizable neural scene representations without camera poses, outperforming prior work on generalizable novel view synthesis without camera poses. We do not claim state-of-the-art camera pose estimation, but provide an analysis of camera pose quality nevertheless, demonstrating robust performance on sequences that are challenging to state-of-the-art SLAM algorithms, ORB-SLAM3 [22] and Droid-SLAM [43].
Neural Depth and Camera Pose Estimation. Prior work has demonstrated joint self-supervised learning of camera pose and monocular depth [20, 21, 49-51] or multi-plane images [52]. These approaches leverage a neural network to directly regress camera poses with the primary goal of training high-quality monocular depth predictors. They are empirically limited to sequences with simple camera trajectories, such as self-driving datasets, and do not enable dense, large-baseline novel view synthesis. We ablate our flow-based camera pose estimation with a similar neural network-based approach. Most closely related to our work are approaches that infer per-timestep 3D voxel grids and train a CNN to regress frame-to-frame poses [53, 54]. We benchmark with the most recent approach in this line of work, Video Autoencoder [53]. Lastly, we strongly encourage the reader to peruse impressive concurrent work DBARF [55], which also regresses camera poses alongside a generalizable NeRF. Key differences are that we leverage a pose solver based on 3D-lifted optical
Figure 1: Method Overview. Given a set of video frames, our method first computes frame-to-frame camera poses (left) and then re-renders the input video (right). To estimate pose between two frames, we compute off-the-shelf optical flow to establish 2D correspondences. Using single-view pixelNeRF [1], we obtain a surface point cloud as the expected 3D ray termination point for each pixel,  $\mathbf{X}$ ,  $\mathbf{Y}$  respectively. Because  $\mathbf{X}$  and  $\mathbf{Y}$  are pixel-aligned, optical flow allows us to compute 3D scene flow as the difference of corresponding 3D points. We then find the camera pose  $\mathbf{P} \in \mathrm{SE}(3)$  that best explains the 3D flow field by solving a weighted least-squares problem with flow confidence weights  $\mathbf{W}$ . Using all frame-to-frame poses, we re-render all frames. We enforce an RGB loss and a flow loss between projected pose-induced 3D scene flow and 2D optical flow. Our method is trained end-to-end, assuming only an off-the-shelf optical flow estimator.
flow for real-time odometry versus predicting iterative updates to pose and depth via a neural network. Further, we extensively demonstrate our method's performance on rotation-dominant video sequences, in contrast to a focus on forward-facing scenes. Lastly, we solely rely on the generalizable scene representation in contrast to leveraging a monocular depth model for pose estimation.
3 Learning 3D Scene Representations from Unposed Videos
Our model learns to map a monocular video with frames  $\{\mathbf{I}_t\}_{t=1}^N$  as well as off-the-shelf optical flow  $\{\mathbf{V}_t\}_{t=1}^{N-1}$  to per-frame camera poses  $\{\mathbf{P}_t\}_{t=1}^N$  and a 3D scene representation  $\Phi$  in a single feed-forward pass. We leverage known intrinsic parameters when available, but may predict them if not. We will first introduce the generalizable 3D scene representation  $\Phi$ . We then discuss how we leverage  $\Phi$  for feed-forward camera pose estimation, where we lift optical flow into 3D scene flow and solve for pose via a weighted least-squares SE(3) solver. Finally, we discuss how we obtain supervision for both the 3D scene representation and pose estimation by re-rendering RGB and optical flow for all frames. An overview of our method is presented in Fig. 1.
Notation. It will be convenient to treat images sometimes as discrete tensors, such as  $\mathbf{I}_t\in \mathbb{R}^{H\times W\times 3}$ , and sometimes as functions  $I:\mathbb{R}^2\to \mathbb{R}^3$  over 2D pixel coordinates  $\mathbf{p}\in \mathbb{R}^2$ . We will denote functions in italic  $I$ , while we denote the corresponding tensors sampled on the pixel grid in bold as  $\mathbf{I}$ .
3.1 Defining Our Image-Conditioned 3D Scene Representation
First, we introduce the generalizable 3D scene representation we aim to train. Our discussion assumes known camera poses; in the subsequent section we will describe how we can use our scene representation to estimate them instead. We parameterize our 3D scene as a Neural Radiance Field (NeRF) [28], such that  $\Phi$  is a function that maps a 3D coordinate  $\mathbf{x}$  to a color  $\mathbf{c}$  and density  $\sigma$  as  $\Phi (\mathbf{x}) = (\sigma ,\mathbf{c})$ . To render the color for a ray  $\mathbf{r}$ , points  $(\mathbf{x}_1,\mathbf{x}_2,\dots,\mathbf{x}_n)$  are sampled along  $\mathbf{r}$  between predefined near and far planes  $[t_1,t_f]$ , fed into  $\Phi$  to produce corresponding color and density tuples  $[( \sigma_{1},\mathbf{c}_{1}),(\sigma_{2},\mathbf{c}_{2}),\dots,(\sigma_{n},\mathbf{c}_{n})]$ , and alpha-composed to produce a final color value  $C(\mathbf{r})$ :
$$
C (\mathbf {r}) = \sum_ {i = 1} ^ {N} T _ {i} \left(1 - \exp \left(- \sigma_ {i} \delta_ {i}\right)\right) \mathbf {c} _ {i}, \text {w h e r e} T _ {i} = \exp \left(- \sum_ {j = 1} ^ {i - 1} \sigma_ {j} \delta_ {j}\right), \tag {1}
$$
Figure 2: Video Reconstruction Results. Our model reconstructs video frames from sparse context frames with higher fidelity than all baselines. While VidAE's renderings often appear with convincing texture, they are often not aligned with the ground truth. RUST's renderings are well aligned but are blurry due to their coarse set latent representation.
Model	CO3D-Hydrants	CO3D-10	RealEstate	KITTI
LPIPS ↓	PSNR ↑	LPIPS ↓	PSNR ↑	LPIPS ↓	PSNR ↑	LPIPS ↓	PSNR ↑
Vid-AE [53]	0.3427	18.56	0.3889	18.50	0.4173	18.03	0.3272	17.65
RUST [32]	0.6126	17.50	0.6145	17.71	0.5692	18.07	0.5584	15.44
Ours	0.2250	23.94	0.2687	24.17	0.2224	24.25	0.1928	23.39
Table 1: Quantitative Comparison on View Synthesis. On the task of view synthesis, our method outperforms other unposed methods by wide margins.
where  $\delta_{i} = t_{i + 1} - t_{i}$  is the distance between adjacent samples. By compositing sample locations instead of colors, we can compute an expected ray-surface intersection point  $S(\mathbf{r})\in \mathbb{R}^3$  ..
$$
S (\mathbf {r}) = \sum_ {i = 1} ^ {N} T _ {i} \left(1 - \exp \left(- \sigma_ {i} \delta_ {i}\right)\right) \mathbf {x} _ {i}. \tag {2}
$$
We require a generalizable NeRF that is not optimized for each scene separately, but instead predicted in a feed-forward pass by an encoder that takes a set of  $M$  context images and corresponding camera poses  $\{(\mathbf{I}_i,\mathbf{P}_i)\}_{i}^{M}$  as input. We denote such a generalizable radiance field reconstructed from images  $\mathbf{I}_i$  as  $\Phi (\mathbf{x}\mid \{(\mathbf{I}_i,\mathbf{P}_i)\}_{i}^{M})$ . Many such models have been proposed [1-10]. We base our model on pixelNeRF [1], which we briefly discuss in the following - please find further details in the supplement. pixelNeRF first extracts per-image features  $\mathbf{F}_i$  from each input image  $\mathbf{I}_i$ . A given 3D coordinate  $\mathbf{x}$  is first projected onto the image plane of each context image  $\mathbf{I}_i$  via the known camera pose and intrinsic parameters to yield pixel coordinates  $\mathbf{p}_i$ . We then retrieve the features  $F_{i}(\mathbf{p}_{i})$  at that pixel coordinate. Color and density  $(\sigma ,\mathbf{c})$  at  $\mathbf{x}$  are then predicted by a neural network that takes as input the features  $\{F_i(\mathbf{p}_i)\}_{i}^M$  and the coordinates of  $\mathbf{x}$  in the coordinate frame of each camera,  $\{\mathbf{P}_i^{-1}\mathbf{x}\}_{i}^{M}$ . Importantly, we can condition pixelNeRF on varying numbers of context images, i.e., we may run pixelNeRF with only a single context image as  $\Phi (\mathbf{x}\mid (\mathbf{I},\mathbf{P}))$ , or with a set of  $M > 1$  context images  $\Phi (\mathbf{x}\mid \{(\mathbf{I}_i,\mathbf{P}_i)\}_{i}^{M})$ .
3.2 Lifting Optical Flow to Scene Flow with Neural Scene Representations
Equipped with our generalizable 3D representation  $\Phi$ , we now describe how we utilize it to lift optical flow into confidence-weighted 3D scene flow. Later, our pose solver will fit a camera pose to the
Figure 3: Qualitative Pose Estimation Comparison. On short sequences, we compare our pose estimation to Video Autoencoder [53], and on long sequences, we compare our method's sliding-window estimations against the per-scene optimization BARF [38]. The trajectory for the bicycle sequence was obtained using a model trained on hydrant sequences: despite never having seen a bicycle before, our model predicts accurate poses.
(a)	(b)
Hydrant	10-Cat.	RE10K	KITTI	Top	Bot.	% Tracked
VidAE [53]	0.100	0.101	0.090	0.084	ORB3 [22]	0.021	0.034	49
Ours	0.041	0.042	0.020	0.010	DROID [43]	0.028	0.049	100
Ours	0.015	0.025	100
Table 2: Quantitative Pose Estimation Comparison. In (a) we compare against VideoAutoencoder [53] on short-sequence odometry estimation (20 frames), reporting the ATE. In (b) we compare against ORB-SLAM3 [22] and DROID-SLAM [43] on long sequences ( $\sim$ 200 frames) from the CO3D 10-Category dataset. We separately report scores on the top and bottom  $50\%$  of sequences ("Top" and "Bot.") in terms of quality of ground-truth poses as indicated by the dataset. We report ATE and percent of sequences tracked ("Tracked"). ORB-SLAM3 fails to track over half of these challenging sequences.
estimated scene flow. Given two sequential frames  $\mathbf{I}_{t-1}$ ,  $\mathbf{I}_t$  we first use an off-the-shelf method [56] to estimate backwards optical flow  $V_t: \mathbb{R}^2 \to \mathbb{R}^2$ . The optical flow  $V_t$  maps a 2D pixel coordinate  $\mathbf{p}$  to a 2D flow vector, such that we can determine the corresponding pixel coordinate in  $\mathbf{I}_{t-1}$  as  $\mathbf{p}' = \mathbf{p} + V_t(\mathbf{p})$ .
We will now lift pixel coordinates  $\mathbf{p}_t$  and  $\mathbf{p}_{t-1}$  to an estimate of the 3D points that they observe in the coordinate frame of their respective cameras. To achieve this, we cast rays from the camera centers through the corresponding pixel coordinates  $\mathbf{p}_t$  and  $\mathbf{p}_{t-1}$  using the intrinsics matrix  $\mathbf{K}$ . Specifically, we compute  $\mathbf{r}_t = \mathbf{K}^{-1}\tilde{\mathbf{p}}_t$  and  $\mathbf{r}_{t-1} = \mathbf{K}^{-1}\tilde{\mathbf{p}}_{t-1}$ , where  $\tilde{\mathbf{p}}$  represents the homogeneous coordinate  $\left( \begin{array}{c} \mathbf{p} \\ 1 \end{array} \right)$ . Next, we sample points along the rays  $\mathbf{r}_t$  and  $\mathbf{r}_{t-1}$  and query our pixelNeRF model in the single-view setting. This involves invoking  $\Phi(\cdot|(\mathbf{I}_t, \mathbb{I}_{4\times 4}))$  and  $\Phi(\cdot|(\mathbf{I}_{t-1}, \mathbb{I}_{4\times 4}))$ , i.e., pixelNeRF is run with only the respective frame as the context view and the identity matrix  $\mathbb{I}$  as the camera pose. Applying the ray-intersection integral defined in Eq. 2 to the pixelNeRF estimates, we obtain a pair of 3D points  $(\mathbf{x}_t, \mathbf{x}_{t-1})$  corresponding to the estimated surfaces observed by pixels  $\mathbf{p}_t$  and  $\mathbf{p}_{t-1}$ , respectively. We repeat this estimation for all optical flow correspondences, resulting in two sets of surface point clouds,  $\mathbf{X}, \mathbf{X}' \in \mathbb{R}^{H\times W\times 3}$ . Equivalently, we may view this as defining the 3D scene flow as  $\mathbf{X}' - \mathbf{X}$ .
Flow confidence weights. We further utilize a confidence weight for each flow correspondence. To accomplish this, we employ a neural network  $\Psi$ , which takes image features  $F_{t}(\mathbf{p})$ ,  $F_{t-1}(\mathbf{p}')$  as input for every pixel correspondence pair  $(\mathbf{p}, \mathbf{p}')$ . The network maps these features to a weight  $\mathbf{w}$  denoted as  $\Psi(F_{t}(\mathbf{p}), F_{t-1}(\mathbf{p})) = \mathbf{w} \in [0,1]$ .  $\Psi$  can importantly overcome several failure modes which lead to faulty pose estimation, including incorrect optical flow, such as in areas of occlusions, dynamic objects, such as pedestrians, or challenging geometry estimates, such as sky regions. We show in Fig. 9 that  $\Psi$  indeed learns such content-based rules.
Predicting Intrinsic Camera Parameters K. Camera intrinsics are often approximately known, either published by the manufacturer, saved in video metadata, or calibrated once. Nevertheless, for purposes of large-scale training, we leverage a simple scheme to predict the camera field-of-view for a video sequence. We feed the feature map of the first frame  $\mathbf{F}_0$  into a convolutional encoder that directly regresses the field of view. We assume that the optical center is at the sensor center. We find that this approach enables us to train on real-world video from YouTube, though more sophisticated schemes [57] will no doubt improve performance further.
Figure 4: Robustness. Our method works even on sequences challenging to ORB-SLAM3, which fails on  $49\%$  of CO3D. We show one successful and one failed sequence.
Figure 5: Learned geometry. Our model's expected ray termination illustrates the unsupervised geometry learned by our model on the challenging CO3D dataset.
3.3 Camera Pose Estimation as Explaining the Scene Flow Field
We will now estimate the camera pose between frame  $\mathbf{I}_t$  and  $\mathbf{I}_{t-1}$ . In the previous section, we lifted the input optical flow into scene flow, producing 3D correspondences  $\mathbf{X}, \mathbf{X}'$ , or, equivalently, 3D scene flow. We cast camera pose estimation as the problem of finding the rigid-body motion that best explains the observed scene flow field, or the transformation mapping points in  $\mathbf{X}$  to  $\mathbf{X}'$ , while considering confidence weights  $\mathbf{W}$ . Note that below, we will refer to the matrices  $\mathbf{X}, \mathbf{X}'$ , and  $\mathbf{W}$  as column vectors, with their spatial dimensions flattened.
We use a weighted Procrustes formulation to solve for the rigid transformation that best aligns the set of points  $\mathbf{X}$  and  $\mathbf{X}'$ . The standard orthogonal Procrustes algorithm solves for the SE(3) pose such that it minimizes the least squares error:
$$
\underset {\mathbf {P} \in \operatorname {S E} (3)} {\arg \min } \| \tilde {\mathbf {X}} - \mathbf {P} \tilde {\mathbf {X}} ^ {\prime} \| _ {2} ^ {2}, \tag {3}
$$
with  $\mathbf{P} = \left( \begin{array}{cc}\mathbf{R} & \mathbf{t}\\ 0 & 1 \end{array} \right)$  as a rigid-body pose with rotation  $\mathbf{R}$  and translation  $\mathbf{t}$  and homogeneous  $\tilde{\mathbf{X}} = \left( \begin{array}{c}\mathbf{X}\\ 1 \end{array} \right)$ . In other words, the minimizer of this loss is the rigid-body transformation that best maps  $\mathbf{X}$  onto  $\mathbf{X}'$ , and, in a static scene, is therefore equivalent to the sought-after camera pose.
As noted by Choy et al. [58], this formulation equally weights all correspondences. As noted in the previous section, however, this would make our pose estimation algorithm susceptible to both incorrect correspondences as well as correspondences that should be down-weighted by nature of belonging to parts of the scene that are specular, dynamic, or have low confidence in their geometry estimate. Following [58], we thus minimize a weighted least-squares problem:
$$
\underset {\mathbf {P} \in \operatorname {S E} (3)} {\arg \min } \| \mathbf {W} ^ {1 / 2} \left(\tilde {\mathbf {X}} - \mathbf {P} \tilde {\mathbf {X}} ^ {\prime}\right) \| _ {2} ^ {2} \tag {4}
$$
with the diagonal weight matrix  $\mathbf{W} = \mathrm{diag}(\mathbf{w})$ . Conveniently, this least-squares problem admits a closed-form solution, efficiently calculated via Singular Value Decomposition, as derived in [58]:
$$
\mathbf {R} = \mathbf {U S V} ^ {T} \text {a n d} \mathbf {t} = \left(\mathbf {X} - \mathbf {R X} ^ {\prime}\right) \mathbf {W 1}, \text {w h e r e} \mathbf {U} \boldsymbol {\Sigma} \mathbf {V} ^ {T} = \operatorname {S V D} \left(\boldsymbol {\Sigma} _ {\mathbf {X} ^ {\prime} \mathbf {X}}\right), \tag {5}
$$
$$
\Sigma_ {\mathbf {X} ^ {\prime} \mathbf {X}} = \mathbf {X} \mathbf {K} \mathbf {W} \mathbf {K} \mathbf {X} ^ {\prime T}, \mathbf {K} = \mathbb {I} - \sqrt {\mathbf {w}} \sqrt {\mathbf {w}} ^ {T}, \text {a n d} \mathbf {S} = \operatorname {d i a g} (1, \dots , \det  (\mathbf {U}) \det  (\mathbf {V})). \tag {6}
$$
Composing frame-to-frame poses. Solving this weighted least-squares problem for each subsequent frame-to-frame pair yields camera transformations  $(\mathbf{P}_2', \mathbf{P}_3', \dots, \mathbf{P}_n')$ , aligning each  $\mathbf{I}_t$  to its predecessor  $\mathbf{I}_{t-1}$ . We compose frame-to-frame transformations to yield camera poses  $(\mathbf{P}_1, \mathbf{P}_2, \dots, \mathbf{P}_n)$  relative to the first frame, such that  $\mathbf{P}_1 = \mathbb{I}_{3 \times 3}$ , concluding our camera pose estimation module.
3.4 Supervision via Differentiable Video and Flow Re-Rendering
We have discussed our generalizable neural scene representation  $\Phi$  and our camera pose estimation module. We will now discuss how we derive supervision to train both modules end-to-end. We have two primary loss signals: First, a photometric loss  $\mathcal{L}_{\mathrm{RGB}}$  scores the visual fidelity of re-rendered video frames. Second, a pose-induced flow loss  $\mathcal{L}_{\mathrm{pose}}$  scores how similar the flow induced by the