SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 6: Wide-Baseline View Synthesis. Given an input video without poses, our model first infers camera poses and can then render wide-baseline novel views of the underlying 3D scene, where we use the first, middle, and final frame of the video as context views.
predicted camera transformations and surface estimations are to optical flow estimated by RAFT [59]. Our model is trained on short ( $\sim 15$  frames) video sequences.
Photometric Loss. Our photometric loss  $\mathcal{L}_{\mathrm{RGB}}$  comprises two terms: A multi-context loss and a single-context loss. The multi-context loss ensures that the full 3D scene is reconstructed accurately. Here, we re-render each frame of the input video sequence  $\mathbf{I}_t$ , using its estimated camera pose  $\mathbf{P}_t$  and multiple context images  $\{(\mathbf{I}_j, \mathbf{P}_j)\}_{j}^J$ . The single-context loss ensures that the single-context pixelNeRF used to estimate surface point clouds  $\mathbf{X}_t$  in the pose estimation module is accurate.
$$
\mathcal {L} _ {\mathrm {R G B}} = \frac {1}{N} \sum_ {i = t} ^ {N} \underbrace {\left\| \mathbf {I} _ {t} - C \left(\mathbf {P} _ {t} \mid \{\left(\mathbf {I} _ {j} , \mathbf {P} _ {j}\right) \} _ {j} ^ {J}\right) \right\| _ {2} ^ {2}} _ {\text {M u l t i - C o n t e x T L o s s}} + \underbrace {\left\| \mathbf {I} _ {t} - C \left(\mathbb {I} _ {4 \times 4} \mid \left(\mathbf {I} _ {t} , \mathbb {I} _ {4 \times 4}\right)\right) \right\| _ {2} ^ {2}} _ {\text {S i n g l e - C o n t e x T L o s s}}, \tag {7}
$$
where, in a slight abuse of notation, we have overloaded the rendering function  $C(\mathbf{P}|\{\left(\mathbf{I}_j,\mathbf{P}_j\right)\}_{j}^J)$  defined in Eq. 1 as rendering out the full image obtained by rendering a pixelNeRF with context images  $\{\left(\mathbf{I}_j,\mathbf{P}_j\right)\}_{j}^J$  from camera pose  $\mathbf{P}$ .  $N$  refers to the number of frames in the video sequence and  $J$  refers to the subset of context frames use for re-rendering the entire sequence. We first attempted picking the first frame only, however, found that this does not converge due to the uncertainty of the 3D scene given only the first frame: single-view pixelNeRF will generate blurry estimates for parts of the scene that have high uncertainty, such as occluded regions or previously unobserved background.
Pose-Induced Flow Loss. An additional, powerful source of supervision for both the estimated geometry and camera poses can be obtained by comparing the optical flow induced by the predicted surface point clouds and pose with the off-the-shelf optical flow. We define this pose-induced flow loss as
$$
\mathcal {L} _ {\text {p o s e}} = \frac {1}{N - 1} \sum_ {t = 1} ^ {N - 1} \left\| \mathbf {V} _ {t} - \left(\pi \left(\mathbf {P} _ {t} ^ {- 1} \cdot \mathbf {P} _ {t + 1} \cdot \mathbf {X} _ {t + 1}\right) - \mathbf {u v}\right) \right\| _ {2} ^ {2}, \tag {8}
$$
with projection operator  $\pi(\cdot)$  and grid of pixel coordinates  $\mathbf{uv} \in \mathbb{R}^2$ . Intuitively, this transforms the surface point cloud of frame  $t + 1$  into the coordinate frame of frame  $t$  and projects it onto that image plane. For every pixel coordinate  $\mathbf{p}$  at timestep  $t + 1$ , this yields a corresponding pixel coordinate  $\mathbf{p}'$  at timestep  $t$ , which we compare against the input optical flow.
3.5 Test-time Inference
After training our model on a large dataset of short video sequences, we may infer both camera poses and a radiance field of such a short sequence in a single forward pass, without test-time optimization.
Sliding Window Inference for Odometry on Longer Trajectories. Our method estimates poses for short ( $\sim 15$  frames) subsequences in a single feed-forward pass. To handle longer trajectories that exceed the capacity of a single batch, we divide a given video into non-overlapping subsequences. We estimate poses for each subsequence individually and compose them to create an aggregated trajectory estimate. This approach allows us to estimate trajectories for longer video sequences.
Test-Time Adaptation. Frame-to-frame camera pose estimation methods, both conventional and the proposed method, accumulate pose estimation error over the course of a sequence. SLAM and SfM methods usually have a mechanism to correct for drift by globally optimizing over all poses and closing loops in the pose graph [60]. We do not have such a mechanism, but propose fine-tuning our model on specific scenes for more accurate feed-forward pose and 3D estimation. For a given video sequence, we may fine-tune our pre-trained model using our standard photometric and flow losses on sub-sequences of the video. Note that this is not equivalent to per-scene optimization or direct
Figure 7: Fine-tuned Pose Estimation and View Synthesis on Large-Scale, Out-of-Distribution Scene. We evaluate our RealEstate10K-trained model on a significantly out-of-distribution scene from the Tanks and Temples dataset [61], first without any scene-adaptation and then with. Even with this significant distribution gap, our method's estimated trajectory captures the looping structure of the ground truth, albeit with accumulated drift. After a scene-adaptation fine-tuning stage (around 7 hours), our model estimates poses which align closely with the ground truth. We also plot the trajectory estimated by BARF [38], which fails to capture the correct pose distribution.
optimization of camera poses and a radiance field, as performed e.g. in BARF [38]: neither camera poses nor the radiance field are free variables. Instead, we fine-tune the weights of our convolutional inference backbone and MLP renderer for more accurate feed-forward prediction.
We benchmark our method on generalizable novel view synthesis on the RealEstate10k [62], CO3D [63], and KITTI [64] datasets. We provide further analysis on the Tanks & Temples dataset and in-the-wild scenes from Ego4D [65] and YouTube. Though we do not claim to perform state-of-the-art camera pose estimation, we nevertheless provide an analysis of the accuracy of our estimated camera poses. Please find more results, as well as precise hyperparameters, implementation, and dataset details, in the supplemental document and video. We utilize camera intrinsic parameters where available, predicting them only for the in-the-wild Ego4D and WalkingTours experiments.
**Pose Estimation.** We first evaluate our method on pose estimation against the closest self-supervised neural network baseline, Video Autoencoder (VidAE) [53]. We then analyze the robustness of our pose estimation with ORB-SLAM3 [66] and DROID-SLAM [43] as references. Finally, we benchmark with BARF [38], a single-scene unposed NeRF baseline. Tab. 2a compares accuracy of estimated poses of our method and VidAE on all four datasets. The performance gap is most pronounced on the challenging CO3D dataset, but even on simpler, forward-moving datasets, RealEstate10k and KITTI, our method significantly outperforms VidAE. Next, we analyse the robustness of our pose estimation module on CO3D, using SfM methods ORB-SLAM3 and DROID-SLAM as references. See Tab. 2b and Fig. 4 for quantitative and qualitative results. To account for inaccuracies in the provided CO3D poses we utilize as ground-truth, we additionally report separate results for the top and bottom  $50\%$  of sequences, ranked based on the pose confidence scores provided by the authors. Although we do not employ any secondary pose method as a proxy ground truth for the bottom half of sequences, this division serves as an approximate indication of the level of difficulty each sequence poses from a SfM perspective. On both subsets, our method outperforms both DROID-SLAM and ORB-SLAM3. Also note that ORB-SLAM3 fails to track poses for over half  $(50.7\%)$  of the sequences. On the sequences where ORB-SLAM3 succeeds, our method predicts poses significantly more accurately. Even on the sequences where ORB-SLAM3 fails, our performance does not degrade (.025 ATE). Lastly, we compare against the single-scene unposed NeRF baseline, BARF. Since BARF requires  $\sim$ one day to reconstruct a single sequence, we evaluate on two representative sequences: a forward-walking sequence on RealEstate10K, and an outside-in trajectory on CO3D. We plot recovered trajectories in Fig. 3. While BARF fails to recover the correct trajectory shape on the CO3D scene, our method produces a trajectory that more accurately reflects the ground-truth looping structure.
Novel View Synthesis. We compare against VidAE [53] and RUST [32] on the task of novel view synthesis. Tab. 1 and Fig. 6 report quantitative and qualitative results respectively. Our method outperforms both baselines significantly. Since VidAE fails to capture the pose distribution on the CO3D datasets, its novel view renderings generally do not align with the ground truth. On RealEstate10K and KITTI, their method successfully captures the pose distribution, but still struggles
Sample Frames
Novel View/ Sa  
Novel Depth Fr
Figure 8: View Synthesis on Ego4D and Walking Tours: We train on a collection of YouTube walking tour videos and Ego4D sequences with unknown camera parameters, and render novel views after a short fine-tuning stage.
Novel View/ Novel Depth
ole Novel View/  
es Novel Depth
Input RGB
Flow-Weight Input Masked RGB RGB
Figure 9: Flow Weights: Our flow-confidence weights allow our model to down-weight unreliable flow correspondences due to occlusions, specular highlights, or dynamic objects, and up-weight well-textured regions.
Input RGB
Flow-Weight Masked RGB
to render high-fidelity images. We similarly outperform RUST, which struggles to capture high-frequency content due to its coarse scene representation. Also note that RUST estimates latent camera poses rather than explicit ones. We further qualitatively evaluate our method on wide-baseline novel view synthesis, where no ground-truth is available - please see Fig. 6 for these results.
Test-Time Adaptation We evaluate the ability of our model to estimate camera poses on out-of-distribution scenes, augmented by test-time optimization as discussed in section 3.5. Fig.7 displays the camera poses estimated by our method on the Bulldozer sequence from the Tanks and Temples dataset[61], using our model trained on the RealEstate10k dataset. Even without any test-time adaptation, our method generalizes impressively, generating a trajectory that follows the outside-in trajectory as determined by COLMAP. We refer to this generalization mode as "zero-shot." Nevertheless, camera poses are not perfect and exhibit some drift. We thus perform test-time optimization, fine-tuning the weights of our feed-forward method on subsequences of the video. Though geometry is plausible after roughly 10 minutes of fine-tuning, we continued optimization for 7 hours for finer details. Our scene-adapted estimates are close to ground truth with little drift, and we find that novel view synthesis results with significant baseline are visually compelling with sound depth estimates. We further show the result of BARF on this sequence, which fails to recover a plausible trajectory.
Results on In-the-Wild Video. We present preliminary results on the Ego4D dataset [65], a recent ego-centric dataset captured with headset cameras, and a new Walking Tours dataset, in which we stream walking tour videos from YouTube. Both datasets comprise some scene motion, such as pedestrians, cars, or people. Images from Ego4D are further subject to radial camera distortion, which we do not model. Nevertheless, after fine-tuning on Walking Tours and Ego4D for 20 minutes and 1.5 hours, we can generate plausible novel views and depth maps, illustrated in Fig. 8. We find that our model is generally robust to dynamic scene content, such as pedestrians or humans, which obtain low flow-confidence flow scores. We further run COLMAP on a subset of the Walking Tours and Ego4D videos, yielding pseudo ground-truth poses for 3 videos from each dataset. The Walking Tour subset is selected randomly, and the Ego4D subset is chosen as the first three relatively-static videos we found (to ensure COLMAP convergence). We divide each video into subsequences of 20 frames at 3fps, corresponding to roughly 7 meters of forward translation per clip. Here, we achieve a competitive ATE of 0.013 and 0.026 on Walking Tours and Ego4D, respectively. We also compute PSNR values on subsequences of 10 frames using two context views, and obtain 18.87dB and 21.59db on Walking Tours and Ego4D, respectively, indicating generally plausible novel view synthesis.
↑ PSNR	↓ LPIPS
MLP-Pose	18.50	0.54
No Flow Weights	17.87	0.51
Full	21.02	0.38
Table 3: Ablation study.
Ablations. In Tab. 3, we ablate key contributions related to our pose formulation, evaluated on the CO3D Hydrant dataset. We first ablate our proposed flow-based pose formulation in favor of concatenating adjacent frames and predicting a pose directly via a CNN, as is common in the Monodepth [67] line of work. We further ablate our weighted flow formulation in favor of a non-weighted
Procrustes estimation. Both methods perform significantly worse than our full flow-based pose formulation, and qualitatively often lead to degenerate geometry estimates. Please see the supplemental material for additional ablations and ablation details.
Limitations. While we believe our method makes significant strides, it still has several limitations. As an odometry method, it accumulates drift and has no loop closure mechanism. Our model further does not currently incorporate scene dynamics, but recent advancements in dynamic NeRF papers [68-70] present promising perspectives for future research.
Conclusion. We have introduced FlowCam, a model capable of regressing camera poses and reconstructing a 3D scene in a single forward pass from a short video. Our key contribution is to factorize camera pose estimation as first lifting optical flow to pixel-aligned scene flow via differentiable rendering, and then solving for camera pose via a robust least-squares solver. We demonstrate the efficacy of our approach on a variety of challenging real-world datasets, as well as in-the-wild videos. We believe that they represent a significant step towards enabling scene representation learning on uncurated, real-world video.
Acknowledgements. This work was supported by the National Science Foundation under Grant No. 2211259, by the Singapore DSTA under DST00OECI20300823 (New Representations for Vision), and by the Amazon Science Hub. The Toyota Research Institute also partially supported this work. This article solely reflects the opinions and conclusions of its authors and no other entity.
[1] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelNeRF: Neural radiance fields from one or few images. In Proc. CVPR, 2021.
[2] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In Proc. CVPR, 2020.
[3] Yilun Du, Cameron Smith, Ayush Tewari, and Vincent Sitzmann. Learning to render novel views from wide-baseline stereo pairs. In Proc. CVPR, 2023.
[4] Alex Trevithick and Bo Yang. Grf: Learning a general radiance field for 3d representation and rendering. In Proc. ICCV, pages 15182-15192, 2021.
[5] Mohammed Suhail, Carlos Esteves, Leonid Sigal, and Ameesh Makadia. Generalizable patch-based neural rendering. In European Conference on Computer Vision. Springer, 2022.
[6] Julian Chibane, Aayush Bansal, Verica Lazova, and Gerard Pons-Moll. Stereo radiance fields (srf): Learning view synthesis for sparse views of novel scenes. In Proc. CVPR, pages 7911-7920, 2021.
[7] Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan T. Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi-view image-based rendering. In CVPR, 2021.
[8] Eric R Chan, Koki Nagano, Matthew A Chan, Alexander W Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein. Generative novel view synthesis with 3d-aware diffusion models. arXiv preprint arXiv:2304.02602, 2023.
[9] Zhizhuo Zhou and Shubham Tulsiani. Sparsefusion: Distilling view-conditioned diffusion for 3d reconstruction. Proc. CVPR, 2022.
[10] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3d-structure-aware neural scene representations. In Proc. NeurIPS, volume 32, 2019.
[11] Philipp Henzler, Jeremy Reizenstein, Patrick Labatut, Roman Shapovalov, Tobias Ritschel, Andrea Vedaldi, and David Novotny. Unsupervised learning of 3d object categories from videos in the wild. In Proc. CVPR, pages 4700-4709, 2021.
[12] Prafull Sharma, Ayush Tewari, Yilun Du, Sergey Zakharov, Rares Andrei Ambrus, Adrien Gaidon, William T Freeman, Fredo Durand, Joshua B Tenenbaum, and Vincent Sitzmann. Neural groundplans: Persistent neural scene representations from a single image. In Proc. ICLR.
[13] Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering. In Proc. NeurIPS, 2021.
[14] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In Proc. ICCV, pages 14124-14133, 2021.
[59] Zachary Teed and Jia Deng. RAFT: Recurrent all-pairs field transforms for optical flow. In Proc. ECCV, 2020.
[60] Johannes Kopf, Xuejian Rong, and Jia-Bin Huang. Robust consistent video depth estimation. In Proc. CVPR, pages 1611-1621, 2021.
[61] Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun. Tanks and temples: Benchmarking large-scale scene reconstruction. Proc. TOG, 36(4), 2017.
[62] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. Proc. TOG, 2018.
[63] Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In Proc. ICCV, pages 10901-10911, 2021.
[64] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? The KITTI vision benchmark suite. In Proc. CVPR, 2012.
[65] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proc. CVPR, pages 18995-19012, 2022.
[66] Carlos Campos, Richard Elvira, Juan J. Gomez Rodriguez, Jose M. M. Montiel, and Juan D. Tardos. ORB-SLAM3: An accurate open-source library for visual, visual-inertial, and multimap SLAM. IEEE Transactions on Robotics, pages 1874-1890, dec 2021.
[67] Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel J. Brostow. Digging into self-supervised monocular depth prediction. In Proc. ICCV, 2019.
[68] Yu-Lun Liu, Chen Gao, Andreas Meuleman, Hung-Yu Tseng, Ayush Saraf, Changil Kim, Yung-Yu Chuang, Johannes Kopf, and Jia-Bin Huang. Robust dynamic radiance fields. In Proc. CVPR, 2023.
[69] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In Proc. CVPR, 2021.
[70] Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. Proc. ICCV, 2021.