FlowCam: Training Generalizable 3D Radiance Fields without Camera Poses via Pixel-Aligned Scene Flow
Authors: Cameron Omid Smith
Yilun Du
Ayush Tewari
Vincent Sitzmann
Venue: NeurIPS 2023 poster (2023)
Keywords: Pose Estimation
Scene Flow Estimation
Scene Representation Learning
Computer Vision
Neural Implicit Representations
Neural Radiance Fields
View Synthesis
Self-Supervised Representation Learning
TL;DR: We propose to train generalizable 3D scene representations without known camera poses

Abstract: Reconstruction of 3D neural fields from posed images has emerged as a promising method for self-supervised representation learning. The key challenge preventing the deployment of these 3D scene learners on large-scale video data is their dependence on precise camera poses from structure-from-motion, which is prohibitively expensive to run at scale. We propose a method that jointly reconstructs camera poses and 3D neural scene representations online and in a single forward pass. We estimate poses by first lifting frame-to-frame optical flow to 3D scene flow via differentiable rendering, preserving locality and shift-equivariance of the image processing backbone. SE(3) camera pose estimation is then performed via a weighted least-squares fit to the scene flow field. This formulation enables us to jointly supervise pose estimation and a generalizable neural scene representation via re-rendering the input video, and thus, train end-to-end and fully self-supervised on real-world video datasets. We demonstrate that our method performs robustly on diverse, real-world video, notably on sequences traditionally challenging to optimization-based pose estimation techniques.

FlowCam: Training Generalizable 3D Radiance Fields without Camera Poses via Pixel-Aligned Scene Flow
Cameron Smith
Yilun Du {camsmith, yilundu, ayusht, ritzmann]-mii .mit.edu edu
Ayush Tewari
Vincent Sitzmann
MIT CSAIL cameronosmith. thub io/flowcam
Abstract
Reconstruction 3D neural fields from posed images has emerged as promising method for self-supervised representation learning. The key challenge preventing the deployment of these 3D scene learners on large-scale video data is their depen- dence on precise camera poses from structure- fromtmmooooommoooionn which is prohibitively expensive to run at scale. We propose a method that jointly reconstructs camera poses and 3D neural scene representations online and in single forward pass. We estimate poses by first lifting frame-to-frame optical flow to 3D scene flow via differentiable rendering, preserving locality and shift-equivariance of the image processing backbone. SE(3) camera pose estimation is then performed via a weighted least-squares fit the scene flow field. This formulation enables us jointly supervise pose estimation and generalizable neural scene representation via re-rendering the input video, and thus, train end-to-end and fully self-supervised on real-world video datasets. We demonstrate that our method performs robustly on diverse, real-world video, notably on sequences traditionally challenging optimization-baser pose estimation techniques.
1 Introduction
Recent learning- bbatn 3D reconstruction techniques show promise in estimating the underlying 3D appearance and geometry from just few posed image observations, single feed-forward pass [1- 19]. These techniques offer an exciting new perspective on computer vision: Instead of making predictions only on pixels, computer vision models might operate directly on the corresponding 3D scenes. This would be a significant step towards a generalist computer vision model, applicable to any task involving interaction with the physical world.
A core challenge to the generality of these methods is that they cannot be trained from just video, but instead require knowledge of per-frame camera poses. Existing methods thus rely on curated datasets that obtain camera poses via Structure-from from-Motion, but this is prohibitively expensive to compute at scale. Lifting this dataset prerequisite would unlock orders of magnitude more training data, making large-scale 3D representation learning tractable. Meanwhile, odometry and SLAM methods offer online camera pose estimation, but may fail track sequences with dominant camera rotation or with sparse visual features, and do not reconstruct dense 3D scene representations. While recent efforts leveraging differentiable rendering have demonstrated impressive results at joint reconstruction of camera poses and 3D scenes, they still require minutes or hours per scene. Further, these optimization- based methods cannot leverage learned priors for camera pose estimation, leaving significant progress in computer vision of the last decade untapped. While prior work has demonstrated self-supervised learning of joint depth and camera pose prediction [20, 21J. these models are constrained to tight video distributions, such as self-driving video, and do not infer full 3D representation, only depth.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

We present a method for jointly training feed-forwer generalizable 3D neural scene representation and camera trajectory estimation, self-supervised only by re-rendering losses on video frames, completely without ground-truth camera poses depth maps. We propose to leverage single-image neural scene representations and differentiable rendering to lift. frame-to-frame optical flow to 31) scene flow. We then estimate SE.(3) camera poses via a robust, weighted least-squares solver on the scene flow field. Regressed poses are used to re-construct the underlying 3D scene from video frames in feed-forward pass, where weights are shared with the neural scene representation leveraged in camera pose estimation.
We validate the efficacy of our model for ferd-forward novel view synthesis and online camera pose estimation on the real-world RealEstate 10K and KITTI datasets, as well as the challenging CO3D dataset. We further demonstrate results on in in-the-wild scenes in Ego4D and Walking Tours streamed from YouTube. We demonstrate generalization of camera pose estimation to out-of-distribution scenes and achieve robust performance on trajectories on which a offe-offthhearr SLAM approach, ORB-SLAM3 [22], struggles.
To summarize, the contributions of our work include:
We present a new formulation of camera pose estimation as weighted least-squares fit of an SE(3)
pose to a 3D scene flow field obtained via differentiable rendering.
We combine our camera pose estimator with multi-frame 3D reconstruction model, unlocking end-to-end, self-supervised training of camera pose estimation and 3D reconstruction.
We demonstrate that our method performs robustly across diverse real-world video datasets, including indoor, self-driving, and object-centric scenes.
2 Related Work
Generalizable Neural Scene Representations. Recent progress in neural fields [23-25] and differentiable rendering [10, 26-31] have enabled novel approaches to 3D reconstruction from few or single images [1-19], but require camera poses both at training and test time. An exception is recently proposed RUST [32], which can be trained for novel view synthesis without access to camera poses, but does not reconstruct 3D scenes explicitly and does not yield explicit control over camera poses. We propose a method that is similarly trained sepeesuperei on real video, but yields explicit camera poses and 3D scenes in the form of radiance fields. We outperform RUST on novel view synthesis and demonstrate strong out-of-distribution generalization by virtue of 3D structure.
(SIM). SfM methods [33-35], and in particular, COLMAP [34], are considered the de-facto standard approach to obtaining accurate geometry and camera poses from video. Recent progress on differentiable rendering has enabled joint estimation of radiance fields and camera poses via gradient descent [36-40], enabling subsequent high-quality novel view synthesis. Both approaches require offline per-scene optimization. In contrast, SLAM methods usually online [22, 41, 42), but notoriously unreliable rotation-heavy trajectories or scenes with sparse visual features. Prior work proposes differentiable SLAM to learn priors over camera poses and geometry [43, 44], but requires ground-truth camera poses for training. Recent work has also explored how differentiable rendering may be directly combined with SLAM [45-48], usually using conventional SLAM algorithm backbone and focusing on the single-scene overfitting case. We propose fully self-supervised method to train generalizable neural scene representations without camera poses, outperforming prior work on generalizable novel view synthesis without camera poses. We do nos claim state-of- satt--of-the-rrd the-hht art camera pose estimation, but provide an analysis of camera pose quality nevertheless, demonstrating robust performance on sequences that are challenging to state-of-the-art SL.AM algorithms, ORB-SLAM3 [22] and Droid-SLAM [43].
SLAM and Structure-from-Morn
Neural Depth and Camera Pose Estimation. Prior work has demonstrated joint self-supervised learning of camera pose and monocular depth [20, 21, 49-51] or multi-plane images [52]. These approaches leverage a neural network to directly regress camera poses with the primary goal of training high-quality monocular depth predictors. They are empirically limited to sequences with simple camera trajectories, such as self-drivigg datasets, and do not enable dense, large-baseline novel view synthesis. We ablate our flow-based camera pose estimation with a similar neural network- based approach. Most closely related to our work are approaches that infer per- timetipttep 3D voxel grids and train a CNN to regress frame-to-frame poses [53, 54]. We benchmark with the most recent approach in this line of work, Video Autoencoder [53]. Lastly, we strongly encourage the reader to peruse impressive concurrent work DBARF [55], which also regresses camera poses alongside a generalizable NcRF. Key differences are that we leverage pose solver based on 3D-lifted optical

---

Video
Pose Estimation Module
Video Re-Rendering with
Stream
Estimated Camera Poses
2D 3D; Vinwle-View Neural Scene Rep.
Scene Flow: 3D Correspondence
Wilmn Renderine
Feature Feature
Volume Ressering
Encoder map
EILA XILA-Xala Xally X A
Optical Flow: 2D Correspondena
Posc
P
Volume Wendering
GT
Confidence Weights W
Induced
Pose-
Re- Rendcred
GT
Pose-Induced Flow Loss
Video Re- Rendering Loss
Conreponning 2D Features
Figure I: Method Overview. Given set of video frames, our method first computes frame-to-frame camera poses (left) and then re-renders the input video (right). To estimate pose between two frames, compute off-the-shelf optical flow establish 2D correspondences. Using single-view pixelNeRF [1], we obtain surface point cloud as the expected 3D ray termination point for each pixel. X, respectively. Because and Y are pixel-aligned, optical flow allows us to compute 3D scene flow as the difference of corresponding 3D points. We then find the camera pose P E SE(3) that best explains the 3D flow field by solving weighted least-squares problem with flow confidence weights W. Using all an RGB loss and a flow loss between projected pose-induced 3D scene flow and 2D optical flow. Our method trained end-to-end, assuming only an off the-shelf optical flow estimator.
poses, we re-render all frames. We enforce
flow for real-time odometry versus predicting iterative updates to pose and depth via a neural network. Further, we extensively demonstrate our method's performance on rotation-dominant video sequences, in contrast to focus on horward-facing scenes. Lastly, we solely rely on the generalizable scene representation in contrast to leveraging a monocular depth model for pose estimation.
3 Learning 3D Scene Representations from Unposed Videos
Our model learns to map monocular video with frames {Te) as well as off-the-shelf optical flow [V.) per-frame camera poses {P.) and 3D scene representation & in a single feed-forward pass. We leverage known intrinsic parameters when available, but may predict them not. We will first introduce the generalizable 3D scene representation $. We then discuss how we leverage & for feed-forward camera pose estimation, where we lift optical flow into 3D scene flow and solve for pose via a weighted least-squares SE(3) solver. Finally, we discuss how we obtain supervision for both the 3D scene representation and pose estimation by re-rendering RGB and optical flow for all frames. An overview of our method is presented in Fig. 1.
Notation. It will be convenient to treat images sometimes discrete tensors, such as I E "Rxx'3, and sometimes as functions : 2 - R over 2D pixel coordinates p E R.2. We will denote functions in italic I, while we denote the corresponding tensors sampled on the pixel grid in bold as I.
3.1 Defining Our Image-Condiitoneed 3D Scene Representation
First, introduce the generalizable 3D scene representation aim to train. Our discussion assumes known camera poses; in the subsequent section we will describe how we can use our scene representation to estimate them instead. We parameterize our 3D scene (NeRF) [28], such that is a function that maps a 3D coordinate x to a color e and density a as ((x)= (a, c). To render the color for ray r, points (X1, X2...., x,,) are sampled along r between predefined near and far planes [41, ty], fed into & to produce corresponding color and density tuples [()), c1), (a2, o2), (On, C, and alpha-composited to produce final color value C(r):
Neural Radiance Field
NN
C(r) T:(1 exp(-o,,))) where T; =exp(-
Σ
Σ ozbz), j=l
(I)
3

---

CO3D Hydrant
CO3D 10-Category
RealEstate OK
KITTI
Context
Video
AE
RUST
Ours
Ours
Fine-tune
Target
Figure Video Reconstruction Results. Our model reconstructs vidco frames from sparse context frames with higher fidelity than all baselines. While VidAE's renderings often appear with convincing texture, they are often not aligned with the ground truth. RUST's renderings well aligned but are blurry due to their coarse set latent
representation.
CO3D- Hydrants
CO3D-10 10
RealEstate
KITTI
Model
LPIPS |
PSNR
LPIPS
PSNR
LPIPS
PSNR
LPIPS
PSNR f
Vid-AE [53]
0.3427
18.56
0.3889
18.50
0.4173
18.03
0.3272
17.65
RUST [32]
0.6126
17.50
0.6145
17.71
0.5692
18.07
0.5584
15.44
Ours
0.2250
23.94
0.2687
24.17
0.2224
24.25
0.1928
23.39
Table 1: Quantitative Comparison on View Synthesis. On the task of view synthesis, our method outperforms
other unposed methods by wide margins.
where 6; = f++1 i is the distance between adjacent samples. By compositing sample locations instead of colors, we can compute an expected ray-surface intersection point S(r) e R²
S(r) = M T1(1 exp(-or6)))x.
==1
We require generalizable NeRF that is not optimized for each scene separately, but instead predicted in heed-forward pass by an encoder that takes set of M context images and corresponding camera poses {LL, P.) Ad input. We denote such a generalizable radiance field reconstructed from images ((x {LL P.)..)) Many such models have been proposed [1-10]. We base our model on piXelNeRF [1]. which we briefly discuss in the following please find further details in the supplement. pixelNeRF first extracts per-image features F; from each input image I, given 3D coordinate x is first projected onto the image plane of each context image I via the known camera pose and intrinsic parameters to yield pixel coordinates p.. We then retrieve the features (p.) at that pixel coordinate. Color and density (a, c) at x are then predicted by neural network that takes input the features (F:():)) and the coordinates of in the coordinate frame of each camera, condition pixeINeRF varying numbers of context images, i.e.,
(P, (P-'x)." X Importantly, we may run pixeINeRF with only a single context image as ( (I, or with set of M > 1 context images ((x P.),. P.)
3.2 Lifting Optical Flow to Scene Flow with Neural Scene Representations
Equipped with our generalizable 3D representation 6 we now describe how we utilize lift optical flow into confidence-weieee weighted 3D scene flow. Later, our pose solver will fit camera pose to the

---

Sample Frames
Ours
Video AF
Sample Frames
Ours
BARF
20 frame Videos
50+ frame Videos
Figure 3: Qualitative Pose Estimation Comparison. On short sequences, compare pose estimation to Video Autocncoder [53], and on long sequences, compare our method's sliding wwiddwwwwwdwww estimations against the per-scene optimization BARF [38]. The trajectory for the bicycle sequence was obtained using model trained on hydrant sequences: despite never having seen a bicycle before, our model predicts accurate poses.
(a)
(b)
Hydrant
10-Cat.
REIOK
KITTI
Top
Bot.
% Tracked
VidAE [53] Ours
0.100
0.101
0.090
0.084
ORB3 [22]
0.021
0.034
49
0.041
0.042
0.020
0.010
DROID [431
0.028
0.049
100
Ours
0.015
0.025
100
Table 2: Quantitative Pose Estimation Comparison. (a) compare against Autonnnuteeee [53] on short-sequence odometry estimation (20 frames), reporting the ATE. (b) we compare against
ORB-SLAM3 [22] and DROID-SLAM [43] on long sequences (~~200 frames) from the CO3D 10-Category dataset. We separately report scores on the and bottom 50% of sequences ("Top" and "Bot.") in terms of quality of ground-truth poses indicated by the dataset. We report ATE and percent of sequences tracked ("Tracked"). ORB-SLAM3 fails to track over half these challenging sequences.
estimated scene flow. Given two sequential frames 1,-1, I, we first use an off-the-shelf method [56] to estimate backwards optical flow V, R22 RR2 The optical flow V, maps 2D pixel coordinate to a 2D flow vector, such that we can determine the corresponding pixel coordinate I as p' - V:(p).
will now lift pixel coordinates p: and P:-1 to an estimate the 3D points that they observe the coordinate frame of their respective cameras. To achieve this, cast rays from the camera centers through the corresponding pixel coordinates Pa and P--1 using the intrinsics matrix K. Specifically, compute r, = K 'P: and r = K PP-11 where p represents the homogeneous coordinate (P). Next, we sample points along the rays H, and r;-1 and query our pixeINeRF model in the single-view setting. This involves invoking (-I (xx4) and (,, (4).)), i.e., pixelNeRF is with only the respective frame as the context view and the identity matrix as the camera pose. Applying the ray-intersection integral defined in Eq. to the pixelNeRE estimates, we obtain a pair of 3D points (x,, x1-1) corresponding the estimated surfaces observed by pixels P. and Pi-1. respectively. We repeat this estimation for all optical flow correspondences, resulting in two sets of surface point clouds, X, X' @HHWWx Equivalently, may view this as defining the 3D scene flow as X' X.
Flow confidence weights. We further utilize a confidence weight for each flow correspondence. To accomplish this, we employ a neural network V, which takes image features F((p), F (p') as input for every pixel correspondence pair (')p'). The network maps these features to a weight w, denoted as V (F), Fi-(("))) = w E [0, 1]. v/ can importantly overcome several failure modes which lead to faulty pose estimation, including incorrect optical flow, such as in areas of occlusions, dynamic objects, such as pedestrians, or challenging geometry estimates, such as sky regions. We show in Fig. 9 that V indeed learns such content- based rules.
Predicting Intrinsic Camera Parameters K. Camera intrinsics are often upproximately known, either published by the manufacturer, saved in video metadata, or calibrated once. Nevertheless, for purposes of large-scale training, we leverage a simple scheme to predict the camera field-of-view for video sequence. We feed the feature map of the first frame F, into a convolutional encoder that directly regresses the field of view. We assume that the optical center is at the sensor center. We find that this approach enables us train real-world video from YouTube, though more sophisticated schemes [57] will no doubt improve performance further.
5

---

Sumple Frames
Our
ORB ORB-SLAN -B--LL RRB-SLAM3 SLAM3
RGB
Geanetry
Figure 4: Robustness. Our methad works sequences challenging to ORB-SLAM3, which fails on 49% of CO3D. We show one successful and one failed sequence.
Figure 5: Learned geometry. Our model's expected ray termination illustrates the unsupervised geometry learned by our model on the challenging CO3D dataset.
3.3 Camera Pose Estimation as Explaining the Scene Flow Field
We will now estimate the camera pose between frame I, and I In the previous section, we lifted the input optical flow into scene flow, producing 3D correspondences X, X', or, equivalently, 3D scene flow. We cast camera pose estimation as the problem of finding the rigid-body body-body motion that best explains the observed scene flow field, or the transformation mapping points in X to X', while considering confidence weights W. Note that below, we will refer to the matrices X, X', and W as column vectors, with their spatial dimensions flattened.
We use a weighted Procrustes formulation solve for the rigid transformation that best aligns the set of points X and X', The standard orthogonal Procrustes algorithm solves for the SE(3) pose such that it minimizes the least squares error:
arg min X PX
PESE(3)
with P (B () as a rigid-body pose with rotation and translation and homogeneous X In other words, the minimizer of this loss is the rigid-body transformation that best maps X onto X', and, in a static scene, is therefore equivalent to the sought-after camera pose.
As noted by Choy et al. [58], this formulation equally weights all correspondences. As noted in the previous section, however, this would make our pose estimation algorithm susceptible to both incorrect correspondences as well as correspondences that should be down-weed wein-weighted by nature of belonging to parts of the scene that are specular, dynamic, or have low confidence their geometry estimate. Following [58], we thus minimize weighted least-squares problem:
arg min| W 1/22 1 PESE(3)
with the diagonal weight matrix W diag(w). (w). Conveniently, this least-squares problem admits closed-form solution, efficiently calculated via Singular Value Decomposition, as derived in [58]:
R = USV and t (X - RX')W1, where UΣV" SVD(Exxx),
(5)
Σx' = XKWKX K =1 wwwww and S = diag(l, det det((U)ee((V))
(6)
Composing frame-to-frame poses. Solving this weighted least-squares problem for each subsequent frame-to-frame pair yields camera transformations (P/, P'..... .., P,,).
aligning each I, to its prede- cessor I-1. We compose frame-to-fran transformations to yield camera poses (P,, P....., P..)
relative to the first frame, such that P T3x3. concluding our camera pose estimation module.
3.4 Supervision via Differentiable Video and Flow Re-Rendering
have discussed our generalizable neural scene representation and our camera pose estimation module. We will now discuss how we derive supervision to train both modules end-to-end. We have two primary loss signals: First, photometric loss LRGB scores the visual fidelity of re-rendered video frames. Second, pose-induced flow loss Lpose scores how similar the flow induced by the

---

Context Views
Novel Views
Context Views
Novel Views
Figure 6:
View Synthesis. Given an input video without poses, our model first infers camera poses and can then render wide-baseline novel views of the underlying 3D scene, where we use the first, middle, and final frame of the video context views.
predicted camera transformations and surface estimations are to optical flow estimated by RAFT [59]. Our model is trained on short (~15 frames) video sequences.
Photometric Loss. Our photometric loss CRGB comprises two terms: multi-context loss and a single-context loss. The multi-context loss ensures that the full 3D scene is reconstructed accurately, Here, we re-render each frame of the input video sequence I,, using its estimated camera pose P. and multiple context images {(I,, P,)); The single-context loss ensures that the single-context pixeINeRF used to estimate surface point clouds X₃ in the pose estimation module is accurate.
N
LRGB =
1 N
Σ 1=1
- C 1 (CT PID + IL, - C(Txx4 | (L, 14x
Mulli Molli-Coniexi Loss
Single-Context Loss
where, in a slight abuse of notation, we have overloaded the rendering function C(P((,, P3), defined in Eq. as rendering out the full image obtained by rendering a pixelNeRE with context images {[T, p from camera pose P. N refers to the number of frames in the video sequence and refers to the subset of context frames use for re-rendering the entire sequence. We first attempted picking the first frame only, however, found that this does not converge due to the uncertainty of the 3D scene given only the first frame: single-view pixelNeRF will generate blurry estimates for parts of the scene that have high uncertainty, such occluded regions previously unobserved background.
Pose-Induced Flow Loss, An additional, powerful source of supervision for both the estimated geometry and camera poses can be obtained by comparing the optical flow induced by the predicted surface point clouds and pose with the fff-the-shelf optical flow. We define this pose- pose-induced flow
loss
N 1
pose
V. - (T(,') P4+1 X - u
with projection operator (-) and grid of pixel coordinates uv E R Intuitively, this transforms the surface point cloud of frame / into the coordinate frame of frame / and projects it onto that image plane. For every pixel coordinate at timestep + 1, this yields corresponding pixel coordinate p' timestep t, which we compare against the input optical flow.
3.5 Test-time Inference
After training our model on a large dataset of short video sequences, we may infer both camera poses and radiance field of such short sequence in single forward pass, without test-time optimization.
Sliding Window Inference for Odometry on Longer Trajectories. Our method estimates poses for short (~15 frames) subsequences in a single feed-forward pass. To handle longer trajectories that exceed the capacity of a single batch, we divide a given video into non-overlapping subsequences. We estimate poses for each subsequence individually and compose them to create aggregated trajectory estimate. This approach allows us to estimate trajectories for longer video sequences.
Test-Time Adaptation.
comne-to-frame camera pose estimation methods, both conventional and the proposed method, accumulate pose estimation error over the course of sequence. SLAM and SfM methods usually have mechanism to correct for drift by globally optimizing over all poses and closing loops the pose graph [60]. We not have such a mechanism, but propose fine-tuning our model on specific scenes for more accurate feed-forward pose and 3D estimation. For a given video sequence, we may fine-tune our pre-trained model using our standard photometric and flow losses on sub-sequences of the video. Note that this is not equivalent to per-scene optimization direct
7

---

Novel Rendering
BARF
Durs Zeto Zeto-Shot
Ours Scene Adapated
Trajectoryy
Context
Frames
Figure 7: Fine-tuned Pose Estimation and View Synthesis on Large-Scale, Out-of-Distribution Scene. We evaluate our RealEstate10K 10Kttriined model on significantly out-of-distribution scene from the Tanks and Temples dataset [6]], first without any scene-adantation and then with. Even with this significant distribution gap. our method's estimated trajectory captures the looping structure of the ground truth. albeit with accumulated drift. After scene-adaptatie fine-tuning stage (around hours), our model estimates poses which align closely with the ground truth. We also plot the trajectory estimated by BARF [38]. which fails to capture the correct pose distribution.
optimization of camera poses and radiance field, performed e.g. in BARF [38]: neither camera poses nor the radiance field are free variables. Instead, we fine-tune the weights of our convolutional inference backbone and MLP renderer for more accurate feed-forward prediction.
Experiments
We benchmark our method generalizable novel view synthesis on the RealEstate [62], CO3D [63], and KITTI [64] datasets. We provide further analysis on the Tanks & Temples dataset and in-the-wild scenes from Ego4D [65] and YouTuub. Though do not claim to perform state-of- the-art camera pose estimation, we nevertheless provide an analysis of the accuracy of our estimated camera poses. Please find more results, as well as precise hyperparameters, implementation, and dataset details, in the supplemental document and video. We utilize camera intrinsic parameters where available, predicting them only for the in-the-will wild Ego4D and WalkingTTuur Tours experiments.
Pose Estimation. We first evaluate method pose estimation against the closest self- supervised neural network baseline. Video Autoencoder (VidAE) [53]. We then analyze the robustness of our pose estimation with ORB-SLANS
[66] and DROID-SLAM SLAM [43] references. Finally, we benchmark with BARF [38], a single-scene unposed NeRF baseline. Tab. 2a compares accuracy of estimated poses of our method and VidAE all four datasets. The performance gap is most pronounced the challenging CO3I) dataset, but even on simpler, forward-moving datasets, RealEstate 10k and KITTI, our method significantly outperforms VidAE. Next, we analyse the robustness of our pose estimation module on CO3D, using SIM methods ORB-SLAM3 and DROID-SLAM as references. See Tab. 2b and Fig. for quantitative and qualitative results. To account for inaccuracies in the provided CO3D poses we utilize as soound-truth, we additionally report separate results for the top and bottom 50% of sequences, ranked based on the pose confidence scores provided by the authors. Although we do not employ any secondary pose method division serves as an approximate indication of the level of difficulty each sequence poses from SfM perspective. On both subsets, our method outperforms both DROID-SI.A note that ORB-SLAM3 fails track poses for over half (50.7%) of the sequences. On the sequences where ORB-SLAMS succeeds, our method predicts poses significantly more accurately. Even on the sequences where ORB-SLAM3 fails, our performance does not degrade (.025 ATE). Lastly, we compare against the single-scene unposed NeRF baseline, BARF. Since BARF requires ~-one day to reconstruct a single sequence, we evaluate on two representative sequences: a forward-walking sequence RealEstate10K, and outside-in trajectory CO3D. We plot recovered trajectories in Fig. 3. While BARF fails to recover the correct trajectory shape on the CO3D scene, our method produces trajectory that more accurately reflects the ground-truth looping structure.
proxy ground truth for the bottom half of sequences, this
and DRB-SLAMM.. Also
Novel View Synthesis. We compare against VidAE [53] and RUST [32] on the task of novel view synthesis. Tab. and Fig. 6 report quantitative and qualitative results respectively. Our method outperforms both baselines significantly. Since VidAE fails to capture the pose distribution on the CO3D datasets, its novel view renderings generally do not align with the ground truth. On RealEstate 10K and KITTI, their method successfully captures the pose distribution, but still struggles
8

---

Sample Frames
Novel View! Novel Depth
Sample Frames
Novel View! Novel Depth
Input RGB
Flow- Weigh Masked RGB
Input RGB
Flow- Flow-Weight
Masked RGB
Figure 8: View Synthesis on Ego4D and Walking Tours: We train on collection of YouTube walking tour videos and Ego4D sequences with unknown camera parameters, and render novel views after a short fine-tuning stage.
Figure 9: Flow Weights: Our flow-confidence weights allow our model to down-weight unreliable flow correspondences due to occlusions, specular highlights, or dynamic objects, and up up-weight well well-textured regions.
to render high-fidelity images. We similarly outperform RUST, which struggles to capture high- frequency content due to its coarse scene representation. Also note that RUST estimates latent camera poses rather than explicit ones. We further qualitatively evaluate our method on wide-baseline novel view synthesis, where no ground ground-truth available please see Fig. 6 for these results.
Test-Time Adaptation We evaluate the ability of our model to estimate camera poses on out- of-distribution scenes, augmented by test-time optimization as discussed in section 3.5. Fig.7 displays the camera poses estimated by our method on the Bulldozer sequence from the Tanks and Temples dataset[61], ], using our model trained on the RealEstate 10k dataset. Even without any test-time adaptation, our method generalizes impressively, generating a trajectory that follows the outside-in trajectory as determined by COLMAP. We refer to this generalization mode as "zero- shot." Nevertheless, camera poses are not perfect and exhibit some drift. We thus perform test-time optimization, fine-tuning the weights of our feed-forward method on subsequences of the video. continued optimization Though geometry is plausible after roughly 10 minutes of fine-tuning, for 7 hours for finer details. Our scene- eddated estimates are close to ground truth with little drift, and we find that novel view synthesis results with significant bascline visually compelling with sound depth estimates. We further show the result of BARF on this sequence, which fails to recover a plausible trajectory.
Results on In-the-Wiit Video. We present preliminary results on the Ego4D dataset [65], a recent ego-centric dataset captured with headset cameras, and new Walking Tours dataset, in which stream walking tour videos from YouTube. Both datasets comprise some scene motion, such as pedestrians, cars, or people. Images from Ego4D further subject to radial camera distortion, which we do not model. Nevertheless, after fine-tuning on Walking Tours and Ego4D for 20 minutes and 1.5 hours, that our model is generally robust to dynamic scene content, such as pedestrians or humans, which obtain low flowccofiidence flow scores. We further run COLMAP on a subset of the Walking Tours and Ego4D videos, yielding pseudo ground-truth poses for 3 videos from each dataset. The Walking Tour subset selected randomly, and the Ego4D subset is chosen as the first three relatively-statie videos we found (to ensure COLMAP convergence). We divide each video into subsequences of 20 frames at 3fps, corresponding to roughly meters of forward translation per clip. Here, we achieve competitive ATE of 0.013 and 0.026 on Walking Tours and Ego4D, respectively. We also compute PSNR values on subsequences of frames using two context views, and obtain 18.87dB and 21. 21.59db on Walking Tours and Ego4D, respectively, indicating generally plausible novel view synthesis.
can generate plausible novel views and depth maps, illustrated in Fig. 8. We find
Ablations. In Tab. we ablate key contributions related to our pose formulation, evaluated on the CO3D Hydrant dataset. We first ablate our proposed flow-based pose for- mulation in favor of concatenating adjacent frames and predicting a pose directly via a CNN, as is common in the Monodepth [67] line of work. We further ablate weighted flow formulation in favor of a non-weighted Procrustes estimation. Both methods perform significantly worse than our full flow-based pose for- mulation, and qualitatively often lead to degenerate geometry estimates. Please see the supplemental material for additional ablations and ablation details.
PSNR
J LPIPS
ML.P-Pose
18.50
0.54
No Flow Weights
17.87
0.51
Full
21.02
0.38
Table 3: Ablation study.

---

5 Discussion
Limitations. While we believe our method makes significant strides, still has several limitations. As an odometry method, accumulates drift and has no loop closure mechanism. Our model further does not currently incorporate scene dynamics, but recent advancements in dynamic NeRF papers [68-70] present promising perspectives for future research.
Conclusion. We have introduced FlowCam, model capable of regressing camera poses and reconstructing a 3D scene in a single forward pass from a short video. Our key contribution is to factorize camera pose estimation as first lifting optical flow to pixel-aligned scene flow via differentiable rendering, and then solving for camera pose via a robust least-squares solver. We demonstrate the efficacy of approach as in-the-wild videos. We believe that they represent a significant step towards enabling scene representation learning uncurated, real-world video.
variety of challenging real-world datasets, well
Acknowledgements. This work was supported by the National Science Foundation under Grant No. 221 1259, by the Singapore DSTA under DST000EC120300823 (New Representations for Vision), and by the Amazon Science Hub. The Toyota Research Institute also partially supported this work. This article solely reflects the opinions and conclusions of its authors and no other entity.
References
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelNeRI: Neural radiance fields from or few images. In Proc. CVPR, 2021.
[2] Michael Niemeyer, Lars Mescheder, Michael Occhsle, and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without supervision. In Proc. CVPR, 2020.
[3] Yilun Du, Cameron Smith, Ayush Tewari, and Vincent Sitzmann. Learning to render novel views from wide-baseline stereo pairs. In Proc. CVPR, 2023.
[4] Alex Trevithick and Bo Yang. Grf: Learning general radiance field for 3d representation and rendering. In Proc. ICCV, pages 15182-155 15192, 2021.
[5] Mohammed Suhail, Carlos Esteves, Leonid Sigal, and Ameesh Makadia. Generalizable patch-based neural rendering. In European Conference Computer Vision. Springer, 2022.
[6] Julian Chibane, Aayush Bansal, Verica Lazova, and Gerard Pons-Moll. Stereo radiance fields (srf): Learning view synthesis for sparse views of novel scenes. In Proc. CVPR, pages 7911-7920. 2021.
[7] Qianqian Wang, Zhicheng Wang. Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser. Ihrnet: Learning multi-view image-based rendering. In CVPR, 2021.
[8] Eric R Chan, Koki Nagano, Matthew A Chan, Alexander W Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein. Generative novel view synthesis with 3d awwa diffusion models, arXiv preprint arXiv:2304.02602, 2023.
[9] Zhizhuo Zhou and Shubham Tulsiani. Sparsefusion: Distilling view-conditioned diffusion for 3d recon- struction. Proc. CVPR, 2022.
[10] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous
3d structure-awat neural scene representations. In Proc. NeurIPS, volume 32, 2019.
[11] Philipp Henzler, Jeremy Reizenstein, Patrick Labatut, Roman Shapovalov, Tobias Ritschel, Andrea Vedaldi, and David Novotny. Unsupervised learning of 3d object categories from videos in the wild. In Proc. CVPR, pages 4700-4709, 2021.
[12] Prafull Sharma, Ayush Tewari, Yilun Du Sergey Zakharov, Rares Andrei Ambrus, Adrien Gaidon, William T Freeman, Fredo Durand, Joshua B Tenenbaum, and Vincent Sitzmann. Neural groundplans: Persistent neural seene representations from single image. In Proc. ICLR.
[13] Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering, In Proc. NewrIPS, 2021.
[14] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang. Fanbo Xiang, Jingyi Yu, and Hao Su. Mvsnerf: Fast generalizable radiance field reconstruction from nult stereo. In Proc. ICCV, pages 2021.
10

---

[15] Mehdi SM Sajjadi, Henning Meyer, Etienne Pot. Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lucic, Daniel Duckworth, Alexey Dosovitskiy, et al. Scene representation transformer: Geometry- free novel view synthesis through set-latent scene representations. arXiv preprint arXiv:2111.1312.. 2021.
[16] Shamit Lal, Mihir Prabhudesai, Ishita Mediratta, Adam W Harley, and Katerina Fragkiadaki. Coconets: Continuous contrastive 3d scene representations. Proc. CVPR, pages 12487-12496, 2021.
[17] Jonáš Kulhánek. Erik Dernrr, Torsten Sattler, and Robert Babuška. Viewformer: Nerf-free neural rendering from few images using transforerss In European Conference on Computer Vision (ECCV), 2022.
[18] Jiaxin Li, Zijian Feng, Qi She, Henghui Ding, Changhu Wang, and Gim Hee Lee. MINE: Towards continuous depth MPI with NeRF for novel view synthesis. In Proc. ICCV, pages 12578-12588, 2021.
[19] Kai-En Lin, Lin Yen-Chen, Wei-Sheng Lai, Tsung- Yi Lin, Yi-Chang Shih, and Ravi Ramamoorthi. Vision transformer for nerf-based view synthesis from single input image. arXiv preprint arXiv:2207.05736. 2022.
[20] Clément Godard, Oisin Mac Aodha, and Gabriel Brostow. Unsupervised monocular depth estimation with left-right consistency. In Proc. CVPR, pages 270-279, 2017.
[21] Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel Brostow. Digging into self-supervised monocular depth estimation. In CVPR, 2019.
[22] Carlos Campos, Richard Elvira, Juan J Gómez Rodríguez, José MM Monticl, and Juan Tardós. Orb- slam3: An accurate open-source library for visual, Robotics, 37(6):1874-1888 2021.
and multimap slam. /EEE Transactions
[23] Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. Neural fields in visual computing and beyond. In Proc. EUROGRAPHICS STAR, 2022.
[24] Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In Proc. NeurIPS, 2020.
[25] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich- Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Proc. NeurIPS, 2020.
[26] Thu H Bruyen-Phuoce Chuan Li, Stephen Balaban, and Yongliang Yang. Rendernet: A deep convolutional network for differentiable rendering from 3d shapes. In Proc. NeurIPS, volume 31, 2018
[27] Vincent Sitzmann, Justus Thics, Felix Heide, Matthias Nicßner, Gordon Wetzstein, and Michael Zollhöfer. Deepvoxels: learning persistent 3d feature emboddings. In Proc. CVPR, 2019.
[28] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In Proc. ECCV, pages 405-421, 2020.
[29] Philipp Henzler, Niloy J Mitra, and Tobias Ritschel. Escaping plato's 's cave: 3d shape from adversarial rendering. In Proc. ICCV, pages 9984-9993, 2019.
[30] Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Brualla, Tomas Simon, Jason Saragih, Matthias Nicßner, et al. State of the art on neural rendering. In Computer Graphics Forum, volume 39, pages 701-727. Wilcy Online Library, 2020.
[31] Lior Yariv, Yoni Kasten, Dror Moran. Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. Proc. NeurIPS, 33: 2492-2502, 2020.
[32] Mehdi S. M. Sajjadi, Aravindh Mahendran, Thomas Kipf, Etienne Pot, Daniel Duckworth, Mario Lučié, and Klaus Greff. RUST: Latent Neural Scene Representations from Unposed Imagery. CVPR, 2023.
[33] Richard Hartley and Andrew Zisserman. Multiple view geometry in computer vision. Cambridge university press, 2003.
[34] Johannes L Schonberger and Jan-Michael Frahm. feucture-from-mootee revisited. In Proc. CVPR, 2016.
[35] Sameer Agarwal, Yasutaka Furukawa, Noah Snavely, Ian Simon, Brian Curless, Steven M Seitz, and Richard Szeliski. Building rome in a day. Communications the ACM, 14(10))005-123 2011.
11

---

[36] Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. Nerf-: Neural radiance fields without known camera parameters. arXiv preprint arXiv:2102.07064. 2021.
[37] Lin Yen-Chen, Pete Florence, Jonathan T Barron, Alberto Rodriguez, Phillip Isola, and Tsung- Yi Lin. inerf:: Inverting neural radiance fields for pose estimation. In Proc. IROS, pages 1323-133 1330. IEEE, 2021.
[38] Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey. Barf: Bundle-adjusting neural radiance fields. In Proc. CVPR, 2021.
[39] Axel Levy, Mark Matthews, Matan Sela, Gordon Wetzstein, and Dmitry Lagun. Melon: Nerf with unposed images using equivalence class estimation. arXiv preprint arXiv:2303. 2023.
[40] Wenjing Bian, Zirui Wang, Kejie Li, Jiawang Bian, and Victor Adrian Prisacariu. Nope-nerf: Optimising neural radiance field with no pose prior. 2023.
[41] Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos. Orb-slam: versatile and accurate monocular slam system. 2015.
[42] Jakob Engel, Thomas Schöps, and Daniel Cremers. Lsd-slam: Large-scale direct monocular slam. In Proc. ECCV, 2014.
[43] Zachary Teed and Deng. Droid-slam: Deep visual slam for monocular, stereo, and rgh-d cameras, 2021.
[44] Krishna Murthy Jatavallabhula, Ganesh lyer, and Liam Paull. Grad-slam: Dense slam meets automatic diffcrentiation. In Proc. ICRA. IEEE, 2020.
[45] Zihan Zhu, Songyou Peng. Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R Oswald, and Mare Pollefeys. Nice-slam: Neural implicit scalable encoding for slam. In Proc. CVPR, pages 12786-12796, 2022.
[46] Antoni Rosinol. John Leonard, and Luca Carlone. Nerf-slam: Real-time dense monocular slam with neural radiance fields. arXiv preprint arXiv2221011641,, 2022.
[47] Chi-Ming Chung, Yang-Che Tseng, Ya-Ching Hsu, Xiang-Qian Shi, Yun-Hung Hua, Jia-Fong Yeh, Wen- Chin Chen, Yi-Ting Chen, and Winston H Hsu. Orbeez-slamt slam: A real-time monocular visual slam with orb features and nerf-realized mapping. arXiv preprint arXiv:2209. 220.2222991177,,.. 2022.
[48] Edgar Sucar, Shikun Liu, Joseph Ortiz, and Andrew Davison. imap: Implicit mapping and positioning in real-time. In Proc. ICCV, pages 6229 6238, 2021.
[49] Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe. Unsupervised learning of depth and ego-motion from video. In Proc. CVPR, 2017.
[50] Zhichao Yin and Jianping Shi. Geonet: Unsupervised learning of dense depth, optical flow and camera pose. In Proc. CVPR, 2018.
[51] Vitor Guizilini, Kuan-Hui Lee, Rarey Ambruş, and Adrien Gaidon. Learning optical flow, depth, and scene flow without real-world labels. Proc. ICRA, 2022.
[52] Yang Fu, Ishan Misra, and Xiaolong Wang. Multiplane nerf-supervised disentanglement of depth and camera pose from videos. arXiv preprint 2rXiv:2210.07181, 2022.
[53] Zihang Lai, Sifei Liu, Alexei A Efros, and Xiaolong Wang. Video autoencoder: self-supervised disentan- glement of static 3d structure and motion. In Proc. ICCV, 2021.
[54] Hsiao- Yu Fish Tung, Ricson Cheng, and Katerina Fragkiadaki. Learning spatial common sense with geometry-awes recurrent networks. In Proc. CVPR, 2019.
[55] Chen and Gim Hee Lee. Dbarf: Deep bundle-adjusting generalizable neural radiance fields, arXiv preprint Xiv::2303. 14478, 2023.
[56] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Proc. CVPR, pages (19.419. Springer, 2020.
[57] Linyi Jin, Jiamming Zhang, Yannick Gold-Geofrroy, Oliver Wang, Kevin Matzen, Matthew Sticha, and David F. Foubcy. Perspective fields for single image camera calibration. Proc. CVPR, 2023.
[58] Christopher Choy, Wei Dong, and Vladlen Koltun. Deep global registration. In Proc. CVPR, 2020.
12

---

[59] Zachary Teed and Jia Deng. RAFT: Recurrent all all-pairs field transforms for optical flow. In Proc. ECCV, 2020.
[60] Johannes Kopf, Xuejian Rong, and Jia-Bin Huang. Robust consistent video depth estimation. In Proc. CVPR, pages 1611-1621, 2021.
[61] Arno Knapitsch, Jaesik Park, Qian-Yi Yi Zhou, and Vladlen Koltun. Tanks and temples: Benchmarking large-scale scene reconstruction. Proc. TOG, 36(4), 2017.
[62] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. Proc. TOG, 2018
[63] Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Shordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In Proc. ICCV. pages (0901-10911) 2021.
[64] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? The KITTI vision benchmark suite. Proc. CVPR, 2012.
[65] Kristen Grauman, Andrew Westhury, Fugene Byme, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu. Xingyu Liu, Ego4d: Around the world 3,000 hours of egocentric video. In Proc. CVPR, pages 18995-19012, 2022.
[66] Carlos Campos, Richard Elvira, Juan Gomez Rodriguez, Jose M. M. Montiel, and Juan D. Tardos. ORB-SLAM3: An accurate open-source library for visual, visual-inertial, and multimap SLAM. IEEE Transactions on Robotics, pages 1874-1890) 1890, dec 2021.
[67] Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel J. Brostow. Digging into self-supervised monocular depth prediction. In Proc. ICCV, 2019.
[68] Yu-Lan Liu, Chen Gao, Andreas Meuleman, Hung-Yt Yu Tseng, Ayush Saraf, Changil Kim, Yung-Yo Yu Chuang, Johannes Kopf. and Jia-Bin Huang. Robust dynamic radiance fields. In Proc. CVPR, 2023.
[69] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In Proc. CVPR, 2021.
[70] Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo Martin- Brualla. Nerfies: Deformable neural radiance fields. Proc. 2021.
13


Reviews:
[{"review_id": "2XESsAjIrk", "reviewer": "Reviewer_oZ25", "summary": "The paper proposes using scene flow to optimize for camera poses to produce generalizable 3D radiance field. The key contribution is the joint optimization of the camera poses and 3D neural scene representations in an single forward pass. The method has been evaluated on multiple datasets and performs well on datasets where traditional pose estimation fails. ", "questions": "in Fig3 results were shown using BARF for pose estimation, why are there no quantitative results using the same?", "limitations": "Limitations have been well discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The motivation for using an joint optimization of camera poses and reconstruction has wide applications and easy to scale.\n\nAs proposed by the current method multiple views of a scene inherently produce good correspondences, which leads to reasonable optical flow estimates. Using and incorporating optical flow into the 3D pose estimation and reconstruction helps overall reconstruction pipeline.\n\nUsing optical flow for reconstruction and Neural radiance fields without cameras poses have been well independently. Combining them into a single formulation is still less explored. the current method lifts the optical flow to scene flow using neural scene representations and uses them to optimize. This is interesting direction to explore.\n\nThe results shows that the method is able to perform better than traditional pose estimation methods like ORB-SLAM in some of their failure cases. ", "weaknesses": "The current method proposes a tradeoff by stating instead of using camera poses first and neural radiance field later as a two stage process, joint optimization is a more promising direction. but using joint optimization adds additional optimization constraints which have not been extensively discussed. It would be nice to see what are the failure cases of using the joint optimization compared to the two-stage optimization process.\n\nThe camera pose estimation has only been evaluated with ORB-SLAM3 or VidAE both of which do not incorporate neural radiance fields for 3D pose estimation. Recently there have been better methods which optimize for both radiance fields and pose simultaneously. having such baseline will give a better understanding of the accuracy of the algorithm.\n\nThe number of frames used for error computation is very small. i.e. 20 frames and 200 frames. The advantage of the method states that the joint optimization is beneficial to scale easily. Showing the method accuracy compared to orb on larger datasets or sequences like ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Pose estimation is only compared with non-NeRF based methods, and unposed NeRF methods have been proposed which yield better results with a more accurate comparison\nWe add a NoPe-NeRF comparison (see author response PDF, Fig. 1b and Tab. 1b); we succeed where BARF and NoPe-NeRF fail. Also recall that the unposed NeRF family of methods has an orthogonal goal to ours, in that those methods jointly optimize poses and NeRF for a single scene, whereas our aim is to train a generalizable scene representation.\n\n### Pros and Cons compared to two-stage approach, failure cases of proposed joint optimization of poses and NeRF\nUsing a two stage approach usually involves an non-differentiable pose estimator, either running in near real-time or offline. For the offline two-stage approach, such as using Colmap, the benefits include that a) it’s less expensive, b) there exists a more developed ecosystem of optimization tools for it, such as intrinsics solvers and loop closure mechanisms, and c) it can optimize on longer sequences with greater accuracy. The drawbacks are that it is prohibitively slow and can not be embedded in a real-time training loop and is known to still regularly fail, particularly on rotation-dominant sequences. Online non-differentiable systems, such as ORB-SLAM, have the benefit of being fast enough to run alongside the training of a downstream task, with the main drawback being that they often fail due to their lack of a scene prior. See Tab. 1a of the author response PDF where we show that ORB-SLAM loses tracking on over half of the CO3D sequences. \nJoint optimization methods have the benefit of being differentiable, meaning we can resolve the scale ambiguity between poses and renderer (discussed extensively in overview discussion), are real-time, and improve with the rendering loss as well. The main failure mode is that the poses are less accurate than an offline-optimization method, such as Colmap, and the renderings are proportionately blurry to the pose inaccuracy. Please see the supplemental document (Figures. 2 and 6) for blurred renderings and non-perfect pose estimations. Joint optimization further also incurs additional cost to a two-stage approach, as we have to additionally estimate geometry and poses for each timestep in each forward pass.\n\n### Number of frames is very small, which contradicts the claim that our contribution is to scale up training\nNote we are referring to the goal of scaling up the size of the datasets, e.g. moving from Co3D or ShapeNet  to all of YouTube, rather than scaling up the length of the videos. Though our experiments indeed usually span 20-30 frames instead of the 1000 frames used for odometry evaluations, note that we often use a considerable frameskip between images, and more sophisticated pose estimation algorithms in future work will likely be able to accommodate even larger frame skips. For instance, the 200 frames cited for the Tanks and Temples are downsampled from the original video to just 1 FPS, spanning a full circle around the Excavator, and that CO3D long-trajectory evaluations also use a significantly low framerate, such that the 200 frames typically similarly circles the full object. The resulting camera baseline from our framerate downsampling is appropriate for our goal of training a generalizable pixelNeRF and within the pose distribution typically used to train these methods. \n\n### BARF comparisons should have quantitative pose evaluations\nWe have added quantitative comparison to BARF and NoPe-NeRF (Tab. 1b of author response PDF).\n"}, {"review_id": "y8Eq6UuDG5", "reviewer": "Reviewer_3ZaF", "summary": "This paper proposes a general method for 3D neural scene reconstruction and camera pose estimation from a video sequence. The method takes a set of video frames as input and outputs the re-rendered video frames and estimated camera poses. The method is based on PixelNeRF, which is a general NeRF method that takes extracted image features as input.\n\nThe method first uses single-view PixelNeRF to compute per-frame point clouds. Then, the frame-to-frame 3D scene flow is estimated by leveraging the estimated 2D optical flow. The 3D scene flow is then used to constrain the frame-to-frame camera poses, which are solved by minimizing a weighted least-squares problem. Two losses are used to supervise the training: the RGB loss, which forces the PixelNeRF-based rendered images to be close to the input images; and the flow loss, which makes the projection offset of neighbor frames' point clouds close to the 2D optical flow.", "questions": "Overall,  the proposed method is reasonable, but I do have questions:\n\n1. There should be a scale problem when estimating the per-view point cloud by single-view pixelNeRF. It seems that the Eq (4) didn't consider it, why?\n\n2. The confidence weight is output by a network. Therefore, in my understanding, this network can implicitly handle the occlusion and specular by downweighting corresponding flow. But it seems that the Masked RGB images shown in Fig 9 mask out too much content (even for textured diffuse regions). Is this because of the inaccurate 2D optical flow estimation?", "limitations": "The proposed method learns a conditioned generaI NeRF network for multiview images. It is still limited since the images should be kept to accompany the learned model.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A nice solution to learn a general NeRF representation for multiview images without camer pose information.\n2. The quality of the rendering results for testing multi-view images are impressive. ", "weaknesses": "The experiments can be improved to further verified the advantage of the proposed method:\n\n1. Only one quantitative pose estimation comparison is shown, there is no quantitative comparisons with BARF. Besides, Nope-NeRF is a similar method that takes sequent frames as input for pose estimation, it better have one more comparison with Nope-NeRF.\n\n2. The quantitative results in Tab 1 are fine-tuned or not? It should be better to show both results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Only one quantitative pose estimation is reported, and we should compare with a more recent unposed NeRF method (such as NoPe-NeRF) \nPlease note that we extensively benchmark with the appropriate baselines of RUST and VideoAutoencoder, including quantitative evaluations of pose estimation on each dataset in Tab. 2 of the main paper. We also include comparisons to ORB-SLAM in Tab. 2 of the main paper and add DROID-SLAM quantitative comparisons in Tab. 1a of the author response, as well as qualitative pose estimation results to both methods in Fig. 6 of the supplemental PDF. We outperform all baselines considered for pose estimation on these difficult sequences.\nAs for the unposed NeRF family of methods as baselines, note that their goal is orthogonal to ours, in that they aim to optimize a NeRF for a single scene without poses, whereas we are interested in training a generalizable NeRF representation without poses. See the general overview response for further discussion on this point. With that distinction made, we agree that our experiment would be stronger with a more recent and robust unposed NeRF method. See the author response PDF, Fig. 1b and Tab. 1b for quantitative and qualitative evidence that we outperform both BARF and NoPe-NeRF for pose estimation. We will include these results in the experiments section of the main paper.\n### Fine-tuned results ambiguity\nIn general, all results in the paper are zero-shot (not fine-tuned), unless explicitly marked as fine-tuned. We will clarify this distinction in the experiments section of the main paper. Also note that the fine-tuning mechanism is not part of our model pipeline, which aims to remove precomputed poses from the training process of generalizable 3D representations, but rather a demonstration that our self-supervised formulation allows us to quickly optimize on any scene for higher-quality novel view synthesis. See the overview discussion for further elaboration on this point. \n### Scale ambiguity in single-view depth estimation that could have been accounted for in Procrustes estimation\nThis is an insightful point and while your analysis is correct, in practice we omit it to encourage the model to learn a consistent scale, and observe that the model’s depth estimates are generally of the same scale. Surprisingly, we found that constraining the scale to be the same across the video is critical and the model performance suffers with the scale degree of freedom. See Tab. 2 (column “Scale Adjusting Procrustes”) of the author response PDF for an explicit evaluation demonstrating this. \n### Confidence maps are concerningly sparse, even discounting textured diffuse regions\nThis is an acute observation and the weights are indeed relatively sparse, but note that our model theoretically only needs four points to solve for correct pose, and our model can therefore afford to choose fewer but more accurate correspondences. Also note that our model can also mask out parts of the scene where it is not confident about the geometry, such as sky or distant regions, not just bad correspondences. Please see Fig. 1a of the author response PDF for an illustration of our model’s flow weight mask including dynamic objects, as well as the overview discussion for further elaboration on the flow weights.\n### The method is limited since the images should be kept to accompany the learned model\nWhile image-conditioned scene representations, such as pixelNeRF, keep the images during rendering, note that all NeRF-based representations keep around some representation, whether it be the weights of a large global network or the densities and radiance of a voxel grid. Also note that one could potentially exchange an image-conditioned scene representation to a voxel-grid based one via sampling the 3D radiance field at the voxel locations, or alternatively distill the representation into a global network. We agree nonetheless that optimizing a global NeRF, such as BARF, and not keeping keyframes is an exciting and self-contained direction for future work.\n"}, {"review_id": "MYxZhyIQY8", "reviewer": "Reviewer_kV1r", "summary": "This paper proposes a method to address the challenge of reconstructing 3D neural fields from images and learns them in a self-supervised manner. The main contribution of this method is the joint reconstruction of camera poses and 3D neural scene representations within a single forward pass.", "questions": "1. I cannot understand how this paper addresses the generalization issue. Why does PixelNeRF have good generalization capabilities in monocular depth prediction?\n2. How does this paper address the problem of scale ambiguity in monocular depth estimation?", "limitations": "This paper discusses its limitations. However, I would have expected to see more failure cases from internet data.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. This paper achieves state-of-the-art performance in its specific setting.\n2. The writing in this paper is clear. \n3. This paper addresses an important problem. It overcomes the previous limitation in training generalized Neural Radiance Fields that require posed videos. It enables direct training on unposed video data.\n4. This paper is innovative in its methodology and no one has utilized similar approaches to address this problem before.", "weaknesses": "1. The paper lacks clarity in explaining the ablation study, such as the incomplete caption for Table 3. Moreover, the abbreviations used in the table are not defined, making it unclear what they refer to.\n2. The paper lacks ablation experiments on not learning intrinsic camera parameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### The ablation study is unclear in its abbreviations and references\nWe agree and have updated the main paper’s ablation table with expanded ablation references as well as its corresponding caption and experiment text. Please see author response PDF Tab. 2 for expanded reference names and caption. In case it is useful for correspondence, note that in the submission’s ablation table “MLP-Pose” refers to pose regressed by an MLP (as opposed to using a solver); “No Flow Weights” refers to using the pose solver but not using the confidence weights to guide the optimization; “Full” refers to the full model of using the solver with the confidence weights.\n### This paper lacks ablations on not learning intrinsics parameters\nIn this paper we use intrinsics parameters when known and only predict them otherwise, such as on the YouTube videos where no such information is available. We noted this on line 93 but we will make this more explicit in the experiments section as well. We also ran an additional ablation experiment on using constant and predicted intrinsics, reported in Tab. 3 of the author response PDF. Surprisingly, using constant intrinsics performs similarly to using SfM-estimated intrinsics, and predicting them actually outperforms using the SfM-estimated intrinsics. While this is surprising, one reason could be that the dataset intrinsics are not perfect but estimated by SfM or that our model may be predicting intrinsics which better correspond to its geometry estimate.\n### Not sure how this method addresses the generalization issue of pixelNeRF\nPixelNeRF has been shown in several instances to be a robust geometry estimator, used for depth estimation (see “Behind the Scenes: Density Fields for Single View Reconstruction”, e.g.). It has been demonstrated that pixelNeRF generality scales favorably when increasing dataset size  (see “Objaverse-XL: A Universe of 10M+ 3D Objects”). In general, depth estimation is a subset of the 3D geometry that pixelNeRF estimates. Our key contribution is a step towards training of pixelNeRF on internet-scale, uncurated video datasets. This would dramatically improve the generality of pixelNeRF, as the training distribution could now essentially be all natural videos on the internet. While some steps towards that goal are outstanding (such as addressing dynamic 3D reconstruction), we believe that our paper takes a significant step towards that goal.\n### How does this paper address the scale ambiguity in monocular depth estimation?\nMost monocular depth estimators output depth in an arbitrary scale, and normalizing their outputs is notoriously difficult, even for data from the same domain (such as driving scenes). The reason for this is related to the scale ambiguity we describe in the overview discussion, which boils down to inconsistent scale of camera poses from different videos. More specifically, the depth targets used to train those depth estimators are often only known up to scale, which makes training accordingly challenging when treating those as metric targets. Our model naturally resolves this scale ambiguity by predicting camera poses using our model’s rendering geometry. That is, since our camera poses are a function of our model’s predicted geometry, they are inherently of the same scale, removing any ambiguity introduced by depending on SfM per-scene optimized poses.\nWe assume that is the ambiguity you are referring to, but another ambiguity you might be referencing is the inherent scale ambiguity between our scene representation and the metric scale of the real world, which is impossible without a metric reference (such as RGBD). And lastly note that the scale ambiguity we refer to addressing in the text and the overview discussion is rather the difference in scale between the estimated poses (traditionally estimated via COLMAP) and the generalizable scene representation (which is regressed during training). See the overview discussion for an elaboration on this ambiguity and how it otherwise poses a significant challenge for training 3D scene representations such as pixelNeRF, which we overcome. \n### Would expect to see more failure cases on internet data\nIndeed, since we have not trained our model on a large-scale internet dataset, our model does not generalize perfectly yet to random internet videos, for reasons which include non-robust intrinsics prediction and out-of-distribution geometry estimation. Note that we train our pixelNeRF on the same scale of datasets typically used to train pixelNeRF (CO3D, e.g.), and just aim in these experiments to reproduce the same level pixelNeRF, without depending on precomputed poses, rather than increase generalization. The point made in our presentation of fine-tuned results for a few internet videos was to simply show that we can optimize on in-the-wild videos, and that although we do not yet scale up training to internet-scale data (see outstanding bottlenecks mentioned above), our model takes a large step towards enabling such internet-scale training. We will further include an additional set of results in the supplemental material of our model on random internet videos.\n"}, {"review_id": "8vhTfBVYOZ", "reviewer": "Reviewer_52at", "summary": "This paper presents a generalizable framework for estimating both the NeRF model of the scene and the camera pose for video sequence simultiunusly.\n\nThe proposed method first uses the pretrained model to predict the optical flow for each video frame. Then, the optical flow is lifted up to the monodepth to scene flow. The camera poses are solved by SVD with scene flow. After computing the camera pose, the nerf model is further fine-tuned for a better result.", "questions": "I am happy to adjust the rating if the author properly addresses the concerns mentioned during the rebuttal.\n\n----\n\nafter rebuttal:\n\nI adjust the rate to the positive side.", "limitations": "Yes.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- a light step of unposed nerf in RGB video sequence setting.\n- the quality of the rendered RGB image looks good after fine-tune.\n- well written, easy to read and follow.", "weaknesses": "- Statement\n  - Line 27-31\n    - Although the manuscript does acknowledge certain NeRF-based SLAM methods for their ability to provide dense 3D reconstruction, relevant references [1,2,3] are conspicuously absent. Additionally, some NeRF-based SLAM methods that utilize prior for pose estimation have been glossed over.\n  - Line 34\n    - The cited works do not adequately represent the body of literature since both methods focus exclusively on depth map estimation. This oversight should be rectified by including relevant citations [4,5].\n - Line 143\n   - wrong format\n\n\n- Motivation\n  - The motivation for introducing single view generalizable nerf for relative pose estimation is unclear and wired.\n    - Line 142\n    - the definition of the x_t. In equ2, x_i is the sampled points along the ray. Does x_t is the S(r) in equ2?\n    - while pixelnerf can use only a single view to render a novel view, it is much like a mono-depth estimation module. Without optical flow, an alternating solution uses iNeRF to align two single views point cloud/nerf, minimizing photometric loss. However, even most SOTA mono-depth estimation methods are insufficient to estimate accurate relative pose due to cross-view scale misalignment. Some unposed/posed Nerf only use mono-depth as a reg term. This is why the proposed method **HAS TO** introduce the out-of-shelf optical flow estimation for pose estimation. If the single view pixelnerf is removed from the proposed method, optical flow, as the correspondences, is enough to estimate the essential matrix/relative pose between two neighbor frames. Furthermore, the deep-sfm module, like banet, droidslam, deepv3d, all of these methods take multiview images to estimate both multiview depth map and the camera pose at the same time. The author should carefully address this issue during the rebuttal. \n  - the motivation of flow confidence weights.\n    - Line 143\n    - This part could be easily solved by bi-direction consistency check with optical flow. Introducing another pretrained network for weight prediction will make the generalization issue worse.\n    - Figure 9 does not show the results with dynamic objects.\n\n  - Line 178\n    - RGB VO/SLAM confronts scale ambiguity problem. The proposed method doesn't handle it at all.\n\n  - Line 194\n    - where is the definition of N, J?\n\n  - Line 203 Equ8\n  - The mono-depth is enforced by this loss term during fine-tune. As mentioned above, the optical flow for relative pose estimation is enough.\n  - This loss term does not handle singular solutions.\n\n- Experiment\n  - Line 211\n    - the training detail is missing.\n    - the proposed method cannot run on a long video clip.\n    - Most of the experiments show the object-center case. It will be great to see the results on the inside-out case either.\n  - Although the authors mention that they do not claim sota accuracy of pose estimation, the metrics of the pose are still necessary to be reported on the main paper, including droid slam, droid vo, orb slam, etc.\n  - Line 224\n    - test-time optimization on the weight of CNN and MLP on a single video clip in a self-supervised manner is extremely hard. The problem will become worse if the precomputed optical flow is removed. The author should provide more detail about this section. \n\n  - Line 250\n    - more baseline methods are required, such as nicer-slam[2], dim-slam[3], droid-slam.\n    - some unposed nerf methods with mono depth estimation module, such as nope-nerf[6]\n    - and another test-time optization method, such as CVD[4], RCVD[5].\n\n  - The pose trajectories in the figure are misaligned in both the main paper and the supplementary, which are very confusing.\n\n- reference\n  - [1] Sucar, E., Liu, S., Ortiz, J., & Davison, A. J. (2021). iMAP: Implicit mapping and positioning in real-time. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 6229-6238).\n  - [2] Zhu, Z., Peng, S., Larsson, V., Cui, Z., Oswald, M. R., Geiger, A., & Pollefeys, M. (2023). Nicer-slam: Neural implicit scene encoding for rgb slam. arXiv preprint arXiv:2302.03594.\n  - [3] Li, H., Gu, X., Yuan, W., Yang, L., Dong, Z., & Tan, P. (2023). Dense RGB SLAM with Neural Implicit Maps. The Eleventh International Conference on Learning Representations(ICLR)\n  - [4] Luo, X., Huang, J. B., Szeliski, R., Matzen, K., & Kopf, J. (2020). Consistent video depth estimation. ACM Transactions on Graphics (ToG), 39(4), 71-1.\n  - [5] Kopf, J., Rong, X., & Huang, J. B. (2021). Robust consistent video depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 1611-1621).\n  - [6] Bian, W., Wang, Z., Li, K., Bian, J. W., & Prisacariu, V. A. (2023). Nope-nerf: Optimising neural radiance field with no pose prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 4160-4169).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### iMAP and robust unposed NeRF \nWe added iMAP methods to the related works and NeRF methods. Note that iMAP addresses a separate problem: our contribution enables end-to-end training of pixelNeRF without precomputed poses, whereas iMAP performs offline, gradient-descent based pose and NeRF optimization. \n### Missing references to Consistent Video Depth Estimation and Robust Consistent Video Depth Estimation\nWe will add CVD and RCVD to our related work. CVD/RCVD fine-tune a pre-trained depth network for temporal consistency, whereas we train a generalizable 3D scene representation; they are not significantly related.\n\n### Typo on line 143\nThanks - fixed!\n\n### pixelNeRF for pose estimation is weird\nWe use the same network for pose estimation and rendering since a) it forces our pixelNeRF to learn good geometry in order to predict correct poses b) it avoids scale inconsistency between poses and rendering. As for why we involve geometry in pose estimation, instead of a 2D solver, see our response below on that point. We can also use a separate depth network instead of re-using pixelNeRF; see Tab. 2, “Depth Regression” of the response PDF to see that it works almost as well.\n\n### 142 ambiguity\nx_t and x_i both refer to 3D points, where x_i in eq2 refers to the 3D point at the i’th sample along the ray, and x_t refers to the 3D point observed at pixel p_t. We will amend the text accordingly to make this notation more explicit.\n\n### iNeRF formulation \niNeRF requires a pre-trained NeRF for pose inversion, which we do not have, and requires many gradient descent steps to align the model for accurate poses, which we cannot afford to embed in the training loop.\n\n### 2D correspondences sufficient for pose estimation\nSolving for the essential matrix from 2D correspondences introduces scale ambiguity between the 3D scene representation and estimated poses: the pose from the 2D correspondences is extremely unlikely to be the scale appropriate for the pixelNeRF’s constant near and far plane. See Tab. 2, column “2D-Only Pose Solver” of the response PDF; renderings catastrophically fail with this approach.\n\n### DROID-SLAM uses just 2D correspondences\nUsing any of these methods incurs the same scale ambiguity between generalizable neural scene representation and pose estimation described above and in the overview text. Also note that none of these methods solve for a 3D scene representation, but rather solve for depth maps and poses. \n\n### Flow confidence weights; bidirectional flow check simpler; generalization is now worse\nSee our overview discussion on flow confidence weights and dynamic object masking. We ablate this bidirectional flow check in Tab. 2 (“Bidirectional Consistency Flow Weights”) of the response PDF and report a significant decrease in rendering quality. Also consider this estimation asks how similar two features are and tends to generalize well. \n\n### Proposed method doesn't resolve scale ambiguity\nPlease see the overview text for expanded discussion on the scale ambiguity we address, which is between the poses and 3D scene representation, and how our method resolves it by using the same geometry estimation for pose estimation and rendering. \n\n### Missing definitions (194,203)\nThanks! We’ll amend the text to be more explicit. N refers to the number of timesteps or frames and J refers to the subset of context images used for rendering.\n\n### Scale resolved with fine-tuning\nThe fine-tuning mechanism is not involved during training of the scene representation; see the overview discussion on fine-tuning and scale ambiguity. We cannot afford to perform fine-tuning during the forward pass.\n\n### Our loss term does not handle singular solutions\nIf there is a singular solution due to textureless images, it should not affect the rendering or 3D scene representation, and is not a priority since our interest is the 3D scene representation. We will add this discussion to our limitation section.\n\n### Sliding window experiments details; Cannot run on long sequences\nThe sliding window method is an offline extension for explicitly chaining predicted camera poses on subsequent video subsequences to accommodate longer sequences. There is no optimization when using the sliding window approach; we simply query our trained model (such as on CO3D).\nWhile our method does not accommodate long sequences, the ~30 frames with considerable frameskip we use is within the pose distribution typically used for training such 3D scene representations. \n\n### Inside-out scenes \nOnly CO3D results are outside-in; please see are results on RealEstate10K and KITTI which are inside-out.\n\n### Comparison with ORB-SLAM and DROID-SLAM\nWe compare with ORB-SLAM and DROID-SLAM in Tab. 1a of the author response PDF and Fig. 6 of the supplemental PDF. We outperform both ORB-SLAM and DROID-SLAM in this setting.\n\n### Test time optimization of CNN and MLP difficult and expensive\nPlease see the overview discussion on the application of fine-tuning, which is not core to our method pipeline. Note that a) the fine-tuning results we demonstrate are on our pretrained model, not from scratch, b) the optimization is empirically robust, and c) it is not considerably more expensive to fine-tune the image features as well.\n\n### DROID-SLAM/Dim-SLAM/NoPe-NeRF/RCVD baselines\nSee Tab. 1 and Fig. 1b for comparison to NoPe-NeRF and Tab. 1a of the author response and Fig. 6 of the supplement for comparisons to DROID-SLAM. Since CVD aims to fine-tune a monocular depth estimator to be temporally consistent for a single video, instead of training a 3D scene representation, we respectfully disagree with CVD or RCVD being apt comparisons, but if you feel otherwise we can discuss it and evaluate the comparison.\n\n### Misaligned pose plots\nWe assume you refer to the reversed captions of figure 4, which we will amend in the paper revision, but otherwise if you can be more specific about which poses are misaligned we will be happy to correct it.\n"}]

Author rebuttal:
We appreciate the time and energy the reviewers have invested in reviewing our paper and for offering insightful and constructive feedback, which will make our paper clearer and stronger. We are glad that reviewers recognized our paper’s contribution in addressing an “important problem” (kV1r) in training a “general NeRF representation without camera pose information” (3ZaF) which has “wide applications and is easy to scale” (oZ25) and “achieves state-of-the-art performance in its specific setting” (kV1r).

## General Clarifications
### Paper goals and their difference to conventional Odometry & SLAM methods.


Our goal is to build a neural network that can jointly predict a 3D radiance field as well as camera poses in a single forward pass, without the need of off-line pre-processing training videos with Structure-from-Motion. We succeed at this goal and train pixelNeRF end-to-end on challenging real-world video with significant camera rotation. 

On the challenging CO3D 10-Category video dataset, our method obtains more accurate poses than ORB-SLAMv3 and DroidSLAM (see author reply, Table 2a). Thus, we do claim - and demonstrate - that training pixelNeRF with poses obtained from either DroidSLAM or ORB-SLAMv3 would perform worse than our method. However, we explicitly (Lines 78, 219) do not claim that our method is competitive with offline Structure-from-Motion, such as Colmap, especially on long-horizon pose estimation, where loop closures and global pose graph optimization are critical. However, this is irrelevant to our goal of removing the SfM pose bottleneck from training these methods. To illustrate this point of the computational infeasibility of scaling up SfM methods, consider that the CO3D dataset totaled over 5 GPU years to estimate poses for, and CO3D is an insignificant fraction of the size of internet-scale datasets. 

### Fine-tuning mechanism and comparison with single-scene unposed NeRF methods, such as BARF and NoPe-NeRF.
While our key contribution is training of pixelNeRF with a feedforward pose estimate, the fine-tuning serves to demonstrate that for longer sequences, our self-supervised loss yields gradients that lead to high-quality novel view synthesis within minutes. We have added comparisons to both BARF and NoPe-NeRF to Section 4; please see the author response PDF, Fig. 1b and Tab. 1b. BARF and NoPE-NeRF both fail to capture the rotation-dominant pose distribution; our method outperforms both of them.

### Scale ambiguity
The scale ambiguity we refer to resolving is the discrepancy between the world scale estimated by the offline pose-estimation method (usually COLMAP), and the world scale estimated by the generalizable 3D scene representation. To train these 3D scene representations, it is assumed that there is a consistent scale of the poses which the scene representation can learn to predict geometry in. Several view synthesis methods (GenVS, GPNR, Diffusion with Forward Models) introduce their own nontrivial scale-normalization steps and note that they are critical for stable training.
 
## Additional Experiments
In addition to reviewer-specific experiments, we highlight a few experiments common to all reviewer requests below.
### More robust and recent unposed NeRF methods (NoPe-NeRF)
We added a NoPe-NeRF comparison in addition to BARF; see Fig. 1b and Tab. 1b of the author response PDF. Despite NoPe-NeRF’s monocular depth prior addition to BARF, both do not capture the correct pose distribution.
### Scale ambiguity demonstration
To demonstrate the difficulties involved with the mentioned scale ambiguity, we randomly scale the poses estimated by our model before rendering and report the corresponding decrease in rendering quality in Tab. 4 of the author response PDF. 
### Confidence weights clarification and additional demonstration
As reviewer 3ZaF mentions, the flow weights that our model estimates are quite sparse. Since our pose estimation theoretically only requires 4 3D correspondences, our model can afford to be selective for good correspondences. Also note that our model not only masks out bad correspondences and nonrigid scene elements, but also areas where it has geometry uncertainty, such as in the sky region. Note that this flow weighting is critical to robust pose estimation, which we ablate in Tab. 2 of the author response PDF (“No Flow Weights” column). We include an instance in Fig. 1a of the author response PDF where our model indeed masks out a dynamic object (a bicyclist), but note that because the flow weights are relatively sparse, we are not suggesting that our model produces a tight “dynamic object mask”, but rather mention dynamic objects being masked to provide intuition about why the flow weights are critical.


Meta-review: The paper proposes using scene flow to optimize for camera poses to produce a generalizable 3D radiance field. The key contribution is the joint optimization of the camera poses and 3D neural scene representations in a single forward pass. After rebuttal, all reviewers recommended acceptance. The key strengths are good paper writing, a significant novelty in combining optical flow with NeRF, and impressive results. AC agrees with the reviewers and recommends acceptance.
Decision: Accept (poster)