Title: FlowCam: Training Generalizable 3D Radiance Fields without Camera Poses via Pixel-Aligned Scene Flow
Venue: NeurIPS 2023
Accepted: yes
Keywords: Pose Estimation

Scene Flow Estimation

Scene Representation Learning

Computer Vision

Neural Implicit Representations

Neural Radiance Fields

View Synthesis

Self-Supervised Representation Learning
Abstract: Reconstruction of 3D neural fields from posed images has emerged as a promising method for self-supervised representation learning. The key challenge preventing the deployment of these 3D scene learners on large-scale video data is their dependence on precise camera poses from structure-from-motion, which is prohibitively expensive to run at scale. We propose a method that jointly reconstructs camera poses and 3D neural scene representations online and in a single forward pass. We estimate poses by first lifting frame-to-frame optical flow to 3D scene flow via differentiable rendering, preserving locality and shift-equivariance of the image processing backbone. SE(3) camera pose estimation is then performed via a weighted least-squares fit to the scene flow field. This formulation enables us to jointly supervise pose estimation and a generalizable neural scene representation via re-rendering the input video, and thus, train end-to-end and fully self-supervised on real-world video datasets. We demonstrate that our method performs robustly on diverse, real-world video, notably on sequences traditionally challenging to optimization-based pose estimation techniques.

FlowCam: Training Generalizable 3D Radiance Fields without Camera Poses via Pixel-Aligned Scene Flow

Abstract

Reconstruction of 3D neural fields from posed images has emerged as a promising method for self-supervised representation learning. The key challenge preventing the deployment of these 3D scene learners on large-scale video data is their dependence on precise camera poses from structure-from-motion, which is prohibitively expensive to run at scale. We propose a method that jointly reconstructs camera poses and 3D neural scene representations online and in a single forward pass. We estimate poses by first lifting frame-to-frame optical flow to 3D scene flow via differentiable rendering, preserving locality and shift-equivariance of the image processing backbone. SE (3) camera pose estimation is then performed via a weighted least-squares fit to the scene flow field. This formulation enables us to jointly supervise pose estimation and a generalizable neural scene representation via re-rendering the input video, and thus, train end-to-end and fully self-supervised on real-world video datasets. We demonstrate that our method performs robustly on diverse, real-world video, notably on sequences traditionally challenging to optimization-based pose estimation techniques.

1 Introduction

Recent learning-based 3D reconstruction techniques show promise in estimating the underlying 3D appearance and geometry from just a few posed image observations, in a single feed-forward pass [119]. These techniques offer an exciting new perspective on computer vision: Instead of making predictions only on pixels, computer vision models might operate directly on the corresponding 3D scenes. This would be a significant step towards a generalist computer vision model, applicable to any task involving interaction with the physical world.

A core challenge to the generality of these methods is that they cannot be trained from just video, but instead require knowledge of per-frame camera poses. Existing methods thus rely on curated datasets that obtain camera poses via Structure-from-Motion, but this is prohibitively expensive to compute at scale. Lifting this dataset prerequisite would unlock orders of magnitude more training data, making large-scale 3D representation learning tractable. Meanwhile, odometry and SLAM methods offer online camera pose estimation, but may fail to track sequences with dominant camera rotation or with sparse visual features, and do not reconstruct dense 3D scene representations. While recent efforts leveraging differentiable rendering have demonstrated impressive results at joint reconstruction of camera poses and 3D scenes, they still require minutes or hours per scene. Further, these optimizationbased methods cannot leverage learned priors for camera pose estimation, leaving significant progress in computer vision of the last decade untapped. While prior work has demonstrated self-supervised learning of joint depth and camera pose prediction [20, 21], these models are constrained to tight video distributions, such as self-driving video, and do not infer a full 3D representation, only depth.

We present a method for jointly training feed-forward generalizable 3D neural scene representation and camera trajectory estimation, self-supervised only by re-rendering losses on video frames, completely without ground-truth camera poses or depth maps. We propose to leverage single-image neural scene representations and differentiable rendering to lift frame-to-frame optical flow to 3D scene flow. We then estimate SE(3) camera poses via a robust, weighted least-squares solver on the scene flow field. Regressed poses are used to re-construct the underlying 3D scene from video frames in a feed-forward pass, where weights are shared with the neural scene representation leveraged in camera pose estimation.

We validate the efficacy of our model for feed-forward novel view synthesis and online camera pose estimation on the real-world RealEstate10K and KITTI datasets, as well as the challenging CO3D dataset. We further demonstrate results on in-the-wild scenes in Ego4D and Walking Tours streamed from YouTube. We demonstrate generalization of camera pose estimation to out-of-distribution scenes and achieve robust performance on trajectories on which a state-of-the-art SLAM approach, ORB-SLAM3 [22], struggles.

To summarize, the contributions of our work include:

We present a new formulation of camera pose estimation as a weighted least-squares fit of an SE(3) pose to a 3D scene flow field obtained via differentiable rendering.

We combine our camera pose estimator with a multi-frame 3D reconstruction model, unlocking end-to-end, self-supervised training of camera pose estimation and 3D reconstruction.

We demonstrate that our method performs robustly across diverse real-world video datasets, including indoor, self-driving, and object-centric scenes.

2 Related Work

Generalizable Neural Scene Representations. Recent progress in neural fields [23-25] and differentiable rendering [10, 26-31] have enabled novel approaches to 3D reconstruction from few or single images [1-19], but require camera poses both at training and test time. An exception is recently proposed RUST [32], which can be trained for novel view synthesis without access to camera poses, but does not reconstruct 3D scenes explicitly and does not yield explicit control over camera poses. We propose a method that is similarly trained self-supervised on real video, but yields explicit camera poses and 3D scenes in the form of radiance fields. We outperform RUST on novel view synthesis and demonstrate strong out-of-distribution generalization by virtue of 3D structure.

SLAM and Structure-from-Motion (SfM). SfM methods [33-35], and in particular, COLMAP [34], are considered the de-facto standard approach to obtaining accurate geometry and camera poses from video. Recent progress on differentiable rendering has enabled joint estimation of radiance fields and camera poses via gradient descent [36-40], enabling subsequent high-quality novel view synthesis. Both approaches require offline per-scene optimization. In contrast, SLAM methods usually run online [22, 41, 42], but are notoriously unreliable on rotation-heavy trajectories or scenes with sparse visual features. Prior work proposes differentiable SLAM to learn priors over camera poses and geometry [43, 44], but requires ground-truth camera poses for training. Recent work has also explored how differentiable rendering may be directly combined with SLAM [45-48], usually using a conventional SLAM algorithm as a backbone and focusing on the single-scene overfitting case. We propose a fully self-supervised method to train generalizable neural scene representations without camera poses, outperforming prior work on generalizable novel view synthesis without camera poses. We do not claim state-of-the-art camera pose estimation, but provide an analysis of camera pose quality nevertheless, demonstrating robust performance on sequences that are challenging to state-of-the-art SLAM algorithms, ORB-SLAM3 [22] and Droid-SLAM [43].

Neural Depth and Camera Pose Estimation. Prior work has demonstrated joint self-supervised learning of camera pose and monocular depth [20, 21, 49-51] or multi-plane images [52]. These approaches leverage a neural network to directly regress camera poses with the primary goal of training high-quality monocular depth predictors. They are empirically limited to sequences with simple camera trajectories, such as self-driving datasets, and do not enable dense, large-baseline novel view synthesis. We ablate our flow-based camera pose estimation with a similar neural network-based approach. Most closely related to our work are approaches that infer per-timestep 3D voxel grids and train a CNN to regress frame-to-frame poses [53, 54]. We benchmark with the most recent approach in this line of work, Video Autoencoder [53]. Lastly, we strongly encourage the reader to peruse impressive concurrent work DBARF [55], which also regresses camera poses alongside a generalizable NeRF. Key differences are that we leverage a pose solver based on 3D-lifted optical

x

σ

Φ

x

σ

Φ

c

σ

Φ

Video

Stream

Ψ

Video Re-Rendering with

Estimated Camera Poses

Pose Estimation Module

Feature

Encoder

0

1

Corresponding 2D Features

Pose-Induced

Flow Loss

Video Re-

Rendering Loss

Confidence

Weights

Pose

Optical Flow: 2D Correspondence

2D      3D: Single-View Neural Scene Rep.

f1

f2

Feature

map

VolumeRendering

X

VolumeRendering

Y

↑

VolumeRendering

Pose-

Induced

Rendered

Re-

flow for real-time odometry versus predicting iterative updates to pose and depth via a neural network. Further, we extensively demonstrate our method's performance on rotation-dominant video sequences, in contrast to a focus on forward-facing scenes. Lastly, we solely rely on the generalizable scene representation in contrast to leveraging a monocular depth model for pose estimation.

3 Learning 3D Scene Representations from Unposed Videos

Our model learns to map a monocular video with frames { I t } N t =1 as well as off-the-shelf optical flow { V t } N -1 t =1 to per-frame camera poses { P t } N t =1 and a 3D scene representation Φ in a single feed-forward pass. We leverage known intrinsic parameters when available, but may predict them if not. We will first introduce the generalizable 3D scene representation Φ . We then discuss how we leverage Φ for feed-forward camera pose estimation, where we lift optical flow into 3D scene flow and solve for pose via a weighted least-squares SE(3) solver. Finally, we discuss how we obtain supervision for both the 3D scene representation and pose estimation by re-rendering RGB and optical flow for all frames. An overview of our method is presented in Fig. 1.

Notation. It will be convenient to treat images sometimes as discrete tensors, such as I t ∈ R H × W × 3 , and sometimes as functions I : R 2 → R 3 over 2D pixel coordinates p ∈ R 2 . We will denote functions in italic I , while we denote the corresponding tensors sampled on the pixel grid in bold as I .

3.1 Defining Our Image-Conditioned 3D Scene Representation

First, we introduce the generalizable 3D scene representation we aim to train. Our discussion assumes known camera poses; in the subsequent section we will describe how we can use our scene representation to estimate them instead. We parameterize our 3D scene as a Neural Radiance Field (NeRF) [28], such that Φ is a function that maps a 3D coordinate x to a color c and density σ as Φ( x ) = ( σ, c ) . To render the color for a ray r , points ( x 1 , x 2 , ..., x n ) are sampled along r between predefined near and far planes [ t 1 , t f ] , fed into Φ to produce corresponding color and density tuples [( σ 1 , c 1 ) , ( σ 2 , c 2 ) , ..., ( σ n , c n )] , and alpha-composited to produce a final color value C ( r ) :

CO3D Hydrant

CO3D 10-Category

RealEstate10K

KITTI

Context

Video

AE

Ours

Ours

Fine-tune

Target

RUST

where δ i = t i +1 -t i is the distance between adjacent samples. By compositing sample locations instead of colors, we can compute an expected ray-surface intersection point S ( r ) ∈ R 3 :

We require a generalizable NeRF that is not optimized for each scene separately, but instead predicted in a feed-forward pass by an encoder that takes a set of M context images and corresponding camera poses { ( I i , P i ) } i M as input. We denote such a generalizable radiance field reconstructed from images I i as Φ( x | { ( I i , P i ) } i M ) . Many such models have been proposed [1-10]. We base our model on pixelNeRF [1], which we briefly discuss in the following - please find further details in the supplement. pixelNeRF first extracts per-image features F i from each input image I i . A given 3D coordinate x is first projected onto the image plane of each context image I i via the known camera pose and intrinsic parameters to yield pixel coordinates p i . We then retrieve the features F i ( p i ) at that pixel coordinate. Color and density ( σ, c ) at x are then predicted by a neural network that takes as input the features { F i ( p i ) } i M and the coordinates of x in the coordinate frame of each camera, { P -1 i x } i M . Importantly, we can condition pixelNeRF on varying numbers of context images, i.e., we may run pixelNeRF with only a single context image as Φ( x | ( I , P )) , or with a set of M > 1 context images Φ( x | { ( I i , P i ) } i M ) .

3.2 Lifting Optical Flow to Scene Flow with Neural Scene Representations

Equipped with our generalizable 3D representation Φ , we now describe how we utilize it to lift optical flow into confidence-weighted 3D scene flow. Later, our pose solver will fit a camera pose to the

Video AE

Ours

Ours

BARF

Sample Frames

Sample Frames

20 frame Videos

50+ frame Videos

(a)

Hydrant

10-Cat.

KITTI

VidAE [53]

0.100

0.101

0.090

0.084

Ours

0.041

0.042

0.020

0.010

(b)

Top

Bot.

%Tracked

ORB3 [22]

0.021

0.034

49

DROID [43]

0.028

0.049

100

Ours

0.015

0.025

100

estimated scene flow. Given two sequential frames I t -1 , I t we first use an off-the-shelf method [56] to estimate backwards optical flow V t : R 2 → R 2 . The optical flow V t maps a 2D pixel coordinate p to a 2D flow vector, such that we can determine the corresponding pixel coordinate in I t -1 as p ′ = p + V t ( p ) .

We will now lift pixel coordinates p t and p t -1 to an estimate of the 3D points that they observe in the coordinate frame of their respective cameras. To achieve this, we cast rays from the camera centers through the corresponding pixel coordinates p t and p t -1 using the intrinsics matrix K . Specifically, we compute r t = K -1 ˜ p t and r t -1 = K -1 ˜ p t -1 , where ˜ p represents the homogeneous coordinate ( p 1 ) . Next, we sample points along the rays r t and r t -1 and query our pixelNeRF model in the single-view setting. This involves invoking Φ( ·| ( I t , I 4 × 4 )) and Φ( ·| ( I t -1 , I 4 × 4 )) , i.e., pixelNeRF is run with only the respective frame as the context view and the identity matrix I as the camera pose. Applying the ray-intersection integral defined in Eq. 2 to the pixelNeRF estimates, we obtain a pair of 3D points ( x t , x t -1 ) corresponding to the estimated surfaces observed by pixels p t and p t -1 , respectively. We repeat this estimation for all optical flow correspondences, resulting in two sets of surface point clouds, X , X ′ ∈ R H × W × 3 . Equivalently, we may view this as defining the 3D scene flow as X ′ -X .

Flow confidence weights. We further utilize a confidence weight for each flow correspondence. To accomplish this, we employ a neural network Ψ , which takes image features F t ( p ) , F t -1 ( p ′ ) as input for every pixel correspondence pair ( p , p ′ ) . The network maps these features to a weight w , denoted as Ψ( F t ( p ) , F t -1 ( p ′ )) = w ∈ [0 , 1] . Ψ can importantly overcome several failure modes which lead to faulty pose estimation, including incorrect optical flow, such as in areas of occlusions, dynamic objects, such as pedestrians, or challenging geometry estimates, such as sky regions. We show in Fig. 9 that Ψ indeed learns such content-based rules.

Predicting Intrinsic Camera Parameters K . Camera intrinsics are often approximately known, either published by the manufacturer, saved in video metadata, or calibrated once. Nevertheless, for purposes of large-scale training, we leverage a simple scheme to predict the camera field-of-view for a video sequence. We feed the feature map of the first frame F 0 into a convolutional encoder that directly regresses the field of view. We assume that the optical center is at the sensor center. We find that this approach enables us to train on real-world video from YouTube, though more sophisticated schemes [57] will no doubt improve performance further.

ORB-SLAM3

Ours

Sample Frames

RGB

Geometry

3.3 Camera Pose Estimation as Explaining the Scene Flow Field

We will now estimate the camera pose between frame I t and I t -1 . In the previous section, we lifted the input optical flow into scene flow, producing 3D correspondences X , X ′ , or, equivalently, 3D scene flow. We cast camera pose estimation as the problem of finding the rigid-body motion that best explains the observed scene flow field, or the transformation mapping points in X to X ′ , while considering confidence weights W . Note that below, we will refer to the matrices X , X ′ , and W as column vectors, with their spatial dimensions flattened.

We use a weighted Procrustes formulation to solve for the rigid transformation that best aligns the set of points X and X ′ . The standard orthogonal Procrustes algorithm solves for the SE(3) pose such that it minimizes the least squares error:

with P = ( R t 0 1 ) as a rigid-body pose with rotation R and translation t and homogeneous ˜ X = ( X 1 ) . In other words, the minimizer of this loss is the rigid-body transformation that best maps X onto X ′ , and, in a static scene, is therefore equivalent to the sought-after camera pose.

As noted by Choy et al. [58], this formulation equally weights all correspondences. As noted in the previous section, however, this would make our pose estimation algorithm susceptible to both incorrect correspondences as well as correspondences that should be down-weighted by nature of belonging to parts of the scene that are specular, dynamic, or have low confidence in their geometry estimate. Following [58], we thus minimize a weighted least-squares problem:

with the diagonal weight matrix W = diag ( w ) . Conveniently, this least-squares problem admits a closed-form solution, efficiently calculated via Singular Value Decomposition, as derived in [58]:

Composing frame-to-frame poses. Solving this weighted least-squares problem for each subsequent frame-to-frame pair yields camera transformations ( P ′ 2 , P ′ 3 , . . . , P ′ n ) , aligning each I t to its predecessor I t -1 . We compose frame-to-frame transformations to yield camera poses ( P 1 , P 2 , . . . , P n ) relative to the first frame, such that P 1 = I 3 × 3 , concluding our camera pose estimation module.

3.4 Supervision via Differentiable Video and Flow Re-Rendering

We have discussed our generalizable neural scene representation Φ and our camera pose estimation module. We will now discuss how we derive supervision to train both modules end-to-end. We have two primary loss signals: First, a photometric loss L RGB scores the visual fidelity of re-rendered video frames. Second, a pose-induced flow loss L pose scores how similar the flow induced by the

Context Views

Novel Views

Context Views

Novel Views

predicted camera transformations and surface estimations are to optical flow estimated by RAFT [59]. Our model is trained on short ( ∼ 15 frames) video sequences.

Photometric Loss. Our photometric loss L RGB comprises two terms: A multi-context loss and a single-context loss. The multi-context loss ensures that the full 3D scene is reconstructed accurately. Here, we re-render each frame of the input video sequence I t , using its estimated camera pose P t and multiple context images { ( I j , P j ) } J j . The single-context loss ensures that the single-context pixelNeRF used to estimate surface point clouds X t in the pose estimation module is accurate.

where, in a slight abuse of notation, we have overloaded the rendering function C ( P |{ ( I j , P j ) } J j ) defined in Eq. 1 as rendering out the full image obtained by rendering a pixelNeRF with context images { ( I j , P j ) } J j from camera pose P . N refers to the number of frames in the video sequence and J refers to the subset of context frames use for re-rendering the entire sequence. We first attempted picking the first frame only, however, found that this does not converge due to the uncertainty of the 3D scene given only the first frame: single-view pixelNeRF will generate blurry estimates for parts of the scene that have high uncertainty, such as occluded regions or previously unobserved background.

Pose-Induced Flow Loss. An additional, powerful source of supervision for both the estimated geometry and camera poses can be obtained by comparing the optical flow induced by the predicted surface point clouds and pose with the off-the-shelf optical flow. We define this pose-induced flow loss as

with projection operator π ( · ) and grid of pixel coordinates uv ∈ R 2 . Intuitively, this transforms the surface point cloud of frame t +1 into the coordinate frame of frame t and projects it onto that image plane. For every pixel coordinate p at timestep t +1 , this yields a corresponding pixel coordinate p ′ at timestep t , which we compare against the input optical flow.

3.5 Test-time Inference

After training our model on a large dataset of short video sequences, we may infer both camera poses and a radiance field of such a short sequence in a single forward pass, without test-time optimization.

Sliding Window Inference for Odometry on Longer Trajectories. Our method estimates poses for short ( ∼ 15 frames) subsequences in a single feed-forward pass. To handle longer trajectories that exceed the capacity of a single batch, we divide a given video into non-overlapping subsequences. We estimate poses for each subsequence individually and compose them to create an aggregated trajectory estimate. This approach allows us to estimate trajectories for longer video sequences.

Test-Time Adaptation. Frame-to-frame camera pose estimation methods, both conventional and the proposed method, accumulate pose estimation error over the course of a sequence. SLAM and SfM methods usually have a mechanism to correct for drift by globally optimizing over all poses and closing loops in the pose graph [60]. We do not have such a mechanism, but propose fine-tuning our model on specific scenes for more accurate feed-forward pose and 3D estimation. For a given video sequence, we may fine-tune our pre-trained model using our standard photometric and flow losses on sub-sequences of the video. Note that this is not equivalent to per-scene optimization or direct

Novel

Rendering

Context

Frames

GT Trajectory

BARF

TOFE

optimization of camera poses and a radiance field, as performed e.g. in BARF [38]: neither camera poses nor the radiance field are free variables. Instead, we fine-tune the weights of our convolutional inference backbone and MLP renderer for more accurate feed-forward prediction.

4 Experiments

We benchmark our method on generalizable novel view synthesis on the RealEstate10k [62], CO3D [63], and KITTI [64] datasets. We provide further analysis on the Tanks & Temples dataset and in-the-wild scenes from Ego4D [65] and YouTube. Though we do not claim to perform state-ofthe-art camera pose estimation, we nevertheless provide an analysis of the accuracy of our estimated camera poses. Please find more results, as well as precise hyperparameters, implementation, and dataset details, in the supplemental document and video. We utilize camera intrinsic parameters where available, predicting them only for the in-the-wild Ego4D and WalkingTours experiments.

Pose Estimation. Wefirst evaluate our method on pose estimation against the closest self-supervised neural network baseline, Video Autoencoder (VidAE) [53]. We then analyze the robustness of our pose estimation with ORB-SLAM3 [66] and DROID-SLAM [43] as references. Finally, we benchmark with BARF [38], a single-scene unposed NeRF baseline. Tab. 2a compares accuracy of estimated poses of our method and VidAE on all four datasets. The performance gap is most pronounced on the challenging CO3D dataset, but even on simpler, forward-moving datasets, RealEstate10k and KITTI, our method significantly outperforms VidAE. Next, we analyse the robustness of our pose estimation module on CO3D, using SfM methods ORB-SLAM3 and DROID-SLAM as references. See Tab. 2b and Fig. 4 for quantitative and qualitative results. To account for inaccuracies in the provided CO3D poses we utilize as ground-truth, we additionally report separate results for the top and bottom 50% of sequences, ranked based on the pose confidence scores provided by the authors. Although we do not employ any secondary pose method as a proxy ground truth for the bottom half of sequences, this division serves as an approximate indication of the level of difficulty each sequence poses from a SfM perspective. On both subsets, our method outperforms both DROID-SLAM and ORB-SLAM3. Also note that ORB-SLAM3 fails to track poses for over half (50.7%) of the sequences. On the sequences where ORB-SLAM3 succeeds, our method predicts poses significantly more accurately. Even on the sequences where ORB-SLAM3 fails, our performance does not degrade (.025 ATE). Lastly, we compare against the single-scene unposed NeRF baseline, BARF. Since BARF requires ∼ one day to reconstruct a single sequence, we evaluate on two representative sequences: a forward-walking sequence on RealEstate10K, and an outside-in trajectory on CO3D. We plot recovered trajectories in Fig. 3. While BARF fails to recover the correct trajectory shape on the CO3D scene, our method produces a trajectory that more accurately reflects the ground-truth looping structure.

Novel View Synthesis. We compare against VidAE [53] and RUST [32] on the task of novel view synthesis. Tab. 1 and Fig. 6 report quantitative and qualitative results respectively. Our method outperforms both baselines significantly. Since VidAE fails to capture the pose distribution on the CO3D datasets, its novel view renderings generally do not align with the ground truth. On RealEstate10K and KITTI, their method successfully captures the pose distribution, but still struggles

Sample

Frames

Novel View/

Novel Depth

Sample

Frames

Novel View/

Novel Depth

Input

RGB

Flow-Weight

Masked RGB

Input

RGB

Flow-Weight

Masked RGB

to render high-fidelity images. We similarly outperform RUST, which struggles to capture highfrequency content due to its coarse scene representation. Also note that RUST estimates latent camera poses rather than explicit ones. We further qualitatively evaluate our method on wide-baseline novel view synthesis, where no ground-truth is available - please see Fig. 6 for these results.

Test-Time Adaptation We evaluate the ability of our model to estimate camera poses on outof-distribution scenes, augmented by test-time optimization as discussed in section 3.5. Fig.7 displays the camera poses estimated by our method on the Bulldozer sequence from the Tanks and Temples dataset[61], using our model trained on the RealEstate10k dataset. Even without any test-time adaptation, our method generalizes impressively, generating a trajectory that follows the outside-in trajectory as determined by COLMAP. We refer to this generalization mode as 'zeroshot.' Nevertheless, camera poses are not perfect and exhibit some drift. We thus perform test-time optimization, fine-tuning the weights of our feed-forward method on subsequences of the video. Though geometry is plausible after roughly 10 minutes of fine-tuning, we continued optimization for 7 hours for finer details. Our scene-adapted estimates are close to ground truth with little drift, and we find that novel view synthesis results with significant baseline are visually compelling with sound depth estimates. We further show the result of BARF on this sequence, which fails to recover a plausible trajectory.

Results on In-the-Wild Video. We present preliminary results on the Ego4D dataset [65], a recent ego-centric dataset captured with headset cameras, and a new Walking Tours dataset, in which we stream walking tour videos from YouTube. Both datasets comprise some scene motion, such as pedestrians, cars, or people. Images from Ego4D are further subject to radial camera distortion, which we do not model. Nevertheless, after fine-tuning on Walking Tours and Ego4D for 20 minutes and 1.5 hours, we can generate plausible novel views and depth maps, illustrated in Fig. 8. We find that our model is generally robust to dynamic scene content, such as pedestrians or humans, which obtain low flow-confidence flow scores. We further run COLMAP on a subset of the Walking Tours and Ego4D videos, yielding pseudo ground-truth poses for 3 videos from each dataset. The Walking Tour subset is selected randomly, and the Ego4D subset is chosen as the first three relatively-static videos we found (to ensure COLMAP convergence). We divide each video into subsequences of 20 frames at 3fps, corresponding to roughly 7 meters of forward translation per clip. Here, we achieve a competitive ATE of 0.013 and 0.026 on Walking Tours and Ego4D, respectively. We also compute PSNR values on subsequences of 10 frames using two context views, and obtain 18.87dB and 21.59db on Walking Tours and Ego4D, respectively, indicating generally plausible novel view synthesis.

Ablations. In Tab. 3, we ablate key contributions related to our pose formulation, evaluated on the CO3D Hydrant dataset. We first ablate our proposed flow-based pose formulation in favor of concatenating adjacent frames and predicting a pose directly via a CNN, as is common in the Monodepth [67] line of work. We further ablate our weighted flow formulation in favor of a non-weighted

Procrustes estimation. Both methods perform significantly worse than our full flow-based pose formulation, and qualitatively often lead to degenerate geometry estimates. Please see the supplemental material for additional ablations and ablation details.

5 Discussion

Limitations. While we believe our method makes significant strides, it still has several limitations. As an odometry method, it accumulates drift and has no loop closure mechanism. Our model further does not currently incorporate scene dynamics, but recent advancements in dynamic NeRF papers [68-70] present promising perspectives for future research.

Conclusion. We have introduced FlowCam, a model capable of regressing camera poses and reconstructing a 3D scene in a single forward pass from a short video. Our key contribution is to factorize camera pose estimation as first lifting optical flow to pixel-aligned scene flow via differentiable rendering, and then solving for camera pose via a robust least-squares solver. We demonstrate the efficacy of our approach on a variety of challenging real-world datasets, as well as in-the-wild videos. We believe that they represent a significant step towards enabling scene representation learning on uncurated, real-world video.

Acknowledgements. This work was supported by the National Science Foundation under Grant No. 2211259, by the Singapore DSTA under DST00OECI20300823 (New Representations for Vision), and by the Amazon Science Hub. The Toyota Research Institute also partially supported this work. This article solely reflects the opinions and conclusions of its authors and no other entity.

References

Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelNeRF: Neural radiance fields from one or few images. In Proc. CVPR , 2021.

Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In Proc. CVPR , 2020.

Yilun Du, Cameron Smith, Ayush Tewari, and Vincent Sitzmann. Learning to render novel views from wide-baseline stereo pairs. In Proc. CVPR , 2023.

Alex Trevithick and Bo Yang. Grf: Learning a general radiance field for 3d representation and rendering. In Proc. ICCV , pages 15182-15192, 2021.

Mohammed Suhail, Carlos Esteves, Leonid Sigal, and Ameesh Makadia. Generalizable patch-based neural rendering. In European Conference on Computer Vision . Springer, 2022.

Julian Chibane, Aayush Bansal, Verica Lazova, and Gerard Pons-Moll. Stereo radiance fields (srf): Learning view synthesis for sparse views of novel scenes. In Proc. CVPR , pages 7911-7920, 2021.

Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan T. Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi-view image-based rendering. In CVPR , 2021.

Eric R Chan, Koki Nagano, Matthew A Chan, Alexander W Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein. Generative novel view synthesis with 3d-aware diffusion models. arXiv preprint arXiv:2304.02602 , 2023.

Zhizhuo Zhou and Shubham Tulsiani. Sparsefusion: Distilling view-conditioned diffusion for 3d reconstruction. Proc. CVPR , 2022.

Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3d-structure-aware neural scene representations. In Proc. NeurIPS , volume 32, 2019.

Philipp Henzler, Jeremy Reizenstein, Patrick Labatut, Roman Shapovalov, Tobias Ritschel, Andrea Vedaldi, and David Novotny. Unsupervised learning of 3d object categories from videos in the wild. In Proc. CVPR , pages 4700-4709, 2021.

Prafull Sharma, Ayush Tewari, Yilun Du, Sergey Zakharov, Rares Andrei Ambrus, Adrien Gaidon, William T Freeman, Fredo Durand, Joshua B Tenenbaum, and Vincent Sitzmann. Neural groundplans: Persistent neural scene representations from a single image. In Proc. ICLR .

Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering. In Proc. NeurIPS , 2021.

Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In Proc. ICCV , pages 14124-14133, 2021.

Mehdi SM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lucic, Daniel Duckworth, Alexey Dosovitskiy, et al. Scene representation transformer: Geometryfree novel view synthesis through set-latent scene representations. arXiv preprint arXiv:2111.13152 , 2021.

Shamit Lal, Mihir Prabhudesai, Ishita Mediratta, Adam W Harley, and Katerina Fragkiadaki. Coconets: Continuous contrastive 3d scene representations. In Proc. CVPR , pages 12487-12496, 2021.

Jonáš Kulhánek, Erik Derner, Torsten Sattler, and Robert Babuška. Viewformer: Nerf-free neural rendering from few images using transformers. In European Conference on Computer Vision (ECCV) , 2022.

Jiaxin Li, Zijian Feng, Qi She, Henghui Ding, Changhu Wang, and Gim Hee Lee. MINE: Towards continuous depth MPI with NeRF for novel view synthesis. In Proc. ICCV , pages 12578-12588, 2021.

Kai-En Lin, Lin Yen-Chen, Wei-Sheng Lai, Tsung-Yi Lin, Yi-Chang Shih, and Ravi Ramamoorthi. Vision transformer for nerf-based view synthesis from a single input image. arXiv preprint arXiv:2207.05736 , 2022.

Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow. Unsupervised monocular depth estimation with left-right consistency. In Proc. CVPR , pages 270-279, 2017.

Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel J Brostow. Digging into self-supervised monocular depth estimation. In CVPR , 2019.

Carlos Campos, Richard Elvira, Juan J Gómez Rodríguez, José MM Montiel, and Juan D Tardós. Orbslam3: An accurate open-source library for visual, visual-inertial, and multimap slam. IEEE Transactions on Robotics , 37(6):1874-1890, 2021.

Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. Neural fields in visual computing and beyond. In Proc. EUROGRAPHICS STAR , 2022.

Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In Proc. NeurIPS , 2020.

Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Proc. NeurIPS , 2020.

Thu H Nguyen-Phuoc, Chuan Li, Stephen Balaban, and Yongliang Yang. Rendernet: A deep convolutional network for differentiable rendering from 3d shapes. In Proc. NeurIPS , volume 31, 2018.

Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhöfer. Deepvoxels: Learning persistent 3d feature embeddings. In Proc. CVPR , 2019.

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In Proc. ECCV , pages 405-421, 2020.

Philipp Henzler, Niloy J Mitra, and Tobias Ritschel. Escaping plato's cave: 3d shape from adversarial rendering. In Proc. ICCV , pages 9984-9993, 2019.

Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Brualla, Tomas Simon, Jason Saragih, Matthias Nießner, et al. State of the art on neural rendering. In Computer Graphics Forum , volume 39, pages 701-727. Wiley Online Library, 2020.

Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. Proc. NeurIPS , 33: 2492-2502, 2020.

Mehdi S. M. Sajjadi, Aravindh Mahendran, Thomas Kipf, Etienne Pot, Daniel Duckworth, Mario Luˇ ci´ c, and Klaus Greff. RUST: Latent Neural Scene Representations from Unposed Imagery. CVPR , 2023.

Richard Hartley and Andrew Zisserman. Multiple view geometry in computer vision . Cambridge university press, 2003.

Johannes L Schonberger and Jan-Michael Frahm. Structure-from-motion revisited. In Proc. CVPR , 2016.

Sameer Agarwal, Yasutaka Furukawa, Noah Snavely, Ian Simon, Brian Curless, Steven M Seitz, and Richard Szeliski. Building rome in a day. Communications of the ACM , 54(10):105-112, 2011.

Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. Nerf-: Neural radiance fields without known camera parameters. arXiv preprint arXiv:2102.07064 , 2021.

Lin Yen-Chen, Pete Florence, Jonathan T Barron, Alberto Rodriguez, Phillip Isola, and Tsung-Yi Lin. inerf: Inverting neural radiance fields for pose estimation. In Proc. IROS , pages 1323-1330. IEEE, 2021.

Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey. Barf: Bundle-adjusting neural radiance fields. In Proc. CVPR , 2021.

Axel Levy, Mark Matthews, Matan Sela, Gordon Wetzstein, and Dmitry Lagun. Melon: Nerf with unposed images using equivalence class estimation. arXiv preprint arXiv:2303.08096 , 2023.

Wenjing Bian, Zirui Wang, Kejie Li, Jiawang Bian, and Victor Adrian Prisacariu. Nope-nerf: Optimising neural radiance field with no pose prior. 2023.

Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos. Orb-slam: a versatile and accurate monocular slam system. 2015.

Jakob Engel, Thomas Schöps, and Daniel Cremers. Lsd-slam: Large-scale direct monocular slam. In Proc. ECCV , 2014.

Zachary Teed and Jia Deng. Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras. 2021.

Krishna Murthy Jatavallabhula, Ganesh Iyer, and Liam Paull. Grad-slam: Dense slam meets automatic differentiation. In Proc. ICRA . IEEE, 2020.

Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R Oswald, and Marc Pollefeys. Nice-slam: Neural implicit scalable encoding for slam. In Proc. CVPR , pages 12786-12796, 2022.

Antoni Rosinol, John J Leonard, and Luca Carlone. Nerf-slam: Real-time dense monocular slam with neural radiance fields. arXiv preprint arXiv:2210.13641 , 2022.

Chi-Ming Chung, Yang-Che Tseng, Ya-Ching Hsu, Xiang-Qian Shi, Yun-Hung Hua, Jia-Fong Yeh, WenChin Chen, Yi-Ting Chen, and Winston H Hsu. Orbeez-slam: A real-time monocular visual slam with orb features and nerf-realized mapping. arXiv preprint arXiv:2209.13274 , 2022.

Edgar Sucar, Shikun Liu, Joseph Ortiz, and Andrew J Davison. imap: Implicit mapping and positioning in real-time. In Proc. ICCV , pages 6229-6238, 2021.

Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe. Unsupervised learning of depth and ego-motion from video. In Proc. CVPR , 2017.

Zhichao Yin and Jianping Shi. Geonet: Unsupervised learning of dense depth, optical flow and camera pose. In Proc. CVPR , 2018.

Vitor Guizilini, Kuan-Hui Lee, Rare¸ s Ambru¸ s, and Adrien Gaidon. Learning optical flow, depth, and scene flow without real-world labels. Proc. ICRA , 2022.

Yang Fu, Ishan Misra, and Xiaolong Wang. Multiplane nerf-supervised disentanglement of depth and camera pose from videos. arXiv preprint arXiv:2210.07181 , 2022.

Zihang Lai, Sifei Liu, Alexei A Efros, and Xiaolong Wang. Video autoencoder: self-supervised disentanglement of static 3d structure and motion. In Proc. ICCV , 2021.

Hsiao-Yu Fish Tung, Ricson Cheng, and Katerina Fragkiadaki. Learning spatial common sense with geometry-aware recurrent networks. In Proc. CVPR , 2019.

Yu Chen and Gim Hee Lee. Dbarf: Deep bundle-adjusting generalizable neural radiance fields. arXiv preprint arXiv:2303.14478 , 2023.

Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Proc. CVPR , pages 402-419. Springer, 2020.

Linyi Jin, Jianming Zhang, Yannick Hold-Geoffroy, Oliver Wang, Kevin Matzen, Matthew Sticha, and David F. Fouhey. Perspective fields for single image camera calibration. Proc. CVPR , 2023.

Christopher Choy, Wei Dong, and Vladlen Koltun. Deep global registration. In Proc. CVPR , 2020.

Zachary Teed and Jia Deng. RAFT: Recurrent all-pairs field transforms for optical flow. In Proc. ECCV , 2020.

Johannes Kopf, Xuejian Rong, and Jia-Bin Huang. Robust consistent video depth estimation. In Proc. CVPR , pages 1611-1621, 2021.

Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun. Tanks and temples: Benchmarking large-scale scene reconstruction. Proc. TOG , 36(4), 2017.

Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. Proc. TOG , 2018.

Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In Proc. ICCV , pages 10901-10911, 2021.

Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? The KITTI vision benchmark suite. In Proc. CVPR , 2012.

Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proc. CVPR , pages 18995-19012, 2022.

Carlos Campos, Richard Elvira, Juan J. Gomez Rodriguez, Jose M. M. Montiel, and Juan D. Tardos. ORB-SLAM3: An accurate open-source library for visual, visual-inertial, and multimap SLAM. IEEE Transactions on Robotics , pages 1874-1890, dec 2021.

Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel J. Brostow. Digging into self-supervised monocular depth prediction. In Proc. ICCV , 2019.

Yu-Lun Liu, Chen Gao, Andreas Meuleman, Hung-Yu Tseng, Ayush Saraf, Changil Kim, Yung-Yu Chuang, Johannes Kopf, and Jia-Bin Huang. Robust dynamic radiance fields. In Proc. CVPR , 2023.

Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In Proc. CVPR , 2021.

Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. Proc. ICCV , 2021.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a generalizable framework for estimating both the NeRF model of the scene and the camera pose for video sequence simultiunusly.\n\nThe proposed method first uses the pretrained model to predict the optical flow for each video frame. Then, the optical flow is lifted up to the monodepth to scene flow. The camera poses are solved by SVD with scene flow. After computing the camera pose, the nerf model is further fine-tuned for a better result.\n\nSTRENGTHS:\n- a light step of unposed nerf in RGB video sequence setting.\n- the quality of the rendered RGB image looks good after fine-tune.\n- well written, easy to read and follow.\n\nWEAKNESSES:\n- Statement\n  - Line 27-31\n    - Although the manuscript does acknowledge certain NeRF-based SLAM methods for their ability to provide dense 3D reconstruction, relevant references [1,2,3] are conspicuously absent. Additionally, some NeRF-based SLAM methods that utilize prior for pose estimation have been glossed over.\n  - Line 34\n    - The cited works do not adequately represent the body of literature since both methods focus exclusively on depth map estimation. This oversight should be rectified by including relevant citations [4,5].\n - Line 143\n   - wrong format\n\n\n- Motivation\n  - The motivation for introducing single view generalizable nerf for relative pose estimation is unclear and wired.\n    - Line 142\n    - the definition of the x_t. In equ2, x_i is the sampled points along the ray. Does x_t is the S(r) in equ2?\n    - while pixelnerf can use only a single view to render a novel view, it is much like a mono-depth estimation module. Without optical flow, an alternating solution uses iNeRF to align two single views point cloud/nerf, minimizing photometric loss. However, even most SOTA mono-depth estimation methods are insufficient to estimate accurate relative pose due to cross-view scale misalignment. Some unposed/posed Nerf only use mono-depth as a reg term. This is why the proposed method **HAS TO** introduce the out-of-shelf optical flow estimation for pose estimation. If the single view pixelnerf is removed from the proposed method, optical flow, as the correspondences, is enough to estimate the essential matrix/relative pose between two neighbor frames. Furthermore, the deep-sfm module, like banet, droidslam, deepv3d, all of these methods take multiview images to estimate both multiview depth map and the camera pose at the same time. The author should carefully address this issue during the rebuttal. \n  - the motivation of flow confidence weights.\n    - Line 143\n    - This part could be easily solved by bi-direction consistency check with optical flow. Introducing another pretrained network for weight prediction will make the generalization issue worse.\n    - Figure 9 does not show the results with dynamic objects.\n\n  - Line 178\n    - RGB VO/SLAM confronts scale ambiguity problem. The proposed method doesn't handle it at all.\n\n  - Line 194\n    - where is the definition of N, J?\n\n  - Line 203 Equ8\n  - The mono-depth is enforced by this loss term during fine-tune. As mentioned above, the optical flow for relative pose estimation is enough.\n  - This loss term does not handle singular solutions.\n\n- Experiment\n  - Line 211\n    - the training detail is missing.\n    - the proposed method cannot run on a long video clip.\n    - Most of the experiments show the object-center case. It will be great to see the results on the inside-out case either.\n  - Although the authors mention that they do not claim sota accuracy of pose estimation, the metrics of the pose are still necessary to be reported on the main paper, including droid slam, droid vo, orb slam, etc.\n  - Line 224\n    - test-time optimization on the weight of CNN and MLP on a single video clip in a self-supervised manner is extremely hard. The problem will become worse if the precomputed optical flow is removed. The author should provide more detail about this section. \n\n  - Line 250\n    - more baseline methods are required, such as nicer-slam[2], dim-slam[3], droid-slam.\n    - some unposed nerf methods with mono depth estimation module, such as nope-nerf[6]\n    - and another test-time optization method, such as CVD[4], RCVD[5].\n\n  - The pose trajectories in the figure are misaligned in both the main paper and the supplementary, which are very confusing.\n\n- reference\n  - [1] Sucar, E., Liu, S., Ortiz, J., & Davison, A. J. (2021). iMAP: Implicit mapping and positioning in real-time. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 6229-6238).\n  - [2] Zhu, Z., Peng, S., Larsson, V., Cui, Z., Oswald, M. R., Geiger, A., & Pollefeys, M. (2023). Nicer-slam: Neural implicit scene encoding for rgb slam. arXiv preprint arXiv:2302.03594.\n  - [3] Li, H., Gu, X., Yuan, W., Yang, L., Dong, Z., & Tan, P. (2023). Dense RGB SLAM with Neural Implicit Maps. The Eleventh International Conference on Learning Representations(ICLR)\n  - [4] Luo, X., Huang, J. B., Szeliski, R., Matzen, K., & Kopf, J. (2020). Consistent video depth estimation. ACM Transactions on Graphics (ToG), 39(4), 71-1.\n  - [5] Kopf, J., Rong, X., & Huang, J. B. (2021). Robust consistent video depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 1611-1621).\n  - [6] Bian, W., Wang, Z., Li, K., Bian, J. W., & Prisacariu, V. A. (2023). Nope-nerf: Optimising neural radiance field with no pose prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 4160-4169).\n\nQUESTIONS:\nI am happy to adjust the rating if the author properly addresses the concerns mentioned during the rebuttal.\n\n----\n\nafter rebuttal:\n\nI adjust the rate to the positive side.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-29"}

comments: SUMMARY:
This paper proposes a method to address the challenge of reconstructing 3D neural fields from images and learns them in a self-supervised manner. The main contribution of this method is the joint reconstruction of camera poses and 3D neural scene representations within a single forward pass.

STRENGTHS:
1. This paper achieves state-of-the-art performance in its specific setting.
2. The writing in this paper is clear. 
3. This paper addresses an important problem. It overcomes the previous limitation in training generalized Neural Radiance Fields that require posed videos. It enables direct training on unposed video data.
4. This paper is innovative in its methodology and no one has utilized similar approaches to address this problem before.

WEAKNESSES:
1. The paper lacks clarity in explaining the ablation study, such as the incomplete caption for Table 3. Moreover, the abbreviations used in the table are not defined, making it unclear what they refer to.
2. The paper lacks ablation experiments on not learning intrinsic camera parameters.

QUESTIONS:
1. I cannot understand how this paper addresses the generalization issue. Why does PixelNeRF have good generalization capabilities in monocular depth prediction?
2. How does this paper address the problem of scale ambiguity in monocular depth estimation?

LIMITATIONS:
This paper discusses its limitations. However, I would have expected to see more failure cases from internet data.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a general method for 3D neural scene reconstruction and camera pose estimation from a video sequence. The method takes a set of video frames as input and outputs the re-rendered video frames and estimated camera poses. The method is based on PixelNeRF, which is a general NeRF method that takes extracted image features as input.\n\nThe method first uses single-view PixelNeRF to compute per-frame point clouds. Then, the frame-to-frame 3D scene flow is estimated by leveraging the estimated 2D optical flow. The 3D scene flow is then used to constrain the frame-to-frame camera poses, which are solved by minimizing a weighted least-squares problem. Two losses are used to supervise the training: the RGB loss, which forces the PixelNeRF-based rendered images to be close to the input images; and the flow loss, which makes the projection offset of neighbor frames' point clouds close to the 2D optical flow.\n\nSTRENGTHS:\n1. A nice solution to learn a general NeRF representation for multiview images without camer pose information.\n2. The quality of the rendering results for testing multi-view images are impressive.\n\nWEAKNESSES:\nThe experiments can be improved to further verified the advantage of the proposed method:\n\n1. Only one quantitative pose estimation comparison is shown, there is no quantitative comparisons with BARF. Besides, Nope-NeRF is a similar method that takes sequent frames as input for pose estimation, it better have one more comparison with Nope-NeRF.\n\n2. The quantitative results in Tab 1 are fine-tuned or not? It should be better to show both results.\n\nQUESTIONS:\nOverall,  the proposed method is reasonable, but I do have questions:\n\n1. There should be a scale problem when estimating the per-view point cloud by single-view pixelNeRF. It seems that the Eq (4) didn't consider it, why?\n\n2. The confidence weight is output by a network. Therefore, in my understanding, this network can implicitly handle the occlusion and specular by downweighting corresponding flow. But it seems that the Masked RGB images shown in Fig 9 mask out too much content (even for textured diffuse regions). Is this because of the inaccurate 2D optical flow estimation?\n\nLIMITATIONS:\nThe proposed method learns a conditioned generaI NeRF network for multiview images. It is still limited since the images should be kept to accompany the learned model.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes using scene flow to optimize for camera poses to produce generalizable 3D radiance field. The key contribution is the joint optimization of the camera poses and 3D neural scene representations in an single forward pass. The method has been evaluated on multiple datasets and performs well on datasets where traditional pose estimation fails.\n\nSTRENGTHS:\nThe motivation for using an joint optimization of camera poses and reconstruction has wide applications and easy to scale.\n\nAs proposed by the current method multiple views of a scene inherently produce good correspondences, which leads to reasonable optical flow estimates. Using and incorporating optical flow into the 3D pose estimation and reconstruction helps overall reconstruction pipeline.\n\nUsing optical flow for reconstruction and Neural radiance fields without cameras poses have been well independently. Combining them into a single formulation is still less explored. the current method lifts the optical flow to scene flow using neural scene representations and uses them to optimize. This is interesting direction to explore.\n\nThe results shows that the method is able to perform better than traditional pose estimation methods like ORB-SLAM in some of their failure cases.\n\nWEAKNESSES:\nThe current method proposes a tradeoff by stating instead of using camera poses first and neural radiance field later as a two stage process, joint optimization is a more promising direction. but using joint optimization adds additional optimization constraints which have not been extensively discussed. It would be nice to see what are the failure cases of using the joint optimization compared to the two-stage optimization process.\n\nThe camera pose estimation has only been evaluated with ORB-SLAM3 or VidAE both of which do not incorporate neural radiance fields for 3D pose estimation. Recently there have been better methods which optimize for both radiance fields and pose simultaneously. having such baseline will give a better understanding of the accuracy of the algorithm.\n\nThe number of frames used for error computation is very small. i.e. 20 frames and 200 frames. The advantage of the method states that the joint optimization is beneficial to scale easily. Showing the method accuracy compared to orb on larger datasets or sequences like\n\nQUESTIONS:\nin Fig3 results were shown using BARF for pose estimation, why are there no quantitative results using the same?\n\nLIMITATIONS:\nLimitations have been well discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}