Title: UP-NeRF: Unconstrained Pose Prior-Free Neural Radiance Field
Venue: NeurIPS 2023
Accepted: yes
Keywords: neural radiance field

pose estimation
Abstract: Neural Radiance Field (NeRF) has enabled novel view synthesis with high fidelity given images and camera poses. Subsequent works even succeeded in eliminating the necessity of pose priors by jointly optimizing NeRF and camera pose. However, these works are limited to relatively simple settings such as photometrically consistent and occluder-free image collections or a sequence of images from a video. So they have difficulty handling unconstrained images with varying illumination and transient occluders. In this paper, we propose **UP-NeRF** (**U**nconstrained **P**ose-prior-free **Ne**ural **R**adiance **F**ields) to optimize NeRF with unconstrained image collections without camera pose prior. We tackle these challenges with surrogate tasks that optimize color-insensitive feature fields and a separate module for transient occluders to block their influence on pose estimation. In addition, we introduce a candidate head to enable more robust pose estimation and transient-aware depth supervision to minimize the effect of incorrect prior. Our experiments verify the superior performance of our method compared to the baselines including BARF and its variants in a challenging internet photo collection, *Phototourism dataset*. The code of UP-NeRF is available at https://github.com/mlvlab/UP-NeRF.

UP-NeRF: Unconstrained Pose-Prior-Free Neural Radiance Fields

Abstract

Neural Radiance Field (NeRF) has enabled novel view synthesis with high fidelity given images and camera poses. Subsequent works even succeeded in eliminating the necessity of pose priors by jointly optimizing NeRF and camera pose. However, these works are limited to relatively simple settings such as photometrically consistent and occluder-free image collections or a sequence of images from a video. So they have difficulty handling unconstrained images with varying illumination and transient occluders. In this paper, we propose UP-NeRF ( U nconstrained P ose-prior-free Ne ural R adiance F ields) to optimize NeRF with unconstrained image collections without camera pose prior. We tackle these challenges with surrogate tasks that optimize color-insensitive feature fields and a separate module for transient occluders to block their influence on pose estimation. In addition, we introduce a candidate head to enable more robust pose estimation and transient-aware depth supervision to minimize the effect of incorrect prior. Our experiments verify the superior performance of our method compared to the baselines including BARF and its variants in a challenging internet photo collection, Phototourism dataset. The code of UP-NeRF is available at https://github.com/mlvlab/UP-NeRF .

1 Introduction

Neural Radiance Fields (NeRF) [1] opened up a new chapter in novel view synthesis by demonstrating its capacity to generate high-quality novel view images given only a set of 2D images. Its powerful performance has enabled many practical applications including virtual/augmented reality (VR/AR) [2, 3], autonomous systems [4, 5], and robotics [6-8]. Various follow-up studies have developed NeRF by addressing its limitations, such as the necessity of dense input views [9-11] and long training time [12-17]. Also, there have been efforts [18, 19] to deal with unconstrained images with varying illuminations and transient occluders to relieve the burden of collecting clean images or establishing a specialized setting to capture them. For instance, NeRF-W [18] introduces an appearance embedding to handle varying illumination and a transient head to filter transient occluders.

One of the popular research topics in NeRF is joint optimization for pose and neural scene representation. Originally, NeRF requires accurate camera pose priors which are generally obtained with classic methods like structure from motion (SfM) [20]. Although the poses obtained from such a method are treated as ground truth in many works, it does not always give optimal results or even fails to converge. Hence recent works [21-23] have proposed a method so-called unposed-NeRF that simultaneously trains camera poses and NeRF. This eliminates the necessity of pose priors and burdensome preprocessing. While these methods have only succeeded in forward-facing scenes, NoPe-NeRF [24] further improved the method to even outdoor scenes with continuous images from videos. These previous works, however, are based on the photometric loss which is not reliable for

Surrogate

Color

Feature

X

X

O

unconstrained images, so they have difficulty in training with unconstrained images. In addition, the unconstrained images have complex camera poses, which is hard to optimize if a prior like continuity of images from a video is not available. Therefore, training unposed-NeRF with unconstrained images causes contradiction: To estimate camera poses accurately, images must be photometrically consistent without transient occluders; to treat unconstrained images without photometric consistency and transient occluders, accurate camera poses are needed. Previous unposed-NeRFs in the literature are not capable of learning in an unconstrained setting, leading to suboptimal pose estimation and poor rendering quality.

To tackle the problem, we propose UP-NeRF ( U nconstrained P ose-prior-free Ne ural R adiance F ield) that jointly optimizes pose and neural scene representations while minimizing the influence of photometric inconsistency and transient occluders on pose estimation. Our contributions are as follows: First, we introduce the novel architecture that enables robust pose optimization with images of complex camera poses. Second, we propose feature-surrogate bundle adjustment by adopting the deep feature as a descriptor. Third, we enable transient-free pose optimization with the isolated network. Lastly, our transient-aware depth loss suggests an effective way to impose depth prior only to the static objects. Consequently, our model enables successful training of unposed-NeRF with unconstrained images, showing its effectiveness over a naive combination of current methods.

2 Related Work

Neural Radiance Fields (NeRF). With the advent of techniques adopting neural fields to reconstruct 3D representations [25-29], Neural Radiance Field (NeRF) [1] proposed by Mildenhall et al. showed a promising result without the requirement of exact 3D geometry of each object in the scene. Compared to previous approaches which require a pre-defined representation of the scene by meshes [30, 31], voxels [25, 26], or point clouds [27, 32], NeRF only requires a set of densely captured 2D images and corresponding camera parameters of each image. Despite its light requirements, it demonstrates quality reconstruction of the complex geometry of scenes. The key idea of NeRF is that even though given are only 2D images, an implicit function represented by a multilayer perceptron can be utilized to parameterize the color and density of each 3D point on a ray, which is subsequently synthesized to decide the color of a pixel from which the ray was shot. Thanks to its simple but effective architecture, many other researches [10, 12-14, 33-35] have followed the trail of NeRF to ameliorate its weakness and improve performance.

Joint optimization of camera pose. Among a lot of interesting NeRF-based works, one major group of following works deals with the requirement of accurate camera pose of each image. Accurate camera pose including intrinsic and extrinsic parameters of a camera is indispensable for NeRF in that sampling accurate 3D coordinates in the world space heavily depends on them. However, such an accurate camera pose is not easily equipped unless the dataset is custom-made like a synthetic dataset, so structure from motion (SfM) [36] has been a classical solution to obtain such parameters. Although a lot of works have postulated camera poses from SfM to be ground truth, they can be sub-optimal depending on properties of scenes like monotonous texture. To solve this problem, other methods [22-24, 37-40] propose joint optimization of a neural field and poses.

Expansion to unconstrained scenes. Several researches [18, 19] dealt with unconstrained outdoor images where color consistency is not guaranteed owing to various environments (sunshine, time, weather, etc.) and so-called transient entities (pedestrian, cars, sundries, etc.) appear in images.

S-NeRF [41] and CF-NeRF [42] proposed a Bayesian network and normalizing flow for measuring the uncertainty of transient objects. D 2 NeRF [43] and RobustNeRF [44] demonstrated that transient objects can be sharply decoupled from static ones without blurry floaters. Some works [45, 46] utilized mask supervision to jointly optimize camera pose with object-centric images of different illumination. In contrast to previous works which require either color consistency or the absence of transient objects to jointly optimize the pose, our method is capable of optimizing camera pose without both requirements.

3 Method

We present a framework dubbed UP-NeRF to optimize unposed-NeRF under challenging conditions. We first briefly summarize Neural Radiance Fields (NeRF) [1] followed by joint optimization of camera poses and NeRF. In Section 3.1, we propose a new architecture that handles different convergence rates of camera poses, which can enable robust training of unposed-NeRF. Next, we explain our methods in Section 3.2 and 3.3 that handle the difficulty of unconstrained image collections like different light conditions ( e.g ., weather, time, etc.) and transient occluders. In addition, we will discuss how monocular depth priors can be used to optimize unposed-NeRF with unconstrained images in Section 3.4.

Neural Radiance Fields (NeRF). Given N images { I i } N i =1 and camera parameters, Neural Radiance Field (NeRF) Π θ learns a continuous volumetric radiance field which maps a 3D point x = ( x, y, z ) and a viewing direction d = ( d x , d y , d z ) to volume density σ and color c = ( r, g, b ) . The mapping functions parameterized by simple neural networks, e.g ., multi-layer perceptrons (MLPs), are defined as

where γ x ( · ) is a positional encoding function for 3D coordinates of each sample point on the ray r ( t ) = o + t d , which is parameterized by the origin point o and the direction d . Color value c ( t ) is encoded with an intermediate feature z ( t ) and a positional encoding function for view directions γ d ( · ) . The synthesized RGB value ˆ C ( r ) is calculated by volume rendering as

where δ k = t k +1 -t k is the distance between adjacent points for approximated rendering via quadrature on discrete points and α ( t k ) = 1 -exp( -σ ( t k ) δ k ) . To improve sampling efficiency, NeRF employs coarse and fine models for hierarchical sampling. The coarse model is optimized using stratified sampling, and then the fine model is optimized with samples biased toward the relevant parts of the volume. The parameters are optimized by minimizing photometric loss L rgb = ∑ ∥ C ( r ) -ˆ C ( r ) ∥ 2 2 with given ground-truth color C ( r ) . Note that even if it is not a color map, any information that represents the scene can be used for training a neural field. In this paper, we exploit deep feature map F as well as color map C for training, which have been studied by Kobayashi et al. [47] to learn selectively editable neural fields.

NeRF with camera pose estimation. Some prior works optimize NeRF with a joint estimation of camera pose. We parameterize the camera poses { p i } N i =1 as 6 degrees of freedom and assume that the camera intrinsics are known as several prior works [23, 24, 39, 48]. Then, the problem can be formulated as

where ˆ p is learnable camera pose. BARF [23] shows that a coarse-to-fine positional encoding strategy facilitates more stable joint optimization of NeRF and camera pose. Following BARF, we parameterize camera poses with the se (3) Lie algebra and use a coarse-to-fine strategy for pose optimization.

3.1 Candidate head for robust pose optimization

Training unposed-NeRF is a joint optimization process with a chick-and-egg problem; estimation of camera pose ˆ p requires accurate scene representation Π θ and vice versa. During the initial phase of joint training, we observed that inaccurate scene representation, which is represented by predicted density σ and color c in Eq. (1), causes specific images to be optimized in the wrong poses. We refer to these images as 'hard-pose' images, which may include images that capture the details of the scene or have little overlap with other images like the image in Fig. 4. To mitigate this problem, we introduce an additional MLP head called candidate head . It yields image-dependent (and tentative) scene representations by density σ ( c ) i and color c ( c ) i given candidate embeddings { ℓ ( c ) i } N i =1 as:

Note that ( · ) ( c ) denotes that it is related to the candidate embeddings. The synthesized RGB value ˆ C ( c ) i is generated by a joint volumetric rendering of the shared representation ( σ and c ) and the image dependent representation ( σ ( c ) i and color c ( c ) i ):

The intuition is to consider easy pieces first to complete a large jigsaw puzzle and handle hard pieces later. In the initial stages of training, shared representation is predominantly learned by 'easypose' images. Meanwhile, candidate representation is primarily trained by hard-pose images. This remedy effectively prevents the shared representation from being distracted by the wrong supervision introduced by hard-pose images.

As the shared representation becomes accurate enough to facilitate the pose estimation of hard-pose images, the influence of the candidate head is gradually reduced so that hard-pose images can be assimilated into the shared representation, and the final model uses only the shared representation, σ and c . To achieve that, we propose a loss scheduling given as:

where w u,v is defined as:

Here l ∈ [0 , 1] denotes training progress and u, v ∈ [0 , 1] are hyperparameters. Then, w u,v tunes the influence of ˆ C ( c ) i and ˆ C . Specifically, for learning during the initial stage of training ( l < u ), only ˆ C ( c ) i is utilized, whereas only ˆ C is employed in the later stage ( l > v ). With the loss scheduling, a high-quality final NeRF model can be trained. In our final training pipeline, we will use features instead of colors, as shown in Fig. 2, which will be covered in Section 3.2.

The volume rendering method shown in Eq. (5) is similar to NeRF-W [18] in that it has an additional head for color and density. In our architecture, however, we do not have a regularization term and uncertainty field that hinder the candidate head from handling hard-pose images. The size of embedding ℓ ( c ) i is adjusted by how challenging the poses of the given images are, which can be viewed as adjusting the regularizers for 'slack' variables in the constrained optimization problem, e.g ., support vector machine (SVM). The quantitative analysis of the candidate embedding size is provided in Table 4.

Feature Encoder

❄

Mono-depth

network

❄

Volumetric rendering

Depth rendering

Alpha blending

Pretrained network

MLP

MLP

C

F(C)

2

(P)PL

e(a)

Lfeat

F

2

中

L

oth

a

a

3.2 Feature-surrogate bundle adjustment

The motivation of this section is shown in Fig. 1. Given images with diverse appearances due to different weather and times, unposed-NeRF cannot be stably trained with a photometric loss since RGB values of corresponding pixels between images are not guaranteed to be the same. We adopt the deep features from ViT [49] as a surrogate, which are less sensitive to appearance changes, and we train the neural field with these features to achieve better robustness under challenging conditions.

With a slight change in Eq. (1), our feature field is defined as:

Unlike the color c ( t ) encoding in Eq. (1), the feature f ( t ) encoding is performed without a viewing direction because the features ought to be independent of viewing directions. Furthermore, ˆ F ( c ) i ( r ) is rendered similarly to Eq. (5) by volume rendering with f ( t ) , σ ( t ) , f ( c ) i ( t ) and σ ( c ) i ( t ) . It is then optimized by minimizing L feat ( r ) = ∥ F i ( r ) -ˆ F ( c ) i ( r ) ∥ 2 2 with 2D deep feature F i ( r ) from a pretrained model. We replace the loss L rgb ( ˆ C ( c ) i | C i ) in Eq. (6) with the loss L feat ( ˆ F ( c ) i | F i ) for the early training. Meanwhile, the unconstrained images can have different colors even though they have the same semantic object. As a result, the feature f ( t ) can be used to encode image-dependent static color c ( a ) i along with viewing direction d i and appearance embedding { ℓ ( a ) i } N i =1 for each image as:

Note that ( . ) ( a ) means it is related to the appearance embeddings. The predicted RGB value ˆ C ( a ) i ( r ) is calculated by volume rendering Eq. (2) with c ( a ) i ( t ) and σ ( t ) . This static color ˆ C ( a ) i ( r ) is optimized jointly with the transient color ˆ C ( τ ) i ( r ) that will be introduced in the next section, Section 3.3. In summary, through candidate scheduling Eq. (6), the feature field is trained only for the early stage ( l < u ), Π θ 3 is not involved in learning in the later stage of training ( l ≥ v ), and the radiance field starts training as feature field gradually stops training ( u ≤ l < v ). See Fig. 2 for the final NeRF model Π θ architecture.

3.3 Isolated network for transient-free pose optimization

The next challenge is the occlusion of transient objects. To optimize the poses accurately, their influence must be minimized in the training process as much as possible. The previous work NeRFW[18], which tackles this problem, models it in 3D space by adding a transient head. If we naively adopt it in our model, it causes a noisy gradient flowing into the learnable pose parameters, with the pose being learned in the wrong direction due to transient occluders. So we mitigate this problem by handling transients occluders at the separated TransientNet T ϕ in 2D image level. It maps deep feature F i ( r ) to opacity α ( τ ) , which is intended to filter out the transient based on semantic information. Then, along with transient embedding { ℓ ( τ ) i } N i =1 , it encodes image-dependent transient color ˆ C ( τ ) i and uncertainty ˜ β ( τ ) i :

Note that ( . ) ( τ ) means it is related to the transient objects. The final predicted color ˆ C i ( r ) is obtained by alpha blending with ˆ C ( a ) i ( r ) from Section 3.2:

The RGB loss for ray r with given color C ( r ) is:

where β ( τ ) i ( r ) = α ( τ ) ( r ) ˜ β ( τ ) i ( r ) + β min. As in NeRF-W, the uncertainty β ( τ ) i ( r ) is modeled as the variance of an isotropic normal distribution with mean ˆ C i ( r ) , resulting the negative log-likelihood loss of C i ( r ) . It reduces the impact of transient occluders as noise when training static objects. β min > 0 is a hyperparameter for ensuring a minimum importance. The last term suppresses α ( τ ) as possible to prevent TransientNet from rendering static objects with λ α > 0 . We set β min to 0.1 and λ α to 1.0.

3.4 Transient-aware depth prior

This section is about incorporating geometric prior from monocular depth into NeRF, which is highly inspired by NoPe-NeRF [24]. The mono-depth prior provides strong geometry cues for jointly training pose and NeRF. Even though NoPe-NeRF proposes inter-frame loss to leverage a depth prior, it can only be applied to the successive images from a video so we cannot adopt such loss for the unconstrained images. So, we apply only the depth loss Eq. (13). The predicted inverse depth { D inv i } N i =1 is acquired from depth network DPT [50]. Since this is a monocular prediction, it is not an absolute inverse depth value. Therefore learnable parameters { η i } N i =1 and { ξ i } N i =1 , which denote a scale and shift factor, were introduced for each mono-depth to learn undistorted depth map D ∗ i . It is optimized with rendered NeRF depth ˆ D i :

By optimizing the L1 loss L depth ( r ) = ∥ D ∗ i ( r ) -ˆ D i ( r ) ∥ , NeRF learns a strong geometry prior. However, since NeRF should exclude transient objects, we need to apply depth prior only to the areas without transient objects. In Section 3.1, the uncertain parts are generated from the candidate density σ ( c ) i of the candidate head, while certain parts are generated from the shared density σ . This implies that the portion of candidate density also filters the transient parts as in Fig. 5. Based on this observation, we can measure transient confidence weight W depth i ( r ) ∈ [0 , 1] from the ratio of candidate density σ ( c ) i to shared density σ . In consequence, the depth loss is defined as follows:

3.5 Training pipeline

The overall training pipeline is shown in Fig. 2. Integrating the depth loss into Eq. (6) modified from Section 3.2 and Section 3.3, the overall loss is as follows:

The loss implies that depth loss L depth decays as the training progresses as feature loss L feat does. This is because we exploit depth supervision only for initial pose estimation, and using it further can degenerate the rendering quality in that the geometry prior we utilize is not a ground truth but like a pseudo-label from a depth network. A hyperparameter λ d for depth loss is set to 0 . 001 .

We use a hierarchical sampling strategy, and there is a slight difference in the coarse model loss with Eq. (12). We optimize the TransientNet with the fine model alone, and the coarse model uses detached α ( τ ) and ˆ C ( τ ) i ( r ) to render ˆ C i ( r ) in Eq. (11), and does not use uncertainty. Thus, the final RGB loss for the coarse model is as follows.

In addition, the way of composing a sample set for the fine model is slightly different from NeRF [1], which only uses the density σ of the coarse model. Along with σ , we also need to use candidate density σ ( c ) until scheduling eliminates the effect of candidate head. Depending on the scheduling factor w u,v , both density σ and σ ( c ) are used at the beginning, and only σ is used in the end.

4 Experiments

Datasets. To demonstrate our methods work well in the unconstrained images. We report results on the Phototourism dataset. It consists of internet photo collections of famous landmarks and we select 4 scenes, Brandenburg Gate , Sacre Coeur , Taj Mahal , and Trevi fountain , which are also used in NeRF-W. We follow the split used by NeRF-W [18] and downsample each image by 2 times. All the initial camera poses are set to the identity transformation.

Implementation details. All the models are trained for 600K iterations with randomly sampled 2048 pixel rays at each step with a learning rate of 5 × 10 -4 decaying to 5 × 10 -5 for NeRF and transient network T , and 2 × 10 -3 decaying to 1 × 10 -3 for pose p and two factors s i and t i of depth. We use Adam optimizer [51] across all the experiments except test-time appearance optimization, where AdamW [52] is used instead. The number of sampling points in each ray for volumetric rendering is set to 128 for both coarse and fine models. We use the default coarse-to-fine strategy

Taj-Mahal

N/A

N/A

ground-truth

Trevi-Fountain

BARF

BARF-W*

BARF-WD*

Ours

ref. NeRF-W

of BARF which starts from training progress 0.1 to 0.5. We set the scheduling parameters u and v to be 0.1 and 0.5, respectively, the same as the parameters of coarse-to-fine. Note that both play the same role in helping robust pose estimation in the early stages. The transient confidence weight W depth is manually detached to prevent those parameters from affecting NeRF directly. We extract deep features from the DINO model [49, 53], whose powerful 2D correspondence representation has been proven in various works [47, 54, 55], and monocular depth from DPT [50] offline. The detailed process of extracting features follows Amir et al. [49]. Full implementation details are provided in the supplementary material.

Evaluation. We implement the evaluation process which consists of two stages, test-time pose optimization, and appearance optimization. In NeRF-W, they do just appearance optimization since it trains with given poses. But in our case, we need to optimize pose either to evaluate novel view synthesis. Therefore, we optimize both pose and appearance on test images for pose test time optimization and then initialize appearance again and optimize it with the optimized pose. In conclusion, the results of novel view synthesis are evaluated using PSNR, SSIM [56], and LPIPS [57]. The train camera poses are Procrustes-aligned for comparison with ground truth poses in the same way as BARF [23].

4.1 Results

To verify the difficulty of our task, we compare our model with BARF [23] and its two variants. First, we implement BARF-W as a baseline by applying latent appearance modeling and the transient head of NeRF-W to BARF. The other baseline is relevant to NoPe-NeRF [24]. NoPe-NeRF succeeded in training NeRF with no pose prior in the outdoor scene. But it uses a sequence of images which is from video, so most of their principal components are using near-frame images. It is a strong prior to estimate camera poses, but in-the-wild images do not have such a prior so only the mono-depth loss is applicable to our setting. Therefore, we add the loss term Eq. (14) without W depth i to BARF-W (BARF-WD) to check whether the depth prior alone is enough to guide the model to robust pose estimation with unconstrained photos.

Table 2 shows the quantitative result of camera pose estimation, and translation errors are scaled by 10. As expected, BARF fails to estimate the poses due to the lack of photometric consistency in the unconstrained image collection. BARF-W also fails to optimize, which means that it is a hard problem that cannot be solved simply by adding appearance modeling and transient head. Although

3,000 iterations

15,000 iterations

45,000 iterations

Results

Ours

Ground Truth

w/o candid. head

Wdepth

2

TPT

1AT

BARF-WD showed slightly better results, it can be seen that the depth prior alone is not enough to solve the photometric inconsistency. In contrast, our model succeeds in pose estimation and shows a quality result.

Table 1 and Fig. 3 show the novel view synthesis results. As shown in the figure, baselines have poor rendering quality, which must be rooted in a failure of pose estimation. Even though variants of BARF (columns 3-4) show a slight improvement over BARF, its sub-optimal quality and pose estimation gives strong evidence that a mere combination of current methods or imposing depth supervision is not enough to resolve the problem. In contrast, our model shows comparable performance against reference NeRF-W and a clear appearance that implies its accurate pose estimation.

4.2 Ablation study

In this section, we verify the efficacy of each component in pose estimation with wild photos and provide a visualization of how these help the model optimize pose correctly.

Feature-surrogate bundle adjustment. In Section 3.2, we optimize the radiance field by minimizing L feat for the early training since the unconstrained images are not guaranteed color consistency. When training only with photometric loss L rgb without feature-surrogate, it can be seen from Table 3 that it fails to find accurate camera poses and render the novel views. This result shows that the feature-surrogate method is important for the early stage of training.

Transient-free pose optimization. We demonstrate the effect of separating the transient network from NeRF by comparing our model with a baseline where the transient network remains the same as NeRF-W, meaning that transient occluders are rendered from 3D space as static objects. Table 3 (column 2) clearly shows that gradient flows of transient objects adversely affect both pose estimation and NeRF optimization.

Transient-aware depth prior. As shown in Fig. 5, confidence score filters transient occluders while imposing depth prior to the static correctly as we intended. Even though the static ( e.g ., the gate) is slightly masked, it is justifiable in that the figure is visualized at the early of the training, meaning that the pose is not fully optimized and the model is still being trained to discern the static from the transient.

Candidate head. The last row of Table 3 shows that candidate head plays a crucial role in optimizing camera pose accurately. Additionally, we provide quantitative analysis on the candidate embedding ℓ ( c ) i size in Table 4. We observed that setting the candidate embedding to an appropriate size is of great help to camera pose estimation, and eventually succeeded in synthesizing novel view images.

4.3 Analysis

Candidate head. Fig. 4 shows that candidate embedding facilitates appropriate pose estimation for a hard pose image, inducing the model not to vainly struggle to overfit the scene without updating the pose. In contrast to the baseline without the candidate head which is stuck in a local minima, our model successfully estimates the pose. The figure implies that if the rendered scene at the early stage is similar to the ground truth with an incorrect pose, the baseline tries to fit the scene by optimizing NeRF, rather than pursuing accurate pose optimization. However, our model is more robust against being trapped into pose local minima for the hard pose images.

Overall training time We compared overall learning time between NeRF-W [18] and UP-NeRF in Table 5. Before training NeRF-W, it is necessary to run COLMAP to get camera poses. The time required for COLMAP depends on the number of images, and the execution time becomes non-negligible as the number increases. For example, Trevi fountain with about 3200 images, takes nearly 1 week to finish. However, UP-NeRF takes just 36 minutes to prepare training, which includes extracting DINO feature and DPT depth maps. The training time of UP-NeRF is also faster than NeRF-W because it renders transient objects directly from the feature maps without a volume rendering process. UP-NeRF only requires about 30 minutes of preprocessing time, so we expect using faster models such as Instant-NGP [58] to reduce the training time greatly.

5 Conclusion

We propose UP-NeRF, a robust unposed-NeRF that can learn with an unconstrained image collection with variable illumination and transient occluders. Thanks to the color-insensitive feature field, separated transient network, candidate head for robust pose optimization, and transient-aware depth prior, our model has less difficulty in estimating poses even in such challenging conditions. The experiment shows the promising results of our model compared to baselines, even comparable to the reference model with accurate poses.

Acknowledgments and Disclosure of Funding

This work was partly supported by ICT Creative Consilience program (IITP-2023-2020-0-01819) supervised by the IITP, and the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (NRF-2023R1A2C2005373).

References

Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV , 2020.

Nianchen Deng, Zhenyi He, Jiannan Ye, Praneeth Chakravarthula, Xubo Yang, and Qi Sun. Foveated neural radiance fields for real-time and egocentric virtual reality. arXiv preprint arXiv:2103.16365 , 2021.

Sixu Li, Chaojian Li, Wenbo Zhu, Boyang Yu, Yang Zhao, Cheng Wan, Haoran You, Huihong Shi, and Yingyan Lin. Instant-3d: Instant neural radiance field training towards on-device ar/vr 3d reconstruction. In ISCA , 2023.

Ziyang Xie, Junge Zhang, Wenye Li, Feihu Zhang, and Li Zhang. S-nerf: Neural radiance fields for street views. In ICLR , 2023.

Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srinivasan, Jonathan T Barron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. In CVPR , 2022.

Jeffrey Ichnowski, Yahav Avigal, Justin Kerr, and Ken Goldberg. Dex-nerf: Using a neural radiance field to grasp transparent objects. In CoRL , 2021.

Michal Adamkiewicz, Timothy Chen, Adam Caccavale, Rachel Gardner, Preston Culbertson, Jeannette Bohg, and Mac Schwager. Vision-only robot navigation in a neural radiance world. IEEE Robotics and Automation Letters , 2022.

Kunlong Hong, Hongguang Wang, and Bingbing Yuan. Inspection-nerf: Rendering multi-type local images for dam surface inspection task using climbing robot and neural radiance field. Buildings , 2023.

Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelNeRF: Neural radiance fields from one or few images. In CVPR , 2021.

Ajay Jain, Matthew Tancik, and Pieter Abbeel. Putting nerf on a diet: Semantically consistent few-shot view synthesis. In ICCV , 2021.

Julian Chibane, Aayush Bansal, Verica Lazova, and Gerard Pons-Moll. Stereo radiance fields (srf): Learning view synthesis from sparse views of novel scenes. In CVPR , 2021.

Sara Fridovich-Keil and Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In CVPR , 2022.

Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. PlenOctrees for real-time rendering of neural radiance fields. In ICCV , 2021.

Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In ICCV , 2021.

David B. Lindell, Julien N.P. Martel, and Gordon Wetzstein. Autoint: Automatic integration for fast neural volume rendering. In CVPR , 2021.

Stephan J. Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin. Fastnerf: High-fidelity neural rendering at 200fps. In ICCV , 2021.

Cheng Sun, Min Sun, and Hwann-Tzong Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In CVPR , 2022.

Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth. NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. In CVPR , 2021.

Xingyu Chen, Qi Zhang, Xiaoyu Li, Yue Chen, Ying Feng, Xuan Wang, and Jue Wang. Hallucinated neural radiance fields in the wild. In CVPR , 2022.

Johannes L Schonberger and Jan-Michael Frahm. Structure-from-motion revisited. In CVPR , 2016.

Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. NeRF --: Neural radiance fields without known camera parameters. arXiv preprint arXiv:2102.07064 , 2021.

Yoonwoo Jeong, Seokjun Ahn, Christopher Choy, Animashree Anandkumar, Minsu Cho, and Jaesik Park. Self-calibrating neural radiance fields. In ICCV , 2021.

Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey. Barf: Bundle-adjusting neural radiance fields. In ICCV , 2021.

Wenjing Bian, Zirui Wang, Kejie Li, Jia-Wang Bian, and Victor Adrian Prisacariu. Nope-nerf: Optimising neural radiance field with no pose prior. In CVPR , 2023.

Yue Jiang, Dantong Ji, Zhizhong Han, and Matthias Zwicker. Sdfdiff: Differentiable rendering of signed distance fields for 3d shape optimization. In CVPR , 2020.

Xinchen Yan, Jimei Yang, Ersin Yumer, Yijie Guo, and Honglak Lee. Perspective transformer nets: Learning single-view 3d object reconstruction without 3d supervision. In NeurIPS , 2016.

Riccardo Roveri, A Cengiz Öztireli, Ioana Pandele, and Markus Gross. Pointpronets: Consolidation of point clouds with convolutional neural networks. Computer Graphics Forum , 2018.

Shichen Liu, Shunsuke Saito, Weikai Chen, and Hao Li. Learning to infer implicit surfaces without 3d supervision. In NeurIPS , 2019.

Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In CVPR , 2020.

Hsueh-Ti Derek Liu, Michael Tao, and Alec Jacobson. Paparazzi: surface editing by way of multi-view image processing. ACM Trans. Graph. , 2018.

Chia-Yin Tsai, Aswin C Sankaranarayanan, and Ioannis Gkioulekas. Beyond volumetric albedo-a surface optimization framework for non-line-of-sight imaging. In CVPR , 2019.

Eldar Insafutdinov and Alexey Dosovitskiy. Unsupervised learning of shape and pose with differentiable point clouds. In NeurIPS , 2018.

Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In ICCV , 2021.

Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In CVPR , 2022.

Mijeong Kim, Seonguk Seo, and Bohyung Han. Infonerf: Ray entropy minimization for few-shot neural volume rendering. In CVPR , 2022.

Alex M Andrew. Multiple view geometry in computer vision. Kybernetes , 2001.

Yitong Xia, Hao Tang, Radu Timofte, and Luc Van Gool. Sinerf: Sinusoidal neural radiance fields for joint pose estimation and scene reconstruction. In BMVC , 2022.

Shin-Fang Chng, Sameera Ramasinghe, Jamie Sherrah, and Simon Lucey. Gaussian activated neural radiance fields for high fidelity reconstruction and pose estimation. In ECCV , 2022.

Prune Truong, Marie-Julie Rakotosaona, Fabian Manhardt, and Federico Tombari. Sparf: Neural radiance fields from sparse and noisy poses. In CVPR , 2023.

Quan Meng, Anpei Chen, Haimin Luo, Minye Wu, Hao Su, Lan Xu, Xuming He, and Jingyi Yu. GNeRF: GAN-based Neural Radiance Field without Posed Camera. In ICCV , 2021.

Jianxiong Shen, Adria Ruiz, Antonio Agudo, and Francesc Moreno-Noguer. Stochastic neural radiance fields: Quantifying uncertainty in implicit 3d representations. In 3DV , 2021.

Jianxiong Shen, Antonio Agudo, Francesc Moreno-Noguer, and Adria Ruiz. Conditional-flow nerf: Accurate 3d modelling with reliable uncertainty quantification. In ECCV , 2022.

Tianhao Wu, Fangcheng Zhong, Andrea Tagliasacchi, Forrester Cole, and Cengiz Oztireli. D 2 nerf: Self-supervised decoupling of dynamic and static objects from a monocular video. In NeurIPS , 2022.

Sara Sabour, Suhani Vora, Daniel Duckworth, Ivan Krasin, David J Fleet, and Andrea Tagliasacchi. Robustnerf: Ignoring distractors with robust losses. In CVPR , 2023.

Mark Boss, Andreas Engelhardt, Abhishek Kar, Yuanzhen Li, Deqing Sun, Jonathan Barron, Hendrik Lensch, and Varun Jampani. Samurai: Shape and material from unconstrained realworld arbitrary image collections. In NeurIPS , 2022.

Zhengfei Kuang, Kyle Olszewski, Menglei Chai, Zeng Huang, Panos Achlioptas, and Sergey Tulyakov. Neroic: neural rendering of objects from online image collections. TOG , 2022.

Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann. Decomposing nerf for editing via feature field distillation. In NeurIPS , 2022.

Yue Chen, Xingyu Chen, Xuan Wang, Qi Zhang, Yu Guo, Ying Shan, and Fei Wang. Local-toglobal registration for bundle-adjusting neural radiance fields. In CVPR , 2023.

Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features as dense visual descriptors. arXiv preprint arXiv:2112.05814 , 2021.

René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In ICCV , 2021.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR , 2014.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR , 2017.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV , 2021.

Chun-Han Yao, Wei-Chih Hung, Yuanzhen Li, Michael Rubinstein, Ming-Hsuan Yang, and Varun Jampani. LASSIE: Learning Articulated Shape from Sparse Image Ensemble via 3d part discovery. In NeurIPS , 2022.

Shangzhe Wu, Ruining Li, Tomas Jakab, Christian Rupprecht, and Andrea Vedaldi. Magicpony: Learning articulated 3d animals in the wild. In CVPR , 2023.

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. TIP , 2004.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR , 2018.

Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph. , 2022.

Peer reviews:
comments: SUMMARY:
This paper solves an interesting problem — joint pose and NeRF optimization on in-the-wild image collections. Unlike prior works such as BARF, this work aims at handling unconstrained images with varying illumination and transient occluders. To tackle this problem, this work incorporates learnable camera parameters, depth prior, and semantic features from DINO as supervision into the NeRF-W framework. Experiments demonstrate that the proposed method outperforms BARF and its variants on unstructured Internet images.

STRENGTHS:
1.  This work identifies an interesting and important research problem — joint pose and NeRF optimization on in-the-wild images. Prior works on the joint pose and NeRF optimization are limited to controlled settings. 
2. The proposed method is intuitive and combines the NeRF-W framework and several components (e.g., depth prior, semantic features, etc.) from other works (such as NoPe-NeRF).

WEAKNESSES:
1. The main contribution of this paper is the problem setting — pose and NeRF estimation in the wild. The proposed method combines NeRF-W and BARF, which has limited technical novelty. 
2. The presentation may be improved. Figure 1 and Figure 2 have not been referred to and explained in the main text. 
3. Missing references:  Meng et al. GNeRF: GAN-based Neural Radiance Field without Posed Camera. ICCV 2021

QUESTIONS:
What is the effect of using semantic features as supervision, i.e. L_feat?

LIMITATIONS:
I did not find discussions on the limitations of this paper. It would be great to discuss the limitations and analyze the possible reasons and future works.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles NeRF training from in-the-wild Internet photos without pre-computed camera poses. The main idea is to leverage (self-supervised) image features and a carefully designed optimization strategy, together with ideas from existing work, including modeling transient regions and using mono-depth supervision. These components work in combination to avoid local minima and enable joint optimization of both pose and NeRF from scratch.\n\nExperiments show that this joint optimization pipeline successfully recovers camera poses in several in-the-wild scenes and leads to good NVS results comparable to those with COLMAP preprocessing, whereas the previous work of BARF and its variants all fail.\n\nSTRENGTHS:\n### S1 - Technically sound approach to a challenging problem\n- The task of estimating camera poses on raw in-the-wild Internet photos using a gradient-descent-based optimization is challenging, due to noisy correspondences and local minima.\n- The proposed method incorporates many good ideas from existing work and works well on a number challenging scenes.\n-- For instance, the use of self-supervised image features for registering semantic correspondences via rendering;\n-- The mechanism to model transient objects using learned opacity. This paper further adapts the original volumetric opacity in NeRF-W to per-image opacity, which is claimed to be more effective.\n-- The use of depth supervision from pre-trained models.\n\n### S2 - Promising results\n- The paper demonstrates good results on a number of challenging scenes using Internet photos, where existing methods clearly fails.\n\nWEAKNESSES:\n### W1 - Complicated pipeline\n- The resulting pipeline is complicated, involving many components, eg, 6 MLPs in total, and easily becomes confusing. It took me several passes back and forth to understand the exact implementation.\n- It also requires a heavily crafted training schedule, gradually activating and deactivating some of the components.\n- A critical concern on such a pipeline is its robustness across various scenes. Would one need to fight against all the hyperparameters and the training schedule when training on other scenes. I strongly suggest the authors also present results on other standard datasets (W3).\n\n### W2 - Unclear motivation and potential redundancy of some technical designs\n- It is unclear to me why the per-image \"candidate embeddings\" can help with the pose optimization. It seems the motivation is to allow some of the per-image variations (multi-view inconsistencies) to be factored into these per-image embeddings. The paper presents ablation results which shows numerical benefits of this component, but why is it useful for pose estimation?\n- Are they still useful in general in the case of a multi-view static scene without transient entities?\n- Isn't this the job of the separate mechanism for handling transient regions?\n- Also, it is confusing to me why the transient confidence weights $\\mathcal{W}_i^{\\text{depth}}$ are calculated from the candidate densities $\\sigma^{(c)}$, rather than from the transient opacities $\\alpha^{(\\tau)}$. Are there some redundancy between the two?\n\n### W3 - Only on one dataset\n- The paper only presents results on one dataset, consisting of Internet photo collections of 4 scenes, which the method is specifically tailored to.\n- However, the paper claims to solve general \"unposed NeRF\". I strongly recommend the authors to also test it on standard multi-view datasets, eg DTU, CO3D etc, to assess the robustness of this complicated pipeline.\n\n### (minor) W4 - Do intrinsics matter?\n- How are the camera intrinsics obtained? From COLMAP? If so, this would undermine the value of avoiding cumbersome COLMAP preprocessing.\n- Or are they estimated or assumed to be some value for all images?\n\n### Other comments\n- There are a few existing work that leverages DINO for pose estimation of objects, which the authors should consider referencing, eg: Zero-Shot Category-Level Object Pose Estimation [1], LASSIE [2], MagicPony [3].\n- Line 125: when the \"candidate embeddings\" are first introduced, it was not immediately obvious to me they are per-image embedding vectors. Also, are they jointly optimized? The term \"candidate head\" is also quite obscure to me.\n- Fig 2: the blue arrow in the middle connecting $\\hat{\\mathbf{F}}_i^{(c)}$ to $\\theta_4$ is slightly inaccurate, as the $\\hat{\\mathbf{F}}_i^{(c)}$ is the result of ray integration, whereas the input to $\\theta_4$ should be the raw per-point feature $\\hat{\\mathbf{f}}_i$, if I understand correctly.\n- What is the computational overheads compared to vanilla NeRF?\n\n### References:\n- [1] Zero-Shot Category-Level Object Pose Estimation. ECCV 2022.\n- [2] LASSIE: Learning Articulated Shape from Sparse Image Ensemble via 3D Part Discovery. NeurIPS 2022.\n- [3] MagicPony: Learning Articulated 3D Animals in the Wild. CVPR 2023.\n\nQUESTIONS:\n1. Clarify the motivation for the \"candidate embeddings\";\n2. If possible, present more evaluation results on other datasets **with the same hyperparameters and training schedule**, to validate the robustness;\n3. It might be difficult as the pipeline is indeed complicated, but if possible, it would be helpful to further simplify the notations, in particular, the super- and sub-scriptions.\n\nLIMITATIONS:\nThe authors included a brief paragraph on the limitation of the image features. I would expect some discussion on the robustness of the proposed pipeline.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper produces a novel approach for optimisation of pose in NeRF scenarios. The core novelty is the addition of a candidate head that improves network stability when the images poses are not yet converged, along with some other tweaks like a transiency inference head or a feature field. \n\nEdit: I have read the rebuttal, and given the scores from other reviewers and I would like to keep my score.\n\nSTRENGTHS:\n* The idea is novel, interesting and timely.\n* The results show good improvements over the current s-o-t-a.\n* The experiments section is thorough enough to demonstrate the qualities of the proposed approach. \n\nOverall, while my negatives might sound long and would seen to outweigh the positives, I think it brings a potentially valuable contribution to the community, so I very much support the acceptance of the paper. My main worry is about the somewhat confusing and short explanation of the core contribution of the “candidate” heads, which should be expanded and improved.\n\nWEAKNESSES:\n* The paper is somewhat confusing to read at times. For example “unconstrained images” are introduced and used without introducing the term. The meaning does become later in the paper, but it would be good to introduce things earlier on. Similarly, I do not see much point for Figure 1, as features have been used for this type of matching for years.\n* Even though the literature review section of the paper is quite good, I think the paper does anchor things a bit too much on BARF, and ignores other works such as NoPe-NERF / GARF / etc in the comparisons. I do understand that some of these works were arxiv at the time of the submission (but are publications now), so this should not penalise the paper too much. That being said, it’d be nice if some of these comparisons could be added.\n* I also found some of the notation difficult to follow initially as I found not find any outline of it’s meaning. An example is the (c) in line 126 which is not explained. \n* Looking at the core contribution of the paper (i) the intuition (i.e. the “roughly speaking … “ part at line 125+) could be expanded and (ii) the size of the embedding should have been ablated in the results section.\n* The results section could have been expanded e.g. with (i) the ablation noted above, (ii) extra ablations where, e.g., the feature matching part is turned off.\n\nQUESTIONS:\n* Figure 5 seems to show a case where the transiency is wrong … should the horse really be transient?\n\nLIMITATIONS:\n* To some extent, but the paper could benefit from a more clear failure case section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel method for optimizing NeRF without a pose-prior and on in-the-wild image collections containing transient occluders and varying lightings. The main contributions are four fold. Firstly, the authors propose a candidate head for NeRFs that uses image-level representations via a learned embedding for compensating inaccurate poses during the early optimization. Secondly, learning a view-independent feature field based on DINO features as intermediate representation increases the robustness of the joint optimization wrt Varying lightings, weather and time. To reduce the impact of transient occluders the authors suggest using a separate network that predicts occluder in 2D image level based on the feature maps. Additionally, to achieve higher accuracy in the geometry the authors apply monocular depth supervision on regions without occluders. Experiments are conducted on four scenes of the Phototourism dataset with initial camera poses set to the identity.\n\nSTRENGTHS:\n1) The paper is very nicely presented and easy to read.\n2) The paper tackles the challenging and relevant problem of in-the-wild NeRF reconstruction without given posen. The contribution is clear and solves different subproblems that emerge in this domain, e.g. transient occluders.\n3) Most parts are well motivated, the methodology is technical sound and experimentally justified e.g.:\n - Candidate head sounds plausible and improves pose optimization and image quality significantly, see Figure 4 and Table 3.\n - Feature field optimization seems to help for in-the-wild images, see Table 3.\n - Depth supervision improves performance, see Table 3 and Figure 5.\n4) The experimental section contains a comparison to BARF, adapted variants with additional supervision cues and numbers for NeRF-in-the-wild. It appears to be a plausible choice and supports the contribution of the paper.\n5) The authors provide code in the supplementary materials.\n\nWEAKNESSES:\n1) The candidate head is introduced to output color and density, equation 4, however it is later used to predict features, see Figure 2. This appears to be misleading for the readers in the beginning.\n2) One of the main limitations of NeRF based methods is the optimization time and an analysis is missing on that. It would be good to discuss the optimization time for the method and the baselines. To get a sense if the time and computational effort is comparable to COLMAP + NeRF-W and others.\n\nQUESTIONS:\n1) The explanation of the loss schedule applied to the candidate head training makes sense however there is no ablation study. What would happen if the weight is not reduced? Why should there be a negative impact if the loss?\n2) Can you explain the z(t) in equation 1?\n3) Would it make sense to introduce a candidate head with feature output in equation 4?\n4) Please provide evidence about the computation time, see Weakness 2.\n\nLIMITATIONS:\nLimitations are discussed in the supplementary.\nI'd suggest that the authors discuss the overall optimization time as a general limitation of the method, if applicable. There are state-of-the-art NeRF architecture, such as Instant-NGP that facilitate optimization in a few minutes instead of hour or days, which might be a good follow up.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

comments: SUMMARY:
The paper proposes a joint camera pose and NeRF optimisation method that can handle transient scenes, including moving objects and various light conditions, by integrating NeRF-W, BARF, NoPe-NeRF, and DINO-based feature-metric loss in a sophisticated way. 

The method is evaluated on four scenes in the Phototourism dataset, showing good performance compared with (modified) BARF baselines.

---
**After rebuttal**: I have read authors' rebuttal and it addresses my concerns.

STRENGTHS:
It’s a challenging task to estimate camera poses in scenes with moving objects and different light conditions, especially in a NeRF setup. This method proposes to make the pipeline more robust by considering
* Moving objects and lighting conditions similar to NeRF-W;
* Pose optimisation similar to BARF;
* Monocular depth un-distortion similar to NoPe-NeRF; and 
* DINO-based Feature-metric.

In short, the proposed method is novel and leads to promising results. It’s also a plus that code is also provided in supplementary.

WEAKNESSES:
I think a primary concern is how robust the pose estimation is in more scenes. The method is only evaluated in 4 scenes. Is it possible to have an experiment, especially for pose estimation successful rate on more scenes?

I understand that it takes a long time for NeRF to reach the best rendering quality, but we should be able to tell if the pose optimisation is successful far before NeRF converges. For example, we can consider pose estimation successful if rotation error is lower than $\theta$ degrees in $x$ epochs. In this way, we can see the pose estimation success rate in many more scenes.

QUESTIONS:
See the weakness section.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06