SlowGuess/NeurIPS_2023_OCR huggingface dataset:
By optimizing the L1 loss  $\mathcal{L}_{\mathrm{depth}}(\mathbf{r}) = \| \mathbf{D}_i^* (\mathbf{r}) - \hat{\mathbf{D}}_i(\mathbf{r})\|$ , NeRF learns a strong geometry prior. However, since NeRF should exclude transient objects, we need to apply depth prior only to the areas without transient objects. In Section 3.1, the uncertain parts are generated from the candidate density  $\sigma_{i}^{(c)}$  of the candidate head, while certain parts are generated from the shared density  $\sigma$ . This implies that the portion of candidate density also filters the transient parts as in Fig. 5. Based on this observation, we can measure transient confidence weight  $\mathcal{W}_i^{\mathrm{depth}}(\mathbf{r})\in [0,1]$  from the ratio of candidate density  $\sigma_i^{(c)}$  to shared density  $\sigma$ . In consequence, the depth loss is defined as follows:
$$
\mathcal {L} _ {\mathrm {d e p t h}} (\mathbf {r}) = \left(1 - \mathcal {W} _ {i} ^ {\mathrm {d e p t h}} (\mathbf {r})\right) \left\| \mathbf {D} _ {i} ^ {*} (\mathbf {r}) - \hat {\mathbf {D}} _ {i} (\mathbf {r}) \right\|,
$$
$$
\text {w h e r e} \mathcal {W} _ {i} ^ {\text {d e p t h}} (\mathbf {r}) = \sum_ {k = 1} ^ {K} T _ {i} ^ {(c)} \left(t _ {k}\right) \alpha \left(\sigma_ {i} ^ {(c)} \left(t _ {k}\right) \delta_ {k}\right)). \tag {14}
$$
Table 1: Novel view synthesis results on PhotoTourism Dataset.
PSNR ↑	SSIM ↑	LPIPS ↓
ref. NeRF-W	BARF	BARF -W	BARF -WD	UP-NeRF	ref. NeRF-W	BARF	BARF -W	BARF -WD	UP-NeRF	ref. NeRF-W	BARF	BARF -W	BARF -WD	UP-NeRF
Brandenburg Gate	26.50	8.94	11.98	12.45	26.29	0.887	0.221	0.629	0.619	0.884	0.116	1.027	0.549	0.630	0.126
Trevi Fountain	22.26	11.59	14.34	16.10	22.17	0.706	0.152	0.442	0.576	0.691	0.211	1.096	0.746	0.487	0.235
Taj Mahal	25.15	7.73	14.13	12.90	24.87	0.858	0.124	0.633	0.624	0.834	0.165	1.053	0.677	0.659	0.209
Sacre Coeur	22.33	8.25	12.51	9.33	21.59	0.827	0.229	0.679	0.651	0.791	0.123	1.027	0.571	0.453	0.157
Mean	24.06	9.13	13.24	12.70	23.73	0.820	0.182	0.596	0.618	0.800	0.154	1.051	0.636	0.557	0.182
Table 2: Camera pose estimation on PhotoTourism Dataset.
Rotation (°) ↓	Translation ↓
BARF	BARF-W	BARF-WD	UP-NeRF	BARF	BARF-W	BARF-WD	UP-NeRF
Brandenburg Gate	128.32	21.42	159.00	0.621	4.479	2.890	3.334	0.070
Trevi Fountain	114.63	33.09	23.08	1.590	6.627	4.763	4.292	0.105
Taj Mahal	94.06	143.30	51.28	0.549	6.054	5.252	3.875	0.120
Sacre Coeur	99.33	112.19	113.50	1.452	12.353	9.875	9.073	0.222
Mean	109.09	77.50	86.72	1.053	7.378	5.695	5.144	0.129
3.5 Training pipeline
The overall training pipeline is shown in Fig. 2. Integrating the depth loss into Eq. (6) modified from Section 3.2 and Section 3.3, the overall loss is as follows:
$$
\mathcal {L} = w _ {u, v} \mathcal {L} _ {\mathrm {r g b}} + (1 - w _ {u, v}) \left(\mathcal {L} _ {\text {f e a t}} + \lambda_ {d} \mathcal {L} _ {\text {d e p t h}}\right). \tag {15}
$$
The loss implies that depth loss  $\mathcal{L}_{\mathrm{depth}}$  decays as the training progresses as feature loss  $\mathcal{L}_{\mathrm{feat}}$  does. This is because we exploit depth supervision only for initial pose estimation, and using it further can degenerate the rendering quality in that the geometry prior we utilize is not a ground truth but like a pseudo-label from a depth network. A hyperparameter  $\lambda_{d}$  for depth loss is set to 0.001.
We use a hierarchical sampling strategy, and there is a slight difference in the coarse model loss with Eq. (12). We optimize the TransientNet with the fine model alone, and the coarse model uses detached  $\alpha^{(\tau)}$  and  $\hat{\mathbf{C}}_i^{(\tau)}(\mathbf{r})$  to render  $\hat{\mathbf{C}}_i(\mathbf{r})$  in Eq. (11), and does not use uncertainty. Thus, the final RGB loss for the coarse model is as follows.
$$
\mathcal {L} _ {\mathrm {r g b}} ^ {\text {c o a r s e}} = \frac {1}{2} \| \hat {\mathbf {C}} _ {i} (\mathbf {r}) - \mathbf {C} _ {i} (\mathbf {r}) \| _ {2} ^ {2}. \tag {16}
$$
In addition, the way of composing a sample set for the fine model is slightly different from NeRF [1], which only uses the density  $\sigma$  of the coarse model. Along with  $\sigma$ , we also need to use candidate density  $\sigma^{(c)}$  until scheduling eliminates the effect of candidate head. Depending on the scheduling factor  $w_{u,v}$ , both density  $\sigma$  and  $\sigma^{(c)}$  are used at the beginning, and only  $\sigma$  is used in the end.
Datasets. To demonstrate our methods work well in the unconstrained images. We report results on the Phototourism dataset. It consists of internet photo collections of famous landmarks and we select 4 scenes, Brandenburg Gate, Sacre Coeur, Taj Mahal, and Trevi fountain, which are also used in NeRF-W. We follow the split used by NeRF-W [18] and downsample each image by 2 times. All the initial camera poses are set to the identity transformation.
Implementation details. All the models are trained for 600K iterations with randomly sampled 2048 pixel rays at each step with a learning rate of  $5 \times 10^{-4}$  decaying to  $5 \times 10^{-5}$  for NeRF and transient network  $\mathcal{T}$ , and  $2 \times 10^{-3}$  decaying to  $1 \times 10^{-3}$  for pose  $\mathbf{p}$  and two factors  $\mathbf{s}_i$  and  $\mathbf{t}_i$  of depth. We use Adam optimizer [51] across all the experiments except test-time appearance optimization, where AdamW [52] is used instead. The number of sampling points in each ray for volumetric rendering is set to 128 for both coarse and fine models. We use the default coarse-to-fine strategy
Figure 3: Qualitative results of baselines and our model, where  $(^{*})$  denotes BARF-variant baselines we implement. As shown in the figure (columns 2-4), a naive combination of BARF and other methods failed to converge, resulting in poor synthesis quality. In contrast, our model achieves comparable quality to reference NeRF with perfect camera poses.
of BARF which starts from training progress 0.1 to 0.5. We set the scheduling parameters  $u$  and  $v$  to be 0.1 and 0.5, respectively, the same as the parameters of coarse-to-fine. Note that both play the same role in helping robust pose estimation in the early stages. The transient confidence weight  $\mathcal{W}^{\mathrm{depth}}$  is manually detached to prevent those parameters from affecting NeRF directly. We extract deep features from the DINO model [49, 53], whose powerful 2D correspondence representation has been proven in various works [47, 54, 55], and monocular depth from DPT [50] offline. The detailed process of extracting features follows Amir et al. [49]. Full implementation details are provided in the supplementary material.
Evaluation. We implement the evaluation process which consists of two stages, test-time pose optimization, and appearance optimization. In NeRF-W, they do just appearance optimization since it trains with given poses. But in our case, we need to optimize pose either to evaluate novel view synthesis. Therefore, we optimize both pose and appearance on test images for pose test time optimization and then initialize appearance again and optimize it with the optimized pose. In conclusion, the results of novel view synthesis are evaluated using PSNR, SSIM [56], and LPIPS [57]. The train camera poses are Procrustes-aligned for comparison with ground truth poses in the same way as BARF [23].
To verify the difficulty of our task, we compare our model with BARF [23] and its two variants. First, we implement BARF-W as a baseline by applying latent appearance modeling and the transient head of NeRF-W to BARF. The other baseline is relevant to NoPe-NeRF [24]. NoPe-NeRF succeeded in training NeRF with no pose prior in the outdoor scene. But it uses a sequence of images which is from video, so most of their principal components are using near-frame images. It is a strong prior to estimate camera poses, but in-the-wild images do not have such a prior so only the mono-depth loss is applicable to our setting. Therefore, we add the loss term Eq. (14) without  $\mathcal{W}_i^{\mathrm{depth}}$  to BARF-W (BARF-WD) to check whether the depth prior alone is enough to guide the model to robust pose estimation with unconstrained photos.
Table 2 shows the quantitative result of camera pose estimation, and translation errors are scaled by 10. As expected, BARF fails to estimate the poses due to the lack of photometric consistency in the unconstrained image collection. BARF-W also fails to optimize, which means that it is a hard problem that cannot be solved simply by adding appearance modeling and transient head. Although
Figure 4: Visual comparison between our model with and without candidate head for a hard pose image during the training process. The small patch images compare the ground-truth pose (black) with the optimized poses (red).
Figure 5: Visualization of transient confidence weight.
Table 3: Quantitative results of ablation study. All metrics are an average of 4 scene results.
Pose	Novel View Synthesis
Rot (°) ↓	Trans↓	PSNR ↑	SSIM ↑	LPIPS ↓
UP-NeRF	1.053	0.129	23.73	0.800	0.182
w/o Lfeat	143.4	6.193	13.01	0.568	0.644
w/o Tφ	2.329	0.314	22.51	0.781	0.224
w/o Widepth	1.226	0.170	23.60	0.795	0.215
w/o Widepth, ℓi(c)	4.972	1.042	21.23	0.744	0.283
BARF-WD showed slightly better results, it can be seen that the depth prior alone is not enough to solve the photometric inconsistency. In contrast, our model succeeds in pose estimation and shows a quality result.
Table 1 and Fig. 3 show the novel view synthesis results. As shown in the figure, baselines have poor rendering quality, which must be rooted in a failure of pose estimation. Even though variants of BARF (columns 3-4) show a slight improvement over BARF, its sub-optimal quality and pose estimation gives strong evidence that a mere combination of current methods or imposing depth supervision is not enough to resolve the problem. In contrast, our model shows comparable performance against reference NeRF-W and a clear appearance that implies its accurate pose estimation.
In this section, we verify the efficacy of each component in pose estimation with wild photos and provide a visualization of how these help the model optimize pose correctly.
Feature-surrogate bundle adjustment. In Section 3.2, we optimize the radiance field by minimizing  $\mathcal{L}_{\mathrm{feat}}$  for the early training since the unconstrained images are not guaranteed color consistency. When training only with photometric loss  $\mathcal{L}_{\mathrm{rgb}}$  without feature-surrogate, it can be seen from Table 3 that it fails to find accurate camera poses and render the novel views. This result shows that the feature-surrogate method is important for the early stage of training.
Transient-free pose optimization. We demonstrate the effect of separating the transient network from NeRF by comparing our model with a baseline where the transient network remains the same as NeRF-W, meaning that transient occluders are rendered from 3D space as static objects. Table 3 (column 2) clearly shows that gradient flows of transient objects adversely affect both pose estimation and NeRF optimization.
Table 4: Quantitative analysis on the candidate embedding size.
size of ℓi(c)	0	8	16	32
Rotation (°) ↓	4.972	1.286	1.053	3.162
Translation ↓	1.042	0.184	0.129	0.474
PSNR ↑	21.23	23.54	23.73	23.11
SSIM ↑	0.744	0.802	0.800	0.779
LPIPS ↓	0.283	0.184	0.182	0.201
Table 5: Overall learning time comparison with NeRF-W.
NeRF-W	UP-NeRF
Preprocessing (COLMAP)	Training	Preprocessing (DINO&DPT)	Training
Brandenburg Gate	29h 29m	50h 34m	0h 16m	42h 02m
Trevi Fountain	150h 06m	51h 24m	0h 36m	42h 15m
Taj Mahal	21h 33m	49h 22m	0h 20m	42h 13m
Sacre Coeur	25h 27m	50h 34m	0h 18m	42h 51m
Transient-aware depth prior. As shown in Fig. 5, confidence score filters transient occluders while imposing depth prior to the static correctly as we intended. Even though the static (e.g., the gate) is slightly masked, it is justifiable in that the figure is visualized at the early of the training, meaning that the pose is not fully optimized and the model is still being trained to discern the static from the transient.
Candidate head. The last row of Table 3 shows that candidate head plays a crucial role in optimizing camera pose accurately. Additionally, we provide quantitative analysis on the candidate embedding  $\ell_i^{(c)}$  size in Table 4. We observed that setting the candidate embedding to an appropriate size is of great help to camera pose estimation, and eventually succeeded in synthesizing novel view images.
Candidate head. Fig. 4 shows that candidate embedding facilitates appropriate pose estimation for a hard pose image, inducing the model not to vainly struggle to overfit the scene without updating the pose. In contrast to the baseline without the candidate head which is stuck in a local minima, our model successfully estimates the pose. The figure implies that if the rendered scene at the early stage is similar to the ground truth with an incorrect pose, the baseline tries to fit the scene by optimizing NeRF, rather than pursuing accurate pose optimization. However, our model is more robust against being trapped into pose local minima for the hard pose images.
Overall training time We compared overall learning time between NeRF-W [18] and UP-NeRF in Table 5. Before training NeRF-W, it is necessary to run COLMAP to get camera poses. The time required for COLMAP depends on the number of images, and the execution time becomes non-negligible as the number increases. For example, Trevi fountain with about 3200 images, takes nearly 1 week to finish. However, UP-NeRF takes just 36 minutes to prepare training, which includes extracting DINO feature and DPT depth maps. The training time of UP-NeRF is also faster than NeRF-W because it renders transient objects directly from the feature maps without a volume rendering process. UP-NeRF only requires about 30 minutes of preprocessing time, so we expect using faster models such as Instant-NGP [58] to reduce the training time greatly.
We propose UP-NeRF, a robust unposed-NeRF that can learn with an unconstrained image collection with variable illumination and transient occluders. Thanks to the color-insensitive feature field, separated transient network, candidate head for robust pose optimization, and transient-aware depth prior, our model has less difficulty in estimating poses even in such challenging conditions. The experiment shows the promising results of our model compared to baselines, even comparable to the reference model with accurate poses.
This work was partly supported by ICT Creative Consilience program (IITP-2023-2020-0-01819) supervised by the IITP, and the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (NRF-2023R1A2C2005373).
[41] Jianxiong Shen, Adria Ruiz, Antonio Agudo, and Francesc Moreno-Noguer. Stochastic neural radiance fields: Quantifying uncertainty in implicit 3d representations. In 3DV, 2021.
[42] Jianxiong Shen, Antonio Agudo, Francesc Moreno-Noguer, and Adria Ruiz. Conditional-flow nef: Accurate 3d modelling with reliable uncertainty quantification. In ECCV, 2022.
[43] Tianhao Wu, Fangcheng Zhong, Andrea Tagliasacchi, Forrester Cole, and Cengiz Oztireli. D  $^2$  nerf: Self-supervised decoupling of dynamic and static objects from a monocular video. In NeurIPS, 2022.
[44] Sara Sabour, Suhani Vora, Daniel Duckworth, Ivan Krasin, David J Fleet, and Andrea Tagliasacchi. Robustnerf: Ignoring distractors with robust losses. In CVPR, 2023.
[45] Mark Boss, Andreas Engelhardt, Abhishek Kar, Yuanzhen Li, Deqing Sun, Jonathan Barron, Hendrik Lensch, and Varun Jampani. *Samurai: Shape and material from unconstrained real-world arbitrary image collections*. In *NeurIPS*, 2022.
[46] Zhengfei Kuang, Kyle Olszewski, Mengelei Chai, Zeng Huang, Panos Achlioptas, and Sergey Tulyakov. Neroic: neural rendering of objects from online image collections. TOG, 2022.
[47] Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann. Decomposing nerf for editing via feature field distillation. In NeurIPS, 2022.
[48] Yue Chen, Xingyu Chen, Xuan Wang, Qi Zhang, Yu Guo, Ying Shan, and Fei Wang. Local-to-global registration for bundle-adjusting neural radiance fields. In CVPR, 2023.
[49] Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features as dense visual descriptors. arXiv preprint arXiv:2112.05814, 2021.
[50] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In ICCV, 2021.
[51] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2014.
[52] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2017.
[53] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
[54] Chun-Han Yao, Wei-Chih Hung, Yuanzhen Li, Michael Rubinstein, Ming-Hsuan Yang, and Varun Jampani. LASSIE: Learning Articulated Shape from Sparse Image Ensemble via 3d part discovery. In NeurIPS, 2022.
[55] Shangzhe Wu, Ruining Li, Tomas Jakab, Christian Rupprecht, and Andrea Vedaldi. Magicpony: Learning articulated 3d animals in the wild. In CVPR, 2023.
[56] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. TIP, 2004.
[57] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.
[58] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph., 2022.