SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Quantitative comparison on ScanNet [10] and Tanks and Temples [19]. The best results are highlighted in bold, while the second best results are underlined.
Methods	Depth prior	ScanNet [10]	Tanks and Temples [19]
9 - 10 views	18 - 20 views	10 views
PSNR ↑	SSIM ↑	LPIPS ↓	PSNR ↑	SSIM ↑	LPIPS ↓	PSNR ↑	SSIM ↑	LPIPS ↓
NerfingMVS [47]	✓	N/A	N/A	N/A	16.29	0.626	0.502	N/A	N/A	N/A
K-planes [13]	✗	16.01	0.618	0.494	18.70	0.708	0.400	12.57	0.453	0.607
RegNeRF [29]	✗	16.38	0.624	0.493	18.93	0.676	0.450	14.12	0.469	0.580
DS-NeRF [11]	✓	N/A	N/A	N/A	20.85	0.713	0.344	N/A	N/A	N/A
DDP-NeRF [37]	✓	N/A	N/A	N/A	19.29	0.695	0.368	N/A	N/A	N/A
SCADE [45]	✓	18.83	0.646	0.375	21.54	0.732	0.292	13.46	0.402	0.607
DäRF (Ours)	✓	18.29	0.690	0.412	21.58	0.765	0.325	15.70	0.514	0.583
Table 2: Evaluation of depth quality: (a) quantitative evaluation of the adapted MDE, compared with other monocular depth estimation models and (b) visualization of depth distributions. The adapted MDE by our method shows a similar distribution to that of the ground truth.
Methods	AbsRel ↓	SqRel ↓	RMSE ↓	RMSE log ↓
LeRes [50]	0.391	0.472	0.999	0.661
MiDaS [35]	0.152	0.095	0.452	0.183
DPT [34]	0.191	0.135	0.563	0.220
DäRF (9 - 10 views)	0.154	0.074	0.361	0.171
DäRF (18 - 20 views)	0.151	0.071	0.356	0.168
(a) Quantitative comparison
(b) Depth distribution comparison
In addition, the loss for the adaptation of the MDE module can be redefined considering  $M$ :
$$
\mathcal {L} _ {\mathrm {M D E}} = \sum_ {I _ {i} \in \mathcal {S}} \sum_ {\mathbf {p} \in \mathcal {P}} M _ {i} (\mathbf {p}) \left(\| \mathbf {s g} \left(\bar {D} _ {i} (\mathbf {p})\right) - \bar {D} _ {i} ^ {*} (\mathbf {p}) \| + \| \left(w _ {i} \mathbf {s g} \left(\bar {D} _ {i} (\mathbf {p})\right) + q _ {i}\right) - \bar {D} _ {i} ^ {*} (\mathbf {p}) \|\right). \tag {12}
$$
With these losses, we train both NeRF and MDE simultaneously, enhancing both models by complementing each other. MDE provides a strong geometric prior to NeRF while having the inherent limitation of obliviousness to the scene-specific prior, whereas NeRF provides it with its absolute geometry.
Implementation details. DärF is implemented based on  $K$ -planes [32] as NeRF. We use DPT-hybrid [34] as MDE model. We use Adam [18] as an optimizer, with a learning rate of  $1 \cdot 10^{-2}$  for NeRF and  $1 \cdot 10^{-5}$  for the MDE, along with a cosine warmup learning rate scheduling. See supplementary material for more details. The code and pre-trained weights will be made publicly available.
Datasets. We evaluate our method in real-world scenes captured at both indoor and outdoor locations. Following previous works [37, 45], we use a subset of sparse-view ScanNet data [10] comprised with three indoor scenes, each consisting of 18 to 20 training images and 8 test images. We also conduct evaluations on more challenging setting with 9 to 10 train images. For outdoor reconstruction, we further test on 5 challenging scenes from the Tanks and Temples dataset [19]. The scenes are real-world outdoor dataset, with a wide variety of scene scales and lighting conditions. Note that these setups are extremely sparse compared to full image setups, where we use approximately 0.5 to 5 percent of the whole training inputs.
Baselines. We adopt the following six recently proposed methods as baselines: standard neural radiance field method:  $K$ -planes [13], few-shot NeRF method: RegNeRF [29], and depth prior based methods: NerfingMVS [47], DS-NeRF [11], DDP-NeRF [37], and SCADE [45].
Evaluation metrics. For quantitative comparison, we follow the NeRF [27] and report the PSNR, SSIM [46], LPIPS [56]. We report standard evaluation metrics for depth estimation [12], absolute relative error (Abs Rel), squared relative error (SqRel), root mean squared error (RMSE), root mean
(a)  $K$  -planes [13]
(b) DS-NeRF [11]
(c) DDP-NeRF [37]
(d) DàRF
(e) Ground truth
Figure 5: Qualitative results of on ScanNet [10] with 18 - 20 input views.
(a) Baseline [13]
(b) Baseline- Depth
(c) DàRF
(d) DaRF-Depth
Figure 6: Qualitative results on ScanNet [10] with 9 - 10 input views.
squared log error (RMSE log). To evaluate view consistency, we utilize a single scaling factor  $s$  for each scene, which is the median scaling [57] value averaged across all test views.
Indoor scene reconstruction. We conducted experiments in two settings: (1) a standard few-shot setup as described in literature [37, 45], and (2) an extreme few-shot setup with approximately 0.5 percent of the full images. As shown in Tab. 1, our approach outperforms the baseline methods in both settings in most of the metrics. Additionally, we provide quantitative results of the adapted MDE model in ScanNet dataset in Tab. 2, and qualitative results in Fig. 4. As shown in Fig. 5 for the setting of standard few-shot, DS-NeRF [11] and DDP-NeRF [37] still show floating artifacts in the novel view and show limitation in capturing details in the chair, smoothing into nearby object. Our method shows better qualitative results compared to other baselines, showing better geometry understanding and detailed view synthesis in the small objects near the chair. In the extreme few-shot setup, we conducted
a visual comparison between our method and a baseline [13] in Fig. 6. This is a more complex setting than standard, but our method outperforms most of the baselines, showing better geometric understanding. It should be noted that SCADE [45] fine-tunes its depth network on an indoor dataset, Taskonomy dataset [54]; whereas our model does not undergo any additional fine-tuning. More qualitative images are included in the supplementary material.
(a) Frozen MDE
Figure 4: Error map visualization. MDE adaptation results in a reduction of errors.
(b) Adapted MDE
Outdoor scene reconstruction. We conduct the qualitative and quantitative comparisons on the Tanks and Temples dataset in Tab. 1 and Fig. 7. Since COLMAP [39] with sparse images is not available, we provide comparisons with baselines without explicit depth prior. The quantitative results show that our approach outperforms the baseline methods on this complex outdoor dataset in all
(b) Baseline-Depth
(c) DaRF
(d) DaRF-Depth
Figure 7: Qualitative results on Tanks and Temples [19].
(a) Baseline [13]
Figure 8: Visualization of ablation studies on ScanNet [10].
(b) (a)  $+ \mathcal{L}_{\text{seen}}$
(c) (b) +  $\mathcal{L}_{\mathrm{unseen}}$
(d) (c) +  $\mathcal{L}_{\mathrm{MDE}}$
(e)  $(\mathrm{d}) + M$  (DaRF)
metrics. As shown in Fig. 7, our baseline shows limited performance, despite its feasible results of view synthesis in novel viewpoint, its depth results show that the network totally fails to understand 3D geometry. Our method shows rich 3D understanding, even in this real-world outdoor setting which is more complicated than other scenes. More qualitative images are included in the supplementary material.
In-the-wild dataset Reconstruction. We further investigate the robustness of our method on in-the-wild dataset, provided from SCADE [45]. This dataset was captured using an iPhoneX, with varying intrinsic parameters for each view. As shown in Tab. 3, our model outperforms all other baselines and this demonstrates that our model performs well in large scenes, such as outdoor scenes and in-the-wild dataset.
Table 3: Comparison on In-the-wild dataset [45].
Methods	PSNR↑	SSIM↑	LPIPS↓
K-planes [13]	20.71	0.732	0.408
DDP-NeRF [37]	21.28	0.727	0.366
SCADE [45]	22.82	0.743	0.347
DäRF (Ours)	22.92	0.760	0.390
Ablation on core components. In Tab. 4 and Fig. 8, we evaluate the effect of each proposed component. The quantitative results show effectiveness of each component. For qualitative results, we found out that  $\mathcal{L}_{\mathrm{unseen}}$  suppresses the artifacts in novel viewpoint, compared to when only  $\mathcal{L}_{\mathrm{seen}}$  is given. With
adaptation of the MDE network to this scene, red basket in the background shows more accurate results and artifacts near the table are removed. In our model, with confidence modeling, view synthesis results show to be more structurally confident in the overall scene.
Table 4: Ablation study.
Components	PSNR↑	SSIM↑	LPIPS↓
(a)	Baseline [13]	18.70	0.708	0.400
(b)	(a) + Lseen	19.71	0.730	0.380
(c)	(b) + Lunseen	21.21	0.758	0.333
(d)	(c) + MDE Adapt. (LCME)	21.39	0.763	0.327
(e)	(d) + Conf. Modeling (DàRF)	21.58	0.765	0.325
Analysis of local fitting. In Tab. 5, we further investigate the effectiveness of local scale-shift fitting and global scale-fitting. For global scale-shift fitting, we give learnable scale and shift parameters for each input image and convert MDE's output to the absolute value in a global manner. For a fair comparison, we compare
our model only with MDE distillation on seen viewpoints. The results show that our method local scale-shift fitting is more effective on giving accurate depth supervision.
Table 5: Local fitting ablation.
Components	PSNR↑	SSIM↑	LPIPS↓
Baseline [13]	18.65	0.706	0.502
w/ global fitting	19.05	0.698	0.399
w/ local fitting (DàRF)	19.71	0.730	0.380
Table 6: Ablation study on MDE Adaptation Loss.
Components	PSNR↑	SSIM↑	LPIPS↓	AbsRel ↓	SqRel ↓	RMSE ↓	RMSE log ↓
scale-shift loss	21.31	0.757	0.343	0.182	0.109	0.484	0.205
l1 loss	21.48	0.758	0.337	0.157	0.079	0.386	0.176
DäRF (Ours)	21.58	0.765	0.325	0.151	0.071	0.356	0.168
Analysis of MDE Adaptation Loss. In Tab. 6, we further investigate the effectiveness of scale-shift loss and  $l1$  loss at MDE adaptation. Equation 6 revolves around the idea of adapting MDE toward predicting a scene-specific absolute geometry, which is achieved by the first addend term: this first term forces itself MDE to adapt towards multiview consistency so that its ill-posed nature is reduced and its initial global depth prediction grows to be more in accordance with the absolute geometry captured by NeRF. In contrast, the second addend term, which takes into account patch-wise scale-shift fitting, is designed to aid the modeling of fine, detailed, local geometry which the model has difficulty modeling without such local fitting. As shown in the results, when only one term is used for optimization (scale-shift or  $l1$ ), it performs worse in every metric than in both are used in conjunction (ours). This justifies our strategy of using both losses as effective.
We propose DāRF, a novel method that addresses the limitations of NeRF in few-shot settings by fully leveraging the ability of monocular depth estimation networks. By integrating MDE's geometric priors, DāRF achieves robust optimization of few-shot NeRF, improving geometry reconstruction and artifact removal in both unseen and seen viewpoints. We further introduce patch-wise scale-shift fitting for accurate mapping of local depths to 3D space, and adapt MDE to NeRF's absolute scaling and multiview consistency, by distilling NeRF's absolute geometry to monocular depth estimation. Through complementary training, DāRF establishes a strong synergy between MDE and NeRF, leading to a state-of-the-art performance in few-shot NeRF. Extensive evaluations on real-world scene datasets demonstrate the effectiveness of DāRF.
This research was supported by the MSIT, Korea (IITP2022-2020-0-01819, No. 2021-0-02068, No.2021-0-00155), and National Research Foundation of Korea (NRF-2021R1C1C1006897). This research was partially supported by Culture, Sports, and Tourism R&D Program through the Korea Creative Content Agency grant funded by the Ministry of Culture, Sports and Tourism in 2023 (4D Content Generation and Copyright Protection with Artificial Intelligence, R2022020068).
[1] Benjamin Attal, Eliot Laidlaw, Aaron Gokaslan, Changil Kim, Christian Richardt, James Tompkin, and Matthew O'Toole. Törf: Time-of-flight radiance fields for dynamic scene view synthesis. Advances in neural information processing systems, 34, 2021.
[2] Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
[3] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mipnerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5470–5479, 2022.
[4] Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller. Zoedepth: Zero-shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302.12288, 2023.
[5] Amlaan Bhoi. Monocular depth estimation: A survey. arXiv preprint arXiv:1901.09402, 2019.
[6] Wenjing Bian, Zirui Wang, Kejie Li, Jia-Wang Bian, and Victor Adrian Prisacariu. Nope-nerf: Optimising neural radiance field with no pose prior. arXiv preprint arXiv:2212.07388, 2022.