Title: DäRF: Boosting Radiance Fields from Sparse Input Views with Monocular Depth Adaptation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Neural Radiance Fields

3D Reconstruction

Few-shot NeRF

Monocular Priors
Abstract: Neural radiance field (NeRF) shows powerful performance in novel view synthesis and 3D geometry reconstruction, but it suffers from critical performance degradation when the number of known viewpoints is drastically reduced. Existing works attempt to overcome this problem by employing external priors, but their success is limited to certain types of scenes or datasets. Employing monocular depth estimation (MDE) networks, pretrained on large-scale RGB-D datasets, with powerful generalization capability may be a key to solving this problem: however, using MDE in conjunction with NeRF comes with a new set of challenges due to various ambiguity problems exhibited by monocular depths. In this light, we propose a novel framework, dubbed DäRF, that achieves robust NeRF reconstruction with a handful of real-world images by combining the strengths of NeRF and monocular depth estimation through online complementary training. Our framework imposes the MDE network's powerful geometry prior to NeRF representation at both seen and unseen viewpoints to enhance its robustness and coherence. In addition, we overcome the ambiguity problems of monocular depths through patch-wise scale-shift fitting and geometry distillation, which adapts the MDE network to produce depths aligned accurately with NeRF geometry. Experiments show our framework achieves state-of-the-art results both quantitatively and qualitatively, demonstrating consistent and reliable performance in both indoor and outdoor real-world datasets.

D¨ aRF: Boosting Radiance Fields from Sparse Inputs with Monocular Depth Adaptation

Abstract

Neural radiance field (NeRF) shows powerful performance in novel view synthesis and 3D geometry reconstruction, but it suffers from critical performance degradation when the number of known viewpoints is drastically reduced. Existing works attempt to overcome this problem by employing external priors, but their success is limited to certain types of scenes or datasets. Employing monocular depth estimation (MDE) networks, pretrained on large-scale RGB-D datasets, with powerful generalization capability would be a key to solving this problem: however, using MDE in conjunction with NeRF comes with a new set of challenges due to various ambiguity problems exhibited by monocular depths. In this light, we propose a novel framework, dubbed D¨ aRF , that achieves robust NeRF reconstruction with a handful of real-world images by combining the strengths of NeRF and monocular depth estimation through online complementary training. Our framework imposes the MDE network's powerful geometry prior to NeRF representation at both seen and unseen viewpoints to enhance its robustness and coherence. In addition, we overcome the ambiguity problems of monocular depths through patch-wise scaleshift fitting and geometry distillation, which adapts the MDE network to produce depths aligned accurately with NeRF geometry. Experiments show our framework achieves state-of-the-art results both quantitatively and qualitatively, demonstrating consistent and reliable performance in both indoor and outdoor real-world datasets. Project page is available at https://ku-cvlab.github.io/DaRF/ .

1 Introduction

Neural radiance field (NeRF) [27] has gained significant attention for its powerful performance in reconstructing 3D scenes and synthesizing novel views. However, despite its impressive performance, NeRF often comes with a considerable limitation in that its performance highly relies on the presence of densely well-calibrated input images which are difficult to acquire. As the number of input images is reduced, NeRF's novel view synthesis quality drops significantly, displaying failure cases such as erroneous overfitting to the input images [15, 29], artifacts clouding empty spaces [29], or degenerate geometry that yields incomprehensible jumble when rendered at unseen viewpoints [16]. These challenges derive from its under-constrained nature, causing it to have extreme difficulty mapping a pixel in input images to a correct 3D location. In addition, NeRF's volume rendering allows the model to map a pixel to multiple 3D locations [11], exacerbating this problem.

Previous few-shot NeRF methods attempt to solve these issues by imposing geometric regularization [29, 16, 20] or exploiting external 3D priors [11, 37] such as depth information extracted from

NeRF

Representa+on

DäRF (Ours)

MDE

Network

❄

Unseen

Viewpoint

Seen

Viewpoint

Ground Truth

Seen

Viewpoint

Ground Truth

Rendered Depth

Rendered Depth

Rendered Depth

Rendered Color

MDE

Network

🔥

Adapta+on

Seen

View Depth Dis+lla+on

Only

Patch-wise

Depth Dis+lla+on

Unseen

View

Depth Dis+lla+on

❄

🔥

: Adapted

○--○

↑0---O-to

input images by COLMAP [39]. However, these methods have weaknesses in that they use 3D priors extracted from a few input images only, which prevents such guidance from encompassing the entire scene. To effectively tackle all the issues mentioned above, pretrained monocular depth estimation (MDE) networks with strong generalization capability [35, 34, 4] could be used to inject an additional 3D prior into NeRF that facilitates robust geometric reconstruction. Specifically, geometry prediction by MDE can constrain NeRF into recovering smooth and coherent geometry, while their bias towards predicting smooth geometry helps to filter out fine-grained artifacts that clutter the scene. More importantly, NeRF's capability to render any unseen viewpoints enables fully exploiting the capability of the MDE, as MDE could provide depth prior to the numerous renderings of unseen viewpoints as well as the original input viewpoints. This allows injecting additional 3D prior to effectively covering the entire scene instead of being constrained to a few input images.

However, applying MDE to few-shot NeRF is not trivial, as there are ambiguity problems that hinder the monocular depth from serving as a good 3D prior. Primarily, relative depths predicted by MDEs are not multiview-consistent [5]. Moreover, MDEs perform poorly in estimating depth differences between multiple objects: this prevents global scale-shift fitting [55, 26] from being a viable solution, as alignment to one region of the scene inevitably leads to misalignment in many other regions. There also exists a convexity problem [26], in which the MDE has difficulty determining whether the surface is planar, convex, or concave, are also present. To overcome these challenges, we introduce a novel method to adapt MDE to NeRF's absolute scaling and multiview consistency as NeRF is regularized by MDE's powerful 3D priors, creating a complementary cycle.

In this paper, we propose D¨ aRF , short for Monocular D epth A daptation for boosting R adiance F ields from Sparse Input Views, which achieves robust optimization of few-shot NeRF through MDE's geometric prior, as well as MDE adaptation for alignment with NeRF through complementary training (see Fig. 1). We exploit MDE for robust geometry reconstruction and artifact removal in both unseen and seen viewpoints. In addition, we leverage NeRF to adapt MDE toward multiview-consistent geometry prediction and introduce novel patch-wise scale-shift fitting to more accurately map local depths to NeRF geometry. Combined with a confidence modeling technique for verifying accurate depth information, our method achieves state-of-the-art performance in few-shot NeRF optimization. We evaluate and compare our approach on real-world indoor and outdoor scene datasets, establishing new state-of-the-art results for the benchmarks.

2 Related Work

Neural radiance field. Neural radiance field (NeRF) [27] represents photo-realistic 3D scenes with MLP. Owing to its remarkable performance, there has been a variety of follow-up studies [2, 51, 25].

These studies improve NeRF such as dynamic and deformable scenes [31, 44, 33, 1], real-time rendering [51, 36, 28], unbounded scene [3, 42, 48] and generative modeling [40, 30, 7]. However, these works still encounter challenges in synthesizing novel views with a limited number of images in a single scene, limiting their applicability in real-world scenarios.

Few-shot NeRF. Numerous few-shot NeRF works attempted to address few-shot 3D reconstruction problem through various techniques, such as pretraining external priors [52, 9], meta-learning [43], regularization [15, 29, 16, 20] or off-the-shelf modules [15, 29]. Recent approaches [29, 16, 20] emphasize the importance of geometric consistency and apply geometric regularization at unknown viewpoints. However, these regularization methods show limitations due to their heavy reliance on geometry information recovered by NeRF. Other works such as DS-NeRF [11], DDP-NeRF [37] and SCADE [45] exploit additional geometric information, such as COLMAP [39] 3D points or monocular depth estimation, for geometry supervision. However, these works have critical limitations of only being able to provide geometry information corresponding to existing input viewpoints. Unlike these works, our work demonstrates methods to provide geometric prior even at unknown viewpoints with MDE for more effective geometry reconstruction.

Monocular depth estimation. Monocular depth estimation (MDE) is a task that aims to predict a dense depth map given a single image. Early works on MDE used handcrafted methods such as MRF for depth estimation [38]. After the advent of deep learning, learning-based approaches [14, 17, 21] were introduced to the field. In this direction, the models were trained on ground-truth depth maps acquired by RGB-D cameras or LiDAR sensors to predict absolute depth values [24, 23]. Other approaches trained the networks on large-scale diverse datasets [8, 22, 34, 35], which demonstrates better generalization power. These approaches struggle with depth ambiguity caused by ill-posed problem, so the following works LeRes [50] and ZoeDepth [4] opt to recover absolute depths using additional parameters.

Incorporating MDE into 3D representation. As both NeRF and monocular depth estimation are closely related, there have been some works that utilize MDE models to enhance NeRF's performance. NeuralLift [49], MonoSDF [53] and SCADE [45] leverage depths predicted by pretrained MDE for depth ordering and detailed surface reconstruction, respectively. Other works optimize scene-specific parameters, such as depth predictor utilizing depth recovered by COLMAP [47] or learnable scaleshift values for reconstruction in noisy pose setting [6]. However, these previous approaches were limited in that MDEs were used to provide prior to only the input viewpoints, which constrains their effectiveness when input views are reduced, e.g., in the few-shot setting.

As a concurrent work, SCADE [45] utilizes MDE for sparse view inputs, by injecting uncertainty into MDEthrough additional pretraining so that canonical geometry can be estimated through probabilistic modeling between multiple modes of estimated depths. While the ultimate goal which is to overcome the ambiguity of MDE may be similar, our approach directly removes ambiguity present in MDE by finetuning with canonical geometry captured by NeRF, for effective suppression of artifacts and divergent behaviors of few-shot NeRF.

3 Preliminaries

NeRF [27] represents a scene as a continuous function F θ ( · ) represented by a neural network with parameters θ . During optimization, 3D points are sampled along rays represented by r coming from a set of input images S = { I i } , whose ground truth camera poses are given, for evaluation by the neural network. For each sampled point, F θ ( · ) takes as input its coordinate x ∈ R 3 and viewing direction d ∈ R 2 with a positional encoding γ ( · ) that facilitates learning high-frequency details, and outputs a color c ∈ R 3 and a density σ ∈ R such that { c , σ } = F θ ( γ ( x ) , γ ( d )) . With a ray parameterized as r p ( t ) = o + t d p , starting from camera center o along the direction d p , color and depth value at the pixel p are rendered as follows:

where ¯ I ( p ) and ¯ D ( p ) are rendered color and depth values at the pixel p along the ray r p ( t ) from t n to t f , and T ( t ) denotes an accumulated transmittance along the ray from t n to t as follows:

(a)

(b)

(c)

Based on this volume rendering, F θ ( · ) is optimized by the reconstruction loss L recon that compares rendered color ¯ I ( p ) to corresponding ground-truth I ( p ) , with R as a set of pixels for training rays:

Our work explores the setting of few-shot optimization with NeRF [16, 20]. Whereas the number of input viewpoints |S| is normally higher than one hundred in the standard NeRF setting [27], the task of few-shot NeRF considers scenarios when |S| is drastically reduced to a few viewpoints (e.g., |S| < 20 ). With such a small number of input viewpoints, NeRF shows high divergent behaviors such as geometry breakdown, overfitting to input viewpoints, and generation of artifacts that cloud the empty space between the camera and object, which causes its performance to drop sharply [15, 16, 29]. To overcome this problem, existing few-shot NeRF frameworks applied regularization techniques at unknown viewpoints to constrain NeRF with additional 3D priors [37, 11] and enhance the robustness of geometry, but they showed limited performance.

4 Methodology

4.1 Motivation and Overview

Our framework leverages the complementary benefits of few-shot NeRF and monocular depth estimation networks for the goal of robust 3D reconstruction. The benefits that pretrained MDE can provide to few-shot NeRF are clear and straightforward: because they predict dense geometry, they provide guidance for the NeRF to recover more smooth geometry. In cases where few-shot NeRF's geometry undergoes divergent behaviors, MDE provides strong constraints to prevent the global geometry from breaking down.

However, there are difficult challenges that must be overcome if the depths estimated by MDE are to be used as 3D prior to NeRF. These challenges, which can be summarized as depth ambiguity problems [26], stem from the inherent ill-posed nature of the monocular depth estimation. Most importantly, MDE networks only predict relative depth information inferred from an image, meaning it is initially not aligned to NeRF's absolute geometry [4]. Global scaling and shifting may seem to be the answer, but this approach leads us to another depth ambiguity problem, as predicted scales and spacings of each instance are inconsistent with one another, as demonstrated in (b) of Fig. 2. Additionally, MDE's weakness in predicting the convexity of a surface, whether it is flat, convex, or concave - also poses a problem in using this depth for NeRF guidance.

In this light, we adapt a pre-trained monodepth network to a single NeRF scene so that its powerful 3D prior can be leveraged to its maximum capability in regularizing the few-shot NeRF. In the following, we first explain how to distill geometric prior from off-the-shelf MDE model [35] from both seen and unseen viewpoints (Sec. 4.2). We also provide a strategy for adapting the MDE model to handle ill-posed problems to a specific scene, while keeping its 3D prior knowledge (Sec. 4.3). Then, we demonstrate a method to handle inaccurate depths (Sec. 4.4). Fig. 1 shows an overview of our method, compared to previous works using MDE prior [45, 53].

(a) Rendered color

¯

I

(b) Rendered depth

¯

D

(c) Mono-depth from

¯

I

(d) Mono-depth from

I

4.2 Distilling Monocular Depth Prior into Neural Radiance Field

To prevent the degradation of reconstruction quality in few-shot NeRF, we propose to distill monocular depth prior to the neural radiance field during optimization. By exploiting pre-trained MDE networks [34, 35], which have high generalization power, we enforce a dense geometric constraint on both seen and unseen viewpoints by using estimated monocular depth maps as pseudo ground truth depth for training few-shot NeRF. We describe the details of this process below.

Monocular depth regularization on seen views. Weleverage a pre-trained MDE model, denoted as G ϕ ( · ) with parameters ϕ , to predict pseudo depth map from given seen view image I i as D ∗ i = G ϕ ( I i ) . Since D ∗ i is initially a relative depth map, it needs to be scaled and shifted into an absolute depth [55] and aligned with NeRF's rendered depth ¯ D in order for it to be used as pseudo-depth D ∗ . However, the scale and shift parameters inferred from the global statistic may undermine local statistic [55]. For example, as shown in Fig. 2 (b), global scale fitting tends to favor dominant objects in the image, leading to ill-fitted depths in less dominant sections of the scene due to inconsistencies in predicted depth differences between the objects. Na¨ ıvely employing such inaccurately estimated depths for distillation can adversely impact the overall geometry of the NeRF.

To alleviate this issue, we propose a patch-wise adjustment of scale and shift parameters, reducing the impact of erroneous depth differences, as illustrated in Fig. 2 (c). The depth consistency loss is defined as follows:

where w i and q i denote the scale and shift parameters obtained by least square [35] between D ∗ i and ¯ D i , P denotes a set of pixels within a patch, and sg ( · ) denotes stop-gradient operation. Thus patch-based approach also helps to overcome the computational bottleneck of full image rendering.

Monocular depth regularization on unseen views. We further propose to give supervision even at unseen viewpoints. As NeRF has the ability to render any unseen viewpoint of the scene, we render color ¯ I l and depth ¯ D l from a sampled patch of l -th novel viewpoint. Sequentially, we extract a monocular depth map from the rendered image as ¯ D ∗ l = G ϕ ( ¯ I l ) . Then, we enforce consistency between our rendered depth ¯ D l and the monocular depth ¯ D ∗ l of l -th novel viewpoint as follows:

where U denotes a set of unseen view images, w l and q l denotes the scale and shift parameters used to align ¯ D ∗ l towards ¯ D l , and P denotes randomly sampled patch.

A valid concern regarding this approach is that monocular depth obtained from noisy NeRF rendering may be affected by fine-grained rendering artifacts that frequently appear in unseen viewpoints of few-shot NeRF, resulting in noisy and erroneous pseudo-depths. However, we demonstrate in

Fig. 3 that a strong geometric prior within the MDE model exhibits robustness against such artifacts, effectively filtering out the artifacts and thereby providing reliable supervision for the unseen views.

It should be noted that our strategy differs from previous methods [11, 37, 53, 45] that exploit monocular depth estimation [34] and external depth priors such as COLMAP [39]. These methods only impose depth priors upon the input viewpoints, and thus their priors only influence the scene partially due to self-occlusions and sparsity of known views. Our method, on the other hand, enables external depth priors to be applied to any arbitrary viewpoint and thus allows guidance signals to thoroughly reach every location of the scene, leading to more robust and coherent NeRF optimization.

4.3 Adaptation of MDE via Neural Radiance Field

Although the patch-wise distillation of monocular depth provides invariance to depth difference inconsistency in MDE, the ill-posed nature of monocular depth estimation often introduces additional ambiguities, such as the inability to distinguish whether the surface is concavity, convexity, or planar or difficulty in determining the orientation of flat surfaces [26]. We argue that these ambiguities arise due to the MDE lacking awareness of the scene-specific absolute depth priors and multiview consistency. To address this issue, we propose providing the scene priors optimized NeRF to MDE, whose knowledge of canonical space and absolute geometry helps eliminate the ambiguities present within MDE. Therefore, we propose to adapt the MDE to the absolute scene geometry, formally written as:

In addition to the patch-wise loss in Eq. 4, we add an l -1 loss without scale-shift adjustment to adapt the MDE with absolute depth prior. We also introduce a regularization term to preserve the local smoothness of MDE, given by:

where D ∗ , init i denotes monocular depth map of I i extracted from MDE with initial pre-trained weight.

4.4 Confidence Modeling

Our framework must take into account the errors present in both few-shot NeRF and estimated monocular depths, which will propagate [41] and intensify during the distillation process if left unchecked. To prevent this, we adopt confidence modeling [20, 41] inspired by self-training approaches [41], to verify the accuracy and reliability of each ray before the distillation process.

The homogeneous coordinates of a pixel p in the seen viewpoint are transformed to p ′ at the target viewpoint using the viewpoint difference R i → l and the camera intrinsic parameter K , as follows:

We generate the confidence map M i by measuring the distance between rendered depth of the unseen viewpoint and MDE output of seen viewpoint such that

where τ denotes threshold parameter, [ · ] is Iverson bracket, and D l ( p ′ ) refers to depth value of the corresponding pixel at l -th unseen viewpoint for reprojected target pixel p of i -th seen viewpoint. We fit D ∗ i to absolute scale, where scale and shift parameters, w i and q i , are obtained by least square [35] between D ∗ i and ¯ D i .

4.5 Overall Training

With the incorporation of confidence modeling, the loss functions for both the radiance field and MDE can redefined. L seen and L unseen can be redefined as:

(a) Quantitative comparison

(b) Depth distribution comparison

In addition, the loss for the adaptation of the MDE module can be redefined considering M :

With these losses, we train both NeRF and MDE simultaneously, enhancing both models by complementing each other. MDE provides a strong geometric prior to NeRF while having the inherent limitation of obliviousness to the scene-specific prior, whereas NeRF provides it with its absolute geometry.

5 Experiments

5.1 Experimental Settings

Implementation details. D¨ aRF is implemented based on K -planes [32] as NeRF. We use DPThybrid [34] as MDE model. We use Adam [18] as an optimizer, with a learning rate of 1 · 10 -2 for NeRF and 1 · 10 -5 for the MDE, along with a cosine warmup learning rate scheduling. See supplementary material for more details. The code and pre-trained weights will be made publicly available.

Datasets. We evaluate our method in real-world scenes captured at both indoor and outdoor locations. Following previous works [37, 45], we use a subset of sparse-view ScanNet data [10] comprised with three indoor scenes, each consisting of 18 to 20 training images and 8 test images. We also conduct evaluations on more challenging setting with 9 to 10 train images. For outdoor reconstruction, we further test on 5 challenging scenes from the Tanks and Temples dataset [19]. The scenes are real-world outdoor dataset, with a wide variety of scene scales and lighting conditions. Note that these setups are extremely sparse compared to full image setups, where we use approximately 0.5 to 5 percent of the whole training inputs.

Baselines. We adopt the following six recently proposed methods as baselines: standard neural radiance field method: K -planes [13], few-shot NeRF method: RegNeRF [29], and depth prior based methods: NerfingMVS [47], DS-NeRF [11], DDP-NeRF [37], and SCADE [45].

Evaluation metrics. For quantitative comparison, we follow the NeRF [27] and report the PSNR, SSIM [46], LPIPS [56]. We report standard evaluation metrics for depth estimation [12], absolute relative error (Abs Rel), squared relative error (SqRel), root mean squared error (RMSE), root mean

(a)

K

-planes [13]

(b) DS-NeRF [11]

(c) DDP-NeRF [37]

(d)

D¨

aRF

(e) Ground truth

(a) Baseline [13]

(b) Baseline- Depth

(c)

D¨

aRF

(d)

D¨

aRF

- Depth

(e) Ground truth

371

squared log error (RMSE log). To evaluate view consistency, we utilize a single scaling factor s for each scene, which is the median scaling [57] value averaged across all test views.

5.2 Comparisons

Indoor scene reconstruction. We conducted experiments in two settings: (1) a standard few-shot setup as described in literature [37, 45], and (2) an extreme few-shot setup with approximately 0.5 percent of the full images. As shown in Tab. 1, our approach outperforms the baseline methods in both settings in most of the metrics. Additionally, we provide quantitative results of the adapted MDE model in ScanNet dataset in Tab. 2, and qualitative results in Fig. 4. As shown in Fig. 5 for the setting of standard few-shot, DS-NeRF [11] and DDPNeRF [37] still show floating artifacts in the novel view and show limitation in capturing details in the chair, smoothing into nearby object. Our method shows better qualitative results compared to other baselines, showing better geometry understanding and detailed view synthesis in the small objects near the chair. In the extreme few-shot setup, we conducted

(b) Adapted MDE

a visual comparison between our method and a baseline [13] in Fig. 6. This is a more complex setting than standard, but our method outperforms most of the baselines, showing better geometric understanding. It should be noted that SCADE [45] fine-tunes its depth network on an indoor dataset, Taskonomy dataset [54]; whereas our model does not undergo any additional fine-tuning. More qualitative images are included in the supplementary material.

Outdoor scene reconstruction. We conduct the qualitative and quantitative comparisons on the Tanks and Temples dataset in Tab. 1 and Fig. 7. Since COLMAP [39] with sparse images is not available, we provide comparisons with baselines without explicit depth prior. The quantitative results show that our approach outperforms the baseline methods on this complex outdoor dataset in all

(a) Baseline [13]

(b) Baseline- Depth

(c)

D¨

aRF

(d)

D¨

aRF

- Depth

(a) Baseline [13]

(b) (a) + L seen

(c) (b) +

L

unseen

(d) (c) + L MDE

(e) (d) + M ( D¨ aRF )

metrics. As shown in Fig. 7, our baseline shows limited performance, despite its feasible results of view synthesis in novel viewpoint, its depth results show that the network totally fails to understand 3D geometry. Our method shows rich 3D understanding, even in this real-world outdoor setting which is more complicated than other scenes. More qualitative images are included in the supplementary material.

In-the-wild dataset Reconstruction. We further investigate the robustness of our method on in-the-wild dataset, provided from SCADE [45]. This dataset was captured using an iPhoneX, with varying intrinsic parameters for each view. As shown in Tab. 3, our model outperforms all other baselines and this demonstrates that our model performs well in large scenes, such as outdoor scenes and in-the-wild dataset.

5.3 Ablation Study

Ablation on core components. In Tab. 4 and Fig. 8, we evaluate the effect of each proposed component. The quantitative results show effectiveness of each component. For qualitative results, we found out that L unseen suppresses the artifacts in novel viewpoint, compared to when only L seen is given. With

adaptation of the MDE network to this scene, red basket in the background shows more accurate results and artifacts near the table are removed. In our model, with confidence modeling, view synthesis results show to be more structurally confident in the overall scene.

Analysis of local fitting. In Tab. 5, we further investigate the effectiveness of local scale-shift fitting and global scale-fitting. For global scale-shift fitting, we give learnable scale and shift parameters for each input image and convert MDE's output to the absolute value in a global manner. For a fair comparison, we compare

our model only with MDE distillation on seen viewpoints. The results show that our method local scale-shift fitting is more effective on giving accurate depth supervision.

Analysis of MDE Adaptation Loss. In Tab. 6, we further investigate the effectiveness of scale-shift loss and l 1 loss at MDE adaptation. Equation 6 revolves around the idea of adapting MDE toward predicting a scene-specific absolute geometry, which is achieved by the first addend term: this first term forces itself MDE to adapt towards multiview consistency so that its ill-posed nature is reduced and its initial global depth prediction grows to be more in accordance with the absolute geometry captured by NeRF. In contrast, the second addend term, which takes into account patch-wise scaleshift fitting, is designed to aid the modeling of fine, detailed, local geometry which the model has difficulty modeling without such local fitting. As shown in the results, when only one term is used for optimization (scale-shift or l 1), it performs worse in every metric than in both are used in conjunction (ours). This justifies our strategy of using both losses as effective.

6 Conclusion

We propose D¨ aRF , a novel method that addresses the limitations of NeRF in few-shot settings by fully leveraging the ability of monocular depth estimation networks. By integrating MDE's geometric priors, D¨ aRF achieves robust optimization of few-shot NeRF, improving geometry reconstruction and artifact removal in both unseen and seen viewpoints. We further introduce patch-wise scale-shift fitting for accurate mapping of local depths to 3D space, and adapt MDE to NeRF's absolute scaling and multiview consistency, by distilling NeRF's absolute geometry to monocular depth estimation. Through complementary training, D¨ aRF establishes a strong synergy between MDE and NeRF, leading to a state-of-the-art performance in few-shot NeRF. Extensive evaluations on real-world scene datasets demonstrate the effectiveness of D¨ aRF .

References

Benjamin Attal, Eliot Laidlaw, Aaron Gokaslan, Changil Kim, Christian Richardt, James Tompkin, and Matthew O'Toole. T¨ orf: Time-of-flight radiance fields for dynamic scene view synthesis. Advances in neural information processing systems , 34, 2021.

Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021.

Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mipnerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5470-5479, 2022.

Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias M¨ uller. Zoedepth: Zero-shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302.12288 , 2023.

Amlaan Bhoi. Monocular depth estimation: A survey. arXiv preprint arXiv:1901.09402 , 2019.

Wenjing Bian, Zirui Wang, Kejie Li, Jia-Wang Bian, and Victor Adrian Prisacariu. Nope-nerf: Optimising neural radiance field with no pose prior. arXiv preprint arXiv:2212.07388 , 2022.

Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometryaware 3d generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16123-16133, 2022.

Weifeng Chen, Zhao Fu, Dawei Yang, and Jia Deng. Single-image depth perception in the wild, 2017.

Julian Chibane, Aayush Bansal, Verica Lazova, and Gerard Pons-Moll. Stereo radiance fields (srf): Learning view synthesis for sparse views of novel scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7911-7920, 2021.

Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Niessner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , July 2017.

Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan. Depth-supervised NeRF: Fewer views and faster training for free. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2022.

David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep network. Advances in neural information processing systems , 27, 2014.

Sara Fridovich-Keil, Giacomo Meanti, Frederik Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. arXiv preprint arXiv:2301.10241 , 2023.

Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao. Deep ordinal regression network for monocular depth estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2002-2011, 2018.

Ajay Jain, Matthew Tancik, and Pieter Abbeel. Putting nerf on a diet: Semantically consistent few-shot view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5885-5894, 2021.

Mijeong Kim, Seonguk Seo, and Bohyung Han. Infonerf: Ray entropy minimization for fewshot neural volume rendering. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) , 2022.

Seungryong Kim, Kihong Park, Kwanghoon Sohn, and Stephen Lin. Unified depth prediction and intrinsic image decomposition from a single image via joint convolutional neural fields. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14 , pages 143-159. Springer, 2016.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. iclr. 2015. arXiv preprint arXiv:1412.6980 , 9, 2015.

Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun. Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Transactions on Graphics (ToG) , 36(4):1-13, 2017.

Minseop Kwak, Jiuhn Song, and Seungryong Kim. Geconerf: Few-shot neural radiance fields via geometric consistency. arXiv preprint arXiv:2301.10941 , 2023.

Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, and Nassir Navab. Deeper depth prediction with fully convolutional residual networks. In 2016 Fourth international conference on 3D vision (3DV) , pages 239-248. IEEE, 2016.

Jae-Han Lee and Chang-Su Kim. Monocular depth estimation using relative depth maps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9729-9738, 2019.

Jin Han Lee, Myung-Kyu Han, Dong Wook Ko, and Il Hong Suh. From big to small: Multi-scale local planar guidance for monocular depth estimation. arXiv preprint arXiv:1907.10326 , 2019.

Bo Li, Chunhua Shen, Yuchao Dai, Anton Van Den Hengel, and Mingyi He. Depth and surface normal estimation from monocular images using regression on deep features and hierarchical crfs. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1119-1127, 2015.

Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. NeurIPS , 2020.

S Mahdi H Miangoleh, Sebastian Dille, Long Mai, Sylvain Paris, and Yagiz Aksoy. Boosting monocular depth estimation models to high-resolution via content-adaptive multi-resolution merging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9685-9694, 2021.

Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV , 2020.

Thomas M¨ uller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. arXiv preprint arXiv:2201.05989 , 2022.

Michael Niemeyer, Jonathan T. Barron, Ben Mildenhall, Mehdi S. M. Sajjadi, Andreas Geiger, and Noha Radwan. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) , 2022.

Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional generative neural feature fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11453-11464, 2021.

Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5865-5874, 2021.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10318-10327, 2021.

Ren´ e Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1217912188, 2021.

Ren´ e Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE transactions on pattern analysis and machine intelligence , 44(3):1623-1637, 2020.

Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 14335-14345, 2021.

Barbara Roessle, Jonathan T Barron, Ben Mildenhall, Pratul P Srinivasan, and Matthias Nießner. Dense depth priors for neural radiance fields from sparse input views. arXiv preprint arXiv:2112.03288 , 2021.

Ashutosh Saxena, Min Sun, and Andrew Y Ng. Make3d: Learning 3d scene structure from a single still image. IEEE transactions on pattern analysis and machine intelligence , 31(5):824840, 2008.

Johannes L. Schonberger and Jan-Michael Frahm. Structure-from-motion revisited. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2016.

Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. Advances in Neural Information Processing Systems , 33:20154-20166, 2020.

Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semisupervised learning with consistency and confidence. Advances in neural information processing systems , 33:596-608, 2020.

Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srinivasan, Jonathan T Barron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8248-8258, 2022.

Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P. Srinivasan, Jonathan T. Barron, and Ren Ng. Learned initializations for optimizing coordinate-based neural representations. In CVPR , 2021.

Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollh¨ ofer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 12959-12970, 2021.

Mikaela Angelina Uy, Ricardo Martin-Brualla, Leonidas Guibas, and Ke Li. Scade: Nerfs from space carving with ambiguity-aware depth estimates. arXiv preprint arXiv:2303.13582 , 2023.

Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing , 13(4):600-612, 2004.

Yi Wei, Shaohui Liu, Yongming Rao, Wang Zhao, Jiwen Lu, and Jie Zhou. Nerfingmvs: Guided optimization of neural radiance fields for indoor multi-view stereo. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5610-5619, 2021.

Yuanbo Xiangli, Linning Xu, Xingang Pan, Nanxuan Zhao, Anyi Rao, Christian Theobalt, Bo Dai, and Dahua Lin. Bungeenerf: Progressive neural radiance field for extreme multi-scale scene rendering. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXII , pages 106-122. Springer, 2022.

Dejia Xu, Yifan Jiang, Peihao Wang, Zhiwen Fan, Yi Wang, and Zhangyang Wang. Neurallift360: Lifting an in-the-wild 2d photo to a 3d object with 360 {\ deg } views. arXiv preprint arXiv:2211.16431 , 2022.

Wei Yin, Jianming Zhang, Oliver Wang, Simon Niklaus, Long Mai, Simon Chen, and Chunhua Shen. Learning to recover 3d scene shape from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 204-213, 2021.

Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. PlenOctrees for real-time rendering of neural radiance fields. In ICCV , 2021.

Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4578-4587, 2021.

Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger. Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. arXiv preprint arXiv:2206.00665 , 2022.

Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3712-3722, 2018.

Chi Zhang, Wei Yin, Billzb Wang, Gang Yu, Bin Fu, and Chunhua Shen. Hierarchical normalization for robust monocular depth estimation. Advances in Neural Information Processing Systems , 35:14128-14139, 2022.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR , 2018.

Zhoutong Zhang, Forrester Cole, Richard Tucker, William T Freeman, and Tali Dekel. Consistent depth of moving objects in video. ACM Transactions on Graphics (TOG) , 40(4):1-12, 2021.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method to better use monocular depth in a few-shot NeRF setup. There are mainly two technical contributions to me:\n1. Applying mono depth constraint to unseen view;\n2. Un-distorting (scale and shift) monocular depth in a per-patch manner, rather than per-image.\n\n---\n**After rebuttal**: I have read authors' rebuttal and it addresses my concerns.\n\nSTRENGTHS:\n1. The motivation makes a lot of sense. Monocular depth estimation suffers from various distortions, and un-distorting it with a global scale and shift per image is only a rough approximation. Exploring better ways to un-distort monocular depth is definitely valid motivation.\n2. It’s interesting to see that monocular depth networks actually perform decently on rendered images from half-trained NeRF models.\n\nWEAKNESSES:\nGeneral\n1. L118-L120: I disagree with this definition of few-shot, i.e. $| S < 20 |$. The definition should be based on view-angle and scene coverage, not on an absolute number of images. For example, in LLFF, a forward-facing scene, NeRF can be trained very well for $| S < 20 |$, because 20 images cover the simple forward-facing scenes very well. In contrast, 50 images might be challenging for 360-degree scenes.\n2. Following point 1, how do the few-shot views cover the evaluated scenes? It would be good to have a top-view visualisation of selected cameras' positions and orientations.\n3. How many patches are used in training? From the supp mat I can see the patch size is 64x64, i.e. 4096 rays. This means there is only one patch used during training? In this case, modelling patch-wise scale/shift is same as image-wise scale/shift?\n\nWriting\n1. Figure 1 caption: unclear, seems like should be … by _applying_ pretrained MDE … (missing a word _applying_)\n2. Figure 2: \n    1. what is the input RGB and its monocular depth? I can see this is a top-view image of back-projected point clouds, but I cannot imagine what’s the input. \n    2. What is colour coding?\n    3. I don’t see why Fig 2b is better than Fig 2a from this image. I understand the motivation and it is supposed to be better when un-distort with patch-wise scale/shift, but I cannot see that from this visualisation.\n3. Sec 4.1 is kind of repeating intro. \n4. Symbol $M_l$ is undefined. I suppose it’s similar to $M_i$, but projected in $l \\rightarrow i$ direction?\n5. Symbol $s_i, t_i$ are undefined.\n\nQUESTIONS:\nSee the weaknesses section.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles the problem of sparse view NeRF reconstruction by leveraging on monocular depth estimation networks as a prior. The main difference between DaRF and existing work is it also computes for a depth loss on unseen views, in contrast to prior works that only constrain depth on the training views. Moreover, they also adapt, i.e. fine tune, the monocular depth estimation network to agree with the depth produced by the NeRF. They also introduce a confidence term that determines which pixels to use the depth loss on. They conduct experiments on Scannet and Tanks and Temples dataset.\n\nSTRENGTHS:\nThe paper proposes to use monocular depth estimation as a prior to constrain NeRF optimization under the sparse view regime. Different from existing works, DaRF found that they can also use the monocular depth prior on unseen views, and even on NeRF rendered depth on early stage of the training, the monocular depth network is able to produce reasonable and cleaner depth maps that can supervise and constrain the NeRF loss (as shown in Figure 3). This is the main contribution and finding of this submission. The authors also showed an ablation study to justify the different components that were introduced.\n\nWEAKNESSES:\nThe concerns I have for the paper is whether 1) the contribution introduced is enough for paper acceptance, and 2) how it positions itself and it claims in contrast to existing works. For the first point, the main contribution and distinction it has is the finding that the monocular depth prior can also be used to constrain unseen viewpoints. As shown in Figure 3, even on noisy NeRF renderings, the MDE prior can produce clean depth maps. An add-on to this is the slight improvement in results (as shown in the ablation study) by also fine-tuning the MDE network as NeRF is being optimized. However, the way the paper was written and motivated is the ambiguity in monocular depth estimation -- as written in the intro, Sec 4.1 and Sec 4.3, which is the main premise and story of the existing (though somewhat concurrent) work SCADE. The paper did cite SCADE, but however only mentions the difference that DaRF is able to train on unseen views. Differentiating with existing work and clarifying contributions can be improved for both paper presentation as well as the claims made for the submission. Now, the concern and question is whether 1) being the contribution of DaRF is enough as a contribution to meet the bar of Neurips. And for this reason I am giving an initial rating of borderline reject and would like to hear from the other reviewers.\n\nQUESTIONS:\n1. How are the viewpoints selected for the unseen views that was used for L_{unseen}? How many unseen viewpoints are selected and how is the loss balanced with the seen training views? This in my opinion is an important detail to justify and back its main claim.\n2. What is the intuition behind using two terms in Eq 6? Specifically, wouldn't the second addend term be enough? It seems redundant and not too intuitive.\n3. What dataset was used for the Ablation study (Table 3)? Is it the average across all scenes in Scannet? Figure 8 shows one example from scene781 of scannet.\n\nLIMITATIONS:\nThe authors included the limitations in their supplementary material. However, according to the checklist, they claim to include the error bars. However, I don't think I saw error bars reported in the submission.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of few-shot NeRF reconstruction. The authors propose using monocular depth estimation (MDE) networks to provide geometry prior to NeRF at both seen and unseen viewpoints. They propose overcoming the ambiguity problems associated with monocular depths by MDE adaption. Experimental results deomonstrate improvements in both rendered novel views and rendered depths.\n\nSTRENGTHS:\n+ Unlike previous works which only exploit depth information at seen viewpoints, this work also exploits MDE to constraint NeRF at unseen viewpoints, leading to more robust and coherent NeRF optimization. The authors demonstrate through an example that the strong geometric prior within the MDE model enables it to generate reliable depth from noisy NeRF rendering. This makes MDE at unseen viewpoints feasible.\n+ The proposed patch-wise scale-shift fitting helps reducing the impact of erroneous depth differences generated by MDE networks in distilling the monoclular depth prior to the NeRF.\n+ Adapting MDE to absolute scene geometry embedded in NeRF further helps to resolve ambiguities in surface curvature and orientation, and improve multiview consistency.\n+ The proposed method demonstrates sota results, both qualitatively and quantitatively, on two real-data sets, particularly showing superb results in rendered depths in few-shot NeRF.\n+ Ablation study has been included to demonstrate the effectiveness of each major deisgn component.\n+ Overall this paper is well-written and well-organized. It is easy to follow.\n\nWEAKNESSES:\n- In MDE adaption, it is not clear why the monocular depths predicted from the rendered input views instead of that predicted from the input views are adopted in (6). There is no explanations or disucssions on the effect of choosing between these two. There is also no explanations or discussions on why only input views are considered.\n- In confidence modeling, why the predicted depth of a point in an input view (after scaling ana shifting) is directly compared with its rendered depth in a target view in (9)? Note both predicted depth and rendered depth are measured with respect to the viewpoints.\n\nQUESTIONS:\n- Figure 2 is rather confusing. It is not clear how one should interpret / understand the renderings. More descriptive caption should be provided.\n- Are there any failure cases? Any analysis for the causes of failure?\n- What is the minimum number of viewpoints for the proposed model to produce reasonable reconstruction?\n\nLIMITATIONS:\nLimitations and impact are only included in the supplementary material, but not in the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper presents a new few-shot neural radience field approach based on joint monocular depth adaption. The main idea of the proposed approach is to utilize the monocular depth estimator to improve the geometry prior of NeRF representation. The motivation is reasonable. Also, it presents attractive performance on both indoor and outdoor scenes.

STRENGTHS:
1. The idea of utilizing monocular depth estimator is interesing and the depth estimator can provide reasonable geometry information for the scene reference. 

2. The proposed approach has provided attractive performance on the real-world benchmarks. Sufficient ablations have been conducted to validate the design of the proposed algorithm. 

3. The presentation of the paper is good.

WEAKNESSES:
1. The paper is based on SCADE[49]. According the results in Table 1, the proposed approach is lower than [49] on the LPIPS score. Is there any potential explanation of this result? Also, How about the results on the "in-the-wild benchmark" proposed in [49]?

2. How is case when the testing data is out of the distribution of the depth estimator? For example, in the case when the depth estimator failed to predict the accurate depth information.

3. How about the inference speed of the proposed algorithm?

QUESTIONS:
Please address the comments released in the weakness section.

LIMITATIONS:
The authors have not full discussed the limitations of the proposed approach.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
The paper proposes a sparse-view NeRF framework that jointly trains a NeRF with a monocular depth estimator. By adapting the MDE network to the target scene, the predicted depth will provide better geometry prior for the NeRF model.

STRENGTHS:
The joint training of MDE and NeRF improves the model's ability compared to existing methods.
After training both models, we'll have a better NeRF that renders more accurate novel views and also a more accurate mono depth estimator.

WEAKNESSES:
- For the few-shot NeRF setting, since we hope to train the NeRF directly, we shall have access to camera poses for multi-views, right?
If that's the case, what's the benefit of using monocular depth estimators, compared to multiview stereo networks such as MVSNet?
- If we use COLMAP to obtain the camera poses, we should also be able to construct a point cloud. What's the benefit of using the proposed distillation mechanism, compared to overcoming the scale and shift ambiguity problem of mono depth with depths extracted from point clouds?
- How is Tab. 2 calculated? Are they evaluated on the trained views of ScanNet?

QUESTIONS:
Please refer to weaknesses.

LIMITATIONS:
Please refer to weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-10