DäRF: Boosting Radiance Fields from Sparse Input Views with Monocular Depth Adaptation
Authors: Jiuhn Song
Seonghoon Park
Honggyu An
Seokju Cho
Min-Seop Kwak
Sungjin Cho
Seungryong Kim
Venue: NeurIPS 2023 poster (2023)
Keywords: Neural Radiance Fields
3D Reconstruction
Few-shot NeRF
Monocular Priors

Abstract: Neural radiance field (NeRF) shows powerful performance in novel view synthesis and 3D geometry reconstruction, but it suffers from critical performance degradation when the number of known viewpoints is drastically reduced. Existing works attempt to overcome this problem by employing external priors, but their success is limited to certain types of scenes or datasets. Employing monocular depth estimation (MDE) networks, pretrained on large-scale RGB-D datasets, with powerful generalization capability may be a key to solving this problem: however, using MDE in conjunction with NeRF comes with a new set of challenges due to various ambiguity problems exhibited by monocular depths. In this light, we propose a novel framework, dubbed DäRF, that achieves robust NeRF reconstruction with a handful of real-world images by combining the strengths of NeRF and monocular depth estimation through online complementary training. Our framework imposes the MDE network's powerful geometry prior to NeRF representation at both seen and unseen viewpoints to enhance its robustness and coherence. In addition, we overcome the ambiguity problems of monocular depths through patch-wise scale-shift fitting and geometry distillation, which adapts the MDE network to produce depths aligned accurately with NeRF geometry. Experiments show our framework achieves state-of-the-art results both quantitatively and qualitatively, demonstrating consistent and reliable performance in both indoor and outdoor real-world datasets.

DäRF: Boosting Radiance Fields from Sparse Inputs with Monocular Depth Adaptation
Jiuhn Song"
Seonghoon Park
Honggyu An"
Seokju Cho
Min-Seop Kwak
Sungjin Cho
Seungryong Kim
Korea University
Abstract
Neural radiance field (NeRF) shows powerful performance novel view synthesis and 3D geometry reconstruction, but it suffers from critical performance degrada- tion when the number of known viewpoints drastically reduced. Existing works attempt to overcome this problem by employing external priors, but their success is limited to certain types of scenes or datasets. Employing monocular depth esti- mation (MDE) networks, pretrained on large-scale RGB-D datasets, with powerful generalization capability would be key to solving this problem: however, using MDE in conjunction with NeRF comes with new set of challenges due various ambiguity problems exhibited by monocular depths. In this light, we propose a novel framework, dubbed DåRF, that achieves robust NeRF reconstruction with handful of real-worle images by combining the strengths of NeRF and monocular depth estimation through online complementary training. Our framework imposes the MDE network's powerful geometry prior to NeRF representation at both seen and unseen viewpoints to enhance its robustness and coherence. In addition, we overcome the ambiguity problems of monocular depths through patch-wise scale- shift fitting and geometry distillation, which adapts the MDE network to produce depths aligned accurately with NeRF geometry. Experiments show our framework achieves state-of-the-ar results both quantitatively and qualitatively, demonstrating consistent and reliable performance in both indoor and outdoor real-world datasets. Project page is available at https: ://ku- cvlab. github. io/DaRF/
Introduction
Neural radiance field (NcRF) (27 has gained significant attention for its powerful performance in reconstructing 3D scenes and synthesizing novel views. However, despite its impressive performance, NeRF often comes with a considerable limitation in that performance highly relies on the presence of densely well-calibrated input images which are difficult to acquire. As the number of input images reduced, NeRF's novel view synthesis quality drops significantly, displaying failure cases such as erroneous overfitting to the input images [1 129], artifacts clouding empty spaces [29]. or degenerate
geometry that yields incomprehensible jumble when rendered at unseen viewpoints [66]. These challenges derive from its under-constrained nature, causing it to have extreme difficulty mapping a pixel in input images to a correct 3D location. addition, NeRF's volume rendering allows the model map pixel to multiple 3D locations
exacerbating this problem.
Previous seo- NeRF methods attempt solve these issues by imposing geometric regulariza- exploiting external 3D priors [1] 37 such as depth information extracted from
tion 29 16 20
Equal contribution Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Rendered Depth
Seen
Rendered Depth
Patch wise Depth Distillation
Viewpoint
Adaptation
Seen View Depth Distiliatio Only
Ground Truth
Seen
Viewpoint
MDE Network
Rendered Color
Network
NeRF Representation
Rendered Depth
Unseen
Depth Distilation
Unseen View
Viewpoint
! Adapted
Previous Work
DäRF (Ours)
Figure I: Overview. DäRF shows robust optimization of few-shot NeRF through MDE's geometric prior, removing inherent ambiguity from MDE through novel patch-wise distillation loss and MDE adaptation. Unlike existing work [45] that distills depths by applying pretrained MDE to NeRF at seen view only, our DäRF fully exploits the ability of MDE by jointly optimizing NeRF and MDE at a specific scene, and distilling the monocular depth prior to NeRF at both seen and unseen views.
input images by COLMAP [39]. However, these methods have weaknesses in that they use 3D priors extracted from few input images only, which prevents such guidance from encompassing the entire scene. To effectively tackle all the issues mentioned above, pretrained monocular depth estimation (MDE) networks with strong generalization capability 35 3D prior into NeRF that facilitates robust geometric reconstruction. Specifically, geometry prediction by MDE can constrain NeRF into recovering smooth and coherent geometry, while their bias towards predicting smooth geometry helps to filter out fine-grained artifacts that clutter the scene. More importantly, NeRF's capability to render any unseen viewpoints enables fully exploiting the capability of the MDE, MDE could provide depth prior to the numerous renderings of unseen viewpoints as well as the original input viewpoints. This allows injecting additional 3D prior to effectively covering the entire scene instead of being constrained few input images.
could used inject an additional
However, applying MDE few-shot NeRF not trivial, there are ambiguity problems that hinder the monocular depth from serving as a good 3D prior. Primarily, relative depths predicted by MDEs are not multiview--onsistent [5) Moreover, MDEs perform poorly estimating depth differences between multiple objects: this prevents global scale-shift fitting (55 26 from being viable solution, as alignment to one region of the scene inevitably leads misalignment in many other regions. There also exists a convexity problem 226, in which the MDE has difficulty determining whether the surface planar, convex, or concave, are also present. To overcome these challenges, we introduce novel method to adapt MDE to NeRF's absolute scaling and multiview consistency as NeRF is regularized by MDE's powerful 3D priors, creating a complementary cycle.
In this paper, we propose DäRF, short for Monocular Depth Adaptation for boosting Radiance Fields from Sparse Input Views, which achieves robust optimization of few-shot NeRF through MDE's geometric prior, well as MDE adaptation for alignment with NeRF through complementary training (see Fig. D. We exploit MDE for robust geometry reconstruction and artifact removal in both unseen and seen viewpoints. In addition, we leverage NeRF to adapt MDE toward multiview-consistee geometry prediction and introduce novel patch-wise scale-shift fitting to more accurately map local depths to NeRF geometry. Combined with confidence modeling technique for verifying accurate depth information, our method achieves state-of-the-an performance few-shot NeRF optimization. We evaluate and compare approach real-world indoor and outdoor scene datasets, establishing new state-of-the -hethherrt results for the benchmarks.
2 Related Work
Neural radiance field. Neural radiance field (NeRF) 27 represents photo-realistic 3D scenes with MLP. Owing to its remarkable performance, there has been variety of follow-up studies 2 51 25

---

These studies improve NeRF such as dynamic and deformable scenes [31
44 33
real-time
rendering [5T 36 28,, unbounded scene B
42 48 and generative modeling 40 30
However,
these works still encounter challenges in synthesizing novel views with limited number of images in single scene, limiting their applicability in real-world scenarios.
Few-shot NeRF. Numerous few-shor NeRF works attempted to address few-shot 3D reconstruction
problem through various techniques, such as pretraining external priors [52 99. meta-learning 43
regularization [13] [2 16 [0] off-the- shelf modules [[5] 29] Recent approaches [29] 16 20
emphasize the importance of geometric consistency and apply geometric regularization at unknown
viewpoints. However, these regularization methods show limitations due to their heavy reliance on
geometry information recovered by NeRF. Other works such as DS-NeRF [ DDP-NeRF (37
and SCADE [45] exploit additional geometric information, such as COLMAP [3]] 3D points or monocular depth estimation, for geometry supervision. However, these works have critical limitations of only being able provide geometry information corresponding to existing input viewpoints.
Unlike these works, our work demonstrates methods to provide geometric prior even at unknown viewpoints with MDE for more effective geometry reconstruction.
Monocular depth estimation. Monocular depth estimation (MDE) is a task that aims to predict dense depth map given a single image. Early works on MDE used handcrafted methods such as MRF for depth estimation [38]. After the advent of deep learning,
soaming-baseee approaches [14 17 21
were introduced to the field. In this direction, the models were trained on ground-truth depth maps
acquired by RGB-D cameras LiDAR sensors to predict absolute depth values 24 23 Other
approaches trained the networks on large-scale diverse datasets g8 22 34 (5), which demonstrates
better generalization power. These approaches struggle with depth ambiguity caused by ill- ill-posed problem, so the following works LeRes [50) and ZoeDepth [4] opt to recover absolute depths using additional parameters.
Incorporating MDE into 3D representation. As both NeRF and monocular depth estimation are closely related, there have been some works that utilize MDE models to enhance NeRF's performance. NeuralLift [49], MonoSDF [53] and SCADE 45 leverage depths predicted by pretrained MDE for depth ordering and detailed surface reconstruction, respectively. Other works optimize specifie parameters, such as depth predictor utilizing depth recovered by COLMAP [47] or learnable scale- shift values for reconstruction noisy pose setting [6]. However, these previous approaches were limited in that MDEs were used to provide prior to only the input viewpoints, which constrains their
effectiveness when input views are reduced, e.g., in the few-shot setting.
As concurrent work, SCADE 45) utilizes MDE for sparse view inputs, by injecting uncertainty into MDE through additional pretraining so that canonical geometry can be estimated through probabilistic modeling between multiple modes of estimated depths. While the ultimate goal which is to overcome the ambiguity of MDE may be similar, our approach directly removes ambiguity present in MDE by finetuning with canonical geometry captured by NeRF, for effective suppression of artifacts and divergent behaviors of few-shot NeRF.
3 Preliminaries
NeRF 27 represents a scene as a continuous function Fo(.) represented by a neural network with parameters 6. During optimization, 3D points sampled along rays represented by coming from a set of input images = {I+). whose ground truth camera poses are given, for evaluation by the neural network. For each sampled point, F-( takes input its coordinate
IR3 and viewing direction d E 2R2 with a positional encoding 7(-) that facilitates learning high-frequency details, and outputs color parameterized as rp(2 (1) = o + Idp, starting from camera center o along the direction dp. color and depth value at the pixel p are rendered as follows:
IR3 and a density a E R such that (c. = Fo (7(x), (()). With ray
I(p) J"
' (()))( D(p) -
D(p)-
f." ((t)o(r
(I)
where I(p) and D(p) are rendered color and depth values at the pixel along the ray (t) from ty to t,, and T(t) denotes accumulated transmittance along the ray from tn to follows:
T(t) exp (- (Ip(s)))
(2)
3

---

(a)
(b)
Figure 2: Effectiveness of patch-wise scale and shift adjustment: (a) input image, (b) monocular depth with image-level adjustment, and (c) monocular depth with patch-level adjustment. We visualize the error of adjusted monocular depth from input image compared to GT depth value. The proposed patch-level adjustment helps minimize the errors caused by inconsistency in depth differences among objects.
Based on this volume rendering, F-(-) is optimized by the reconstruction loss Crenon that compares rendered color [(p) to corresponding ground-truth /(p), with R. as set of pixels for training rays:
Lieron Σ>
[I(()) (p)
IEES PER
Our work explores the setting of few-shot optimization with NeRF [66 [0). Whereas the number of input viewpoints |S] is normally higher than one hundred in the standard NeRF setting [27]. the task of few-shot NeRF considers scenarios when [S] drastically reduced
few viewpoints (e.g..
|S] 20). With such a small number of input viewpoints, NeRF shows high divergent behaviors such as geometry breakdown, overfitting to input viewpoints, and generation of artifacts that cloud the empty space between the camera and object, which causes its performance to drop sharply [15 [6 29]
To overcome this problem, existing few-shot NeRF frameworks applied regularization techniques at unknown viewpoints constrain NeRF with additional 3D priors [37 11 and enhance the robustness of geometry, but they showed limited performance.
Methodology
4.1 Motivation and Overview
Our framework leverages the complementary benefits of few-shot NeRF and monocular depth estimation networks for the goal of robust 3D reconstruction. The benefits that pretrained MDE can provide to few-shot NeRF are clear and straightforward: because they predict dense geometry, they provide guidance for the NeRF to recover more smooth geometry. In cases where few-shot NeRF's geometry undergoes divergent behaviors, MDE provides strong constraints to prevent the global geometry from breaking down.
However, there are difficult challenges that must be overcome if the depths estimated by MDE are to be used as 3D prior to NeRF These challenges, which can be summarized as depth ambiguity problems 26), stem from the inherent ill ill-posed nature of the monocular depth estimation. Most importantly, MDE networks only predict relative depth information inferred from an image, meaning is initially not aligned to NeRF's absolute geometry [4. Global scaling and shifting may seem to be the answer, but this approach leads us to another depth ambiguity problem, as predicted scales and spacings of each instance are inconsistent with one another, as demonstrated in (b) of Fig. 2 Additionally, MDE's weakness in predicting the convexity of surface, whether it is flat, convex, or concave also poses a problem using this depth for NeRF guidance.
In this light, we adapt a pre-trained monodepth network to single NeRF scene so that its powerful 3D prior can be leveraged to its maximum capability in regularizing the few-shot NeRF. In the following, we first explain how to distill geometric prior from off-the-shelf MDE model [35] from both seen and unseen viewpoints (Sec. 4.2). We also provide a strategy for adapting the MDE model to handle ill-posed problems to a specific scene, while keeping its 3D prior knowledge (Sec. 4.3) Then, we demonstrate method handle inaceurate depths (Sec. Fig. T shows an overview of method, compared to previous works using MDE prior 45 53

---

(a) Rendered color i
(b) Rendered depth
(c) Mono-depth from i
(d) Mono-depth from /
Figure 3: Robustness of MDE model for multi-view scale ambiguity and artifacts: (a-b) color and depth of NeRF rendered in the early stage of the training, (c-d) monocular depths estimated from rendered image I and input image 1. The results show that MDE model ignores the artifacts of rendered images by NeRF, enabling reliable supervision for seen and unseen viewpoint.
4.2 Distilling Monocular Depth Prior into Neural Radiance Field
To prevent the degradation of reconstruction quality in few-shot NeRF, we propose to distill monoc- ular depth prior the neural radiance field during optimization. By exploiting pre-trained MDE networks [34] 35). which have high generalization power, we enforce a dense geometric constraint on both seen and unseen viewpoints by using estimated monocular depth maps as pseudo ground truth depth for training few-shot NeRF. We describe the details of this process below.
Monocular depth regularization on seen views. We leverage pre-trained MDE model, denoted as (-)-) with parameters to predict pseudo depth map from given seen view image J; as D Go Since D is initially relative depth map, it needs to be scaled and shifted into an absolute depth [55] and aligned with NeRF's rendered depth D in order for be used as pseudo-depth D* However, the scale and shift parameters inferred from the global statistic may undermine local statistic (55). For example, as shown in Fig. (b). global scale fitting tends to favor dominant objects in the image, leading to ill-fitted depths in less dominant sections of the scene due to inconsistencies in predicted depth differences between the objects. Naively employing such inaccurately estimated depths for distillation can ad adversely impact the overall geometry of the NeRF.
To alleviate this issue, we propose patch-wise adjustment of scale and shift parameters, reducing the impact of erroneous depth differences, as illustrated in Fig. (c). The depth consistency loss is defined as follows:
Lreen ΣΣ ((w:sg(D;(())) + q:) - D.(p)ll.
LESS PEP
where w; and qi denote the scale and shift parameters obtained by least square 35 between D and D. P denotes set of pixels within patch, and sg(-) denotes stop-gradient operation. Thus patch-based approach also helps to overcome the computational bottleneck of full image rendering.
Monocular depth regularization on unseen views. We further propose give supervision even at unseen viewpoints. As NeRF has the ability to render any unseen viewpoint of the scene, we render color I and depth Dj from sampled patch of 1-th novel viewpoint. Sequentially, we extract a monocular depth map from the rendered image D = G (I). Then, we enforce consistency between our rendered depth D) and the monocular depth D of I-th novel viewpoint as follows:
Lunveen
Σ Σ [[ (D) qt) D.(p)
IEEU DEP
where l4 denotes set of unseen view images, wy and q denotes the scale and shift parameters used to align D towards D,, and P denotes randomly sampled patch.
A valid concern regarding this approach is that monocular depth obtained from noisy NeRF rendering may be affected by fine-grained rendering artifacts that frequently appear unseen viewpoints of sho- NeRF, resulting in noisy and erroncous pseudo-depths. However, we demonstrate in
5

---

Fihtt a strong geometric prior within the MDE model exhibits robustness against such artifacts,
effectively filtering out the artifacts and thereby providing reliable supervision for the unseen views.
should be noted that our strategy differs from previous methods I
37
53
45
that exploit
monocular depth estimation [34] and external depth priors such as COLMAP
39 These methods
only impose depth priors upon the input viewpoints, and thus their priors only influence the scene partially due self-occlusions and sparsity of known views. Our method, on the other hand, enables external depth priors to be applied to any arbitrary viewpoint and thus allows guidance signals to thoroughly reach every location of the scene, leading to more robust and coherent NeRF optimization.
4.3 Adaptation of MDE via Neural Radiance Field
Although the patch-wiss distillation of monocular depth provides invariance depth difference inconsistency in MDE, the ill-posed nature of monocular depth estimation often introduces additional ambiguities, such as the inability to distinguish whether the surface is concavity, convexity, or planar or difficulty in determining the orientation of flat surfaces [26]. We argue that these ambiguities arise due to the MDE lacking awareness of the scene-specific absolute depth priors and multiview consistency. To address this issue, we propose providing the scene priors optimized NeRF to MDE, whose knowledge canonical space and absolute geometry helps eliminate the ambiguities present within MDE. Therefore, propose to adapt the MDE to the absolute scene geometry, formally written as:
MDDE E{ls
{Ilsg (Dr())) (P) +| (w;ssg (D((p) + DJ (p)l))
(6)
JEP
In addition the patch-wis wish loss in Eq. 4 we add I-1 loss without scale-shift adjustment adapt the MDE with absolute depth prior. We also introduce a regularization term to preserve the local smoothness of MDE, given by:
Lreg Σ
Σ DEP
(1 (p::inl((p))))))p 9:)) (p)l,,
L,S
where Dr denotes monocular depth map of I; extracted from MDE with initial pre-trained weight.
4.4 Confidence Modeling
Our framework must take into account the errors present in both few-shot NeRF and estimated monoc- ular depths, which will propagate [41] and intensify during the distillation process if left unchecked. To prevent this, we adopt confidence modeling [20] 41 inspired by self- elf-trriinng approaches [41), to verify the accuracy and reliability of each ray before the distillation process.
The homogeneous coordinates of a pixel p in the
viewpoint are transformed p' at the target
viewpoint using the viewpoint difference R and the camera intrinsic parameter K, as follows:
p'
(p)
We generate the confidence map M, by measuring the distance between rendered depth of the unseen viewpoint and MDE output of seen viewpoint such that
(p) = [I[(, + q)) - Di('')l <T]
(9)
where T denotes threshold parameter, [-] is Iverson bracket, and Di(p') refers to depth value of the corresponding pixel 1-th unseen viewpoint for reprojected target pixel p of i-th seen viewpoint. We fit Dj to absolute scale, where scale and shift parameters, w; and gi. are obtained by least square (35 between D and D..
4.5 Overall Training
With the incorporation of confidence modeling. the loss functions for both the radiance field and MDE can redefined. Lroen and Lunseen can be redefined as:
Escen
Σ
M((p) (w,sg (D) q.) D.(p)]]
(10)
T,E8 PEP
C Lunseen
ΣE Mi(p) (simg (D) (p)) + Φ) -D((p))
Σ
(11)
PEP

---

Table I: Quantitative comparison ScanNet [10] and Tanks and Temples [9]. The best results are highlighted in bold, while the second best results are underlined.
SeamNet
[0]
Tanks and Temples 19
Methods
Depih prior
9. views
18 20
views SSIM
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
LPIPS
NertingMVS
47
NIA
N/A
N/A
16.29
0.626
0.502
NIA
N/A
N/A
K-planes
0.494
0.708
0.400
0.607
RegNeRF
29
16.38
0.624
0.493
18.93
0.676
0.450
14.12
0.469
0,580
DS-NeRF
N/A
N/A
N/A
20.85
0.713
0.344
NIA
N/A
N/A
DDP-NeRF 37
N/A
N/A
0.695
0.368
N/A
N/A
SCADE 45
18.83
0.646
0.375
21.54
0.732
0.292
13:46
0.402
0.607
DARP (Ours)
18.29
0.690
0.412
21.58
0.765
0.325
15.70
0.514
0.583
Table 2: Evaluation depth quality: (a) quantitative evaluation of the adapted MDE, compared with other monocular depth estimation models and (b) visualization of depth distributions. The adapted MDE by our method shows a similar distribution to that the ground truth.
Methods
AbsRel
SqRel
RMSE
RMSE log
Ground truth
Perceniage
DPT
LeRes 50
0.391
0.472
0.999
0.661
DIRF (Ours)
MiDaS 35
0.152
0.095
0.452
0.183
DPT 34
0.191
0.135
0.563
0.220
DäRF (9. 10 views)
0.154
0.074
0.361
0,171
DaRF (18 20 views)
0.151
0.071
0.356
0.168
Depth
(a) Quantitative comparison
(b) Depth distribution comparison
In addition, the loss for the adaptation of the MDE module can be redefined considering M:
LMDE
Σ Σ Mi()) (l|g (D.(p) - (p)ll + [1(w; (Di(p)) + + (P))) (12)
LES
With these losses, we train both NeRF and MDE simultaneously. enhancing both models by com- plementing each other. MDE provides a strong geometric prior NeRF while having the inherent limitation of obliviousness to the scene-specific prior, whereas NeRF provides it with its absolute geometry.
Experiments
5.1 Experimental Settings
Implementation details. DäRF is implemented based on K-planes [32] as NeRF. We use DPT- hybrid (34] as MDE model. We use Adam [18) optimizer, with learning rate of 1 10-2 for NeRF and 1 10-5 for the MDE, along with a cosine warmup learning rate scheduling. See supplementary material for more details. The code and pre-trained weights will be made publicly available.
Datasets. evaluate our method in real-world scenes captured at both indoor and outdoor locations. Following previous works 377 45), we use subset of sparse-view ScanNet data [TO] comprised with three indoor scenes, each consisting of 18 to 20 training images and 8 test images. We also conduct evaluations reconstruction, we further test on 5 challenging scenes from the Tanks and Temples dataset [19]. The scenes real-world outdoor dataset, with wide variety of scene scales and lighting conditions. Note that these setups are extremely sparse compared to full image setups, where we use approximately 0.5 to 5 percent of the whole training inputs.
more challenging setting with to 10 train images. For outdoor
Baselines. We adopt the following six recently proposed methods baselines: standard neural radiance field method: K K-planes [13,, few-shot NeRF method: RegNeRF [29), and depth prior based methods: NerfingMVS [47]. DS-NeRF [Л]] DDP-NeRF [37]. and SCADE 45).
Evaluation metrics. For quantitative comparison, we follow the NeRF 27 and report the PSNR, SSIM [6). LPIPS [56]. We report standard evaluation metrics for depth estimation [12), absolute relative error (Abs Rel), squared relative error (SqRel), root mean squared error (RMSE), root mean
7

---

(a) K-planes [3
(b) DS-NeRF [T]
(c) DDP-NeRF 37
(d) DåRF
(e) Ground truth
Figure 5: Qualitative results of ScanNet [10] with 18 20 input views.
mi
(a) Baseline
13
(b) Baseline- Depth
(c) D&RF
(d) D&RF- Depth
(c) Ground truth
Figure 6: Qualitative results on ScanNet 110 with 9 10 input views.
squared log (RMSE log). To evaluate view consistency, utilize single scaling factor for each scene, which is the median scaling [57] value averaged across all test views.
5.2
Comparisons
Indoor scene reconstruction. We conducted experiments two settings: (1) standard few-shot setup as described in literature 37 45 and (2) an extreme few-shot setup with
approximately 0.5 percent of the full images. As shown in Tab. our approach outperforms the baseline methods in both settings in most the metrics. Additionally, we provide quan- titative results of the adapted MDE model in ScanNet dataset in Tab. 2 and qualitative results Fig. 4 As shown Fig. for the setting of standard few-shot, DS-NeR [I] and DDP.
NeRF (37) still show floating artifacts in the novel view and show limitation in capturing details in the chair, smoothing into nearby object. Our method shows better qualitative re- sults compared other baselines. showing better geometry understanding and detailed view synthesis in the small objects near the chair. the extreme few-shot setup, we conducted a visual comparison between our method and a baseline [13] in Fig. 6 This is more complex setting than standard, but our method outperforms most of the baselines, showing better geometric understanding. It should be noted that SCADE [45) fine-tunes its depth network on an indoor dataset, Taskonomy dataset [ whereas our model does not undergo any additional fine-tuning. More qualitative images are included in the supplementary material.
(a) Frozen MDE
(b) Adapted MDE
Figure 4: Error map visualization. MDE adaptation results in reduc- tion errors.
Outdoor scene reconstruction. We conduct the qualitative and quantitative comparisons on Tanks and Temples dataset in Tab. and Fig. 7 Since COLMAP [39 with sparse images is not available, we provide comparisons with baselines without explicit depth prior. The quantitative results show that our approach outperforms the baseline methods on this complex outdoor dataset in all

---

(a) Baselinc
13
(b) Baseline- Depth
(c) DARF
(d) DåRF- Depth
Figure Qualitative results on Tanks and Temples 19
(a) Baseline 13
(b) (a) + Lisen
(c) (b) + Lunscon
(d) (c) + LMDE
(e) + M (DäRF)
Figure 8: Visualization of ablation studies on ScanNet (10).
metrics. As shown in Fig. baseline shows limited performance, despite its feasible results of view synthesis in novel viewpoint, its depth results show that the network totally fails to understand 3D geometry. Our method shows rich 3D understanding, even in this real-world outdoor setting which is more complicated than other scenes. More qualitative images are included in the supplementary material.
In-the-wild dataset Reconstruction. We further inves- tigate the robustness of our method on in-the-wild dataset, provided from SCADE (45). This dataset was captured using an iPhoneX, with varying intrinsic parameters for each view. As shown Tab. 3 our model outperforms all other baselines and this demonstrates that our model performs well in large scenes, such as outdoor scenes and in-the-wild dataset.
Table 3: Comparison In-the-wild dataset [45].
Methods
PSNR1
SSIM1
LPIPS]
K K-planes [3]
20.71
0.732
0.408
DDP-NoRF 37
21.28
0.727
0.366
SCADE 45
22.82
0.743
0.347
DARF (Ours)
22.92
0.760
0.390
5.3 Ablation Study
Ablation on core components. In Tab, and Fig. 8 evaluate the effect of each pro- posed component. The quantitative results
Table Ablation study,
Components
PSNR
SSIM
LPIPS]
(a) Baseline 113
18.70
0.708
0.400
show effectiveness of each component. For
(b)
Leven
19.71
0.730
0.380
qualitative results, we found out that Luuseen suppresses the artifacts in novel viewpoint, compared to when only Leorn is given. With adaptation of the MDE network to this scene, red basket in the background shows accurate results and artifacts near the table are removed. In our model, with confidence modeling, view synthesis results show to be structurally confident in the overall scene.
(c)
(b) Lumsess.
21.21
0.758
0.333
(d) (c) MDE Adapt. (CsDD)
21.39
0.763
0.327
(d) Conf. Modeling (DáRF)
21.58
0.765
0.325
Analysis of local fitting. In Tab. we further inves- tigate the effectiveness of local scale-shift fitting and global fitting. For global scale-shift fitting, we give learnable scale and shift parameters for each input image and convert MDE's output to the absolute value in global manner. For fair comparison, we compare our model only with MDE distillation on seen viewpoints. The results show that our method local scale-shift fitting is more effective giving accurate depth supervision.
Table 5: Local fitting ablation.
Components
PSNR
SSIMf
LPIPS
Baseline [13]
18.65
0.706
0.502
global litting
19.05
0.698
0.399
w/ local fitting (DGRF)
19.71
0.730
0.380

---

Table 6:
Ablation study MDE Adaptation Loss.
Components
PSNRT
SSIM
LPIPS|
AbsRel
SqRel | RMSE
RMSE log 1
scale-shift loss
21.31
0.757
0,343
0.182
0.109
0,484
0.205
(1 loss
21.48
0.758
0.337
0.157
0.079
0.386
0.176
DåRF (Ours)
21.58
0.765
0.325
0.151
0.071
0.356
0.168
Analysis of MDE Adaptation Loss. In Tab. we further investigate the effectiveness scale-shie loss and (1 loss at MDE adaptation. Equation 6 revolves around the idea of adapting MDE toward predicting a scene-specific absolute geometry, which is achieved by the first addend term: this first term forces itself MDE to adapt towards multiview consistency so that its ill-posed nature is reduced and its initial global depth prediction grows to be more in accordance with the absolute geometry captured by NeRF. In contrast, the second addend term, which takes into account patch-wise scale- shift fitting, is designed aid the modeling of fine, detailed, local geometry which the model has difficulty modeling without such local fitting, As shown in the results, when only one term is used for optimization (scale-shift or (1), performs worse in every metric than both are used conjunction (ours). This justifies our strategy of using both losses as effective.
6 Conclusion
We propose DäRF, a novel method that addresses the limitations of NeRF in few-shot settings by fully leveraging the ability of monocular depth estimation networks. By integrating MDE's geometric priors, DåRF achieves robust optimization of few-shot NeRF, improving geometry reconstruction and artifact removal in both unseen and seen viewpoints. We further introduce patch-wise scale-shift fitting for accurate mapping of local depths 3D space, and adapt MDE NeRF's absolute scaling and multiview consistency, by distilling NeRF's absolute geometry to monocular depth estimation. Through complementary training, DäRF establishes a strong synergy between MDE and NeRF, leading to -fate-of-the-art performance in few-shot NeRF. Extensive evaluations on real-world scene datasets demonstrate the effectiveness of DäRF.
Acknowledgements
This research was supported by the MSIT, Korea (1ITP202-22020000111) No. 2021-0-02068, No.2021-0-00155, and National Research Foundation of Korea ANRF-202RRICICI006399 This research was partially supported by Culture, Sports, and Tourism R&D Program through the Korea Creative Content Agency grant funded by the Ministry of Culture, Sports and Tourism in 2023 (4I) Content Generation and Copyright Protection with Artificial Intelligence, R2022020068).
References
[1] Benjamin Attal, Eliot Laidlaw, Aaron Gokaslan, Changil Kim, Christian Richardt, James Tompkin, and Matthew O'Toole. Törf: Time-of-flight radiance fields for dynamic scene view synthesis. Advances in neural information processing systems, 34, 2021.
[2] Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo
and Pratul P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
[3] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip- nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 5470-5479, 2022.
[4] Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Mathhis Müller. Zoedepth: Zero-shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302. /2. /2288, 2023.
[5] Amlaan Bhoi. Monocular depth estimation: A survey, arXiv preprint arXiv:1901.09400 2019.
[6] Wenjing Bian, Zirui Wang, Kejie Li, Jia- Ha-Wang Bian, and Victor Adrian Prisacariu. Nope-nerf: Optimising neural radiance field with no pose prior. arXiv preprint arXiv:2212.07388. 2022.
10

---

[7] Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, et Efficient geometry- aware 3d generative adversarial networks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 16123-16133, 2022.
[8] Weifeng Chen, Zhao Fu, Dawei Yang, and Jia Deng. Single-image depth perception in the wild, 2017.
[9] Julian Chibane, Aayush Bansal, Verica Lazova, and Gerard Pons-Moll. Stereo radiance fields (srf): Learning view synthesis for sparse views of novel scenes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages
2021.
[10] Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Niessner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
[11] Kangle Deng, Andrew Liu, Jun- Yan Zhu, and Deva Ramanan. Depth-supervised NeRF: Fewer views and faster training for free. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), June 2022.
[12] David Figen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using multi-scale deep network. Advances in neural information processing systems, 27, 2014.
[13] Sara Fridovich-Keil. Giacomo Mcanti, Frederik Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. arXiv preprins arXiv:230.11224/ 2023.
[14] Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao. Deep ordinal regression network for monocular depth estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2002-2011, 2018.
[15] Ajay Jain, Matthew Tancik, and Pieter Abbeel. Putting nerf on a diet: Semantically consis- tent few-shot view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5885-5894, 2021.
[16] Mijeong Kim, Seonguk Seo, and Bohyung Han. Infonerf: Ray entropy minimization for few- shot neural volume rendering. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2022.
[17] Seungryong Kim, Kihong Park, Kwanghoon Sohn, and Stephen Lin. Unified depth prediction and intrinsic image decomposition from a single image via joint convolutional neural fields. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14, pages 159-159. Springer, 2016.
[18] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. iclr. 2015.
arXiv preprinl arXiv: 14/2.6440. 9, 2015.
[19] Arno Knapitsch, Jaesik Park, Qian Yi Zhou, and Vladlen Koltun. Tanks and temples: Bench- marking large-scale scene reconstruction. ACM Transactions on Graphics (ToG), 36(4):1-13, 2017.
[20] Minseop Kwak, Jiuhn Song, and Seungryong Kim. Geconerf: Few-shot neural radiance fields via geometric consistency. arXiv preprint arXiv:2301. 10941, 2023.
[21] Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, and Nassir Navab. Deeper depth prediction with fully convolutional residual networks. In 2016 Fourth international conference on 3D vision (3DV), pages 239-248. IEEE, 2016.
[22] Jae-Han Lee and Chang-Su Kim. Monocular depth estimation using relative depth maps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9738, 2019.
[23] Jin Han Lee, Myung- Kyu Han, Dong Wook Ko, and II Hong Suh. From big to small: Multi-scale local planar guidance for monocular depth estimation. arXiv preprint arXiv: 10326, 2019.
[24] Bo Li, Chunhua Shen, Yuchao Dai, Anton Van Den Hengel, and Mingyi He. Depth and surface normal estimation from monocular images using regression on deep features and hierarchical crfs. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1119- 127, 2015.
11

---

[25] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. NeurlPS, 2020.
[26] S Mahdi Miangolch, Sebastian Dille, Long Mai, Sylvain Paris, and Yagiz Aksoy. Boosting monocular depth estimation models to high-resolution via content-adaptive multi-resolution merging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9685-9694 2021.
[27] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[28] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. arXiv preprint arXiv:2201.05989 2022.
[29] Michael Niemeyer, Jonathan T. Barron, Ben Mildenhall, Mehdi S. M. Sajjadi, Andreas Geiger, and Noha Radwan. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2022.
[30] Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional genera- tive neural feature fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1453-1 1165-11164, 1464, 2021.
[31] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualle Nerfies: Deformable neural radiance fields. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pages 5865-5874, 2021.
[32] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[33] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10318-10327, 2021.
[34] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12179-
12188, 2021.
[35] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE transactions on pattern analysis and machine intelligence, 44(3):1623-16 16337 2020.
[36] Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mips. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 44335-14345, 2021.
[37] Barbara Roessle, Jonathan Barron, Ben Mildenhall, Pratul P Srinivasan, and Matthias Nießner. Dense depth priors for neural radiance fields from sparse input views. arXiv preprint 1rXiv:2112.03388,, 2021.
[38] Ashutosh Saxena, Min Sun, and Andrew Y Ng. Make3d: Learning 3d scene structure from a single still image. IEEE transactions on pattern analysis and machine intelligence, 31(5):824- 840, 2008.
[39] Johannes I.. Schonberger and Jan-Michael Frahm. Structure-from-net motoommmotionnoooo revisited. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
[40] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. Advances in Neural Information Processing Systems, 33:20154-20166 2020.
[41] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raf- fel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi- supervised learning with consistency and confidence. Advances in neural information processing systems, 33:596-608 2020.
12

---

[42] Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srinivasan, Jonathan T Barron, and Henrik Kretzschmar, Block-nerf: Scalable large scene neural view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2248-8258, 2022.
[43] Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P. Srinivasan, Jonathan T. Barron, and Ren Ng. Learned initializations for optimizing coordinate-based neural representa-
tions. In CVPR, 2021.
[44] Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of dynamic scene from monocular video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12970, 2021.
[45] Mikaela Angelina Uy. Ricardo Martin Buraina,, Leonidas Guibas, and Ke Li. Scade: Nerfs from space carving with ambiguity-awart depth estimates. arXiv preprint arXiv:2303. 13582, 2023.
[46] Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600-612 2004.
[47] Yi Wei, Shaohui Liu, Yongming Rao, Wang Zhao, Jiwen Lu, and Jie Zhou. Nerfingmvs: Guided optimization of neural radiance fields for indoor multi-view stereo. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5610-5619, 2021.
[48] Yuanbo Xiangli, Linning Xu, Xingang Pan, Nanxuan Zhao, Anyi Rao, Christian Theobalt, Bo Dai, and Dahua Lin. Bungeenerf: Progressive neural radiance field for extreme multi-scale scene rendering. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23 27, 2022, Proceedings, Part XXXII, pages 106-122. Springer, 2022.
[49] Dejia Xu, Yifan Jiang, Peihao Wang, Zhiwen Fan, Yi Wang, and Zhangyang Wang. Neurallift- 360: Lifting an in-the-wild 2d photo to a 3d object with 360 (Adeg] views. arXiv preprint arXiv:2211.166 164331 2022.
[50] Wei Yin, Jianming Zhang, Oliver Wang, Simon Niklaus, Long Mai, Simon Chen, and Chunhua Shen. Learning to recover 3d scene shape from a single image. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 204-213, 2021.
[51] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. PlenOctrees for real-time rendering of neural radiance fields. In ICCV, 2021.
[52] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3578-4587, 2021.
[53] Zehao Yu, Songyou Peng. Michael Niemeyer, Torsten Sattler, and Andreas Geiger. Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. arXiv preprint arXiv:2206.00M 00665 2022.
[54] Amir R Zamir, Alexander Sax, William Shen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3712-3722, 2018.
[55] Chi Zhang, Wei Yin, Billzb Wang, Gang Yu, Bin Fu, and Chunhua Shen. Hierarchical normal- ization for robust monocular depth estimation. Advances in Neural Information Processing Systems, 35:14128- 1114128-81199, 2022.
[56] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features as perceptual metric. In CVPR, 2018.
[57] Zhoutong Zhang, Forrester Cole, Richard Tucker, William T Freeman, and Tali Dekel. Con- sistent depth of moving objects in video. ACM Transactions on Graphics (TOG), 40(4):1-12, 2021.
13


Reviews:
[{"review_id": "jasht7l3lk", "reviewer": "Reviewer_Eioc", "summary": "The paper proposes a sparse-view NeRF framework that jointly trains a NeRF with a monocular depth estimator. By adapting the MDE network to the target scene, the predicted depth will provide better geometry prior for the NeRF model.", "questions": "Please refer to weaknesses.", "limitations": "Please refer to weaknesses.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The joint training of MDE and NeRF improves the model's ability compared to existing methods.\nAfter training both models, we'll have a better NeRF that renders more accurate novel views and also a more accurate mono depth estimator.", "weaknesses": "- For the few-shot NeRF setting, since we hope to train the NeRF directly, we shall have access to camera poses for multi-views, right?\nIf that's the case, what's the benefit of using monocular depth estimators, compared to multiview stereo networks such as MVSNet?\n- If we use COLMAP to obtain the camera poses, we should also be able to construct a point cloud. What's the benefit of using the proposed distillation mechanism, compared to overcoming the scale and shift ambiguity problem of mono depth with depths extracted from point clouds?\n- How is Tab. 2 calculated? Are they evaluated on the trained views of ScanNet? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n> **Q1. Benefits of MDE compared to MVS networks**\n> \n\nThank you for pointing this out. It is true that MVS networks can be used for better NeRF training, as shown in previous works [1, 2]. However, their approach of leveraging image features combined from multiple viewpoints serves as a critical weakness in more extreme wide-baseline scenarios where the number of input viewpoints is very limited and the distances between known cameras are large, due to increased difficulty in finding consensus between image features - resulting in a sharp drop in performance.\n\nOur methodology of leveraging monocular depth networks’ powerful geometric prior helps overcome such weaknesses of MVS methods. As MDE predicts geometric information for each image independently, which fundamentally bypasses wide-baseline scenario problems of previous MVS methods. In addition, our MDE regularization method can be **directly extended to unseen viewpoints** for stronger regularization and artifact removal, which is impossible for standard MVS methods that only operate upon input viewpoints. \n\nIn this way, our paper demonstrates how these desirable qualities of MDE can be effectively leveraged in sparse-view NeRF settings in a complementary manner, orthogonal to previous MVS methods and effectively improving upon their weaknesses.\n\n> **Q2. Comparison to using COLMAP point cloud for scale-shift fitting**\n> \n\nIt is true that the point cloud acquired through COLMAP provides us with absolute depth information, and is beneficial for aligning the scale-shift value of the predicted depth, especially in the acceleration of the early stages in training.\n\nHowever, since COLMAP point cloud only provides sparse depth information, it falls short in modeling local (as described in our paper, patch-wise) scale-shift values, as the points are unequally distributed in the point cloud and the sparse regions do not provide our model enough geometric information for patch-wise scale-shift value modeling. Since local MDE alignment is precise solution than global scale-shift fitting [3], we find that solely using dense information predicted by MDE allows us to robustly capture patch-wise scale-shift values for depth fitting regardless of where the patch is sampled. \n\nAlso, because COLMAP point clouds are generated from input images, they are severely limited from providing our model with accurate geometric data in unseen regions, which makes it unfit for out unseen viewpoint depth regularization loss. Therefore, we find that using rendered depth data captured by NeRF provides us stronger advantages than naïvely employing COLMAP point cloud to overcome the scale and shift ambiguity problem of monocular depths.\n\n> **Q3. Details of evaluation method in Tab.2**\n> \n\nWe evaluated adapted MDE model only on test views of ScanNet. Table 2 is for evaluating view consistency of MDE model [4], so we utilize a single scaling factor s for each scene, which is the median scaling value averaged across all test views.\n\n[1] Wei et al., NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereo, ICCV 2021\n\n[2] Chibane et al., Stereo Radiance Fields (SRF): Learning View Synthesis for Sparse Views of Novel Scenes, CVPR 2021\n\n[3] Zhang et al., Hierarchical normalization for robust monocular depth estimation, NeurIPS 2022\n\n[4] Zhang et al., Consistent depth of moving objects in video, SIGGRAPH 2021."}, {"review_id": "mWXjtEKGnK", "reviewer": "Reviewer_oMo1", "summary": "The paper presents a new few-shot neural radience field approach based on joint monocular depth adaption. The main idea of the proposed approach is to utilize the monocular depth estimator to improve the geometry prior of NeRF representation. The motivation is reasonable. Also, it presents attractive performance on both indoor and outdoor scenes. ", "questions": "Please address the comments released in the weakness section. ", "limitations": "The authors have not full discussed the limitations of the proposed approach. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of utilizing monocular depth estimator is interesing and the depth estimator can provide reasonable geometry information for the scene reference. \n\n2. The proposed approach has provided attractive performance on the real-world benchmarks. Sufficient ablations have been conducted to validate the design of the proposed algorithm. \n\n3. The presentation of the paper is good. ", "weaknesses": "1. The paper is based on SCADE[49]. According the results in Table 1, the proposed approach is lower than [49] on the LPIPS score. Is there any potential explanation of this result? Also, How about the results on the \"in-the-wild benchmark\" proposed in [49]?\n\n2. How is case when the testing data is out of the distribution of the depth estimator? For example, in the case when the depth estimator failed to predict the accurate depth information.\n\n3. How about the inference speed of the proposed algorithm?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1. Comparison with SCADE**\n> \n\nThank you for your comment. First of all, we would like to emphasize that our paper is **not based** on SCADE [1] nor use it as its baseline but instead suggests a method orthogonal to it. There is a fundamental difference between our work and SCADE’s approach toward handling ambiguity present within MDE: our work proposes an online, scene-specific adaptation of MDE which **directly adapts it to predict canonical geometry** in accordance with NeRF under optimization. On the other hand, SCADE **injects uncertainty into MDE** through additional pretraining so that canonical geometry can be estimated through probabilistic modeling between multiple modes of estimated depths. So, while the ultimate goal regarding our usage of MDEs may be similar (ambiguity removal), the core idea behind them can be seen as more or less opposite.\n\nIn this light, we explain our potential explanation for lower LPIPS score in Table 1: our work’s baseline, K-planes [2], has been shown to perform slightly weaker in LPIPS than **NeRF-pytorch** [3], which is the baseline for SCADE. We have chosen K-planes as our baseline due to its fast optimization speed and more efficient memory, but due to this trait of baseline, we believe that our model show slightly less performance in LPIPS metric than SCADE, while exceeding it in all other metrics. At the time of our submission, the code for SCADE was not revealed, so we could not directly perform experiments with SCADE or its baseline: now it has been revealed, we bring you quantitative experiment results for accurate comparison.\n\nHere are the results of our model, DäRF, for in-the-wild benchmark proposed in SCADE, which shows that our model is slightly better results than SCADE in two metrics.\n\n|  | PSNR | SSIM | LPIPS |\n| --- | --- | --- | --- |\n| DDP | 21.28 | 0.727 | 0.366 |\n| SCADE | 22.82 | 0.743 | **0.347** |\n| Ours | **22.92** | **0.760** | 0.390 |\n\n> **Q2. How to deal with errors in MDE**\n> \n\nDue to MDE’s weakness in predicting geometry for data outside of distribution, this method does show a drop in performance when faced with testing data out of the distribution of depth estimator. However, we also point out that this is a fundamental weakness in all NeRF methods that utilize MDE, and the strength of our method derives from the fact that our method allows the adaptation of off-the-shelf pre-trained MDE to a scene without any additional training or dataset, unlike SCADE.\n\n> **Q3.  Inference speed of our model**\n> \n\nThe inference speed of our model is identical to that of K-planes, which is our baseline model.\n\n[1] Uy et al., SCADE: NeRFs from Space Carving with Ambiguity-Aware Depth Estimates, CVPR 2023\n\n[2] Fridovich-Keil et al., K-Planes: Explicit Radiance Fields in Space, Time, and Appearance, CVPR 2023.\n\n[3] Lin Yen-Chen. Nerf-pytorch. 2020."}, {"review_id": "KpvS1mejGH", "reviewer": "Reviewer_KpuH", "summary": "This paper addresses the problem of few-shot NeRF reconstruction. The authors propose using monocular depth estimation (MDE) networks to provide geometry prior to NeRF at both seen and unseen viewpoints. They propose overcoming the ambiguity problems associated with monocular depths by MDE adaption. Experimental results deomonstrate improvements in both rendered novel views and rendered depths.", "questions": "- Figure 2 is rather confusing. It is not clear how one should interpret / understand the renderings. More descriptive caption should be provided.\n- Are there any failure cases? Any analysis for the causes of failure?\n- What is the minimum number of viewpoints for the proposed model to produce reasonable reconstruction?", "limitations": "Limitations and impact are only included in the supplementary material, but not in the main paper.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ Unlike previous works which only exploit depth information at seen viewpoints, this work also exploits MDE to constraint NeRF at unseen viewpoints, leading to more robust and coherent NeRF optimization. The authors demonstrate through an example that the strong geometric prior within the MDE model enables it to generate reliable depth from noisy NeRF rendering. This makes MDE at unseen viewpoints feasible.\n+ The proposed patch-wise scale-shift fitting helps reducing the impact of erroneous depth differences generated by MDE networks in distilling the monoclular depth prior to the NeRF.\n+ Adapting MDE to absolute scene geometry embedded in NeRF further helps to resolve ambiguities in surface curvature and orientation, and improve multiview consistency.\n+ The proposed method demonstrates sota results, both qualitatively and quantitatively, on two real-data sets, particularly showing superb results in rendered depths in few-shot NeRF.\n+ Ablation study has been included to demonstrate the effectiveness of each major deisgn component.\n+ Overall this paper is well-written and well-organized. It is easy to follow.\n", "weaknesses": "- In MDE adaption, it is not clear why the monocular depths predicted from the rendered input views instead of that predicted from the input views are adopted in (6). There is no explanations or disucssions on the effect of choosing between these two. There is also no explanations or discussions on why only input views are considered.\n- In confidence modeling, why the predicted depth of a point in an input view (after scaling ana shifting) is directly compared with its rendered depth in a target view in (9)? Note both predicted depth and rendered depth are measured with respect to the viewpoints.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1. Details of MDE adaptation**\n> \n\nThank you for pointing this out. Equation 6 has a **notation mistake** on our part, as we do use monocular depths predicted from ground truth input image, $D^*_{i}$, and not the one from depth rendered from NeRF, $\\bar{D}^*_{i}$. Our accurate methodology, in which we use the depth predicted with MDE from ground truth input images, is correctly described in Figure 1. We apologize for this mistake and we promise to revise it in the final version of our paper as the following:\n\n$\n\\mathcal{L}_\\text{MDE} = \\sum\\_{I\\_i\\in\\mathcal{S}} \\sum\\_{\\textbf{p} \\in \\mathcal{P}} \\left(||\\texttt{sg} \\left(\\bar{D}\\_i(\\textbf{p})\\right) - D^*\\_i(\\textbf{p})|| +  ||(w\\_i\\texttt{sg}\\left(\\bar{D}\\_{i}(\\textbf{p})\\right)+q\\_i) - D^*\\_i(\\textbf{p}) || \\right)\n$\n\nWe used only input viewpoints for MDE adaptation, since using rendered color patch in unseen viewpoint as supervision might lead MDE to lose its geometric prior due to its noisy rendering results.\n\n> **Q2. Details of Confidence Modeling**\n> \n\nThank you for pointing this out. Your comment is correct, and upon closer inspection, we have made a mistake in our thresholding equation so it does not correctly describe our masking process. We apologize for our mistake, and we are also very thankful for your careful reading and revision of our paper. The correct thresholding equation is as follows:\n\n$\nM\\_i(\\textbf{p}) = \\big[\\|R\\_{i}(w\\_i{D}^*\\_i(\\textbf{p}) + q\\_i)K^{-1}\\textbf{p} - R\\_{l}\\bar{D}\\_l\\textbf{p}')K^{-1}\\textbf{p}' \\| < \\tau\\big]\n$\n\nWhere $R_{i}$ and  $R_{l}$ stand for camera-to-world extrinsic matrices that transform the 3D points from camera view spaces to the canonical space. In short, using the predicted depth of viewpoint $i$ and NeRF-rendered depth of target viewpoint $l$, we compare the 3D point acquired from both viewpoints to canonical space by calculating the distance between the two. Only if the distance is lower than the threshold (in agreement with each other), do we take the mask as reliable. Again, thank you for your constructive revision, and we promise to revise this mistake in the final version of our paper.\n\n> **Q3.  Details of Figure2**\n> \n\nThank you for your constructive comment. We agree that Figure 2 is difficult to recognize at first sight, and we promise to fix it. \n\nOur intention for Figure 2 was to visualize our point that a single scale-shift value cannot perfectly fit into ground truth geometry due to the ambiguity of distances between object instances. For this reason, we have visualized **error** of a predicted point cloud of a room from the bird’s eye view to show that patch-wise fitting most accurately fits point cloud to ground truth geometry. This is done with jet color coding, so that red color means large error and blue color means small error. This point cloud is projected depth from the input image from the viewpoint stated as red camera. \n\nFollowing your comment, we promise to either replace it with Figure 1 of our supplementary material (which describes the same phenomenon more clearly) or reinforce it with an additional caption that explains it in a more detailed manner.\n\n> **Q4.  Any Failure cases?**\n> \n\nFor failure cases, there are occasions when our confidence modeling fails and unable to completely filter out erroneous predictions. Also, in out-of-domain cases where MDE fundamentally models depth incorrectly, our model also shows drop in performance as well. Failure case is shown in Figure2 of the attached pdf file, showing the case of wrong depth prediction in the window since neither NeRF and MDE model predict accurate depth of the transparent object.\n\n> **Q5. Minimum number of viewpoints?**\n> \n\nWe do not analytically calculate the minimum number of viewpoints required for few-shot reconstruction, as that number will vary scene by scene according to each scene’s shape, geometry and occlusions. Instead, like all other few-shot NeRF methods [1,2,3,4], we assume a setting supposed by all other few-shot NeRF methods - an extreme scenario where the known viewpoints hardly overlap with each other yet all directions of the scene are viewed at least once - which is 10 viewpoints in our case. If we reduce the number of viewpoints from here, scene reconstruction will still happen, but it will result unseen directions / regions of the scene where no information is provided whatsoever. In such cases our model cannot perform reconstruction, as it is not a generative model capable of imagining unseen parts and our model is not designed to do so in the first place. Furthermore, visualization of camera setting for few-shot is visualized at Figure 1 of the attached pdf file.\n\n[1] Jain et al., Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis, ICCV 2021.\n\n[2] Kim et al., InfoNeRF: Ray Entropy Minimization for Few-Shot Neural Volume Rendering, CVPR 2022.\n\n[3] Niemeyer et al., RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs, CVPR 2022.\n\n[4] Yang et al., FreeNeRF: Improving Few-shot Neural Rendering with Free Frequency Regularization, CVPR 2023."}, {"review_id": "zfjaRYjzhX", "reviewer": "Reviewer_Ndsp", "summary": "The paper tackles the problem of sparse view NeRF reconstruction by leveraging on monocular depth estimation networks as a prior. The main difference between DaRF and existing work is it also computes for a depth loss on unseen views, in contrast to prior works that only constrain depth on the training views. Moreover, they also adapt, i.e. fine tune, the monocular depth estimation network to agree with the depth produced by the NeRF. They also introduce a confidence term that determines which pixels to use the depth loss on. They conduct experiments on Scannet and Tanks and Temples dataset.", "questions": "1. How are the viewpoints selected for the unseen views that was used for L_{unseen}? How many unseen viewpoints are selected and how is the loss balanced with the seen training views? This in my opinion is an important detail to justify and back its main claim.\n2. What is the intuition behind using two terms in Eq 6? Specifically, wouldn't the second addend term be enough? It seems redundant and not too intuitive.\n3. What dataset was used for the Ablation study (Table 3)? Is it the average across all scenes in Scannet? Figure 8 shows one example from scene781 of scannet.", "limitations": "The authors included the limitations in their supplementary material. However, according to the checklist, they claim to include the error bars. However, I don't think I saw error bars reported in the submission.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper proposes to use monocular depth estimation as a prior to constrain NeRF optimization under the sparse view regime. Different from existing works, DaRF found that they can also use the monocular depth prior on unseen views, and even on NeRF rendered depth on early stage of the training, the monocular depth network is able to produce reasonable and cleaner depth maps that can supervise and constrain the NeRF loss (as shown in Figure 3). This is the main contribution and finding of this submission. The authors also showed an ablation study to justify the different components that were introduced.", "weaknesses": "The concerns I have for the paper is whether 1) the contribution introduced is enough for paper acceptance, and 2) how it positions itself and it claims in contrast to existing works. For the first point, the main contribution and distinction it has is the finding that the monocular depth prior can also be used to constrain unseen viewpoints. As shown in Figure 3, even on noisy NeRF renderings, the MDE prior can produce clean depth maps. An add-on to this is the slight improvement in results (as shown in the ablation study) by also fine-tuning the MDE network as NeRF is being optimized. However, the way the paper was written and motivated is the ambiguity in monocular depth estimation -- as written in the intro, Sec 4.1 and Sec 4.3, which is the main premise and story of the existing (though somewhat concurrent) work SCADE. The paper did cite SCADE, but however only mentions the difference that DaRF is able to train on unseen views. Differentiating with existing work and clarifying contributions can be improved for both paper presentation as well as the claims made for the submission. Now, the concern and question is whether 1) being the contribution of DaRF is enough as a contribution to meet the bar of Neurips. And for this reason I am giving an initial rating of borderline reject and would like to hear from the other reviewers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1. Contribution of DaRF / comparison with SCADE**\n> \n\nThank you for your comment. While SCADE [1] does have a similar setting to ours (leveraging MDE for few-shot NeRF reconstruction), we would like to emphasize that there are **critical differences** that position our work orthogonal to SCADE.\n\nThere is a fundamental difference between our work and SCADE’s approach toward handling ambiguity present within MDE: our work proposes an online, scene-specific adaptation of MDE which **directly adapts it to predict absolute, canonical geometry** in accordance with few-shot NeRF. Our novel local scale/shift alignment also aids this process. On the other hand, SCADE **injects uncertainty into MDE** through additional pretraining so that canonical geometry can be estimated through probabilistic modeling between multiple modes of estimated depths. So, while the ultimate goal regarding our MDEs may be similar (ambiguity removal), the core idea behind them are more or less opposite: our approach **directly removes ambiguity** present in MDE by leveraging canonical geometry captured by few-shot NeRF, while SCADE handles MDE through ***embracing its uncertainties and ambiguities** (as stated in its introduction)* and solving it with a probabilistic approach.\n\nThrough this key difference, our approach attains many advantages over SCADE:\n\n1. First, our approach does not require **additional pre-training of MDE** that SCADE requires. Because our online MDE adaptation requires no additional dataset nor training stages that SCADE relied upon, but enables a direct usage of an off-the-shelf MDE for few-shot NeRF, it is simpler, faster, and cheaper in aspects of training time and computation cost.\n2. Second, this approach **bypasses SCADE’s process of predicting multiple (20) depths of each input image** for probabilistic modeling, which also costs more training time and computational cost. Since our MDE is adapted in a more precise, scene-specific manner, it only needs to predict a single depth map that can be analytically fitted to NeRF through our **novel patchwise scale-shift fitting**, which is also our major contribution.\n3. Our usage of MDE at **unseen viewpoints** for regularization effect and artifact removal, a contribution you pointed out, is, therefore, a natural extension that is possible due to our adapted MDE’s enhanced ability to predict canonical geometry, as noted in Strength 1 of reviewer KpuH.\n4. Finally, we point out that this cheaper and faster approach achieves **higher performance** than SCADE.\n\nTo summarize, despite having similar objectives, our methodology introduces an orthogonal strategy of leveraging MDE through online training of NeRF for per-scene adaptation. This enables our model to achieve higher performance without SCADE’s MDE pretraining, multiple per-depth predictions, or probabilistic modeling. However, as the two methods are orthogonal, both methods may be able to be used in conjunction for even higher performance and more efficient few-shot NeRF regularization, which would be an exciting direction to expand our research. We are grateful for your comment and suggestion.\n\n> **Q2. Details regarding unseen view selection and loss**\n> \n\nWe generate an unseen camera pose every iteration, adding noise value sampled between an interval of [-6, +6] to the original Euler rotation angles of randomly selected input view pose. As mentioned in training details in our supplementary materials, the weights for the seen and unseen viewpoint loss are the same.\n\n> **Q3. Regarding intuition behind Eq6 / redundant terms**\n> \n\nWe clarify that our methodology revolves around the idea of **adapting MDE toward predicting a scene-specific absolute geometry**, which is achieved by the first addend term: this first term forces itself MDE to adapt towards multiview consistency so that its ill-posed nature is reduced and its initial **global depth prediction** grows to be more in accordance with the absolute geometry captured by NeRF.\n\nIn contrast, the second addend term, which takes into account patchwise scale-shift fitting, is designed to aid the modeling of **fine, detailed, local geometry** which the model has difficulty modeling without such local fitting.\n\nWe emphasize here that the second term alone cannot guide MDE toward multiview consistent geometry, as it solely deals with locally fitted depths whose location and scale have already been largely altered through scale/shift fitting. Therefore, when only the second term is used, MDE has **no incentive to adapt toward canonical geometry at the global prediction level** and only adapts toward local fine details. This leads to a drop in performance, as shown in the results below. When only the second term is used for optimization (scale-shift), it performs worse in every metric than in other cases where only the first term is used (L1) or both are used in conjunction (ours). This justifies our strategy of using both losses as effective and not redundant.\n\n|  | PSNR | SSIM | LPIPS | abs_rel | sq_rel | RMSE | log_RMS |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| scale-shift | 21.43 | 0.763 | 0.330 | 0.182 | 0.109 | 0.484 | 0.205 |\n| l1 | 21.45 | 0.763 | 0.327 | 0.157 | 0.079 | 0.386 | 0.176 |\n| ours | **21.58** | **0.765** | **0.325** | **0.151** | **0.071** | **0.356** | **0.168** |\n\n> **Q4. Dataset used for ablation**\n> \n\nThe quantitative results of the Ablation study in Table 3 contain **average values** of quantitative results yielded by every scene in ScanNet. It is solely for visualization purposes that we use Scene781 of ScanNet as a representative for qualitative results in Figure 8.\n\n> **Q5. Error bars**\n> \n\nWe apologize that we have made a mistake in checking the checklist, as we do not show error bars in our paper. Thank you for pointing this out.\n\n[1] Uy et al., SCADE: NeRFs from Space Carving with Ambiguity-Aware Depth Estimates, CVPR 2023"}, {"review_id": "d0WG0v9c5D", "reviewer": "Reviewer_VhkU", "summary": "The paper proposes a method to better use monocular depth in a few-shot NeRF setup. There are mainly two technical contributions to me:\n1. Applying mono depth constraint to unseen view;\n2. Un-distorting (scale and shift) monocular depth in a per-patch manner, rather than per-image.\n\n---\n**After rebuttal**: I have read authors' rebuttal and it addresses my concerns.", "questions": "See the weaknesses section.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The motivation makes a lot of sense. Monocular depth estimation suffers from various distortions, and un-distorting it with a global scale and shift per image is only a rough approximation. Exploring better ways to un-distort monocular depth is definitely valid motivation.\n2. It’s interesting to see that monocular depth networks actually perform decently on rendered images from half-trained NeRF models.", "weaknesses": "General\n1. L118-L120: I disagree with this definition of few-shot, i.e. $| S < 20 |$. The definition should be based on view-angle and scene coverage, not on an absolute number of images. For example, in LLFF, a forward-facing scene, NeRF can be trained very well for $| S < 20 |$, because 20 images cover the simple forward-facing scenes very well. In contrast, 50 images might be challenging for 360-degree scenes.\n2. Following point 1, how do the few-shot views cover the evaluated scenes? It would be good to have a top-view visualisation of selected cameras' positions and orientations.\n3. How many patches are used in training? From the supp mat I can see the patch size is 64x64, i.e. 4096 rays. This means there is only one patch used during training? In this case, modelling patch-wise scale/shift is same as image-wise scale/shift?\n\nWriting\n1. Figure 1 caption: unclear, seems like should be … by _applying_ pretrained MDE … (missing a word _applying_)\n2. Figure 2: \n    1. what is the input RGB and its monocular depth? I can see this is a top-view image of back-projected point clouds, but I cannot imagine what’s the input. \n    2. What is colour coding?\n    3. I don’t see why Fig 2b is better than Fig 2a from this image. I understand the motivation and it is supposed to be better when un-distort with patch-wise scale/shift, but I cannot see that from this visualisation.\n3. Sec 4.1 is kind of repeating intro. \n4. Symbol $M_l$ is undefined. I suppose it’s similar to $M_i$, but projected in $l \\rightarrow i$ direction?\n5. Symbol $s_i, t_i$ are undefined.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1. Specific definition of few-shot setting and its visualization**\n> \n\nWe agree with your statement that the precise definition of few-shot cannot be defined naively as |S<20|, described in our L118-L120. As you have said, its definition should be based on view-angle and general coverage of the scene. However, due to this very reason, it is difficult to precisely define the minimum number of viewpoints required for few-shot reconstruction, as that number will vary scene by scene according to each scene’s specific shape, geometry, and occlusions. \n\nIn this light, current few-shot NeRF works [1,2,3,4] assume extreme cases where i) the known viewpoints overlap with each other minimally, ii) all directions of the scene are viewed at least once so that there are no regions of the scene where no information is given. Such extreme scenarios cause original NeRF models to struggle heavily with geometric reasoning so that they cannot perform proper reconstruction, which necessitates the introduction of few-shot regularization methods.\n\nConsidering these criteria and the general characteristics of the scenes in the dataset, they decide upon a certain number of viewpoints that result in such extreme setting per dataset. For example, the forward-facing *LLFF dataset* selects equally distant 3 views for few-shot setting, and *NeRF-synthetic dataset*, despite its 360 degree viewpoint, also assumes 3 views as its few-shot setting considering its simplicity.\n\nIn this manner, we constructed few-shot train/test sets of Tanks and Temples dataset, where few-shot standard does not yet exist, considering its complexity, 360 degrees, real-world dataset. Following your constructive suggestion, we visualize how these few viewpoints cover our scenes in a minimal yet sufficient manner in Figure 1 of our attached pdf file. We believe this figure will be beneficial in justifying our few-shot setting to the readers, and we promise to include it in the final version of our paper. Thank you for the helpful comment.\n\n> **Q2. Number of patches used in training**\n> \n\nWe clarify that we sample each patch per iteration, and each patch’s scale/shift value does not equate to global image-wise scale/shift values. \n\nMore specifically, we sample a 64 by 64 patch and a 128 by 128 patch per iteration, but these patches are randomly sampled from a random seen pose and a random unseen pose each other, and it is this corresponding patch-wise region that we regularize with predicted MDE depth. In every iteration, both viewpoint and patch locations are randomly sampled a new, and the patch-wise scale-shift values are modeled accordingly. Since the viewpoints and patches are sampled uniformly, our regularization signal gives coverage of every possible region of the scene throughout the course of optimization. In this way, the local supervision signal influences all regions of the scene as optimization progresses.\n\n> **Q3. Details of Figure 2**\n> \n\nThank you for your constructive comment. We agree that Figure 2 is difficult to recognize at first sight, and we promise to fix it. \n\nOur intention for Figure 2 was to visualize our point that a single scale-shift value cannot perfectly fit into ground truth geometry due to the ambiguity of distances between object instances. For this reason, we have visualized **error** of a predicted point cloud of a room from the bird’s eye view to show that patch-wise fitting most accurately fits point cloud to ground truth geometry. This is done with jet color coding, so that red color means large error and blue color means small error. This point cloud is projected depth from the input image from the viewpoint stated as red camera. \n\nFollowing your comment, we promise to either replace it with Figure 1 of our supplementary material (which describes the same phenomenon more clearly) or reinforce it with an additional caption that explains it in a more detailed manner.\n\n> **Q4. Clarifications in writing**\n> \n\nThank you for pointing these out. We promise to make a modification to the caption of Figure 1 and Section 4.1 in the final version of our paper.\n\nAlso, we clarify that $M_l$ shares a similar definition with $M_i$, which we define as a confidence mask for seen viewpoint $i$ - therefore, $M_l$ corresponds to a confidence mask for unseen viewpoint $l$. Also, the notation $s_i,t_i$ in supplementary materials is a notation mistake: it is supposed to be $w_i, q_i$. Thank you for the careful reading and revision of our paper: we promise to clarify & revise these notations accordingly in the final version of our paper.\n\n[1] Jain et al., Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis, ICCV 2021.\n\n[2] Kim et al., InfoNeRF: Ray Entropy Minimization for Few-Shot Neural Volume Rendering, CVPR 2022.\n\n[3] Niemeyer et al., RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs, CVPR 2022.\n\n[4] Yang et al., FreeNeRF: Improving Few-shot Neural Rendering with Free Frequency Regularization, CVPR 2023."}]

Author rebuttal:
# General Response

We would like to first thank all the reviewers for their helpful suggestions and constructive reviews. We are greatly encouraged by their assessment of our work as **well-motivated** (VhkU), with novel and **interesting** (oMo1, VhkU) findings of effectively **resolving MDE’s ambiguities** (KpuH, VhkU), and its exploitation at **unseen viewpoints** (Ndsp, KpuH, VkhU) thus **improving upon previous models** (Eioc, KpuH). The reviews assess our work as displaying **state-of-the-art** (KpuH) and **attractive performance** (oMo1) on **real-world** (KpuH, oMo1) benchmarks, well-supported by **sufficient ablation** (oMo1, KpuH, Ndsp) and **good presentation** (oMo1, KpuH). We are grateful that they saw significance in our quantitative and qualitative improvement over our baselines, achieving few-shot novel view synthesis quality competitive to current SOTA models. We carefully address each concern given by reviewers with detailed explanations and supporting experimental results.

We clarify our work’s distinction from SCADE [1], which has been brought up by multiple reviewers due to its similar motivation of exploiting MDE for few-shot NeRF. Our work proposes an online, scene-specific adaptation of MDE which **directly adapts it to predict canonical geometry** in accordance with NeRF under optimization. On the other hand, SCADE **injects uncertainty into MDE** so that canonical geometry can be estimated through probabilistic modeling between multiple estimated depths. Therefore, while the two works’ ultimate goal regarding MDEs is similar (ambiguity removal), the core ideas behind them are orthogonal. Another important distinction is that our method of MDE adaption allows it to be leveraged to **predict depths of unknown viewpoints for artifact removal and geometry regularization**, which is a novel contribution of our method. We further elaborate on this distinction with much deeper detail in our first response to Reviewer Ndsp.

[1] Uy et al., SCADE: NeRFs from Space Carving with Ambiguity-Aware Depth Estimates, CVPR 2023

Meta-review: This submission received 5 positive recommendations. Initially, the reviewers were concerned about the limited contribution, the difference to SCADE, and the errors in MDE. The authors addressed most of the concerns in the rebuttal and the responses to reviewers’ additional comments. The reviewers reached a consensus of acceptance after the discussion period. The AC read through the submission, the review, the rebuttal, the confidential comments to the AC and the discussions. The AC agrees with the reviewers that the idea is interesting and novel and the evaluation is convincing. Per this, the AC supports the reviewers’ decision, and accepts this submission. The decision was discussed with and approved by the SAC. Please follow the reviewers’ comments (especially, pay more attention to reviewer Ndsp’s comments and advice) to improve the manuscript.
Decision: Accept (poster)