SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jiuhn Song* Seonghoon Park* Honggyu An*
Seokju Cho Min-Seop Kwak Sungjin Cho Seungryong Kim
Neural radiance field (NeRF) shows powerful performance in novel view synthesis and 3D geometry reconstruction, but it suffers from critical performance degradation when the number of known viewpoints is drastically reduced. Existing works attempt to overcome this problem by employing external priors, but their success is limited to certain types of scenes or datasets. Employing monocular depth estimation (MDE) networks, pretrained on large-scale RGB-D datasets, with powerful generalization capability would be a key to solving this problem: however, using MDE in conjunction with NeRF comes with a new set of challenges due to various ambiguity problems exhibited by monocular depths. In this light, we propose a novel framework, dubbed DäRF, that achieves robust NeRF reconstruction with a handful of real-world images by combining the strengths of NeRF and monocular depth estimation through online complementary training. Our framework imposes the MDE network's powerful geometry prior to NeRF representation at both seen and unseen viewpoints to enhance its robustness and coherence. In addition, we overcome the ambiguity problems of monocular depths through patch-wise scale-shift fitting and geometry distillation, which adapts the MDE network to produce depths aligned accurately with NeRF geometry. Experiments show our framework achieves state-of-the-art results both quantitatively and qualitatively, demonstrating consistent and reliable performance in both indoor and outdoor real-world datasets. Project page is available at https://ku-cvlab.github.io/DaRF/.
Neural radiance field (NeRF) [27] has gained significant attention for its powerful performance in reconstructing 3D scenes and synthesizing novel views. However, despite its impressive performance, NeRF often comes with a considerable limitation in that its performance highly relies on the presence of densely well-calibrated input images which are difficult to acquire. As the number of input images is reduced, NeRF's novel view synthesis quality drops significantly, displaying failure cases such as erroneous overfitting to the input images [15, 29], artifacts clouding empty spaces [29], or degenerate geometry that yields incomprehensible jumble when rendered at unseen viewpoints [16]. These challenges derive from its under-constrained nature, causing it to have extreme difficulty mapping a pixel in input images to a correct 3D location. In addition, NeRF's volume rendering allows the model to map a pixel to multiple 3D locations [11], exacerbating this problem.
Previous few-shot NeRF methods attempt to solve these issues by imposing geometric regularization [29, 16, 20] or exploiting external 3D priors [11, 37] such as depth information extracted from
Figure 1: Overview. DärRF shows robust optimization of few-shot NeRF through MDE's geometric prior, removing inherent ambiguity from MDE through novel patch-wise distillation loss and MDE adaptation. Unlike existing work [45] that distills depths by applying pretrained MDE to NeRF at seen view only, our DärRF fully exploits the ability of MDE by jointly optimizing NeRF and MDE at a specific scene, and distilling the monocular depth prior to NeRF at both seen and unseen views.
input images by COLMAP [39]. However, these methods have weaknesses in that they use 3D priors extracted from a few input images only, which prevents such guidance from encompassing the entire scene. To effectively tackle all the issues mentioned above, pretrained monocular depth estimation (MDE) networks with strong generalization capability [35, 34, 4] could be used to inject an additional 3D prior into NeRF that facilitates robust geometric reconstruction. Specifically, geometry prediction by MDE can constrain NeRF into recovering smooth and coherent geometry, while their bias towards predicting smooth geometry helps to filter out fine-grained artifacts that clutter the scene. More importantly, NeRF's capability to render any unseen viewpoints enables fully exploiting the capability of the MDE, as MDE could provide depth prior to the numerous renderings of unseen viewpoints as well as the original input viewpoints. This allows injecting additional 3D prior to effectively covering the entire scene instead of being constrained to a few input images.
However, applying MDE to few-shot NeRF is not trivial, as there are ambiguity problems that hinder the monocular depth from serving as a good 3D prior. Primarily, relative depths predicted by MDEs are not multiview-consistent [5]. Moreover, MDEs perform poorly in estimating depth differences between multiple objects: this prevents global scale-shift fitting [55, 26] from being a viable solution, as alignment to one region of the scene inevitably leads to misalignment in many other regions. There also exists a convexity problem [26], in which the MDE has difficulty determining whether the surface is planar, convex, or concave, are also present. To overcome these challenges, we introduce a novel method to adapt MDE to NeRF's absolute scaling and multiview consistency as NeRF is regularized by MDE's powerful 3D priors, creating a complementary cycle.
In this paper, we propose DärF, short for Monocular Depth Adaptation for boosting Radiance Fields from Sparse Input Views, which achieves robust optimization of few-shot NeRF through MDE's geometric prior, as well as MDE adaptation for alignment with NeRF through complementary training (see Fig. 1). We exploit MDE for robust geometry reconstruction and artifact removal in both unseen and seen viewpoints. In addition, we leverage NeRF to adapt MDE toward multiview-consistent geometry prediction and introduce novel patch-wise scale-shift fitting to more accurately map local depths to NeRF geometry. Combined with a confidence modeling technique for verifying accurate depth information, our method achieves state-of-the-art performance in few-shot NeRF optimization. We evaluate and compare our approach on real-world indoor and outdoor scene datasets, establishing new state-of-the-art results for the benchmarks.
Neural radiance field. Neural radiance field (NeRF) [27] represents photo-realistic 3D scenes with MLP. Owing to its remarkable performance, there has been a variety of follow-up studies [2, 51, 25].
These studies improve NeRF such as dynamic and deformable scenes [31, 44, 33, 1], real-time rendering [51, 36, 28], unbounded scene [3, 42, 48] and generative modeling [40, 30, 7]. However, these works still encounter challenges in synthesizing novel views with a limited number of images in a single scene, limiting their applicability in real-world scenarios.
Few-shot NeRF. Numerous few-shot NeRF works attempted to address few-shot 3D reconstruction problem through various techniques, such as pretraining external priors [52, 9], meta-learning [43], regularization [15, 29, 16, 20] or off-the-shelf modules [15, 29]. Recent approaches [29, 16, 20] emphasize the importance of geometric consistency and apply geometric regularization at unknown viewpoints. However, these regularization methods show limitations due to their heavy reliance on geometry information recovered by NeRF. Other works such as DS-NeRF [11], DDP-NeRF [37] and SCADE [45] exploit additional geometric information, such as COLMAP [39] 3D points or monocular depth estimation, for geometry supervision. However, these works have critical limitations of only being able to provide geometry information corresponding to existing input viewpoints. Unlike these works, our work demonstrates methods to provide geometric prior even at unknown viewpoints with MDE for more effective geometry reconstruction.
Monocular depth estimation. Monocular depth estimation (MDE) is a task that aims to predict a dense depth map given a single image. Early works on MDE used handcrafted methods such as MRF for depth estimation [38]. After the advent of deep learning, learning-based approaches [14, 17, 21] were introduced to the field. In this direction, the models were trained on ground-truth depth maps acquired by RGB-D cameras or LiDAR sensors to predict absolute depth values [24, 23]. Other approaches trained the networks on large-scale diverse datasets [8, 22, 34, 35], which demonstrates better generalization power. These approaches struggle with depth ambiguity caused by ill-posed problem, so the following works LeRes [50] and ZoeDepth [4] opt to recover absolute depths using additional parameters.
Incorporating MDE into 3D representation. As both NeRF and monocular depth estimation are closely related, there have been some works that utilize MDE models to enhance NeRF's performance. NeuralLift [49], MonoSDF [53] and SCADE [45] leverage depths predicted by pretrained MDE for depth ordering and detailed surface reconstruction, respectively. Other works optimize scene-specific parameters, such as depth predictor utilizing depth recovered by COLMAP [47] or learnable scale-shift values for reconstruction in noisy pose setting [6]. However, these previous approaches were limited in that MDEs were used to provide prior to only the input viewpoints, which constrains their effectiveness when input views are reduced, e.g., in the few-shot setting.
As a concurrent work, SCADE [45] utilizes MDE for sparse view inputs, by injecting uncertainty into MDE through additional pretraining so that canonical geometry can be estimated through probabilistic modeling between multiple modes of estimated depths. While the ultimate goal which is to overcome the ambiguity of MDE may be similar, our approach directly removes ambiguity present in MDE by finetuning with canonical geometry captured by NeRF, for effective suppression of artifacts and divergent behaviors of few-shot NeRF.
NeRF [27] represents a scene as a continuous function  $\mathcal{F}_{\theta}(\cdot)$  represented by a neural network with parameters  $\theta$ . During optimization, 3D points are sampled along rays represented by  $\mathbf{r}$  coming from a set of input images  $S = \{I_i\}$ , whose ground truth camera poses are given, for evaluation by the neural network. For each sampled point,  $\mathcal{F}_{\theta}(\cdot)$  takes as input its coordinate  $\mathbf{x} \in \mathbb{R}^3$  and viewing direction  $\mathbf{d} \in \mathbb{R}^2$  with a positional encoding  $\gamma(\cdot)$  that facilitates learning high-frequency details, and outputs a color  $\mathbf{c} \in \mathbb{R}^3$  and a density  $\sigma \in \mathbb{R}$  such that  $\{\mathbf{c}, \sigma\} = \mathcal{F}_{\theta}(\gamma(\mathbf{x}), \gamma(\mathbf{d}))$ . With a ray parameterized as  $\mathbf{r_p}(t) = \mathbf{o} + t\mathbf{d_p}$ , starting from camera center  $\mathbf{o}$  along the direction  $\mathbf{d_p}$ , color and depth value at the pixel  $\mathbf{p}$  are rendered as follows:
$$
\bar {I} (\mathbf {p}) = \int_ {t _ {n}} ^ {t _ {f}} T (t) \sigma \left(\mathbf {r} _ {\mathbf {p}} (t)\right) \mathbf {c} \left(\mathbf {r} _ {\mathbf {p}} (t)\right) d t, \quad \bar {D} (\mathbf {p}) = \int_ {t _ {n}} ^ {t _ {f}} T (t) \sigma \left(\mathbf {r} _ {\mathbf {p}} (t)\right) t d t, \tag {1}
$$
where  $\bar{I} (\mathbf{p})$  and  $\bar{D} (\mathbf{p})$  are rendered color and depth values at the pixel  $\mathbf{p}$  along the ray  $\mathbf{r_p}(t)$  from  $t_n$  to  $t_f$ , and  $T(t)$  denotes an accumulated transmittance along the ray from  $t_n$  to  $t$  as follows:
$$
T (t) = \exp \left(- \int_ {t _ {n}} ^ {t} \sigma (\mathbf {r _ {p}} (s)) d s\right). \tag {2}
$$
(b)
Figure 2: Effectiveness of patch-wise scale and shift adjustment: (a) input image, (b) monocular depth with image-level adjustment, and (c) monocular depth with patch-level adjustment. We visualize the error of adjusted monocular depth from input image compared to GT depth value. The proposed patch-level adjustment helps to minimize the errors caused by inconsistency in depth differences among objects.
Based on this volume rendering,  $\mathcal{F}_{\theta}(\cdot)$  is optimized by the reconstruction loss  $\mathcal{L}_{\mathrm{recon}}$  that compares rendered color  $\bar{I} (\mathbf{p})$  to corresponding ground-truth  $I(\mathbf{p})$ , with  $\mathcal{R}$  as a set of pixels for training rays:
$$
\mathcal {L} _ {\text {r e c o n}} = \sum_ {I _ {i} \in \mathcal {S}} \sum_ {\mathbf {p} \in \mathcal {R}} \| I _ {i} (\mathbf {p}) - \bar {I} _ {i} (\mathbf {p}) \| _ {2} ^ {2}. \tag {3}
$$
Our work explores the setting of few-shot optimization with NeRF [16, 20]. Whereas the number of input viewpoints  $|S|$  is normally higher than one hundred in the standard NeRF setting [27], the task of few-shot NeRF considers scenarios when  $|S|$  is drastically reduced to a few viewpoints (e.g.,  $|S| < 20$ ). With such a small number of input viewpoints, NeRF shows high divergent behaviors such as geometry breakdown, overfitting to input viewpoints, and generation of artifacts that cloud the empty space between the camera and object, which causes its performance to drop sharply [15, 16, 29]. To overcome this problem, existing few-shot NeRF frameworks applied regularization techniques at unknown viewpoints to constrain NeRF with additional 3D priors [37, 11] and enhance the robustness of geometry, but they showed limited performance.
4.1 Motivation and Overview
Our framework leverages the complementary benefits of few-shot NeRF and monocular depth estimation networks for the goal of robust 3D reconstruction. The benefits that pretrained MDE can provide to few-shot NeRF are clear and straightforward: because they predict dense geometry, they provide guidance for the NeRF to recover more smooth geometry. In cases where few-shot NeRF's geometry undergoes divergent behaviors, MDE provides strong constraints to prevent the global geometry from breaking down.
However, there are difficult challenges that must be overcome if the depths estimated by MDE are to be used as 3D prior to NeRF. These challenges, which can be summarized as depth ambiguity problems [26], stem from the inherent ill-posed nature of the monocular depth estimation. Most importantly, MDE networks only predict relative depth information inferred from an image, meaning it is initially not aligned to NeRF's absolute geometry [4]. Global scaling and shifting may seem to be the answer, but this approach leads us to another depth ambiguity problem, as predicted scales and spacings of each instance are inconsistent with one another, as demonstrated in (b) of Fig. 2. Additionally, MDE's weakness in predicting the convexity of a surface, whether it is flat, convex, or concave - also poses a problem in using this depth for NeRF guidance.
In this light, we adapt a pre-trained monodepth network to a single NeRF scene so that its powerful 3D prior can be leveraged to its maximum capability in regularizing the few-shot NeRF. In the following, we first explain how to distill geometric prior from off-the-shelf MDE model [35] from both seen and unseen viewpoints (Sec. 4.2). We also provide a strategy for adapting the MDE model to handle ill-posed problems to a specific scene, while keeping its 3D prior knowledge (Sec. 4.3). Then, we demonstrate a method to handle inaccurate depths (Sec. 4.4). Fig. 1 shows an overview of our method, compared to previous works using MDE prior [45, 53].
(a) Rendered color  $\bar{I}$
(b) Rendered depth  $\bar{D}$
(c) Mono-depth from  $\bar{I}$
Figure 3: Robustness of MDE model for multi-view scale ambiguity and artifacts: (a-b) color and depth of NeRF rendered in the early stage of the training, (c-d) monocular depths estimated from rendered image  $\bar{I}$  and input image  $I$ . The results show that MDE model ignores the artifacts of rendered images by NeRF, enabling reliable supervision for seen and unseen viewpoint.
(d) Mono-depth from  $I$
4.2 Distilling Monocular Depth Prior into Neural Radiance Field
To prevent the degradation of reconstruction quality in few-shot NeRF, we propose to distill monocular depth prior to the neural radiance field during optimization. By exploiting pre-trained MDE networks [34, 35], which have high generalization power, we enforce a dense geometric constraint on both seen and unseen viewpoints by using estimated monocular depth maps as pseudo ground truth depth for training few-shot NeRF. We describe the details of this process below.
Monocular depth regularization on seen views. We leverage a pre-trained MDE model, denoted as  $\mathcal{G}_{\phi}(\cdot)$  with parameters  $\phi$ , to predict pseudo depth map from given seen view image  $I_{i}$  as  $D_{i}^{*} = \mathcal{G}_{\phi}(I_{i})$ . Since  $D_{i}^{*}$  is initially a relative depth map, it needs to be scaled and shifted into an absolute depth [55] and aligned with NeRF's rendered depth  $\bar{D}$  in order for it to be used as pseudo-depth  $D^{*}$ . However, the scale and shift parameters inferred from the global statistic may undermine local statistic [55]. For example, as shown in Fig. 2 (b), global scale fitting tends to favor dominant objects in the image, leading to ill-fitted depths in less dominant sections of the scene due to inconsistencies in predicted depth differences between the objects. Naively employing such inaccurately estimated depths for distillation can adversely impact the overall geometry of the NeRF.
To alleviate this issue, we propose a patch-wise adjustment of scale and shift parameters, reducing the impact of erroneous depth differences, as illustrated in Fig. 2 (c). The depth consistency loss is defined as follows:
$$
\mathcal {L} _ {\text {s e e n}} = \sum_ {I _ {i} \in \mathcal {S}} \sum_ {\mathbf {p} \in \mathcal {P}} \| \left(w _ {i} \mathbf {s g} \left(D _ {i} ^ {*} (\mathbf {p})\right) + q _ {i}\right) - \bar {D} _ {i} (\mathbf {p}) \|, \tag {4}
$$
where  $w_{i}$  and  $q_{i}$  denote the scale and shift parameters obtained by least square [35] between  $D_{i}^{*}$  and  $\bar{D}_i$ ,  $\mathcal{P}$  denotes a set of pixels within a patch, and  $\mathrm{sg}(\cdot)$  denotes stop-gradient operation. Thus patch-based approach also helps to overcome the computational bottleneck of full image rendering.
Monocular depth regularization on unseen views. We further propose to give supervision even at unseen viewpoints. As NeRF has the ability to render any unseen viewpoint of the scene, we render color  $\bar{I}_l$  and depth  $\bar{D}_l$  from a sampled patch of  $l$ -th novel viewpoint. Sequentially, we extract a monocular depth map from the rendered image as  $\bar{D}_l^* = \mathcal{G}_{\phi}(\bar{I}_l)$ . Then, we enforce consistency between our rendered depth  $\bar{D}_l$  and the monocular depth  $\bar{D}_l^*$  of  $l$ -th novel viewpoint as follows:
$$
\mathcal {L} _ {\text {u n s e e n}} = \sum_ {I _ {l} \in \mathcal {U}} \sum_ {\mathbf {p} \in \mathcal {P}} \| \left(w _ {l} \mathbf {s g} \left(\bar {D} _ {l} ^ {*} (\mathbf {p})\right) + q _ {l}\right) - \bar {D} _ {l} (\mathbf {p}) \|, \tag {5}
$$
where  $\mathcal{U}$  denotes a set of unseen view images,  $w_{l}$  and  $q_{l}$  denotes the scale and shift parameters used to align  $\bar{D}_l^*$  towards  $\bar{D}_l$ , and  $\mathcal{P}$  denotes randomly sampled patch.
A valid concern regarding this approach is that monocular depth obtained from noisy NeRF rendering may be affected by fine-grained rendering artifacts that frequently appear in unseen viewpoints of few-shot NeRF, resulting in noisy and erroneous pseudo-depths. However, we demonstrate in
Fig. 3 that a strong geometric prior within the MDE model exhibits robustness against such artifacts, effectively filtering out the artifacts and thereby providing reliable supervision for the unseen views.
It should be noted that our strategy differs from previous methods [11, 37, 53, 45] that exploit monocular depth estimation [34] and external depth priors such as COLMAP [39]. These methods only impose depth priors upon the input viewpoints, and thus their priors only influence the scene partially due to self-occlusions and sparsity of known views. Our method, on the other hand, enables external depth priors to be applied to any arbitrary viewpoint and thus allows guidance signals to thoroughly reach every location of the scene, leading to more robust and coherent NeRF optimization.
4.3 Adaptation of MDE via Neural Radiance Field
Although the patch-wise distillation of monocular depth provides invariance to depth difference inconsistency in MDE, the ill-posed nature of monocular depth estimation often introduces additional ambiguities, such as the inability to distinguish whether the surface is concavity, convexity, or planar or difficulty in determining the orientation of flat surfaces [26]. We argue that these ambiguities arise due to the MDE lacking awareness of the scene-specific absolute depth priors and multiview consistency. To address this issue, we propose providing the scene priors optimized NeRF to MDE, whose knowledge of canonical space and absolute geometry helps eliminate the ambiguities present within MDE. Therefore, we propose to adapt the MDE to the absolute scene geometry, formally written as:
$$
\mathcal {L} _ {\mathrm {M D E}} = \sum_ {I _ {i} \in \mathcal {S}} \sum_ {\mathbf {p} \in \mathcal {P}} \left\{\| \mathbf {s g} (\bar {D} _ {i} (\mathbf {p})) - D _ {i} ^ {*} (\mathbf {p}) \| + \| (w _ {i} \mathbf {s g} (\bar {D} _ {i} (\mathbf {p})) + q _ {i}) - D _ {i} ^ {*} (\mathbf {p}) \| \right\}. \tag {6}
$$
In addition to the patch-wise loss in Eq. 4, we add an  $l - 1$  loss without scale-shift adjustment to adapt the MDE with absolute depth prior. We also introduce a regularization term to preserve the local smoothness of MDE, given by:
$$
\mathcal {L} _ {\text {r e g}} = \sum_ {I _ {i} \in \mathcal {S}} \sum_ {\mathbf {p} \in \mathcal {P}} \| \left(w _ {i} \mathbf {s g} \left(D _ {i} ^ {*}, \text {i n i t} (\mathbf {p})\right) + q _ {i}\right) - D _ {i} ^ {*} (\mathbf {p}) \|, \tag {7}
$$
where  $D_{i}^{*,\mathrm{init}}$  denotes monocular depth map of  $I_{i}$  extracted from MDE with initial pre-trained weight.
4.4 Confidence Modeling
Our framework must take into account the errors present in both few-shot NeRF and estimated monocular depths, which will propagate [41] and intensify during the distillation process if left unchecked. To prevent this, we adopt confidence modeling [20, 41] inspired by self-training approaches [41], to verify the accuracy and reliability of each ray before the distillation process.
The homogeneous coordinates of a pixel  $\mathbf{p}$  in the seen viewpoint are transformed to  $\mathbf{p}'$  at the target viewpoint using the viewpoint difference  $R_{i\rightarrow l}$  and the camera intrinsic parameter  $K$ , as follows:
$$
\mathbf {p} ^ {\prime} \sim K R _ {i \rightarrow l} D _ {i} (\mathbf {p}) K ^ {- 1} \mathbf {p}. \tag {8}
$$
We generate the confidence map  $M_{i}$  by measuring the distance between rendered depth of the unseen viewpoint and MDE output of seen viewpoint such that
$$
M _ {i} (\mathbf {p}) = \left[ \left\| \left(w _ {i} D _ {i} ^ {*} (\mathbf {p}) + q _ {i}\right) - \bar {D} _ {l} \left(\mathbf {p} ^ {\prime}\right) \right\| <   \tau \right], \tag {9}
$$
where  $\tau$  denotes threshold parameter,  $[\cdot ]$  is Iverson bracket, and  $D_{l}(\mathbf{p}^{\prime})$  refers to depth value of the corresponding pixel at  $l$ -th unseen viewpoint for reprojected target pixel  $\mathbf{p}$  of  $i$ -th seen viewpoint. We fit  $D_{i}^{*}$  to absolute scale, where scale and shift parameters,  $w_{i}$  and  $q_{i}$ , are obtained by least square [35] between  $D_{i}^{*}$  and  $\bar{D}_i$ .
4.5 Overall Training
With the incorporation of confidence modeling, the loss functions for both the radiance field and MDE can be defined.  $\mathcal{L}_{\mathrm{seen}}$  and  $\mathcal{L}_{\mathrm{unseen}}$  can be redefined as:
$$
\mathcal {L} _ {\text {s e e n}} = \sum_ {I _ {i} \in \mathcal {S}} \sum_ {\mathbf {p} \in \mathcal {P}} M _ {i} (\mathbf {p}) \| \left(w _ {i} \mathbf {s g} \left(D _ {i} ^ {*} (\mathbf {p})\right) + q _ {i}\right) - \bar {D} _ {i} (\mathbf {p}) \|, \tag {10}
$$
$$
\mathcal {L} _ {\text {u n s e e n}} = \sum_ {I _ {l} \in \mathcal {U}} \sum_ {\mathbf {p} \in \mathcal {P}} M _ {l} (\mathbf {p}) \| \left(w _ {l} \mathbf {s g} \left(\bar {D} _ {l} ^ {*} (\mathbf {p})\right) + q _ {l}\right) - \bar {D} _ {l} (\mathbf {p}) \|. \tag {11}
$$