SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Injae Kim*
dna9041@korea.ac.kr
Minhyuk Choi*
sodlqnf123@korea.ac.kr
hyunwoojkim@korea.ac.kr
Neural Radiance Field (NeRF) has enabled novel view synthesis with high fidelity given images and camera poses. Subsequent works even succeeded in eliminating the necessity of pose priors by jointly optimizing NeRF and camera pose. However, these works are limited to relatively simple settings such as photometrically consistent and occluder-free image collections or a sequence of images from a video. So they have difficulty handling unconstrained images with varying illumination and transient occluders. In this paper, we propose UP-NeRF (Unconstrained Pose-prior-free Neural Radiance Fields) to optimize NeRF with unconstrained image collections without camera pose prior. We tackle these challenges with surrogate tasks that optimize color-insensitive feature fields and a separate module for transient occluders to block their influence on pose estimation. In addition, we introduce a candidate head to enable more robust pose estimation and transient-aware depth supervision to minimize the effect of incorrect prior. Our experiments verify the superior performance of our method compared to the baselines including BARF and its variants in a challenging internet photo collection, Phototourism dataset. The code of UP-NeRF is available at https://github.com/mlvlab/UP-NeRF.
Neural Radiance Fields (NeRF) [1] opened up a new chapter in novel view synthesis by demonstrating its capacity to generate high-quality novel view images given only a set of 2D images. Its powerful performance has enabled many practical applications including virtual/augmented reality (VR/AR) [2, 3], autonomous systems [4, 5], and robotics [6-8]. Various follow-up studies have developed NeRF by addressing its limitations, such as the necessity of dense input views [9-11] and long training time [12-17]. Also, there have been efforts [18, 19] to deal with unconstrained images with varying illuminations and transient occluders to relieve the burden of collecting clean images or establishing a specialized setting to capture them. For instance, NeRF-W [18] introduces an appearance embedding to handle varying illumination and a transient head to filter transient occluders.
One of the popular research topics in NeRF is joint optimization for pose and neural scene representation. Originally, NeRF requires accurate camera pose priors which are generally obtained with classic methods like structure from motion (SfM) [20]. Although the poses obtained from such a method are treated as ground truth in many works, it does not always give optimal results or even fails to converge. Hence recent works [21-23] have proposed a method so-called unposed-NeRF that simultaneously trains camera poses and NeRF. This eliminates the necessity of pose priors and burdensome preprocessing. While these methods have only succeeded in forward-facing scenes, NoPe-NeRF [24] further improved the method to even outdoor scenes with continuous images from videos. These previous works, however, are based on the photometric loss which is not reliable for
*First two authors have an equal contribution.
Figure 1: Deep feature focuses on semantic information of objects rather than their colors, which makes it an effective descriptor for images with photometric inconsistency.
unconstrained images, so they have difficulty in training with unconstrained images. In addition, the unconstrained images have complex camera poses, which is hard to optimize if a prior like continuity of images from a video is not available. Therefore, training unposed-NeRF with unconstrained images causes contradiction: To estimate camera poses accurately, images must be photometrically consistent without transient occluders; to treat unconstrained images without photometric consistency and transient occluders, accurate camera poses are needed. Previous unposed-NeRFs in the literature are not capable of learning in an unconstrained setting, leading to suboptimal pose estimation and poor rendering quality.
To tackle the problem, we propose UP-NeRF (Unconstrained Pose-prior-free Neural Radiance Field) that jointly optimizes pose and neural scene representations while minimizing the influence of photometric inconsistency and transient occluders on pose estimation. Our contributions are as follows: First, we introduce the novel architecture that enables robust pose optimization with images of complex camera poses. Second, we propose feature-surrogate bundle adjustment by adopting the deep feature as a descriptor. Third, we enable transient-free pose optimization with the isolated network. Lastly, our transient-aware depth loss suggests an effective way to impose depth prior only to the static objects. Consequently, our model enables successful training of unposed-NeRF with unconstrained images, showing its effectiveness over a naive combination of current methods.
Neural Radiance Fields (NeRF). With the advent of techniques adopting neural fields to reconstruct 3D representations [25-29], Neural Radiance Field (NeRF) [1] proposed by Mildenhall et al. showed a promising result without the requirement of exact 3D geometry of each object in the scene. Compared to previous approaches which require a pre-defined representation of the scene by meshes [30, 31], voxels [25, 26], or point clouds [27, 32], NeRF only requires a set of densely captured 2D images and corresponding camera parameters of each image. Despite its light requirements, it demonstrates quality reconstruction of the complex geometry of scenes. The key idea of NeRF is that even though given are only 2D images, an implicit function represented by a multilayer perceptron can be utilized to parameterize the color and density of each 3D point on a ray, which is subsequently synthesized to decide the color of a pixel from which the ray was shot. Thanks to its simple but effective architecture, many other researches [10, 12-14, 33-35] have followed the trail of NeRF to ameliorate its weakness and improve performance.
Joint optimization of camera pose. Among a lot of interesting NeRF-based works, one major group of following works deals with the requirement of accurate camera pose of each image. Accurate camera pose including intrinsic and extrinsic parameters of a camera is indispensable for NeRF in that sampling accurate 3D coordinates in the world space heavily depends on them. However, such an accurate camera pose is not easily equipped unless the dataset is custom-made like a synthetic dataset, so structure from motion (SfM) [36] has been a classical solution to obtain such parameters. Although a lot of works have postulated camera poses from SfM to be ground truth, they can be sub-optimal depending on properties of scenes like monotonous texture. To solve this problem, other methods [22-24, 37-40] propose joint optimization of a neural field and poses.
Expansion to unconstrained scenes. Several researches [18, 19] dealt with unconstrained outdoor images where color consistency is not guaranteed owing to various environments (sunshine, time, weather, etc.) and so-called transient entities (pedestrian, cars, sundries, etc.) appear in images.
S-NeRF [41] and CF-NeRF [42] proposed a Bayesian network and normalizing flow for measuring the uncertainty of transient objects.  $\mathrm{D}^2\mathrm{NeRF}$  [43] and RobustNeRF [44] demonstrated that transient objects can be sharply decoupled from static ones without blurry floaters. Some works [45, 46] utilized mask supervision to jointly optimize camera pose with object-centric images of different illumination. In contrast to previous works which require either color consistency or the absence of transient objects to jointly optimize the pose, our method is capable of optimizing camera pose without both requirements.
We present a framework dubbed UP-NeRF to optimize unposed-NeRF under challenging conditions. We first briefly summarize Neural Radiance Fields (NeRF) [1] followed by joint optimization of camera poses and NeRF. In Section 3.1, we propose a new architecture that handles different convergence rates of camera poses, which can enable robust training of unposed-NeRF. Next, we explain our methods in Section 3.2 and 3.3 that handle the difficulty of unconstrained image collections like different light conditions (e.g., weather, time, etc.) and transient occluders. In addition, we will discuss how monocular depth priors can be used to optimize unposed-NeRF with unconstrained images in Section 3.4.
Neural Radiance Fields (NeRF). Given  $N$  images  $\{\mathbf{I}_i\}_{i=1}^N$  and camera parameters, Neural Radiance Field (NeRF)  $\Pi_\theta$  learns a continuous volumetric radiance field which maps a 3D point  $\mathbf{x} = (x,y,z)$  and a viewing direction  $\mathbf{d} = (d_x,d_y,d_z)$  to volume density  $\sigma$  and color  $\mathbf{c} = (r,g,b)$ . The mapping functions parameterized by simple neural networks, e.g., multi-layer perceptrons (MLPs), are defined as
$$
[ \sigma (t), \mathbf {z} (t) ] = \Pi_ {\theta_ {1}} \left(\gamma_ {\mathbf {x}} (\mathbf {r} (t))\right), \quad \mathbf {c} (t) = \Pi_ {\theta_ {2}} \left(\mathbf {z} (t), \gamma_ {\mathbf {d}} (\mathbf {d})\right), \tag {1}
$$
where  $\gamma_{\mathbf{x}}(\cdot)$  is a positional encoding function for 3D coordinates of each sample point on the ray  $\mathbf{r}(t) = \mathbf{o} + t\mathbf{d}$ , which is parameterized by the origin point  $\mathbf{o}$  and the direction  $\mathbf{d}$ . Color value  $\mathbf{c}(t)$  is encoded with an intermediate feature  $\mathbf{z}(t)$  and a positional encoding function for view directions  $\gamma_{\mathbf{d}}(\cdot)$ . The synthesized RGB value  $\hat{\mathbf{C}}(\mathbf{r})$  is calculated by volume rendering as
$$
\hat {\mathbf {C}} (\mathbf {r}) = \sum_ {k = 1} ^ {K} T \left(t _ {k}\right) \alpha \left(\sigma \left(t _ {k}\right) \delta_ {k}\right) \mathbf {c} \left(t _ {k}\right), \tag {2}
$$
$$
\text {w h e r e} T \left(t _ {k}\right) = \prod_ {k ^ {\prime} = 1} ^ {k - 1} \left(1 - \alpha \left(t _ {k ^ {\prime}}\right)\right) = \exp \left(- \sum_ {k ^ {\prime} = 1} ^ {k - 1} \sigma \left(t _ {k ^ {\prime}}\right) \delta_ {k ^ {\prime}}\right),
$$
where  $\delta_{k} = t_{k + 1} - t_{k}$  is the distance between adjacent points for approximated rendering via quadrature on discrete points and  $\alpha (t_k) = 1 - \exp (-\sigma (t_k)\delta_k)$ . To improve sampling efficiency, NeRF employs coarse and fine models for hierarchical sampling. The coarse model is optimized using stratified sampling, and then the fine model is optimized with samples biased toward the relevant parts of the volume. The parameters are optimized by minimizing photometric loss  $\mathcal{L}_{\mathrm{rgb}} = \sum \| \mathbf{C}(\mathbf{r}) - \hat{\mathbf{C}} (\mathbf{r})\| _2^2$  with given ground-truth color  $\mathbf{C}(\mathbf{r})$ . Note that even if it is not a color map, any information that represents the scene can be used for training a neural field. In this paper, we exploit deep feature map  $\mathbf{F}$  as well as color map  $\mathbf{C}$  for training, which have been studied by Kobayashi et al. [47] to learn selectively editable neural fields.
NeRF with camera pose estimation. Some prior works optimize NeRF with a joint estimation of camera pose. We parameterize the camera poses  $\{\mathbf{p}_i\}_{i=1}^N$  as 6 degrees of freedom and assume that the camera intrinsics are known as several prior works [23, 24, 39, 48]. Then, the problem can be formulated as
$$
\theta^ {*}, \mathbf {p} ^ {*} = \arg \min  _ {\theta , \hat {\mathbf {p}}} \mathcal {L} _ {\mathrm {r g b}} (\hat {\mathbf {C}}, \hat {\mathbf {p}} | \mathbf {C}), \tag {3}
$$
where  $\hat{\mathbf{p}}$  is learnable camera pose. BARF [23] shows that a coarse-to-fine positional encoding strategy facilitates more stable joint optimization of NeRF and camera pose. Following BARF, we parameterize camera poses with the  $\mathfrak{se}(3)$  Lie algebra and use a coarse-to-fine strategy for pose optimization.
3.1 Candidate head for robust pose optimization
Training unposed-NeRF is a joint optimization process with a chick-and-egg problem; estimation of camera pose  $\hat{\mathbf{p}}$  requires accurate scene representation  $\Pi_{\theta}$  and vice versa. During the initial phase of joint training, we observed that inaccurate scene representation, which is represented by predicted density  $\sigma$  and color  $\mathbf{c}$  in Eq. (1), causes specific images to be optimized in the wrong poses. We refer to these images as 'hard-pose' images, which may include images that capture the details of the scene or have little overlap with other images like the image in Fig. 4. To mitigate this problem, we introduce an additional MLP head called candidate head. It yields image-dependent (and tentative) scene representations by density  $\sigma_i^{(c)}$  and color  $\mathbf{c}_i^{(c)}$  given candidate embeddings  $\{\ell_i^{(c)}\}_{i = 1}^N$  as:
$$
\left[ \sigma_ {i} ^ {(c)} (t), \mathbf {c} _ {i} ^ {(c)} (t) \right] = \Pi_ {\theta_ {3}} \left(\mathbf {z} (t), \ell_ {i} ^ {(c)}\right). \tag {4}
$$
Note that  $(\cdot)^{(c)}$  denotes that it is related to the candidate embeddings. The synthesized RGB value  $\hat{\mathbf{C}}_i^{(c)}$  is generated by a joint volumetric rendering of the shared representation  $(\sigma$  and c) and the image dependent representation  $(\sigma_{i}^{(c)})$  and color  $\mathbf{c}_i^{(c)}$ ):
$$
\hat {\mathbf {C}} _ {i} ^ {(c)} (\mathbf {r}) = \sum_ {k = 1} ^ {K} T _ {i} ^ {(c)} \left(t _ {k}\right) \left(\alpha \left(\sigma \left(t _ {k}\right) \delta_ {k}\right) \mathbf {c} \left(t _ {k}\right) + \alpha \left(\sigma_ {i} ^ {(c)} \left(t _ {k}\right) \delta_ {k}\right) \mathbf {c} _ {i} ^ {(c)} \left(t _ {k}\right)\right), \tag {5}
$$
$$
\text {w h e r e} T _ {i} ^ {(c)} (t _ {k}) = \exp \left(- \sum_ {k ^ {\prime} = 1} ^ {k - 1} \left(\sigma \left(t _ {k ^ {\prime}}\right) + \sigma_ {i} ^ {(c)} \left(t _ {k ^ {\prime}}\right)\right) \delta_ {k ^ {\prime}}\right).
$$
The intuition is to consider easy pieces first to complete a large jigsaw puzzle and handle hard pieces later. In the initial stages of training, shared representation is predominantly learned by 'easy-posed' images. Meanwhile, candidate representation is primarily trained by hard-posed images. This remedy effectively prevents the shared representation from being distracted by the wrong supervision introduced by hard-posed images.
As the shared representation becomes accurate enough to facilitate the pose estimation of hard-pose images, the influence of the candidate head is gradually reduced so that hard-pose images can be assimilated into the shared representation, and the final model uses only the shared representation,  $\sigma$  and  $c$ . To achieve that, we propose a loss scheduling given as:
$$
\mathcal {L} = w _ {u, v} \mathcal {L} _ {\mathrm {r g b}} (\hat {\mathbf {C}} | \mathbf {C}) + (1 - w _ {u, v}) \mathcal {L} _ {\mathrm {r g b}} (\hat {\mathbf {C}} _ {i} ^ {(c)} | \mathbf {C}), \tag {6}
$$
where  $w_{u,v}$  is defined as:
$$
w _ {u, v} (l) = \left\{ \begin{array}{l l} 0 & \text {i f} l <   u \\ \frac {1 - \cos (\pi (l - u) / (v - u))}{2} & \text {i f} u \leq l <   v \\ 1 & \text {i f} l \geq v. \end{array} \right. \tag {7}
$$
Here  $l \in [0,1]$  denotes training progress and  $u, v \in [0,1]$  are hyperparameters. Then,  $w_{u,v}$  tunes the influence of  $\hat{\mathbf{C}}_i^{(c)}$  and  $\hat{\mathbf{C}}$ . Specifically, for learning during the initial stage of training  $(l < u)$ , only  $\hat{\mathbf{C}}_i^{(c)}$  is utilized, whereas only  $\hat{\mathbf{C}}$  is employed in the later stage  $(l > v)$ . With the loss scheduling, a high-quality final NeRF model can be trained. In our final training pipeline, we will use features instead of colors, as shown in Fig. 2, which will be covered in Section 3.2.
The volume rendering method shown in Eq. (5) is similar to NeRF-W [18] in that it has an additional head for color and density. In our architecture, however, we do not have a regularization term and uncertainty field that hinder the candidate head from handling hard-pose images. The size of embedding  $\ell_i^{(c)}$  is adjusted by how challenging the poses of the given images are, which can be viewed as adjusting the regularizers for 'slack' variables in the constrained optimization problem, e.g., support vector machine (SVM). The quantitative analysis of the candidate embedding size is provided in Table 4.
Figure 2: Overall training pipeline of our UP-NeRF ( $\Pi_{\theta}$  is based on Section 3.2). To learn the poses robustly, image-dependent scene representation is learned by the candidate head  $\Pi_{\theta_3}$  (Section 3.1). The depth loss  $\mathcal{L}_{\mathrm{depth}}$  provides transient-aware depth prior to the model (Section 3.4). In the early stages of training, the deep feature  $\mathbf{F}$  is used for color-independent surrogate optimization (Section 3.2). After the poses are roughly learned, the NeRF is optimized with a separated TransientNet  $\mathcal{T}_{\phi}$  to prevent unnecessary gradient flowing the pose parameters (Section 3.3).
3.2 Feature-surrogate bundle adjustment
The motivation of this section is shown in Fig. 1. Given images with diverse appearances due to different weather and times, unposed-NeRF cannot be stably trained with a photometric loss since RGB values of corresponding pixels between images are not guaranteed to be the same. We adopt the deep features from ViT [49] as a surrogate, which are less sensitive to appearance changes, and we train the neural field with these features to achieve better robustness under challenging conditions.
With a slight change in Eq. (1), our feature field is defined as:
$$
\mathbf {f} (t) = \Pi_ {\theta_ {2}} (\mathbf {z} (t)),
$$
$$
[ \mathbf {f} _ {i} ^ {(c)} (t), \sigma_ {i} ^ {(c)} (t) ] = \Pi_ {\theta_ {3}} (\mathbf {z} (t), \ell_ {i} ^ {(c)}). \tag {8}
$$
Unlike the color  $\mathbf{c}(t)$  encoding in Eq. (1), the feature  $\mathbf{f}(t)$  encoding is performed without a viewing direction because the features ought to be independent of viewing directions. Furthermore,  $\hat{\mathbf{F}}_i^{(c)}(\mathbf{r})$  is rendered similarly to Eq. (5) by volume rendering with  $\mathbf{f}(t)$ ,  $\sigma(t)$ ,  $\mathbf{f}_i^{(c)}(t)$  and  $\sigma_i^{(c)}(t)$ . It is then optimized by minimizing  $\mathcal{L}_{\mathrm{feat}}(\mathbf{r}) = \|\mathbf{F}_i(\mathbf{r}) - \hat{\mathbf{F}}_i^{(c)}(\mathbf{r})\|_2^2$  with 2D deep feature  $\mathbf{F}_i(\mathbf{r})$  from a pretrained model. We replace the loss  $\mathcal{L}_{\mathrm{rgb}}(\hat{\mathbf{C}}_i^{(c)}|\mathbf{C}_i)$  in Eq. (6) with the loss  $\mathcal{L}_{\mathrm{feat}}(\hat{\mathbf{F}}_i^{(c)}|\mathbf{F}_i)$  for the early training. Meanwhile, the unconstrained images can have different colors even though they have the same semantic object. As a result, the feature  $\mathbf{f}(t)$  can be used to encode image-dependent static color  $\mathbf{c}_i^{(a)}$  along with viewing direction  $\mathbf{d}_i$  and appearance embedding  $\{\ell_i^{(a)}\}_{i=1}^N$  for each image as:
$$
\mathbf {c} _ {i} ^ {(a)} (t) = \Pi_ {\theta_ {4}} \left(\mathbf {f} (t), \gamma_ {\mathbf {d}} (\mathbf {d}), \ell_ {i} ^ {(a)}\right). \tag {9}
$$
Note that  $(.)^{(a)}$  means it is related to the appearance embeddings. The predicted RGB value  $\hat{\mathbf{C}}_i^{(a)}(\mathbf{r})$  is calculated by volume rendering Eq. (2) with  $\mathbf{c}_i^{(a)}(t)$  and  $\sigma(t)$ . This static color  $\hat{\mathbf{C}}_i^{(a)}(\mathbf{r})$  is optimized jointly with the transient color  $\hat{\mathbf{C}}_i^{(\tau)}(\mathbf{r})$  that will be introduced in the next section, Section 3.3. In summary, through candidate scheduling Eq. (6), the feature field is trained only for the early stage  $(l < u)$ ,  $\Pi_{\theta_3}$  is not involved in learning in the later stage of training  $(l \geq v)$ , and the radiance field starts training as feature field gradually stops training  $(u \leq l < v)$ . See Fig. 2 for the final NeRF model  $\Pi_{\theta}$  architecture.
3.3 Isolated network for transient-free pose optimization
The next challenge is the occlusion of transient objects. To optimize the poses accurately, their influence must be minimized in the training process as much as possible. The previous work NeRF-W [18], which tackles this problem, models it in 3D space by adding a transient head. If we naively adopt it in our model, it causes a noisy gradient flowing into the learnable pose parameters, with the pose being learned in the wrong direction due to transient occladers. So we mitigate this problem by handling transients occladers at the separated TransientNet  $\mathcal{T}_{\phi}$  in 2D image level. It maps deep feature  $\mathbf{F}_i(\mathbf{r})$  to opacity  $\alpha^{(\tau)}$ , which is intended to filter out the transient based on semantic information. Then, along with transient embedding  $\{\ell_i^{(\tau)}\}_{i=1}^N$ , it encodes image-dependent transient color  $\hat{\mathbf{C}}_i^{(\tau)}$  and uncertainty  $\tilde{\beta}_i^{(\tau)}$ :
$$
\left[ \alpha^ {(\tau)} (\mathbf {r}), \mathbf {z} ^ {(\tau)} (\mathbf {r}) \right] = \mathcal {T} _ {\phi_ {1}} \left(\mathbf {F} _ {i} (\mathbf {r})\right),
$$
$$
\begin{array}{l} \left[ \hat {\mathbf {C}} _ {i} ^ {(\tau)} (\mathbf {r}), \tilde {\beta} _ {i} ^ {(\tau)} (\mathbf {r}) \right] = \mathcal {T} _ {\phi_ {2}} \left(\mathbf {z} ^ {(\tau)} (\mathbf {r}), \ell_ {i} ^ {(\tau)}\right). \end{array} \tag {10}
$$
Note that  $(.)^{(\tau)}$  means it is related to the transient objects. The final predicted color  $\hat{\mathbf{C}}_i(\mathbf{r})$  is obtained by alpha blending with  $\hat{\mathbf{C}}_i^{(a)}(\mathbf{r})$  from Section 3.2:
$$
\hat {\mathbf {C}} _ {i} (\mathbf {r}) = \left(1 - \alpha^ {(\tau)} (\mathbf {r})\right) \hat {\mathbf {C}} _ {i} ^ {(a)} (\mathbf {r}) + \alpha^ {(\tau)} (\mathbf {r}) \hat {\mathbf {C}} _ {i} ^ {(\tau)} (\mathbf {r}). \tag {11}
$$
The RGB loss for ray  $\mathbf{r}$  with given color  $\mathbf{C}(\mathbf{r})$  is:
$$
\mathcal {L} _ {\mathrm {r g b}} (\mathbf {r}) = \frac {\left\| \hat {\mathbf {C}} _ {i} (\mathbf {r}) - \mathbf {C} _ {i} (\mathbf {r}) \right\| _ {2} ^ {2}}{2 \beta_ {i} ^ {(\tau)} (\mathbf {r}) ^ {2}} + \frac {\log \beta_ {i} ^ {(\tau)} (\mathbf {r}) ^ {2}}{2} + \lambda_ {\alpha} \alpha^ {(\tau)} (\mathbf {r}), \tag {12}
$$
where  $\beta_{i}^{(\tau)}(\mathbf{r}) = \alpha^{(\tau)}(\mathbf{r})\tilde{\beta}_{i}^{(\tau)}(\mathbf{r}) + \beta_{\mathrm{min}}$ . As in NeRF-W, the uncertainty  $\beta_{i}^{(\tau)}(\mathbf{r})$  is modeled as the variance of an isotropic normal distribution with mean  $\hat{\mathbf{C}}_i(\mathbf{r})$ , resulting the negative log-likelihood loss of  $\mathbf{C}_i(\mathbf{r})$ . It reduces the impact of transient occluders as noise when training static objects.  $\beta_{\mathrm{min}} > 0$  is a hyperparameter for ensuring a minimum importance. The last term suppresses  $\alpha^{(\tau)}$  as possible to prevent TransientNet from rendering static objects with  $\lambda_{\alpha} > 0$ . We set  $\beta_{\mathrm{min}}$  to 0.1 and  $\lambda_{\alpha}$  to 1.0.
3.4 Transient-aware depth prior
This section is about incorporating geometric prior from monocular depth into NeRF, which is highly inspired by NoPe-NeRF [24]. The mono-depth prior provides strong geometry cues for jointly training pose and NeRF. Even though NoPe-NeRF proposes inter-frame loss to leverage a depth prior, it can only be applied to the successive images from a video so we cannot adopt such loss for the unconstrained images. So, we apply only the depth loss Eq. (13). The predicted inverse depth  $\{\mathbf{D}_i^{\mathrm{inv}}\}_{i = 1}^N$  is acquired from depth network DPT [50]. Since this is a monocular prediction, it is not an absolute inverse depth value. Therefore learnable parameters  $\{\eta_i\}_{i = 1}^N$  and  $\{\xi_i\}_{i = 1}^N$ , which denote a scale and shift factor, were introduced for each mono-depth to learn undistorted depth map  $\mathbf{D}_i^*$ . It is optimized with rendered NeRF depth  $\hat{\mathbf{D}}_i$ :
$$
\mathbf {D} _ {i} ^ {*} (\mathbf {r}) = \frac {1}{\eta_ {i} \mathbf {D} _ {i} ^ {\mathrm {i n v}} (\mathbf {r}) + \xi_ {i}}, \quad \hat {\mathbf {D}} _ {i} (\mathbf {r}) = \sum_ {k = 1} ^ {K} T (t _ {k}) \alpha (\sigma (t _ {k}) \delta_ {k}) t _ {k}. \tag {13}
$$