SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yixing Lao
yxiao@cs.hku.hk
Xiaogang Xu
Zhejiang Lab, Zhejiang University
xgxu@zhejianglab.com
Zhipeng Cai
zhipeng.cai@intel.com
Xihui Liu
xihuiliu@eee.hku.hk
Hengshuang Zhao*
hszhao@cs.hku.hk
Neural Radiance Fields (NeRFs) have achieved impressive results in novel view synthesis and surface reconstruction tasks. However, their performance suffers under challenging scenarios with sparse input views. We present CorresNeRF, a novel method that leverages image correspondence priors computed by off-the-shelf methods to supervise NeRF training. We design adaptive processes for augmentation and filtering to generate dense and high-quality correspondences. The correspondences are then used to regularize NeRF training via the correspondence pixel reprojection and depth loss terms. We evaluate our methods on novel view synthesis and surface reconstruction tasks with density-based and SDF-based NeRF models on different datasets. Our method outperforms previous methods in both photometric and geometric metrics. We show that this simple yet effective technique of using correspondence priors can be applied as a plug-and-play module across different NeRF variants. The project page is at https://yxlao.github.io/corres-nerf/.
Building on coordinate-based implicit representations [1, 2, 3], Neural Radiance Field (NeRF) [4] has achieved great success in solving the fundamental computer vision problem of reconstructing 3D geometries from RGB images, benefiting various downstream applications. However, training such implicit representations typically requires a large number of input views, especially for objects with complex shapes, which can be costly to collect. Therefore, training NeRFs with sparse input RGB views remains a challenging yet important problem, where solving it can benefit various real-world applications, e.g., 3D portrait reconstruction in the monitoring system [5, 6, 7], the city digital reconstruction [8, 9, 10], etc.
Several works [11, 12, 13, 14, 15, 16] have been proposed to address this problem by optimizing the rendering process or adding training constraints. However, these methods may suffer from poor real-world performance [12], since only sparse 2D input views are an under-constrained problem [17, 13], and the training process is prone to overfitting on the limited input views. Recent works have proposed to utilize extra priors to supervise NeRF training [14, 16, 18]. For example, some work proposed to train a separate network to compute depth priors [14]. However, current priors are not robust enough against the sparse property of the target scene, e.g., DS-NeRF [13] relies on running external SfM module [19] which may not yield sufficient point clouds for supervision, or not have an absolute scale and shift from the monocular depth estimation [16].
Input A
Corres(A,B)
UNISURF Render
NeuS Render
UNISURF Mesh
NeuS Mesh
Input B
Input C
Corres(C, B)
Corres(C, A)
CorresNeRF Render
Figure 1: Novel view synthesis and surface reconstruction from sparse inputs with image correspondence priors. Given a sparse set of input images (column 1), our method leverages the image correspondence priors computed from pre-trained models (column 2) to supervise NeRF training. The color of the highlighted pixels represents the confidence of the correspondence, where higher confidence values are greener. With image correspondence supervision, we achieve much higher quality novel view synthesis (column 3) and surface reconstruction (column 4) compared to the baseline UNISURF [20] and NeuS [21] models. The correspondence priors can be used in both density-based and SDF-based neural implicit representations on both novel-view synthesis and surface reconstruction tasks. Best viewed in color.
CorresNeRF Mesh
Our key observation is that image correspondences can provide strong supervision signals for the training of NeRF. In Figure 2, we visualize the point cloud created from triangulating the image correspondences pre-processed by our automatic augmentation and filtering (Section 3.2). Without training any NeRF networks, this triangulated point cloud clearly captures rich geometrical information about the target scene, which suggests that the image correspondences, along with the given camera parameters, can provide strong supervision signals for NeRF Training. The idea of using image correspondences as priors is widely applicable, since the correspondences can be estimated as long as there are sufficient textured overlapping regions among the input views, regardless of the NeRF variants. Additionally, the acquisition of image correspondence is inexpensive, as they can be directly computed using pre-trained off-the-shelf methods [22, 23, 24].
To improve the robustness of our method, we propose an automatic augmentation and outlier filtering process for the correspondences, ensuring their quantity and quality. Ablation studies demonstrate the effectiveness of our proposed augmentation and filtering strategy. To incorporate the correspondences into the NeRF training, we design novel correspondence loss terms, including pixel reprojection loss and depth loss. The reprojection loss is designed to constrain the distances between the reprojected corresponding points in the 2D pixel coordinates and the pixel-level distances. The depth loss is designed to constrain the relative depth differences between the corresponding points. Moreover, the confidence values from correspondence estimation are adopted as loss weights to avoid the negative impact of mismatching correspondences.
We evaluate our method on novel view synthesis and surface reconstruction tasks on LLFF [25] and DTU [26] datasets. We observe that the combination of correspondence priors via our method leads to significantly improved performance in novel-view synthesis (e.g., the PSNR of NeRF baseline [4] on LLFF is improved more than 3dB, SSIM is enhanced by  $14\%$ ) and surface reconstruction (the DepthMAE of NeRF baseline on LLFF is decreased from 1.66 to 0.91, and Chamfer- $L_{1}$  distance is reduced from 6.16 to 2.63 for NeuS [21] on DTU). Furthermore, we benchmark our method against other state-of-the-art sparse-view reconstruction methods on different datasets, and show that our approach (built based on the simple typical NeRF, i.e., vanilla NeRF [4] for view synthesis and NeuS [21] for surface reconstruction) outperforms the comparative methods in terms of photometric and geometrical metrics.
Points Viewed at A
Input A
Input B
Corres(B, C)
Points Viewed at B
Input C
Figure 2: Reconstructing point cloud by triangulating dense image correspondences. Column 1 shows the input images. Column 2 shows the correspondences between image pairs. With these correspondences and camera parameters, one can directly reconstruct 3D points without any training. Column 3 shows the reconstructed point cloud visualized using the same input cameras. The rightmost column shows the points rendered in novel view, as well as the camera poses. This example showcases the strong supervision signals that correspondences can bring to NeRF training. Best viewed in color.
Corres(A, C)
Points Viewed at C
Points from Corres
- We propose to use image correspondence as priors to supervise NeRF training, leading to better performance under challenging sparse-view scenarios.
- We design an adaptive pipeline to automatically augment and filter image correspondences, ensuring their quantity and quality.
- We propose robust correspondence loss, including pixel reprojection loss and depth loss based on correspondence priors.
- Extensive experiments are conducted on different baselines and datasets, showing the effectiveness of our method under different choices of neural implicit representations.
Nerual implicit representations. Unlike traditional explicit 3D representations such as point cloud, mesh, or voxels, neural implicit representations [2, 1, 3] define a 3D scene with an implicit function that maps 3D coordinates and view directions to the corresponding properties (e.g., color and density) or features. The implicit function is typically modeled with neural networks. These types of representations are more compact and flexible towards different scenes, and they have been successfully applied to various tasks such as 3D reconstruction [4] and novel view synthesis [21].
Sparse-view NeRFs. Due to the practical limitation and the high cost of data collection, the data used for neural implicit representation learning is often sparse. To address this problem, researchers have recently proposed to incorporate geometric [27, 14, 13, 15, 28, 29, 16, 30] or pre-trained priors [12, 11, 31, 32, 33] to supervise the training of neural implicit representations. As a fundamental property of a scene, depth priors are commonly used, including depth generated from an SfM system [13], depth computed via monocular depth estimation [16], and depth predicted from a depth completion network [27, 14]. In contrast to these works, we rely on a more robust and flexible low-level feature, i.e., image correspondences, as supervision. The image correspondences obtained from modern image matchers [24, 23] are typically denser than the depth maps generated from traditional SfM system [19], when given sparse input views, since the matching is completed according to various characteristics in addition to geometrical information. With the known camera parameters, one can also compute absolute depth from the image correspondences, avoiding the ambiguity of shift and scale of depth maps from monocular depth estimation methods [16]. ConsistentNeRF [34] uses the depths from pre-trained MVSNeRF [31] to derive the correspondence mask to emphasize the multi-view appearance consistency, while our method uses the correspondences computed from off-the-shelf image matchers to regularize geometric properties of the scene. Concurrent work
SPARF [35] uses image correspondences to refine noisy poses from sparse input views with a 2D pixel-space correspondence loss, while our method uses both a 2D pixel-space loss and a 3D depth-space loss. Our depth-space loss is normalized to compute the relative depth differences, adapting to different scene scalings and geometry properties.
Geometric matching. The task of finding pixel-level correspondences between two views of a 3D scene, known as geometric matching, is a fundamental computer vision problem. Early works of geometric matching are based on matching measurement via hand-crafted local features [36, 37, 38]. Later, detectors and feature descriptors learned through data-driven processes [39, 40, 41, 22] were proposed to substitute the hand-crafted features, surpassing their performance. Recently, detector free methods [23], transformer-based [42, 43], and dense geometric matching [44, 45, 24] have been proposed to further improve the performance. In this work, we use image correspondences generated by the state-of-the-art dense image matcher [24] as supervision for neural implicit representations.
3.1 Background of Neural Radiance Fields
For a given 3D point  $\mathbf{x} \in \mathbb{R}^3$  and a viewing direction  $\mathbf{d} \in \mathbb{R}^3$ , a neural radiance field [4] predicts the corresponding density  $\sigma \in [0, \infty)$  and RGB color  $\mathbf{c} \in [0, 1]^3$ , modeled by an MLP network, as
$$
f _ {\theta}: (\gamma (\mathbf {x}), \gamma (\mathbf {d})) \mapsto (\mathbf {c}, \sigma), \tag {1}
$$
where  $\gamma$  is the positional encoding function.
A ray is defined as  $\mathbf{r}(t) = \mathbf{o} + t\mathbf{d}, t \in [t_n, t_f]$ , where  $\mathbf{o}$  is the camera center and  $\mathbf{d}$  is the ray direction,  $t_n$  is the near bound and  $t_f$  is the far bound. To render the ray  $\mathbf{r}$  with a pre-defined  $t_n$  and  $t_f$ , we integrate the density  $\sigma$  and color  $\mathbf{c}$  along the ray, as
$$
\hat {\mathbf {c}} _ {\theta} (\mathbf {r}) = \int_ {t _ {n}} ^ {t _ {f}} T (t) \sigma_ {\theta} (\mathbf {r} (t)) \mathbf {c} _ {\theta} (\mathbf {r} (t), \mathbf {d}) d t, \quad T (t) = \exp \left(- \int_ {t _ {n}} ^ {t} \sigma_ {\theta} (\mathbf {r} (t)) d t\right), \tag {2}
$$
where  $T(t)$  is the accumulated transmittance,  $\mathbf{c}_{\theta}(\mathbf{r}(t), \mathbf{d})$  and  $\sigma_{\theta}(\mathbf{r}(t))$  are the predicted color and density output from  $f_{\theta}$ , respectively. The rendering is implemented via stratified sampling approach, where  $M$  points are sampled in  $[t_n, t_f]$ , as  $\{x_1, \dots, x_M\}$ . The density and color can be obtained as
$$
\hat {\mathbf {c}} _ {\theta} (\mathbf {r}) = \sum_ {i = 1} ^ {M} T _ {i} \left(1 - \exp \left(- \sigma_ {\theta} \left(x _ {i}\right) \delta_ {i}\right)\right) \mathbf {c} _ {\theta} \left(x _ {i}, \mathbf {d}\right), \quad T _ {i} = \exp \left(- \sum_ {j = 1} ^ {i - 1} \sigma_ {\theta} \left(x _ {j}\right) \delta_ {j}\right), \tag {3}
$$
where  $\delta_{j} = t_{j + 1} - t_{j}$  is the distance between adjacent samples. Specifically, for a ray  $r$ , its predicted 3D point can be obtained by summing up the weighted depth values along the ray, as
$$
\mathbf {y} = \mathbf {o} + \left(\sum_ {i = 1} ^ {M} T _ {i} \left(1 - \exp \left(- \sigma_ {\theta} \left(x _ {i}\right) \delta_ {i}\right)\right) t _ {i}\right) \mathbf {d}. \tag {4}
$$
To optimize parameter  $\theta$  in the NeRF model, a set of input images and camera parameters are provided, and the mean squared error color loss is minimized for optimization, as
$$
\mathcal {L} _ {\text {c o l o r}} (\theta , \mathcal {R}) = \mathbb {E} _ {\mathbf {r} \in \mathcal {R}} \| \hat {\mathbf {c}} _ {\theta} (\mathbf {r}) - \mathbf {c} (\mathbf {r}) \| _ {2} ^ {2}, \tag {5}
$$
where  $\mathcal{R}$  is the set of rays in the training views, and  $\mathbf{c}(\mathbf{r})$  is the ground-truth color of the ray  $\mathbf{r}$ .
3.2 Generating Correspondences
In this paper, we focus on how to utilize the computed image correspondences to enhance the performance of neural implicit representations in NeRF. Thus, the quality of correspondence is crucial. For each pair of images in the training views, we compute the correspondences using an off-the-shelf SOTA pre-trained image-matching model. In particular, DKMv3 [24] is used because it provides dense matching results, which is suitable for our use case. To improve generalization ability, we fuse the predictions of the indoor and outdoor models, which are pre-trained on ScanNet [46] and MegaDepth [47] respectively. To further enhance the reliability of the correspondences, we propose
Figure 3: Correspondence generation process. The left column illustrates correspondence propagation in a connected component in the correspondence graph  $\mathcal{G}$ . Given correspondence pairs  $(\mathbf{r}_q, \mathbf{r}_m)$  and  $(\mathbf{r}_m, \mathbf{r}_s)$  with confidence  $\alpha_{q,m}$  and  $\alpha_{m,r}$ , we assign correspondence relationship to pair  $(\mathbf{r}_q, \mathbf{r}_s)$  with  $\alpha_{q,s} = \alpha_{q,m} \alpha_{m,s}$ . The right columns show the overall correspondence generation process: (A) input images; (B) correspondence pairs generated by the vanilla matcher; (C) correspondence pairs augmented by image transformations and correspondence propagation; and (D) correspondence pairs after outlier removal. Best viewed in color.
to utilize the correspondence confidence, and also design the automatic and adaptive correspondence process algorithm, increasing the convincing correspondences and removing the outliers.
**Confidence.** For ray  $\mathbf{r}_q$  in training rays  $\mathcal{R}$ , the image matcher computes a set of corresponding rays  $\mathcal{C}(\mathbf{r}_q)$ . For each pair of correspondences  $\mathbf{r}_q$  (query) and  $\mathbf{r}_s$  (support) in  $\{(\mathbf{r}_q, \mathbf{r}_s) \mid \mathbf{r}_q \in \mathcal{R}, \mathbf{r}_s \in \mathcal{C}(\mathbf{r}_q)\}$ , a confidence value  $\alpha_{q,s} \in [0.5, 1]$  is also predicted by DKMv3. Note that a ray  $\mathbf{r}_q$  can have 0, 1, or more corresponding rays  $\mathbf{r}_s$  in different images. Confidence scores are used as the scaling factors for the correspondence losses, which will be described in Sec. 3.3.
Augmentation. To increase the number of correspondences, we perform augmentations for correspondences. The first type of augmentation is image transformations, including flipping, swapping query and support images, and scaling. These image transformations can effectively increase the density of the predicted correspondences since the image transformations can provide various context conditions to generate correspondences. The second type of augmentation propagates correspondences across image pairs, effectively increasing the area coverage of the correspondences. We build an undirected graph  $\mathcal{G} = (\mathcal{V},\mathcal{E})$ , with vertices  $\mathcal{V} = \{\mathbf{r}\mid \mathbf{r}\in \mathcal{R}\}$ , and edges  $\mathcal{E} = \{(\mathbf{r}_q,\mathbf{r}_s)\mid \mathbf{r}_s\in \mathcal{C}(\mathbf{r}_q)\}$ . For each edge  $(\mathbf{r}_q,\mathbf{r}_s)$ , a confidence value  $\alpha_{q,s}$  is assigned. We then propagate the correspondence relationship to vertex pairs within each connected component in  $\mathcal{G}$ . In particular, let  $\mathbf{r}_q$  and  $\mathbf{r}_s$  be two vertices with distance  $d$ , where  $d = 2$ . We assign a correspondence relationship between  $\mathbf{r}_q$  and  $\mathbf{r}_s$  with confidence  $\alpha_{q,s} = \alpha_{q,1}\alpha_{1,2}\dots \alpha_{d - 1,s}$ . In practice, we cap the propagation distance  $d\leq d_{\mathrm{max}}$ , where we use  $d_{\mathrm{max}} = 2$  in our experiments. Figure 3 (B) and (C) show the original and augmented correspondences, respectively.
Outlier filtering. To increase the quality of the correspondences to guide the supervision, we remove outliers after calculating and enhancing the correspondences. First, we remove outliers according to the projected ray distance between corresponding points. Suppose  $\mathbf{p}_q$  and  $\mathbf{p}_s$  be a pair of 2D correspondence in  $I_q$  to  $I_s$ ,  $\pi_q$  and  $\pi_s$  be the world-to-pixel projection for  $I_q$  and  $I_s$ , respectively. Given a pair of correspondences and camera parameters, we compute the closest 3D points  $\mathbf{x}_q$  and  $\mathbf{x}_s$  along the two rays shot from the camera centers through the two correspondences. Then, we project these two 3D points to the correspondence's image plane. The projected ray distance [48] is defined as the averaged Euclidean distance between the projected points and the correspondences:
$$
d _ {\text {p r o j}} = \frac {\left\| \pi_ {q} \left(\mathbf {x} _ {s}\right) - \mathbf {p} _ {q} \right\| _ {2} + \left\| \pi_ {s} \left(\mathbf {x} _ {q}\right) - \mathbf {p} _ {s} \right\| _ {2}}{2}. \tag {6}
$$
We remove the correspondences with projected ray distance  $d_{\mathrm{proj}}$  larger than a threshold. Second, we remove outliers by checking if a point is statistically far from its neighbors. For each pair of correspondences, two 3D points  $\mathbf{x}_q$  and  $\mathbf{x}_s$  can be obtained, which has been indicated in the last paragraph. We then consider  $\frac{1}{2} (\mathbf{x}_q + \mathbf{x}_s)$  to be the 3D point of the correspondence. We do this for all correspondence pairs to obtain a set of 3D points  $\mathcal{P}$ . For each 3D point in  $\mathcal{P}$ , we compute the average distance to its  $k$  nearest neighbors and remove the point (as well as its matched correspondence pair) if the distance is larger than a threshold. This threshold is determined by the standard deviation of all points' average distances in  $\mathcal{P}$ .
Experiments in Sec. 4.5 demonstrate the effects of our proposed augmentation and removal strategy since they can increase the reliability of the correspondences utilized in NeRF's training.
3.3 Correspondence Loss
We incorporate the obtained correspondences into NeRF training from two aspects: correspondence pixel reprojection loss and depth loss, as illustrated in Fig. 4.
Correspondence pixel reprojection loss. Let  $\mathbf{y}_q$  and  $\mathbf{y}_s$  be the corresponding 3D points predicted via NeRF, by accumulating the weighted sum of z-depth values to the ray origin according to Eq.4. We compute the pixel-space reprojection loss as follows:
Figure 4: Correspondence loss of a ray pair  $(\mathbf{r}_q, \mathbf{r}_s)$ .  $\mathbf{o}_q$  and  $\mathbf{o}_s$  are the ray origins.  $\mathbf{x}_q$  and  $\mathbf{x}_s$  are the closest points along the two rays.  $\mathbf{y}_q$  and  $\mathbf{y}_s$  are the 3D points predicted by the NeRF model.  $\pi_q(\mathbf{y}_s)$  and  $\pi_s(\mathbf{y}_q)$  are the 2D projections of  $\mathbf{y}_s$  and  $\mathbf{y}_q$  on the other image. The correspondence losses are defined with the geometric properties as illustrated.
$$
\mathcal {L} _ {\mathrm {p i x e l}} (\theta , \mathcal {R}) = \mathbb {E} _ {\mathbf {r} _ {q} \in \mathcal {R}, \mathbf {r} _ {s} \in \mathcal {C} (\mathbf {r} _ {q})} \alpha_ {q, s} \left(\| \pi_ {q} (\mathbf {y} _ {s}) - \mathbf {p} _ {q} \| _ {2} + \| \pi_ {s} (\mathbf {y} _ {q}) - \mathbf {p} _ {s} \| _ {2}\right). \tag {7}
$$
The pixel-space reprojection loss is similar to the projected ray distance [48]. The difference is that we reproject the predicted 3D points  $\mathbf{y}_q$  and  $\mathbf{y}_s$ , instead of reproducing the closest points along the two rays. Moreover, we utilize the correspondence score  $\alpha_{q,s}$  as weights to provide a more reliable metric. Since the camera parameters are known, the projected ray distance is constant. However, the reprojection of the predicted 3D points can provide a supervision signal to guide the NeRF model.
Correspondence depth loss. The correspondence depth loss term penalizes the difference between the predicted depth and the ground-truth depth. The ground-truth depth is estimated by computing the closest 3D points  $\mathbf{x}_q$  and  $\mathbf{x}_s$  along the two rays shot from the camera centers  $\mathbf{o}_q$  and  $\mathbf{o}_s$  through the two correspondences. We use a relative depth loss, which is adaptive to different scalings and can be described with the following equation:
$$
\mathcal {L} _ {\mathrm {d e p t h}} (\theta , \mathcal {R}) = \mathbb {E} _ {\mathbf {r} _ {q} \in \mathcal {R}, \mathbf {r} _ {s} \in \mathcal {C} (\mathbf {r} _ {q})} \alpha_ {q, s} \left(\left| \frac {\| \mathbf {y} _ {q} - \mathbf {o} _ {q} \| _ {2}}{\| \mathbf {x} _ {q} - \mathbf {o} _ {q} \| _ {2}} - 1 \right| + \left| \frac {\| \mathbf {y} _ {s} - \mathbf {o} _ {s} \| _ {2}}{\| \mathbf {x} _ {s} - \mathbf {o} _ {s} \| _ {2}} - 1 \right|\right), \tag {8}
$$
where  $\alpha_{q,s}$  is the confidence of correspondence to regular the effects of the relative depth loss. In the ideal scenario,  $\mathbf{y}_q$  should overlap with  $\mathbf{x}_q$  and similarly  $\mathbf{y}_s$  should overlap with  $\mathbf{x}_s$ , so the loss will be zero. The relative depth loss is a smooth function and is minimized when the predicted depth is close to the estimated depth of the ground truth.
Training procedure. For each ray in a batch of rays  $\mathcal{R}_{\mathrm{batch}}$ , we query the correspondence map to find the corresponding rays in all training rays  $\mathcal{R}$ . We assume that each ray may have  $B$  corresponding rays,  $B \in [0, +\infty)$ . For each pair of corresponding rays, we run the forward pass of the NeRF model to obtain the predicted 3D points. We then compute the pixel-space reprojection loss and the correspondence depth loss for each pair of corresponding rays. The final loss is the sum of pixel-space correspondence reprojection loss, correspondence depth loss, and the regular NeRF color loss  $\mathcal{L}_{\mathrm{color}}$ :
$$
\mathcal {L} = \mathcal {L} _ {\text {c o l o r}} + \lambda_ {1} \mathcal {L} _ {\text {p i x e l}} + \lambda_ {2} \mathcal {L} _ {\text {d e p t h}}, \tag {9}
$$
where  $\lambda_{1}$  and  $\lambda_{2}$  are the loss weights, and we set them to 0.1 in our experiments.
We compare novel view synthesis results on LLFF [25] with density-field-based NeRF models. We follow the convention to use every 8th image as test images [4], while selecting the training views uniformly from the rest of the images [29]. The selected training views and test views are the same across all methods. Three input views are used for training.