CorresNeRF: Image Correspondence Priors for Neural Radiance Fields
Authors: Yixing Lao
Xiaogang Xu
zhipeng cai
Xihui Liu
Hengshuang Zhao
Venue: NeurIPS 2023 poster (2023)
Keywords: Neural Radiance Fields
3D Reconstruction
Few View

Abstract: Neural Radiance Fields (NeRFs) have achieved impressive results in novel view synthesis and surface reconstruction tasks. However, their performance suffers under challenging scenarios with sparse input views. We present CorresNeRF, a novel method that leverages image correspondence priors computed by off-the-shelf methods to supervise NeRF training. We design adaptive processes for augmentation and filtering to generate dense and high-quality correspondences. The correspondences are then used to regularize NeRF training via the correspondence pixel reprojection and depth loss terms. We evaluate our methods on novel view synthesis and surface reconstruction tasks with density-based and SDF-based NeRF models on different datasets. Our method outperforms previous methods in both photometric and geometric metrics. We show that this simple yet effective technique of using correspondence priors can be applied as a plug-and-play module across different NeRF variants. The project page is at https://yxlao.github.io/corres-nerf/.

CorresNeRF: Image Correspondence Priors for Neural Radiance Fields
Yixing Lao The University of Hong Kong AelaaBesshku.d.e
Xiaogang Xu Zhejiang Lab, Zhejiang University xgxu6 zhe anglab com
Zhipeng Cai Intel Labs
Xihui Liu
Hengshuang Zhao" The University of Hong Kong hszhaolcs. hku.hk
The University of Hong Kong xihuiliuêeee hku.hk
zhipeng. caie intel.com
Abstract
Neural Radiance Fields (NeRFs) have achieved impressive results in novel view synthesis surface reconstruction tasks. However, their performance suffers under challenging scenarios with sparse input views. We present CorresNeRF, novel method that leverages image correspondence priors computed by off-the- shelf methods to supervise NeRF training. We design adaptive processes for augmentation and filtering to generate dense and high-quality correspondences. then used to regularize NeRF training via the corre- spondence pixel reprojection and depth loss terms. We evaluate our methods novel view synthesis and surface reconstruction tasks with density-based and SDF-based NeRF models on different datasets. Our method outperforms pre- vious methods in both photometric and geometric metrics. We show that this simple yet effective technique of using correspondence priors can be applied as a plug-and-play module across different NeRF variants. The project page is https:/ petpps://yllee /ywlao...oo//o///.oo//////////l/ gi github.... io/ corres nerf/.
The correspondences
Introduction
Building coordinate-basee implicit representations [1, 2, 3J,, Neural Radiance Field (NeRF) [4] has achieved great success in solving the fundamental computer vision problem of reconstructing 3D geometries from RGB images, benefiting various downstream applications. However, training such implicit representations typically requires large number of input views, especially for objects with complex shapes, which can be costly collect. Therefore, training NeRFs with sparse input RGB views remains challenging yet important problem, where solving it can benefit various real-world applications, e.g., 3D portrait reconstruction in the monitoring system [5, 6, 7]. the city digital reconstruction [8, 9, 10], etc.
Several works [11, 12, 13, 14, 15, 16] have been proposed address this problem by optimizing the rendering process or adding training constraints. However, these methods may suffer from poor real- world performance [12], since only sparse 2D input views
under-constrained problem [17, 13), and the training process is prone to overfitting on the limited input views. Recent works have proposed utilize extra priors supervise NeRF training [14, 16, 18]. For example, some work proposed to train a separate network to compute depth priors [14]. However, current priors are not robust enough against the sparse property of the target scene, e.g.. DS-NeRF [13] relies on running external SfM module [19] which may not yield sufficient point clouds for supervision, or not have an absolute scale and shift from the monocular depth estimation [16].
"Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Input A
Corres(A, B)
UNISURF Render
UNISURF Mesh
Input B
(Cr B)
NeuS Render
NeuS Mesh
Input C
Corres(C, A)
CorresNeRF Render
CorresNeRF Mesh
Figure I: Novel view synthesis and surface reconstruction from sparse imputs with image correspondence priors. Given a sparse set of input images (column 1), our method leverages the image correspondence priors computed from pre-trained models (column 2) to supervise NeRF training. The color of the highlighted pixels represents the confidence of the correspondence, where higher confidence values are greener. With image correspondence supervision. we achieve much higher quality novel view synthesis (column 3) and surface reconstruction (column 4) compared to the baseline UNISURF [20] and NeuS [21] models. The correspondence priors can be used in both density- -bssed and SDF-based neural implicit representations on both novel-view synthesis and surface reconstruction tasks, Best viewed in color.
Our key observation is that image correspondences can provide strong supervision signals for the training of NeRF, In Figure 2, we visualize the point cloud created from triangulating the image correspondences pre- pre-processed by our automatic augmentation and filtering (Section 3.2). Without training any NeRF networks, this triangulated point cloud clearly captures rich geometrical information about the target scene, which suggests that the image correspondences, along with the given camera parameters, can provide strong supervision signals for NeRF Training. The idea of using image correspondences as priors is widely applicable, since the correspondences can be estimated as long as there are sufficient textured overlapping regions among the input views, regardless of the NeRF variants. Additionally, the acquisition of image correspondence is inexpensive, as they can be directly computed using pre-trained off-the-shelf methods [22, 23, 24).
To improve the robustness of our method, we propose an automatic augmentation and outlier filtering process for the correspondences, ensuring their quantity and quality. Ablation studies demonstrate the effectiveness of our proposed augmentation and filtering strategy. To incorporate the correspondences into the NeRF training, we design novel correspondence loss terms, including pixel reprojection loss and depth loss. The reprojection loss is designed to constrain the distances between the reprojected corresponding points the 2D pixel coordinates and the pixel-level distances. The depth loss is designed to constrain the relative depth differences between the corresponding points. Moreover, the confidence values from correspondence estimation are adopted as loss weights to avoid the negative impact of mismatching correspondences.
We evaluate our method on novel view synthesis and surface reconstruction tasks on LLFF [25] and DTU [26] datasets. We observe that the combination of correspondence priors via our method leads significantly improved performance in novel-view synthesis (e.g., the PSNR of NeRF baseline [4] on LLFF is improved more than 3dB, SSIM is enhanced by 14%) and surface reconstruction (the DepthMAE of NeRF baseline on LLFF is decreased from 1.66 to 0.91, and Chamfer- distance is reduced from 6. 16 to 2.63 for NeuS [21] on DTU). Furthermore, we benchmark our method against sparse-view reconstruction methods on different datasets, and show that our approach (built based on the simple typical NeRF, i.e., vanilla NeRF [4] for view synthesis and NeuS [21] for surface reconstruction) outperforms the comparative methods terms of photometric and geometrical metrics.
other state-of-the-a

---

Input A
Corres(A, B)
Points Viewed at
Input B
Corres(B, C)
Points Viewed at B
Input C
Corres(A, C)
Points Viewed at
Points from Corres
Figure 2: Reconstructing point cloud by triangulating dense image correspondences. Column shows the input images. Column shows the correspondences between image pairs. With these correspondences and camera parameters, one can directly reconstruct 3D points without any training. Column 3 shows the reconstructed point cloud visualized using the same input cameras. The rightmost column shows the points rendered novel view, as well as camera poses. This example showcases the strong supervision signals that correspondences can bring to NeRF training. Best viewed in color.
We summarize our contributions below:
We propose to use image correspondence as priors to supervise NeRF training, leading to better performance under challenging sparse-view scenarios.
We design adaptive pipeline to automatically augment and filter image correspondences, ensuring their quantity and quality.
We propose robust correspondence loss, including pixel reprojection loss and depth loss based on correspondence priors.
Extensive experiments conducted on different baselines and datasets, showing the effectiveness of our method under different choices of neural implicit representations.
Related Work
Nerual implicit representations. Unlike traditional explicit 3D representations such as point cloud, mesh or voxels, neural implicit representations [2, 1, 3] define a 3D scene with an implicit function that maps 3D coordinates and view directions to the corresponding properties (e.g., color and density) or features. The implicit function is typically modeled with neural networks. These types of representations are more compact and flexible towards different scenes, and they have been successfully applied to various tasks such as 3D reconstruction
[4] and novel view synthesis [21].
Sparse-view NeRFs. Due to the practical limitation and the high cost of data collection, the data used for neural implicit representation learning is often sparse. To address this problem, researchers have recently proposed to incorporate geometric [27, 14, 13, 15, 28, 29, 16, 30] pre-trained priors [12, 11, 31, 32, 33] to supervise the training of neural implicit representations. As a fundamental property of scene, depth priors are commonly used, including depth generated from SfM system [13], depth computed via monocular depth estimation [16], and depth predicted from a depth completion network [27, 14]. contrast to these works, we rely on more robust and flexible low- level feature, i.c., image correspondences, as supervision. The image correspondences obtained from modern image matchers [24, 23] are typically denser than the depth maps generated from traditional SfM system [19], when given sparse input views, since the matching is completed according to various characteristics in addition to geometrical information. With the known camera parameters, one can also compute absolute depth from the image correspondences, avoiding the ambiguity of shift and scale of depth maps from monocular depth estimation methods [16]. ConsistentNeRF [34] uses the depths from pre-trained MVSNeRF [31] derive the correspondence mask to emphasize the multi-view appearance consistency, while our method uses the correspondences computed from off-the-shelf image matchers to regularize geometric properties of the scene. Concurrent work
3

---

SPARF [35] uses image correspondences to refine noisy poses from sparse input views with a 2D pixel-space correspondence loss, while our method uses both 2D pixel-space loss and 3D depth- space loss. Our depth-space loss is normalized to compute the relative depth differences, adapting to different scene scalings and geometry properties.
Geometric matching. The task of finding pixel-level correspondences between two views of a 3D scene, known geometric matching, is fundamental computer vision problem. Early works of geometric matching are based on matching measurement via hand-crafted local features [36, 37, 38]. Later, detectors and feature descriptors learned through data-driven processes [39, 40, 41, 22] were proposed to substitute the hand-crafted features, surpassing their performance. Recently, detector free methods [23], transformer-basee [42, 43], and dense geometric matching [44, 45, 24] have been proposed to further improve the performance. In this work, we use image correspondences generated by the state-of-the te-thee-rt dense image matcher [24] as supervision for neural implicit representations.
3 Method
3.1 Background of Neural Radiance Fields
For given 3D point IR' and viewing direction d E 12 neural radiance field [4] predicts the corresponding density E [0, oo) and RGB color с E [0, 1] modeled by an MLP network, as
Jo (h(x), + (c,o),
where y is the positional encoding function.
A ray is defined r(t) = o + id,t t E [y), where o is the camera center and d is the ray direction, tn is the near bound and ty is the far bound. To render the ray r with a pre-defined in and ty, we integrate the density and color along the ray, as
co(r) = S["" T())o(((()))otr(((...
T(t) = exp (-f,, oo(r(t))dt)
(2)
where T(t) is the accumulated transmittance, (r(). d) and (r(r(t) are the predicted color and density output from fe. respectively. The rendering is implemented via stratified sampling approach, where M points are sampled in [L,, (y]. as (aI,..,, FMr. The density and color can be obtained as
(r)
Σ T1(1 (x))-ccp(-(((,,))))))o d), T. exp(- Σ (o)()))0
(3)
where 6j = 1-- 12 is the distance between adjacent samples. Specifically, for ray r, its predicted 3D point can be obtained by summing up the weighted depth values along the ray,
Σ TN(1 - (000-ex(-o0(xx))))) d.
y +
To optimize parameter in the NeRF model, a set of input images and camera parameters are provided, and the mean squared error color loss is minimized for optimization, as
Leoker (0, R) = FTER [r²)(r) - c(r)1l2
(5)
where the set of rays in the training views, and c(r) is the ground-trit truth color of the ray
3.2 Generating Correspondences
In this paper, we focus on how to utilize the computed image correspondences to enhance the performance of neural implicit representations in NeRF. Thus, the quality of correspondence is crucial. For each pair of images in the training views, we compute the correspondences using an off-the-shelf SOTA pre-trained image-matching model. In particular, DKMv3 [24] is used because provides dense matching results, which is suitable for our use case. To improve generalization ability, we fuse the predictions of the indoor and outdoor models, which are pre-trained on ScanNet [46] and MegaDepth [47] respectively. To further enhance the reliability of the correspondences, propose

---

Corres Propagation
(A) Input
(B) Corres
(C) Augmented
(D) Filtered
Figure 3: Correspondence generation process. The left column illustrates correspondence propagation in a connected component in the correspondence graph G. Given correspondence pairs (rg. Fm) and (Tm, rx) with confidence Qq.m and mm we assign correspondence relationship to pair (Fq: with Qq.: = Qq.m Omm.s. The right columns show the overall correspondence generation process: (A) input images; (B) correspondence pairs generated by the vanilla matcher, (C) correspondence pairs augmented by image transformations and correspondence propagation: and (D) correspondence pairs after outlier removal. Best viewed in color.
utilize the correspondence confidence, and also design the automatic and adaptive correspondence process algorithm, increasing the convincing correspondences and removing the outliers.
Confidence. For ray ry in training rays R. the image matcher computes
of corresponding
rays C(r,)- For each pair of correspondences r, (query) and r, (support) in {(r,.. r.) rr E R, r, E C(r,),. confidence value d' [0.5, 1] is also predicted by DKMv3. Note that a ray rg can have 0, I, or more corresponding rays r, in different images. Confidence scores are used as the scaling factors for the correspondence losses, which will be described in Sec. 3.3.
Augmentation. To increase the number of correspondences, we perform augmentations for corre- spondences. The first type of augmentation is image transformations, including flipping, swapping query and support images, and scaling. These image transformations can effectively increase the den- sity of the predicted correspondences since the image transformations ditions to generate correspondences. The second type of augmentation propagates correspondences across image pairs, effectively increasing the undirected graph g = EV,&), with vertices V = {r E R.,, and edges & = {(r,, r.) r. E C(r,))) For each edge (ry rs), confidence value 0g is assigned. We then propagate the correspondence relationship to vertex pairs within each connected component in G. In particular, let r and r, be r') connecting them. We assign a correspondence relationship between r and r, with confidence agg. = 1,,1 Odd-1,a. In practice, we cap the propagation distance S dmax, where we use dmax in our experiments. Figure (B) and (C) show the original and augmented correspondences, respectively.
provide various context con-
coverage of the correspondences. We build an
two vertices with distance d, where is a path (r..
Outlier filtering. To increase the quality of the correspondences to guide the supervision, remove outliers after calculating and enhancing the correspondences. First, we remove outliers according to the projected ray distance between corresponding points. Suppose Py and p. be pair of 2D correspondence in I to I, Tg and т, be the world-to-pixe projection for Ig and Ig. respectively. Given pair of correspondences and camera parameters, compute the closest 3D points X, and x, along the two rays shot from the camera centers through the two correspondences. Then, we project these two 3D points to the correspondence's image plane. The projected ray distance [48] is defined as the averaged Fuclidean distance between the projected points and the correspondences:
(x,)) Pyll2 + (x,g) -all2
dproj
We remove the correspondences with projected ray distance drroj larger than threshold. Second, we remove outliers by checking if point is statistically far from its neighbors. For each pair of correspondences, two 3D points X, and x, be obtained, which has been indicated in the last paragraph. We then consider (x, + x.) be the 3D point of the correspondence. We do this for all correspondence pairs to obtain set of 3D points P. For each 3D point in P. we compute the average distance to its k nearest neighbors and remove the point (as well its matched correspondence pair) the distance is larger than a threshold. This threshold is determined by the standard deviation of all points' average distances in P.
5

---

Experiments in Sec. 4.5 demonstrate the effects of our proposed augmentation and removal strat- egy since they can increase the reliability of the correspondences utilized NeRF's training.
9
3.3 Correspondence Loss
P.
P2
T,YY
We incorporate the obtained correspondences into NeRF training from two aspects: correspondence pixel reprojection loss and depth loss, as illus- trated in Fig. 4.
Figure Correspondence loss a ray pair (xy. rs). og and o, are the ray origins. Xy and x, are the closest points along the two rays. Yy and y. are the 3D points predicted by the NeRF model. (Va) and n. (y,) are the 2D projections of y. and Y. on the other image. The correspondence losses are defined with the geo- metric properties as illustrated.
Correspondence pixel reprojection loss. Let Yy and y, be the corresponding 3D points pre- dicted via NeRF, by accumulating the weighted sum of 7-depth values to the ray origin according Eq. We compute the pixel-space reprojection loss as follows:
(7) The pixel-space reprojection loss is similar to the projected ray distance [48]. The difference is that we reproject the predicted 3D points Yg and ys, instead of reprojecting the closest points along the two rays. Moreover, we utilize the correspondence score aq,s as weights to provide more reliable metric. Since the camera parameters are known, the projected ray distance constant. However, the reprojection of the predicted 3D points can provide supervision signal to guide the NeRF model.
Spi R) Er.r, ag,s (y,T,(y)) - P,lla + || (y,,) -Pl12).
Correspondence depth loss. The correspondence depth loss term penalizes the difference between the predicted depth and the ground-truth depth. The ground-truth depth is estimated by computing the closest 3D points Xg and x, along the two rays shot from the camera centers Og and o, through the two correspondences. We use a relative depth loss, which is adaptive to different scalings and can be described with the following equation:
Vg o X2 1211
1/+
ll. 1y.-0oll: 0,112 1):
Cdepth (0, R) = E SR., aq.a
(8)
||,- 0z112
where Og,s is the confidence of correspondence regular the effects of the relative depth loss. In the ideal scenario, Yg should overlap with X, and similarly ys should overlap with x,, so the loss will be zero. The relative depth loss is smooth function and is minimized when the predicted depth is close to the estimated depth of the ground truth.
Training procedure. For each ray in batch of rays Rhatch, query the correspondence map to find the corresponding rays all training rays R. We assume that each ray may have corresponding rays, B E [0, +oo). For each pair of corresponding rays, model to obtain the predicted 3D points. We then compute the pixel-space reprojection loss and the correspondence depth loss for each pair of corresponding rays. The final loss is the sum of pixel-space correspondence reprojection loss, correspondence depth loss, and the regular NeRF color loss Lcolor'
the forward pass of the NeRF
- Lcolor ò Cpixel A2
where \ and A2 are the loss weights, and we set them 0.1 our experiments.
4
Experiments
4.1 Datasets
We compare novel view synthesis results LLFF [25] with density-field-based NeRF models. We follow the convention to use every 8th image as test images [4], while selecting the training views uniformly from the rest of the images [29]. The selected training views and test views are the same across all methods. Three input views are used for training.
We compare surface reconstruction results on DTU [26] with SDF-based NeRF models. We follow the convention to use the same scenes from DTU as previous works [21, 49]. The selected training views and test views the same across all methods. Three input views are used for training.

---

Table 1:
Quantitative results LLFE We compare novel view synthesis the LLFF dataset for imput views. Our method outperforms the baseline NeRF model and sparse-view optimized models, Our model built with the vamilla NeRF as the direct baseline.
PSNR 1
SSIM
LPIPS +
Depth MAE &
NeRF [4]
16.79
0.56
0.37
1.66
DS-NeRF [13]
17.09
0.57
0.38
1.64
RegNeRF [29]
19.08
0.59
0.34
1.02
Ours
19.83
0.70
0.29
0.91
Table 2: Quantitative results on DTU. We compare novel view synthesis and surface reconstruction with SDF-based methods on the DTU dataset for 3 input views. All models are trained without foreground masks. For evaluation, report photometric metrics for the masked foreground object and the full image. The Chamfer-Li geometric metric computed with the official DTU evaluation code with foreground masks applied. Our model built with NeuS as the direct baseline, achicving the best results across photometric and geometric metrics.
PSNR
SSIM
LPIPS
Chamfer- L1 + Object
Object
Image
Object
Image
Object
Image
UNISURF [20]
12.90
13.85
0.47
0.58
0.28
0.57
7.23
VoISDF [52]
15.68
15.44
0.58
0.65
0.21
0.47
4.44
NeuS [21]
16,06
16.37
0.59
0,66
0.21
0.46
6.16
Ours
20.58
18.23
0.77
0.76
0.13
0.33
2.63
4.2
Evaluation Metries
For the novel view synthesis task, we report photometric metrics, including PSNR, structural similarity index (SSIM) [50], and the learned perceptual metric LPIPS [5]]. In addition to the image metries, we also report the mean absolute еrror (MAE) for depth prediction. We compute the pseudo ground truth depth using the baseline NeRF model given all input views. The depth error calculated as the mean absolute difference between the predicted depths and the pseudo ground truth depths. Both the predicted depths and the pseudo ground truth depths are the normalized coordinates.
For the surface reconstruction task, we report the Chamfer distance with DTU's official evaluation code [26], where the Chamfer distance is computed in the unnormalized world coordinates for the foreground objects. We also report the PSNR, SSIM, and LPIPS metrics for the rendered images with (object) and without (image) the foreground masks.
4.3 Novel View Synthesis
Quantitative comparison. For the novel view synthesis task, we compare our method with the baseline NeRF [4] as well as few-view optimized DS-NeRF [13] and RegNeRF [29]. DS-NeRF uses external SfM module COLMAP [19] to generate sparse point clouds for depth supervision, while RegNeRF applies additional regularizations on depths and colors rendered from unobserved views. The experiments are performed on the LLFF dataset with 3 input views. Our method outperforms all other methods regarding photometric metrics and depth prediction. The results are shown in Table 1. For DS-NeRF, observe that if COLMAP is only given sparse-view inputs with known camera poses for SfM (as opposed to providing all views for SfM and selecting visible points from sparse views), COLMAP does not generate sufficient number of points for depth supervision. In contrast, our method relies on image correspondences as inputs that are computed according to various characteristies. Therefore, it can leverage much denser prior information for supervision, as shown in Fig. 3.
Qualitative comparison. Besides the quantitative comparison, we provide the qualitative comparison to demonstrate the effects of our approach in improving novel view synthesis performance. The visual samples are shown in Fig. 5. Compared with baselines, our results have sharper and more accurate details and fewer artifacts. Such an improvement in performance is most visually evident in challenging cases such as small or thin structures.
7

---

4.4 Surface Reconstruction
Quantitative comparison. We apply the same correspondence loss terms to the SDF-based neural implicit field method NeuS [21] and compare the results with the baseline NeuS model as well as other two SOTA SDF-based surface reconstruction method, UNISURF [20] and VolSDF [52]. The experiments performed on the DTU dataset with input views. The results are shown in Table 2. Our method outperforms all baseline models in terms of Chamfer-Ly distances, i.c., ours generates more accurate surfaces. Moreover, ours also is also significantly better than the baselines in terms of rendering metrics, including PSNR, SSIM, and LPIPS. This further demonstrates that our approach be plugged into different types of network backbones (density- -based and SDF-based) and can be applied to different tasks (novel view synthesis and surface reconstruction).
Qualitative comparison. The visual comparisons between our method and the baseline for surface reconstruction are shown in Fig. 6. It's apparent that our 3D reconstruction results have more precise geometrical shapes. Combined with the correspondence priors, our CorresNeRF achieves clean and high-quality reconstruction even when given sparse input views.
4.5 Ablation Studies
We perform ablation studies of the correspondence generation process, correspondence loss terms, robustness to correspondence noise, and the effect of foreground masks.
Correspondence generation process. We compare the result with augmentation removed, with automatic filtering removed, and with the full pipeline. The results are shown the first two rows in Table 3. The result shows that both the augmentation and the automatic outlier filtering process are able to improve the reconstruction quality measured photometric and geometric metrics. The augmentation step can provide denser supervision signals, while the filtering step is able to remove the noisy correspondences to further improve the reconstruction quality.
Correspondence loss terms. We evaluate the effectiveness of the correspondence pixel reprojection loss and correspondence depth loss terms. We compare the results with only the reprojection loss removed, with only the depth loss removed, and the full pipeline. The results are displayed in Table 3. Our results show that both correspondence reprojection loss and the correspondence depth loss contribute significantly to the reconstruction quality, each adding ~~1 dB PSNR compared to the vanilla NeRF. The combined loss term yields the best performance, adding dB PSNR.
Robustness to correspondence noise. To evaluate the robustness CorresNeRF to noisy corre- spondences, introduce Gaussian noise with standard deviations of 1, 2, and 4 pixels to both the and y pixel coordinates. We measure the performance of CorFesNeRF on L.FFF dataset with 3 input views. The results shown in Table 4. The automatic filtering process is able remove the noisy correspondences, demonstrating the robustness of CorresNeRF to noisy correspondences.
Correspondence with mask supervision. For the surface reconstruction task, we evaluate the performance of CorresNeRF when the foreground mask is provided. When the foreground mask available, the correspondences are filtered by the mask, allowing the model to focus on the foreground region. The results are shown in Table 5. Our model outperforms the NeuS model in both the with-mask and without-mask settings. Qualitative visualization results are shown in Fig. 8 in the supplementary material.
5 Conclusion
We presented CorresNeRF, method that can utilize the image correspondence priors for training neural radiance fields with sparse-view inputs. We propose automatic augmentation and filtering methods to generate dense and high-quality image correspondences from sparse-view inputs. We design reprojection and depth loss terms based the correspondence priors regularize the neural radiance field training. Experiments show that our method can significantly improve the reconstruction quality measured photometric and geometric metrics with only a few input images.
Limitations and future work. In CorresNeRF, convincing correspondences are obtained by using SOTA matching networks and further processed via adaptive augmentation and outlier removal. An

---

NeRF [4]
DS-NeRF [13]
RegNeRF [29]
CorresNeRF
Ground Truth
Figure 5: Qualitative results from the LLFF dataset.
UNISURF [18]
VolSDF [52]
NeuS [21]
CorresNeRF
Ground Truth
Figure Qualitative results from the DTU dataset.

---

Table 3: Ablation study correspondenee generation and loss terms. We verify the effectiveness of correspondence pre-processing and correspondence terms. find both correspondence augmentation and outlier filtering are helpful for the final performance. We also find that both the pixel reprojection loss and correspondence depth loss contribute to the final performance on both photometric and geometric metrics.
PSNR 1
SSIM t
LPIPS
Depth MAE 4
NeRF [4]
16.79
0.56
0.37
1.66
Ours w/o Corres Augment
18.50
0.66
0.32
1.15
Ours w/o Corres Filter
19.32
0.69
0.29
0,93
Ours w/o Pixel Loss
18.16
0.62
0,42
1.33
Ours w/o Depth Loss
17.75
0.59
0.51
1.47
Ours
19.83
0.70
0.29
0.91
Table 4: Ablation study robustness to correspondence noise. We add Gaussian noise the correspondences generated by our method and evaluate performance CorresNeRF on the LLFF dataset with 3 imput views.
Corres % After Filter
PSNR
SSIM
LPIPS
Depth MAE L
NeRF [4]
N/A
16.79
0.56
0.37
1.66
Ours (Noise STD px)
13.96%
18.31
0.61
0.48
1.04
Ours (Noise STD 2 px)
27.04%
19.16
0.66
0.33
1.06
Ours (Noise STD = 1 px)
48.91% %
19.31
0.67
0.28
1.06
Ours (Noise STD = px)
100.00%
19.83
0.70
0.29
0.91
Table 5: Ablation study on the effects of foreground masks. We report the results of NeuS and our model trained with and without foreground masks the DTU dataset with 3 input views. The photometrie metries calculated with the foreground masks applied regardless the training settings.
PSNR
SSIM f
LPIPS 4
Depth MAE
NeuS w/o Mask Ours w/o Mask
16.06
0.59
0.21
6.16
20.58
0.77
0.13
2.63
NeuS w/ Mask
20.85
0.78
0.12
2.36
Ours w/ Mask
21.85
0.81
0.11
1.57
admired number of correspondences can be acquired for most practical scenes (proved by experiments on various benchmarks). However, there are still some extreme cases where few correspondences are computed via matching networks (due to unreasonable camera positions some specifie textures of target scenes). Correspondence generation methods are needed for these cases to synthesize convincing correspondences along with the optimization of NeRF. Some recent works [53, 54] have proved that neural implicit representations can be utilized to learn correspondences in turn. Dealing with such extreme scenes will be future work.
Acknowledgment
This work is supported in part by the National Natural Science Foundation of China (No. 62201484). HKU Startup Fund, and HKU Seed Fund for Basic Research.
10

---

References
[1] Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling, In CVPR, 2019. 1,3
[2] Lars M. Mescheder, Michael Occhsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In CVPR, 2019. 1,3
[3] Jeong Joon Park, Peter Florence, Julian Straub, Richard A. Newcombe, and Steven Lovegrove. DeepSDF: Learning continuous signed distance functions for shape representation. In CVPR, 2019. 1,3
[4] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020. 1, 3, 3,4, 6, 7, 9, 10, 14, 17
[5] Xiaogang Xu, Ying-Cong Chen, and Jiaya Jia. View independent generative adversarial network for novel view synthesis. In ICCV, 2019. 1
[6] Sicheng Xu, Jiaolong Yang, Dong Chen, Fang Wen, Yu Deng, Yunde Jia, and Xin Tong. Deep 3d portrait from a single image. In CVPR, 2020. 1
[7] Anastasiaa Koriloova, Marsel Faizullin, Konstantin Pakulev, Andrey Sadkov, Denis Kukushkin, Azat Akhmetyanov, Timur Akhtyamov, Hekmat Taherinejad, and Gonzalo Ferrer. Smartportraits: Depth powered handheld smartphone dataset human portraits state estimation, reconstruction and synthesis. In CVPR, 2022.
[8] Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P. Srinivasan, Jonathan T. Barron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. In CVPR, 2022.
[9] Haithem Turki, Deva Ramanan, and Mahadev Satyanarayanan. Mega-nerf: Scalable construction of large-scale nerfs for virtual fly-throughs. In CVPR, 2022.
[10] Ricardo Martin- Brualla, Noha Radwan, Mchdi S. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In CVPR, 2021.
[11] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa, pixelNeRF: Neural radiance fields from one or few images. In CVPR, 2021. 1,3
[12] Ajay Jain, Matthew Tancik, and Pieter Abheel. Putting nerf on a dict: Semantically consistent few-shot view synthesis. In ICCV, 2021. 1,3
[13] Kangle Deng. Andrew Liu, Jun-Junnna Yan Zhu, and Deva Ramanan. Depth-supervised nerf: Fewer views and faster training free. CVPR, 2022. 1, 7,7,9 14, 17
[14] Barbara Roessle, Jonathan T. Barron, Ben Mildenhall, Pratul P. Srinivasan, and Matthias Nießner. Dense depth priors for neural radiance fields from sparse input views. In CVPR, 2022. 1,3
[15] Mijeong Kim, Seonguk Seo, and Bohyung Han. InfoNeRF: Ray entropy minimization for few-shot neural volume rendering. In CVPR, 2022. 3,
[16] Zehao Yu, Songyou Peng. Michael Niemeyer, Torsten Sattler, and Andreas Geiger. MonoSDF: Exploring monocular geometric cues for neural implicit surface reconstruction. In NewrlPS, 2022. 1,3
[17] Kai Zhang, Gernot Riegler. Noah Snavely, and Vladlen Koltun. NeRF++: Analyzing and improving neural radiance fields. arXiv:2010.07492, 2020.
[18] Jiepeng Wang, Peng Wang, Xiaoxiao Long, Christian Theobalt, Taku Komura, Lingjie Liu, and Wenping
Wang. Neuris: Neural reconstruction of indoor scenes using normal priors. In ECCV, 2022. 1,9
[19] Johannes Lutz Schönberger and Jan- An-Michacl Frahm. Structure- fromu---rommmoiiinn revisited. CVPR, 2016. 1,
3, 7, 14
[20] Michael Oechsle, Songyou Peng, and Andreas Geiger. UNISURF: unifying neural implicit surfaces and radiance fields for multi-view reconstruction. ICCV, 2021. 2, 8, 17
[21] Peng Wang, Lingje Liu, Yuan Liu, Christian Theohalt, Taku Komura, and Wenping Wang. NeuS: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. In NeurlPS, 2021. 2, 3, 6, 7,
9, 15, 16, 17
[22] Raul-Edouard Sarlin, Daniel DeTone. Tomasz Malisiewicz, and Andrew Rabinovich. superglue: Learning feature matching with graph neural networks. In CVPR, 2020. 2,4
[23] Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. Loftr: Detector-free local feature matching with transformers. In CVPR, 2021. 2,
[24] Johan Edstedt, loannis Athanasiadis, Marten Wadenbück, and Michael Felsberg. Dkm: Dense kernelized feature matching for geometry estimation. arXiv:2202.00667 2022. 3,4

---

[25] Ben Mildenhall, Pratul P. Srinivasan. Rodrigo Ortiz Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi. Ren Ng. and Abhishek Kar. Local light field fusion: practical view synthesis with prescriptive sampling guidelines. TOG, 2019. 2, 6
[26] Rasmus Ramsbøl Jensen, Anders Lindbjerg Dahl, George Vogiatzis, Engin Tola, and Henrik Aanæs. Large
scale multi- view stereopsis evaluation. In CVPR, 2014. 2, 6.
[27] Yi Wei, Shaohui Liu, Yongming Rao, Wang Zhao, Jiwen Lu, and Jie Zhou. Nerfingmvs: Guided optimization of neural radiance fields for indoor multi-view stereo. In ICCV, 2021. 3
[28] Jian Zhang, Yuanqing Zhang, Huan Fu, Xiaowei Zhou, Bowen Cai, Jinchi Huang, Rongfei Jia, Binqiang Zhao, and Xing Tang. Ray priors through reprojection: Improving neural radiance fields for novel view extrapolation. In CVPR, 2022. 3
[29] Michael Niemeyer, Jonathan T. Barron, Ben Mildenhall, Mehdi S. M. Sajjadi, Andreas Geiger, and Noha Radwan, RegNeRF; Regularizing neural radiance fields for view synthesis from sparse inputs. In CVPR, 2022. 3, 6, 7, 9, 14, 17
[30] Haoyu Guo, Sida Peng, Haotong Lin. Qianqian Wang, Guofeng Zhang. Hujun Bao, and Xiaowei Zhou. Neural 3d scene reconstruction with the manhattan-world world assumption. In CVPR, 2022. 3
[31] Anpei Chen, Zexiang Xu, Fugiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In ICCV, 2021. 3
[32] Qianqian Wang. Zhicheng Wang. Kyle Genova, Pratul P. Srinivasan, Howard Zhou, Jonathan T. Barron, Ricardo Martin-Brually Noah Snavely, and Thomas A. Funkhouser. Ibrnet: Learning multi-view image-
based rendering. In CVPR. 2021. 3
[33] Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P. Srinivasan, Jonathan T. Barron, and Ren Ng. Learned initializations for optimizing coordinate baadiinttt---eddddddeeee neural representations. In CVPR, 2021. 3
[34] Shoukang Hu, Kaichen Zhou, Kaiyu Li. Longhui Yu, Lanqing Hong. Tianyang Hu, Zhenguo Li, Gim Hee Lee, and Ziwei Liu. ConsistentNeRFF: Enhancing neural radiance fields with 3d consistency for sparse view synthesis. arXiv:2305. / 1031, 2023. 3
[35] Prune Truong, Maric-Julic Rakotosaona, Fabian Manhardt, and Federico Tombari. SPARF: neural radiance fields from sparse and noisy poses. In CVPR, 2023, 4
[36] David G. Lowe. Distinctive image features from scale-invariant keypoints. IJCV, 2004. 4
[37] Navneet Dalal and Bill Triggs. Histograms of oriented gradients for human detection. In CVPR, 2005. 4
[38] Herbert Bay, Tinne Tuytelaars, and Luc Van Gool. SURF: speeded up robust features. ECCV, 2006.
[39] Kwang Moo Yi, Eduard Trulls, Vincent Lepetit, and Pascal Fua. LIFT: learned invariant feature transform.
In Bastian I Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, ECCV, 2016. 4
[40] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superpoint: Self-supervised interest point detection and description. In CVPR, 2018. 4
[41] Eric Brachmann and Carsten Rother. Neural-guided RANSAC: learning where to sample model hypotheses.
In ICCV, 2019.
[42] Shitao Tang, Jiahui Zhang, Siyu Zhu, and Ping Tan. Quadtree attention for vision transformers. In ICLR, 2022. 4
[43] Qing Wang, Jiaming Zhang, Kailun Yang, Kunyu Peng, and Rainer Stiefelhagen. Matchformer: Interlcaving attention in transformers for feature matching. In ACCV, 2022. 4
[44] laroslav Melekhov, Aleksei Tiulpin, Torsten Sattler, Mare Pollefeys, Esa Rahtu, and Juho Kannala. Dge-net:
Dense geometric correspondence network. In WACV, 2019. 4
[45] Prune Truong, Martin Danelljan, and Radu Timofte. Glu-net: Global-local universal network for dense
flow and correspondences. In CVPR, 2020. 4
[46] Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas A. Funkhouser, and Matthias Nießner. ScanNet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, 2017. 4
[47] Zhengqi Li and Noah Snavely. MegaDepth: Learning single-view depth prediction from internet photos. In CVPR, 2018. 4
[48] Yoonwoo Jeong, Seokjun Alm, Christopher B. Choy, Animashree Anandkumar, Minsu Cho, and Jaesik Park. Self-calibrating neural radiance fields. In ICCV, 2021. 5,
[49] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Ronen Basri, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. In NewrIPS, 2020.
[50] Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: from error visibility structural similarity. TIP, 2004. 7
12

---

[51] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In CVPR. 2018. 7
[52] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. In NewrIPS, 2021. 7, 8, 9, 17
[53] Lin Yen-Chen, Pete Florence. Jonathan Barron, Tsung- Yi Lin, Alberto Rodriguez, and Phillip Isola. Nerf-supervision: Learning dense object descriptors from neural radiance fields. In ICRA, 2022. 10
[54] Jianfei Guo, Zhiyuan Yang, Xi Lin, and Qingfu Zhang. Template nerf: Towards modeling dense shape correspondences from sppegoty-speeific object images, arXiv:2111.04237, 2021. 10
13

---

Appendix
A Discussions on Correspondence
In this section, we provide additional discussions and visualizations on the correspondence generation pipeline, including direct correspondence reconstruction, comparing correspondence with SfM, and the correspondence propagation process.
A.1 Comparing with SIM
We compare the DS-NeRF [13]'s SIM pipeline with our CorresNeRI's correspondence generation procedure in terms of the number of points generated and the number of correspondences generated. The results are reported in Table 6. The table shows that CorresNeRF is able to leverage much denser supervision compared to EiM-based depth supervision methods.
Table 6: Comparison of DS-NeRF's SfM point cloud number, CorresNeRF's correspondence number, and their pixel coverage (%) We compare the number of points the supervision point cloud used DS-NeRF [13] generated by COLMAP SfM [19] with the number of correspondences used CorresNeRF. The numbers are reported in counts (total number of points and total number of correspondences) and pixel coverage percentage (the percentage of pixels that supervised by sparse point cloud in DS-NeRF and the percentage of pixels having correspondences Co))esNeRF), For DS-NeRF, views with camera poses provided (as opposed to providing all views SiM and selecting visible points from input views). The results are reported on the LLFF dataset with 3 input views.
run COLMAP only the selected input
DS-NeRF [13]
Ours
with SfM
Base
Base+Filter
Basc+Filter+Aug
fern
362 (0. 19%)
325,633 (57%)
281,382 (49%)
368,798 (65%)
flower
685 (0.35%)
415,459 (73%)
278,180 (49%)
356,044 (62%)
fortress
609 (0.31%)
435,676 (76%)
377,956 (66%)
430,044 (75%)
horns
512 (0.27%)
270.778 (47%)
224,812 (39%)
271,705 (48%)
leaves
201 (0.11%)
272,158 (48%)
153,024 (27%)
198,412 (35%)
orchids
229 (0.12%)
213,660 (37%)
153,250 (27%)
242,620 (42%)
room
345 (0.18%)
226,156 (40%)
190,302 (33%)
260,308 (46%)
trex
644 (0.34%)
202,650 (35%)
164,047 (29%)
233,950 (41%)
A.2
Correspondence Propagation
We provide additional visualizations to demonstrate the correspondence propagation process. With correspondence propagation, image pairs initially with few no correspondences may still obtain correspondence relationships propagated by other image pairs. For the propagated correspondences, we assign new confidence score as the cumulative product of the confidence scores in the propagation path. These confidence levels are used to weigh the loss terms, as described in the main paper. See Fig. 7 for the visualizations.
B Implementation Details
LLFF dataset. For the novel view synthesis task on LLFF, we based our code on NeRF [4]. For most parameters, we use the same settings as the original NeRF paper.
Number of input views:
Image scale factor: 8
Correspondence pixel reprojection weight (Cpixet): 0.1
Correspondence depth weight (Cdepth): 0,1
Training iterations: 50K²
Learning rate: -4
Similar to RegNeRF [29]'s 44K iterations in 3-view setting. More iterations could enhance results.

---

Input
Corres(A,
Corres(A, with Propagation
Input
Corres(C, B)
Corres(C, B) with Propagation
Input C
Corres(C, A)
Corres(C, A) with Propagation
input images. Column 2 shows the correspondences without correspondence propagation but with all other augmentations and filtering enabled. Column 3 shows the propagated correspondences. The correspondence colors represent the confidence scores, where the higher confidence scores greener. The highlighted area shows that the image pair B) can receive additional correspondence relationships by propagating their correspondences from C.
Figure Correspondence propagation visualizations. Column shows the sparse
DTU dataset. For the novel view synthesis and surface reconstruction task on DTU, based our code on NeuS [21]. Important parameters are listed below. For most parameters, we utilize the same settings in the original NeuS paper.
Number of imput views:
Image scale factor: 4
Correspondence pixel reprojection weight (Cpixel): 0.1
Correspondence depth weight (Edepth): 0.1
Training iterations: 200K
Learning rate: 5e
C
Additional Results
C.1 Effects of Foreground Masks
We provide additional qualitative results to demonstrate the effeets of foreground masks with corre- spondence priors in Fig.
C.2 Per-Scene Results
We provide per-scene results for the novel view synthesis task on the LLFF dataset in Table 7, as well as novel view synthesis and surface reconstruction tasks on DTU dataset Table and Table
15

---

NORR
B0R
GRAAA
NeuS w/o Mask
Ours w/o Mask
NeuS Mask
Ours w/ Mask
Ground Truth
Figure 8: Effects foreground masks. Columns and 2 compare NeuS [21] and our model trained without foreground masks. Our model outperforms NeuS in terms of novel-view rendering quality and mesh reconstruc- tion quality. Columns 3 and 4 compare NeuS and our model trained with foreground masks. Our model trained with foreground masks further improves the rendering quality and geometric reconstruction quality. We observe more geometric details and fewer artifacts in our reconstruction compared NeuS.
16

---

Table 7: Per-scene results on the LLFF dataset. We compare novel view synthesis on the LLFF dataset for 3 input views. Our method outperforms the baseline NeRF model and the sparse-view optimized DS-NeRF and RegNeRF models. Our model built with the vanilla NeRF as the direct baseline. Note that for RegNeRF [29]. the numbers reported here are results from our own evaluation, which are slighty better than the numbers (PSNR: 19.08, SSIM: 0.59, and LPIPS: 0.34) reported in the original author's paper and Table 1.
NeRF [4] DS NeRF /13]/ / ReRN [291/ Ours
PSNR f
SSIM f
LPIPS 4
Depth MAE І
fern
20.47 / 21.28/ / 19.87 /
0.71 /.75 / 0.70/
0.33/ /33//.277 0.27 / 0.30 0.33
0.28 0.28/0.29 / 0.34 / 0.31
flower
17.35/ 19.35 / 19.93 / 20.44
0.66/ / 0.68/ 0066//00 0.69
0.34/0.26/ 0.23 00.5
1.78/0885 / 0,91 / 0,85
fortress
19.48 / 20.87 / 23,33/ / 22.53
0,68/ 0.67 0.74/ 0.76
0.38/0.37 / 0.26/ /0.35
0 / 0,46 / 0.51
horns
14.46 / 14.50/ / 15.64 / 19.18
/ 0.61 0.71
0.51/0 / 0.36 0.32
0.32
3.06
/ 1.68 / 1.09
leaves
12.73 / 14.83 / 16.60 / 16.89
0.28/ 0.61 0.59
0.45/0.41 / 0.22 0.32
2.16/ / 3,08 / 3.10
orchids 13,92/ 14.44/ / 15.56 15.85
0.38 0.50 / 0.48
0.33/ / 0.25 0.38
L.13
/ 0.71 / 0.55
room
19,97 / 17.69 / 21.53 / 22.41
0.82/0.71 0.86 0.87
0.25
0.19
0.77 /2.44 0,45 0.38
trex
15.89/ 13.79 / 20.16/ / 20.18
0.59/0.33
/ 0.77 0.76
0.36
0.22
1.19 / 2.39 / 0.51 /0.51
mean
16.79/17.09/ / 19.08 / 19.83 0,56 1.56/0.3 0.57 / 0.68// / 0.70 0.37 / 0.38 / 0.25 / 0.29 1.66/ / 1.64 / 1.02 / 0.91
Table 8: Per-scene results on the DTU dataset (image-level eval). We compare novel view synthesis and surface reconstruction the DTU dataset for 3 input views. The models are trained without mask supervision. The models are evaluated without foreground masks applied (image-level).
UNISURF [20]/ / VoISDF [52] / NeuS /2urs
PSNR f
SSIM f
LPIPS $
scan24
8.98 / 10.53 / 10.36 / 13.99
0,43 0,46 0.64
0.65 0.65 /0.58 / 0.61 / 0,36
scan37
7.26/ 10.40/ 10.31 / 13.38
/
/ 0.47 / 0.59
0.67 /0.43 / 0.58 / 0,40
scan40
12.14/ 7.48/ 10.61 / 13.51
0.58/0.39 / 0,48/ 0.70
0.59 / / 0.63 / 0,38
scan55
14.68 / 17.51 / 17.75/ 18.99
0.52 /..6 0.73
0.71 /0.45 / 0.52 / 0,41
scan63
4.79/ 5.80 10,92 / 12.66
0.42/0.33/..6 / 0.78
0.75 / 0.50 0.31
scan65
12.78 / 19,93/ 19.15/ 20.13
0.60/ 0,82 /0.6 0.79 0.57 / / 0.62 / 0.75
0.59 0..2 / 0.37 / 0.30 / 0,48 / / 0.31
scan69
17.05 / 16.62/ / 17.87 / 19.71
0.51
scan83
8.37 / 13.19/ / 13.73 / 13.98
0,48 / / 0.69 / 0.74
0.68
0.68
0.39
scan97
13.28 / 10.93 14.57/ / 15.80
0.59/
0.74
0.46
/ 0.39 / 0.31
scan 105
11.78 / 13.69 / 12.91 / 14.68
0.57 / 0.63 / 0.60 0.72
/.56 / 0.51 / 0.37
scan 106
18. / 21.36 / 20.09/ 22.03
0.71 / 0.78 / 0.76 / 0.83
0.44 /0.35 / 0.40 / 0.30
scan 10
18.07 / 16.21 20.51 / 22.05
0.67 / 0.60 0.77 0.81
0.48 /0.62 / 0.39 0.34
scan 14
20.90/ / 23.00/ 23.12 24.25
0.79 / / 0083// / 0.86
0.37 / / 0.28 / 0.24
11
/8.84/ /0.93/ 20.55
25.01
0.66
0.78 / 0.73 0.85
0.57 0.46 / 0.44 0.27
scan122
20.69/ / 24.02/ 23.08 / 23.26
0.70/ 0.85 / 0.80 / 0.81
0.53 / 0.32 / 0.38 0.33
mean
13.85/ 15.44/ 16.37 / 18.23
0.58 / 0.65/ 0.66/ / 0.76 0.57 / 0.47 / 0.46/ 0.33
Table 9: Per-seene results on the DTU dataset (object-level eval). We compare novel view synthesis and surface reconstruction on the DTU dataset for 3 input views. Note that VoISDF fails to reconstruct mesh for one of the scenes (scan1 10), thus its Chamfer- L1 is averaged over the remaining scenes. The models are trained without mask supervision. The models are evaluated with foreground masks applied (object-level).
UNISURF [20]/ / VolSDF [52) / NeuS [21]/ Ours
PSNR
SSIM t
LPIPS +
Chamfer-Ly L1 4
scan24
9.45 / 10.95/ / 10.65 / 16.82
0.37 / 0.36 / 0.40/ / 0.61
0.54 0.49 / 0.49/ / 0.29
7.81 77000/ / 6.06 / 2.73
scan37
7.19/ 11.17 / 10.07 / 13.44
/ 0.30/ 0.45
0.33 153/0.21 0.21 0.27 / 0.22
7.54 / 7.24 / 4.92
scan40)
15.37 / 10.56 / 12.36 / 19.08
0.52/ / 0041 / 0.71
0.49 /0.48/ / 0.47/ / 0.28
6.37 / 7.68 / 3.00
scan55
12.71 16.35 / 15.93 / 19.55
0.30
0.58 / 0.43 / 0.73
/ 0.28 / 0.17
8.38/2.90/ 5.85 / 2,37
scan63
5.62. 12.83 12.05 / 20.37
0.38 /.63 / / 0.82
0.27 /0.17 0. 0.08
8.40/4.58 / 8.84 2.52
scan65
11.23 / 17.76 / 18.10/ 19.00
0.87 / 0.86
0.18 /.08 0.09 / 0.07
5.08 /2.0 / 4.65 / 2.71
scan69
16.44/ 17.01 / 18.21 / 21.94
0.60 0,81
0.27 /0.23 /..3///3333//////////////33 0.13
7.42/ / 6 6,30 2.05
scan83
7.62. / 12.72 / 13.52 / 22.63
0.37 / 0.49 / 0.52//
0.19 /.13 0.05
7.92 /9.62 3.14
scan97
12.13 / 13.32 / 15.03 / 18.36
0.44 0.49 / 0.70
0.27 /0.11 0.18 0.13
8.73 /3500/ 2.227
scan 105
12.12 14.60 15.94 / 21.48
0.46 /.54 / 0.62 0.80
0.29 /0.25 0.21 0.15
8.89 6.52 / 8. 3.61
scan106
15.83 /0.44/ 18.65 / 21.37
0.61 0.77 / 0.73 0.86
/0.33///.. 0.10
5.89 / 1.76
2.08
sc
14.89
13.69/ 17.68 19.79
0.51 0.70 0.79
0.19/0.26/0.10
/.1////0..4//.4 0.11
7.68 /NA/ / 5.75 / 2.03
scan1
18.90 21.1//11111111 21.69
23.44
0.74/ / 0.87
/.21/0.1100.1//
0.10
3.43
3,43 0.81 / 2.01 / 1.37
scan118
16.09 / 21.03 / 18.29
27.44
0.7//0.440..6
/.64//0661//// /
0.24
0.05
/3.9 / 6.16 1,83 6.47
scan122
17.91 21.66 /2.72
24.00
0.52
/ / 0.79
0.15/0.06/007
0.05
8.51 1.45 / 4.25 2.85
mean
/ 15.68 / 16.06/ / 20.58
0.47 / 0.58/ / 0.59 / 0.77
0.28 / 0.21 / 0.21 / 0.13
7.23 / /.44 / 6.16
17


Reviews:
[{"review_id": "TK57w2c9Xh", "reviewer": "Reviewer_yxQE", "summary": "The paper proposes an approach for sparse-view NeRF reconstruction by using image correspondences as a prior. NeRF under the sparse-view regime is overparameterized and under constrained hence requiring a prior to optimize. This paper proposes to use image correspondences that are extracted across the different views, in particular, they use DKMv3. They propose two additional loss functions based on the correspondences from the prior: the first uses the reprojection error by using the expected depth as predicted by the NeRF while the second uses a correspondence-depth based loss by finding the closest 3D points in space based on the correspondence from the prior. Experiments on novel view synthesis and surface reconstruction show the improvement of their proposed approach.", "questions": "1. What happens when other image correspondence priors (not DKMv3) is used? It will be beneficial to see how this would affect results, for example for both another neural network based prior as well as a handcrafted prior, e.g. output of COLMAP, even if it will only give correspondences at sparse pixel locations.\n2. NeRF depth is used for the correspondence pixel reprojection loss, and it is known that in the sparse regime the NeRF depth can be erroneous. Did this cause an issue in the convergence of the network? Would it have helped to add in a depth loss supervision as a prior?", "limitations": "The authors have included limitations in the main paper of the submission.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper proposes to use image correspondences as a prior for sparse view nerf reconstruction which is intuitive and sound. Image correspondences as a prior is generalizable and hence a pretrained model can be used, namely DKMv3. They propose two simple, yet intuitive losses for their approach. Experiments show that the proposed method is able to perform better compared to existing baselines.", "weaknesses": "The effectiveness of the method is relying on the accurate prediction of the correspondences, and it is known that correspondences can be erroneous on texture less regions, illumination changes or wide-baseline cameras. On real scenes, these issues might arise more -- e.g. on sparse scannet images as used by existing benchmarks [53, 54]. The sparse view inputs here have wide-camera baselines as opposed to forward facing scenes in LLFF. It would be more convincing if the method can also perform reasonably in such settings.\n\n\nSome references on sparse view NeRF:\n\n**[53]** Dense Depth Priors for Neural Radiance Fields from Sparse Input Views, CVPR '22\n**[54]** SCADE: NeRFs from Space Carving with Ambiguity-Aware Depth Estimates, CVPR '23", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1: Performance on wide-camera baselines (e.g. ScanNet) in addition to forward-facing LLFF.\n\nWe appreciate the reviewer's inquiry. In our paper, we present evaluations of\nCorresNeRF on the LLFF dataset with forward-facing cameras, as well as on the\nDTU dataset where the cameras have a spherical configuration.\n\nIn \"wide-camera\" scenarios such as the ScanNet dataset, establishing\ncorrespondences between input images necessitates a significant overlap between\nimages containing textured regions. Provided that reasonable correspondences can\nbe achieved through image matching techniques, CorresNeRF remains effective. We\nintend to include additional experiments on the ScanNet dataset in the paper's\nfinal version.\n\nIn essence, CorresNeRF depends on image matchers to ascertain correspondences.\nAs long as these correspondences are established, CorresNeRF can function as a\ngeneric plug-and-play module, enhancing the efficacy of any NeRF model, which\nbodes well for the broader community.\n\n### Q2: What happens when other image correspondence priors is used?\n\nWe thank the reviewer for the question. Indeed, the quality and quantity of\nimage correspondences influence CorresNeRF's performance. We've introduced a new\nexperiment where we compare CorresNeRF's performance when using different image\nmatching techniques, specifically LoFTR [55] and DKMv3 [56], on the LLFF\ndataset. Here are the results:\n\n| Method                  | PSNR↑ | SSIM↑ | LPIPS↓ | Depth MAE↓ |\n| ----------------------- | ----- | ----- | ------ | ---------- |\n| NeRF                    | 16.79 | 0.56  | 0.37   | 1.66       |\n| CorresNeRF (with LoFTR) | 18.13 | 0.64  | 0.30   | 1.10       |\n| CorresNeRF (with DKMv3) | 19.83 | 0.70  | 0.29   | 0.91       |\n\nIt's evident that regardless of the chosen image matching method, if viable\ncorrespondences are established, CorresNeRF can enhance performance beyond the\noriginal NeRF. Additionally, when a superior image matching method is employed,\nCorresNeRF naturally benefits, potentially driving its performance further.\n\nRegarding COLMAP correspondences, we applied COLMAP to the LLFF dataset using\nspecified cameras in 3 sparse views. Here's a summary:\n\n| Scene    | COLMAP: Num of Corres (Pixel Coverage %) | DKMv3: Num of Corres (Pixel Coverage %) |\n| -------- | ---------------------------------------- | --------------------------------------- |\n| fern     | 362 (0.19%)                              | 368,798 (57%)                           |\n| flower   | 685 (0.35%)                              | 356,044 (73%)                           |\n| fortress | 609 (0.31%)                              | 430,044 (76%)                           |\n| horns    | 512 (0.27%)                              | 271,705 (47%)                           |\n| leaves   | 201 (0.11%)                              | 198,412 (48%)                           |\n| orchids  | 229 (0.12%)                              | 242,620 (37%)                           |\n| room     | 345 (0.18%)                              | 260,308 (40%)                           |\n| trex     | 644 (0.34%)                              | 233,950 (35%)                           |\n\nFrom this data, it's clear that COLMAP yields sparser correspondences compared\nto DKMv3. While CorresNeRF's performance might be restricted with only a few\ncorrespondences, it still outperforms the standard NeRF model. We direct the\nreviewer to Table 1 in the supplementary material for more detailed statistics.\n\n- [55] LoFTR: Detector-Free Local Feature Matching with Transformers, CVPR 2022\n- [56] DKM: Dense Kernelized Feature Matching for Geometry Estimation, CVPR 2023\n\n### Q3: NeRF depth can be erroneous in the sparse regime. Did this cause an issue in the convergence of the network? Would it have helped to add in a depth loss supervision as a prior?\n\nWe thank the reviewer for the question. Indeed, NeRF's depth can be unreliable\nin a sparse regime without an auxiliary prior. In CorresNeRF, image\ncorrespondences are used as priors, ensuring the depth values are tethered by\nthe corres depth loss and pixel reprojection loss. This makes the depth values\nmore reliable than those in the standard NeRF model.\n\nRegarding the proposition of introducing depth loss supervision as a prior, we\nbelieve that CorresNeRF's corres depth loss and pixel reprojection loss already\nimplicitly offer such supervision. Given the camera parameters and\ncorrespondences, depth values can be triangulated, and CorresNeRF's loss terms\nappropriately model this relationship."}, {"review_id": "S7JLSFUnvl", "reviewer": "Reviewer_cKcF", "summary": "The paper introduces CorresNeRF, a method that leverages image correspondence priors to improve the performance of Neural Radiance Fields (NeRF) in scenarios with sparse input views. The authors propose a plug-and-play module that incorporates correspondence priors into the training process by adding loss terms on the reprojection error and depth error of the correspondence points. They develop an adaptive algorithm for augmenting and filtering the correspondence priors to enhance their quality. The proposed method is evaluated on novel view synthesis and surface reconstruction tasks using density-based and SDF-based neural implicit representations across different datasets.\n\nThe proposed CorresNeRF utilizes image correspondence priors to supervise the training of NeRF models. This approach addresses the challenge of sparse input views and enhances the performance of NeRF in reconstructing 3D geometries.\nThe authors propose an automatic augmentation and outlier removal process for improving the quality and robustness of the correspondence priors. This process enhances the dense correspondence estimation and mitigates the effects of inaccurate correspondences.\nThe paper formulates a correspondence loss that incorporates reprojection and depth errors based on the correspondence priors. This loss effectively guides the learning of implicit functions in NeRF models and improves their performance.", "questions": "Could you provide a more detailed analysis of the computational efficiency and resource requirements of the CorresNeRF method? Specifically, it would be valuable to include information on training time, inference speed, and memory utilization to understand the practical implications and scalability of the approach.\n\nEstimating accurate and robust image correspondences can be challenging in real-world scenarios. It would be helpful to discuss the performance under inaccurate correspondence.\n\nIt would be beneficial to provide a more comprehensive comparison with existing state-of-the-art methods for sparse-view reconstruction, such as MVSNeRF and GeoNeRF. Including a thorough evaluation and comparison against these methods would strengthen the justification for the superiority and novelty of the proposed CorresNeRF method.", "limitations": "The limitation is discussed.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper demonstrates several strengths across different dimensions:\n\nThe paper introduces the concept of leveraging image correspondence priors to improve the performance of NeRF models in sparse-view scenarios. This novel approach addresses the challenge of reconstructing 3D geometries with limited input views and introduces the use of image correspondences as explicit supervision for learning implicit functions in NeRF. The combination of image correspondence priors and NeRF training is a creative and innovative approach that expands the capabilities of NeRF models.\n\nThe paper addresses a significant problem in the field of 3D reconstruction and view synthesis. Sparse-view scenarios are common in real-world applications, and improving the performance of NeRF models under such conditions has practical implications. The proposed method offers a practical and effective solution by leveraging image correspondence priors, which are readily obtainable and can be computed using standard methods. The experimental results demonstrate the superiority of the proposed approach over previous methods, highlighting its potential for advancing the state-of-the-art in novel view synthesis and surface reconstruction tasks.\n\nThe paper presents a well-designed methodology with clear objectives and a systematic evaluation process. The authors carefully consider the limitations of existing methods and propose solutions to overcome them. The proposed CorresNeRF method incorporates robust correspondence loss and automatic augmentation and filtering of correspondence priors, enhancing the quality and effectiveness of the training process. The experimental evaluation is thorough, encompassing various neural implicit representations and datasets, and the results demonstrate significant improvements in performance metrics.\n\nThe proposed method is described in a structured manner, with detailed explanations of the augmentation and filtering process, formulation of correspondence loss, and evaluation metrics. The figures and equations further enhance the clarity of the paper, aiding in the understanding of the concepts and techniques presented.\n", "weaknesses": "While the paper demonstrates several strengths, there are also a few areas where it could be improved:\n\nExperimental Evaluation: The paper would benefit from a more detailed analysis of the computational efficiency and resource requirements of the CorresNeRF method. Providing insights into the computational demands and resource utilization of the approach would help readers understand the practical implications and scalability of the method.\n\nImage Correspondences: While the paper introduces image correspondences as priors, it is important to acknowledge the potential challenges in estimating accurate and robust image correspondences, especially in scenarios with occluded or noisy images. Conducting a sensitivity analysis of correspondence accuracy would provide a clearer understanding of the method's performance under different conditions and shed light on its robustness and generalization capabilities.\n\nComparison with State-of-the-Art: The paper would benefit from a more comprehensive comparison with existing state-of-the-art methods for sparse-view reconstruction, such as MVSNeRF and GeoNeRF. Providing a thorough evaluation and comparison against these methods would help establish the superiority and novelty of the proposed CorresNeRF method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1: Experimental Evaluation: computational efficiency of CorresNeRF.\n\nWe thank the reviewer for the question. At inference time, CorresNeRF operates at exactly the same runtime as the baseline NeRF model. However, during training, CorresNeRF incurs additional runtime overheads due to the search for correspondences and the computation of the correspondence loss.\n\nWe've conducted a supplementary runtime analysis experiment. Specifically, we evaluated the runtime for both the forward pass (rendering) and backward pass (gradient computation), performing these measurements over 100 iterations with a batch size of 1024 on the fern scene of the LLFF dataset. Testing was conducted on a single NVIDIA RTX 2080Ti GPU, and both average and standard deviation of the runtimes were reported.\n\n| Method     | Training Forward (ms) | Training Backward (ms) |\n| ---------- | --------------------- | ---------------------- |\n| NeRF       | 51.722 ± 0.312        | 70.941 ± 0.429         |\n| CorresNeRF | 126.896 ± 3.254       | 124.160 ± 2.371        |\n\nIt's worth noting that the runtime overhead of CorresNeRF is contingent upon the ratio of pixels possessing valid correspondences. We believe the additional overhead introduced by CorresNeRF is justifiable, especially given the substantial performance enhancement over the baseline NeRF model.\n\n### Q2: Image Correspondences: sensitivity analysis of image correspondences\n\nWe thank the reviewer for the suggestion. We conducted two sets of supplementary experiments to assess the impact of 1) quality and 2) quantity of image correspondences on CorresNeRF's performance. Our results indicate that CorresNeRF maintains impressive performance even when faced with noisy correspondences or when utilizing only a small subset of image correspondences. This underscores the robustness of CorresNeRF; it demonstrates that as long as reasonable correspondences are present, CorresNeRF can amplify the performance over the baseline NeRF model.\n\n**Effects of correspondence quality (robustness to noise)**\n\nWe employed image matching methods to obtain correspondences and subsequently introduced Gaussian noise to these correspondences. Specifically, we added Gaussian noise with standard deviations of 1, 2, and 4 pixels to both x and y pixel coordinates of the correspondences. We then assessed the performance of CorresNeRF using the LLFF dataset.\n\nSection 3.2 describes how CorresNeRF employs an automatic outlier removal process based on camera reprojection error. Column 2 in the table below reports the relative number of correspondences remaining after this filtering process.\n\n| Method                       | Corres # After Auto Filter | PSNR↑ | SSIM↑ | LPIPS↓ | Depth MAE↓ |\n| ---------------------------- | -------------------------- | ----- | ----- | ------ | ---------- |\n| NeRF                         | 0.00%                      | 16.79 | 0.56  | 0.37   | 1.66       |\n| CorresNeRF (noise_std = 4px) | 13.96%                     | 18.31 | 0.61  | 0.48   | 1.04       |\n| CorresNeRF (noise_std = 2px) | 27.04%                     | 19.16 | 0.66  | 0.33   | 1.06       |\n| CorresNeRF (noise_std = 1px) | 48.91%                     | 19.31 | 0.67  | 0.28   | 1.06       |\n| CorresNeRF (noise_std = 0px) | 100.00%                    | 19.83 | 0.70  | 0.29   | 0.91       |\n\nWhen image correspondences are contaminated by noise, the automatic outlier removal process discards more of them, resulting in fewer correspondences for CorresNeRF to utilize. However, the correspondences that remain are deemed higher quality. Consequently, CorresNeRF maintains satisfactory performance even with noisy correspondences.\n\n**Effects of correspondence quantity**\n\nWe obtained correspondences using image matching methods and subsequently sampled a subset (50%, 25%, 12.5%, 6.25%, and 3.125%) of these correspondences to train CorresNeRF. We then assessed CorresNeRF's performance on the LLFF dataset.\n\n| Method                          | PSNR↑  | SSIM↑ | LPIPS↓ | Depth MAE↓ |\n| ------------------------------- | ------ | ----- | ------ | ---------- |\n| NeRF                            | 16.79  | 0.56  | 0.37   | 1.66       |\n| CorresNeRF (with 3.125% corres) | 18.616 | 0.647 | 0.322  | 1.129      |\n| CorresNeRF (with 6.25% corres)  | 18.934 | 0.657 | 0.299  | 1.113      |\n| CorresNeRF (with 12.5% corres)  | 18.854 | 0.66  | 0.287  | 1.108      |\n| CorresNeRF (with 25% corres)    | 19.068 | 0.669 | 0.269  | 1.10       |\n| CorresNeRF (with 50% corres)    | 18.986 | 0.67  | 0.266  | 1.085      |\n| CorresNeRF (with 100% corres)   | 19.83  | 0.70  | 0.29   | 0.91       |\n\nNotably, even with only 3.125% of the correspondences, CorresNeRF significantly outperforms the baseline NeRF model. The performance of CorresNeRF enhances as the correspondence quantity increases. When 100% of the correspondences are used, CorresNeRF achieves its peak performance. Thus, as long as quality correspondences are provided in adequate numbers, CorresNeRF can surpass the regular NeRF's performance.\n\n### Q3: Comparison with the state-of-the-art (MVSNeRF and GeoNeRF)\n\nWe thank the reviewer for the suggestion. In essence, if reasonable image correspondences can be secured, CorresNeRF can serve as a versatile plug-and-play module to enhance any NeRF model, including MVSNeRF and GeoNeRF.\n\nMVSNeRF and GeoNeRF are expansive NeRF models suitable for sparse-view settings. MVSNeRF capitalizes on a plane-sweeping cost volume, while GeoNeRF constructs cost volumes through transformer-based feature aggregation. These methodologies are distinct from the image correspondence priors employed by CorresNeRF. Hence, combining these techniques with CorresNeRF could potentially yield further performance improvements. We plan to incorporate additional experiments, merging CorresNeRF with MVSNeRF and GeoNeRF, in the finalized version of the paper."}, {"review_id": "ffHN8vf25R", "reviewer": "Reviewer_7TAK", "summary": "This paper proposed CorresNeRF, a method that leverages image correspondence priors to improve NeRF training on sparse input views. The correspodence matching is computed by off-the-shelf methods. The authors augue that the introduced inexpensive image correspodence priors can be used to supervise training of arbitrary NeRFs and lead to better performance / faster convergence taking sparse view inputs. Further, a robust correspondence loss is designed, including reprojection loss and depth loss baesd on correspondence priors. Overall, the method demonstrates superior reconstruction quality against baselines like VolSDF and NeuS, etc.", "questions": "Some questions: \n\n- Clarification on the image correspondence methods.The paper mentioned that image correspondences are obtained via off-the-shelf method. Could you provide more details on how they are acquired and how this (i.e. selection of the method) would affect the final reconstruction quality?\n\n- Robustness to the noise/outliers in the correspondence. An automatic augmentation and outlier removal process was designed, which seems to be key to the proposed approach. It would be useful to provide more details on the robustness of the design with some various level of noise/outliers on some scenes. \n\n-  What is the minimum quality of correspondence necessary for the method to outperform traditional NeRF implementations?\n\n- Impact of loss terms on results. How did the inclusion of correspondence pixel reprojection loss and correspondence depth loss affect the final results? Could you clarify how these losses contribute to the reconstruction quality?\n\n- Extreme and failure cases. One such case I'd imagine is glossy/specular or textureless surfaces. \n", "limitations": "Limitations are discussed in the above weakness section. ", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper's main contributions are summarized as follows:\n\n- Introduction of image correspondence as a cost-effective prior to supervise the training of any NeRFs.\n- Design of a pipeline to obtain robust correspondences from standard methods, including automatic augmentation and filtering.\n- Introduce a robust correspondence loss incorporating reprojection loss and depth loss based on correspondence priors.\n- Extensive experiments conducted on various baselines and datasets demonstrating the method's effectiveness across different types of neural implicit representations.\n\nThe authors conduct extensive experiments on various datasets, which demonstrate the effectiveness of their method. They find significant improvements in both novel view synthesis and surface reconstruction metrics. The proposed method outperforms other state-of-the-art sparse-view reconstruction methods and works well with various types of NeRF, including those with other priors.", "weaknesses": "There are some limitations that should be properly discussed:\n\n- Dependence on the quality of image correspodence. The quality of the obtained image correspodence matching significantly impacts the effectiveness of the proposed approach. Less accurate correspondences can negatively affect the supervised training of NeRF, which might lead to suboptimal reconstruction results.\n- Performance in non-sparse scenarios. The method is focused on the advantage of using CorresNeRF in sparse-view configurations, but it doesn't mention how this method would perform in dense-view configurations. It would be interesting to show such an ablation study to verify this.\n- The main comparisons show VolSDF and NeuS results as baselines. However, a more adequate baseline would be SparseNeuS: \nSparseNeuS: Fast Generalizable Neural Surface Reconstruction from Sparse Views. ECCV 2022. \n- The method shows 3 input views on DTU/LLFF datasets. What happen if arbitrary number of input views are given? This is soemwhat related to the 2nd point above. But it would be nice to have such experiments to better evaluate the robustness of the proposed pipeline. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n### Q1 (from weaknesses): Dependence on the quality of image correspondence.\n\nWe thank the reviewer for the suggestion. We employed image matching methods to obtain correspondences and subsequently introduced Gaussian noise to these correspondences. Specifically, we added Gaussian noise with standard deviations of 1, 2, and 4 pixels to both x and y pixel coordinates of the correspondences. We then assessed the performance of CorresNeRF using the LLFF dataset.\n\nSection 3.2 describes how CorresNeRF employs an automatic outlier removal process based on camera reprojection error. Column 2 in the table below reports the relative number of correspondences remaining after this filtering process.\n\n| Method                       | Corres # After Auto Filter | PSNR↑ | SSIM↑ | LPIPS↓ | Depth MAE↓ |\n| ---------------------------- | -------------------------- | ----- | ----- | ------ | ---------- |\n| NeRF                         | 0.00%                      | 16.79 | 0.56  | 0.37   | 1.66       |\n| CorresNeRF (noise_std = 4px) | 13.96%                     | 18.31 | 0.61  | 0.48   | 1.04       |\n| CorresNeRF (noise_std = 2px) | 27.04%                     | 19.16 | 0.66  | 0.33   | 1.06       |\n| CorresNeRF (noise_std = 1px) | 48.91%                     | 19.31 | 0.67  | 0.28   | 1.06       |\n| CorresNeRF (noise_std = 0px) | 100.00%                    | 19.83 | 0.70  | 0.29   | 0.91       |\n\nWhen image correspondences are contaminated by noise, the automatic outlier removal process discards more of them, resulting in fewer correspondences for CorresNeRF to utilize. However, the correspondences that remain are deemed higher quality. Consequently, CorresNeRF maintains satisfactory performance even with noisy correspondences.\n\n### Q2 (from weaknesses): Performance in non-sparse scenarios.\n\nWe tested the robustness of CorresNeRF with varying input view counts. Specifically, we doubled the number of input views from 3 to 6 and then evaluated CorresNeRF's performance on the LLFF dataset.\n\n| Method               | PSNR↑ | SSIM↑ | LPIPS↓ | Depth MAE↓ |\n| -------------------- | ----- | ----- | ------ | ---------- |\n| NeRF (3 views)       | 16.79 | 0.56  | 0.37   | 1.66       |\n| CorresNeRF (3 views) | 19.83 | 0.7   | 0.29   | 0.91       |\n| NeRF (6 views)       | 20.15 | 0.69  | 0.22   | 1.08       |\n| CorresNeRF (6 views) | 21.51 | 0.74  | 0.22   | 0.85       |\n\nThe table indicates that CorresNeRF consistently outshines the baseline NeRF model, regardless of whether 3 or 6 views are used. Given that CorresNeRF is a plug-and-play module that can be added to any NeRF, provided quality image correspondences are available, its addition can boost the performance of NeRF, even in dense-view configurations.\n\n### Q3 (from weaknesses): Comparison with other baselines, such as SparseNeuS.\n\nIn summary, as long as reliable image correspondences can be established, CorresNeRF can serve as a generic plug-and-play module to enhance the performance of any NeRF model, including SparseNeuS.\n\nSparseNeuS learns generalizable priors from image features, encoding coarse-to-fine geometry volumes for generic surface prediction. These methods are distinct from the image correspondence priors used in CorresNeRF. Consequently, SparseNeuS can be integrated with CorresNeRF to further enhance performance. We plan to incorporate additional experiments combining CorresNeRF with SparseNeuS in the final version of the paper.\n\n### Q4 (from questions): Clarification on the image correspondence methods, including the selection of the method.\n\nWe assessed CorresNeRF's performance using correspondences derived from different image matching techniques, namely LoFTR [55] and DKMv3 [56]. The LLFF dataset served as our testing ground.\n\n| Method                  | PSNR↑ | SSIM↑ | LPIPS↓ | Depth MAE↓ |\n| ----------------------- | ----- | ----- | ------ | ---------- |\n| NeRF                    | 16.79 | 0.56  | 0.37   | 1.66       |\n| CorresNeRF (with LoFTR) | 18.13 | 0.64  | 0.30   | 1.10       |\n| CorresNeRF (with DKMv3) | 19.83 | 0.70  | 0.29   | 0.91       |\n\nThe results show that CorresNeRF can outperform the vanilla NeRF, irrespective of the image matching technique employed, as long as quality correspondences are acquired. Additionally, CorresNeRF benefits from a \"free\" performance boost when a superior image matching method is employed, offering avenues for further enhancing CorresNeRF's performance.\n\n### Q5 (from questions): Robustness to the noise/outliers in the correspondence, and what is the minimum quality of correspondence necessary for the method to outperform traditional NeRF implementations?\n\nPlease kindly refer the answer above for Q1.\n\n### Q6 (from questions): Impact of loss terms. How do pixel reprojection loss and correspondence depth loss affect the final results?\n\nWe believe that the pixel reprojection loss is more closely associated with the image matching method. This loss is defined in the 2D image space and is directly tied to the 2D image correspondences. In contrast, the depth loss is defined in the 3D space and has a strong connection to the NeRF model, given that the rendered depth in NeRF is the weighted sum of points along the camera ray. Based on our ablation study (Table 3 in the main paper), both the pixel loss and depth loss contribute significantly to the performance of CorresNeRF.\n\n### Q7 (from questions): Extreme and failure cases\n\nAs described in the limitations section of the paper, CorresNeRF is dependent on the results produced by the image matching method. If the input surface is glossy, specular, or lacks texture, the image matching method may fail to generate accurate correspondences. Moreover, such correspondences could be eliminated by the automated outlier removal process, which is based on camera reprojection errors. In these scenarios, the enhancements offered by CorresNeRF over the baseline NeRF model might be minimal."}, {"review_id": "pS6IaanfTk", "reviewer": "Reviewer_NREe", "summary": "The paper presents NeRF regularization method for few view NeRF.", "questions": "Can the author show the proposed method is better than Neuris?", "limitations": "The image matcher sometimes fails when there are not enough overlap regions.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Using the state-of-the-art image matcher to regularize NeRF training is novel.\n2. The paper is well-written and clear.", "weaknesses": "1. The paper proposes employing the cutting-edge image matcher to enhance NeRF training. However, a similar idea was presented in Neuris [1], which also suggested using patch matching to optimize NeRF training. Intuitively, one could assume that a state-of-the-art image matcher would identify more precise correspondences than patch match, leading to superior results. However, considering that Neuris integrates additional monocular depths and surface normals, the effectiveness of combining these three methods remains uncertain. Therefore, the author appears to have overlooked an essential baseline, Neuris. It is recommended that the author carry out experimental work based on the Neuris setting rather than implementing Neuris in their own setup, which make the conclusion more convincing.\n\n2. The pixel loss and depth loss appear to aim towards the same goal. The concept of using both has been previously explored in DSAC [2], but was later discarded in DSAC++ [3], deemed as unnecessary. While the author provides an ablation study to illustrate the effectiveness of the reprojection loss, its value remains questionable. This is primarily because, in multiview settings, the reprojection loss mirrors the depth loss, making its unique contribution uncertain.\n\n3. Minor, The citation of UNISURF in Figure 5 seems to be wrong. \n\n[1] NeuRIS: Neural Reconstruction of Indoor Scenes Using Normal Priors\n[2] DSAC-Differentiable RANSAC for Camera Localization\n[3] Visual Camera Re-Localization from RGB and RGB-D Images Using DSAC", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1: Comparison with Neuris.\n\nWe have examined the Neuris method and believe its approach of utilizing normal and monocular depth priors complements CorresNeRF. Notably, CorresNeRF serves as a plug-and-play module applicable to any NeRF, provided reasonable image correspondences can be achieved. Consequently, the image correspondence priors in CorresNeRF can integrate seamlessly with the normal and monocular depth priors in Neuris.\n\nMoreover, the Neuris paper predominantly focuses on indoor scenes, having been evaluated solely on the ScanNet dataset. It requires a monocular depth and surface normal models that are specifically trained for indoor settings. Conversely, CorresNeRF's image matching methods are versatile, catering to both indoor and outdoor scenes.\n\n### Q2: Pixel loss and depth loss comparison.\n\nWe thank the reviewer for the question. We believe that the pixel reprojection loss is more closely associated with the image matching method. This loss is defined in the 2D image space and is directly tied to the 2D image correspondences. In contrast, the depth loss is defined in the 3D space and has a strong connection to the NeRF model, given that the rendered depth in NeRF is the weighted sum of points along the camera ray. Based on our ablation study (Table 3 in the main paper), both the pixel loss and depth loss contribute significantly to the performance of CorresNeRF.\n\n### Q3: Minor Citation issue of UNISURF.\n\nWe thank the reviewer for pointing this out. We have fixed the citation issue."}, {"review_id": "UkGzh2AbQh", "reviewer": "Reviewer_1K2a", "summary": "The submission #8302, entitled \"CorresNeRF: Image Correspondence Priors for Neural Radiance Fields\" proposes a novel set of losses to improve the quality of NeRF under challenging conditions. In particular, the developed strategy effectively deals with the problem of sparse images. To achieve these performances, the authors take advantage of an out-of-the-box matching strategy between pairs of images to enforce geometric constraints during the training of the implicit representation. In particular, two types of losses are proposed to improve the quality of the reconstruction, namely the reprojection loss and the depth loss.\nAn extensive series of experiments demonstrates the relevance of these extra losses incorporated into the training. Another advantage of the proposed strategy is that it can easily be integrated into most implicit reconstruction techniques.", "questions": "- An ablation study with an increasing number of images could be interesting to analyze how the method scales and if the demonstrated improvement are also valid with a larger density of images.\n- Analysis with different matcher/keypoints would be a plus.\n- What is the effect of the density of matched points? For instance, if a very sparse SIFT matching was used, what would be the expected effect of that?\n- Exploring other types of loss would be interesting, for instance, some epipolar losses instead of the reprojection loss.", "limitations": "I have very little to say about this paper as it is very clear and straightforward. I would like to kindly recommend additional experiments, as explained in the previous section. ", "rating": 5, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written and straightforward\n- The approach is simple and can easily be integrated into most NeRF-based approaches leading to improved results\n- The proposed technique is very effective under sparse view constraints\n- The computational overhead is very limited as matching strategies are often very fast", "weaknesses": "- The approach is very simple; the losses in themselves are not really new but demonstrate very effective results. The contributions appear to be limited, but the quality of the results might justify an acceptance. For this reason, I would like to express a mixed opinion regarding the acceptance of this work. Note that a relatively similar loss (on 3D structure obtained via correspondences) is applied in \"Structure-Aware NeRF without Posed Camera via Epipolar Constraint\" but with less success than in this manuscript #8302.\n- CorressNeRF demonstrates good performance when few images are used, but it would be interesting to know the effect of these losses in more common scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1: Ablation study with an increasing number of images.\n\nWe thank the reviewer for the suggestion. We tested the robustness of CorresNeRF with varying input view counts. Specifically, we doubled the number of input views from 3 to 6 and then evaluated CorresNeRF's performance on the LLFF dataset.\n\n| Method               | PSNR↑ | SSIM↑ | LPIPS↓ | Depth MAE↓ |\n| -------------------- | ----- | ----- | ------ | ---------- |\n| NeRF (3 views)       | 16.79 | 0.56  | 0.37   | 1.66       |\n| CorresNeRF (3 views) | 19.83 | 0.7   | 0.29   | 0.91       |\n| NeRF (6 views)       | 20.15 | 0.69  | 0.22   | 1.08       |\n| CorresNeRF (6 views) | 21.51 | 0.74  | 0.22   | 0.85       |\n\nThe table indicates that CorresNeRF consistently outshines the baseline NeRF model, regardless of whether 3 or 6 views are used. Given that CorresNeRF is a plug-and-play module that can be added to any NeRF, provided quality image correspondences are available, its addition can boost the performance of NeRF, even in dense-view configurations.\n\n### Q2: Analysis with different matcher/keypoints. Does it work with very sparse correspondence, e.g., SIFT matching?\n\nWe thank the reviewer for the question. We assessed CorresNeRF's performance using correspondences derived from different image matching techniques, namely LoFTR [55] and DKMv3 [56]. The LLFF dataset served as our testing ground.\n\n| Method                  | PSNR↑ | SSIM↑ | LPIPS↓ | Depth MAE↓ |\n| ----------------------- | ----- | ----- | ------ | ---------- |\n| NeRF                    | 16.79 | 0.56  | 0.37   | 1.66       |\n| CorresNeRF (with LoFTR) | 18.13 | 0.64  | 0.30   | 1.10       |\n| CorresNeRF (with DKMv3) | 19.83 | 0.70  | 0.29   | 0.91       |\n\nThe results show that CorresNeRF can outperform the vanilla NeRF, irrespective of the image matching technique employed, as long as quality correspondences are acquired. Additionally, CorresNeRF benefits from a \"free\" performance boost when a superior image matching method is employed, offering avenues for further enhancing CorresNeRF's performance.\n\n- [55] LoFTR: Detector-Free Local Feature Matching with Transformers, CVPR 2022\n- [56] DKM: Dense Kernelized Feature Matching for Geometry Estimation, CVPR 2023\n\nTo study the effect of the density of matched points, we obtained correspondences using image matching methods and subsequently sampled a subset (50%, 25%, 12.5%, 6.25%, and 3.125%) of these correspondences to train CorresNeRF. We then assessed CorresNeRF's performance on the LLFF dataset.\n\n| Method                          | PSNR↑  | SSIM↑ | LPIPS↓ | Depth MAE↓ |\n| ------------------------------- | ------ | ----- | ------ | ---------- |\n| NeRF                            | 16.79  | 0.56  | 0.37   | 1.66       |\n| CorresNeRF (with 3.125% corres) | 18.616 | 0.647 | 0.322  | 1.129      |\n| CorresNeRF (with 6.25% corres)  | 18.934 | 0.657 | 0.299  | 1.113      |\n| CorresNeRF (with 12.5% corres)  | 18.854 | 0.66  | 0.287  | 1.108      |\n| CorresNeRF (with 25% corres)    | 19.068 | 0.669 | 0.269  | 1.10       |\n| CorresNeRF (with 50% corres)    | 18.986 | 0.67  | 0.266  | 1.085      |\n| CorresNeRF (with 100% corres)   | 19.83  | 0.70  | 0.29   | 0.91       |\n\nNotably, even with only 3.125% of the correspondences, CorresNeRF significantly outperforms the baseline NeRF model. The performance of CorresNeRF enhances as the correspondence quantity increases. When 100% of the correspondences are used, CorresNeRF achieves its peak performance. Thus, as long as quality correspondences are provided in adequate numbers, CorresNeRF can surpass the regular NeRF's performance.\n\n### Q3: Exploring other types of loss, e.g. epipolar loss.\n\nWe thank the reviewer for the suggestion. The epipolar loss, as discussed in the Structure-Aware NeRF paper, bears similarities to the reprojection loss used in CorresNeRF. We will delve deeper into the intricacies of the epipolar loss in the finalized version of our CorresNeRF paper."}]

Author rebuttal:
We thank all the reviewers for their insightful comments and questions. In this section, we provide a summary of our responses and present new experimental results. Specifically, we introduce new experiments that examine:

- The robustness of CorresNeRF with noisy correspondences
- The robustness of CorresNeRF when using fewer correspondences
- The performance of CorresNeRF with additional input views (3-view and 6-view)
- The performance of CorresNeRF when using different image matchers

## Robustness of CorresNeRF with Noisy Correspondences

We employed image matching methods to obtain correspondences and subsequently introduced Gaussian noise to these correspondences. Specifically, we added Gaussian noise with standard deviations of 1, 2, and 4 pixels to both x and y pixel coordinates of the correspondences. We then assessed the performance of CorresNeRF using the LLFF dataset.

Section 3.2 describes how CorresNeRF employs an automatic outlier removal process based on camera reprojection error. Column 2 in the table below reports the relative number of correspondences remaining after this filtering process.

| Method                       | Corres # After Auto Filter | PSNR↑ | SSIM↑ | LPIPS↓ | Depth MAE↓ |
| ---------------------------- | -------------------------- | ----- | ----- | ------ | ---------- |
| NeRF                         | 0.00%                      | 16.79 | 0.56  | 0.37   | 1.66       |
| CorresNeRF (noise_std = 4px) | 13.96%                     | 18.31 | 0.61  | 0.48   | 1.04       |
| CorresNeRF (noise_std = 2px) | 27.04%                     | 19.16 | 0.66  | 0.33   | 1.06       |
| CorresNeRF (noise_std = 1px) | 48.91%                     | 19.31 | 0.67  | 0.28   | 1.06       |
| CorresNeRF (noise_std = 0px) | 100.00%                    | 19.83 | 0.70  | 0.29   | 0.91       |

When image correspondences are contaminated by noise, the automatic outlier removal process discards more of them, resulting in fewer correspondences for CorresNeRF to utilize. However, the correspondences that remain are deemed higher quality. Consequently, CorresNeRF maintains satisfactory performance even with noisy correspondences.

## Robustness of CorresNeRF with Reduced Correspondence Quantity

We obtained correspondences using image matching methods and subsequently sampled a subset (50%, 25%, 12.5%, 6.25%, and 3.125%) of these correspondences to train CorresNeRF. We then assessed CorresNeRF's performance on the LLFF dataset.

| Method                          | PSNR↑  | SSIM↑ | LPIPS↓ | Depth MAE↓ |
| ------------------------------- | ------ | ----- | ------ | ---------- |
| NeRF                            | 16.79  | 0.56  | 0.37   | 1.66       |
| CorresNeRF (with 3.125% corres) | 18.616 | 0.647 | 0.322  | 1.129      |
| CorresNeRF (with 6.25% corres)  | 18.934 | 0.657 | 0.299  | 1.113      |
| CorresNeRF (with 12.5% corres)  | 18.854 | 0.66  | 0.287  | 1.108      |
| CorresNeRF (with 25% corres)    | 19.068 | 0.669 | 0.269  | 1.10       |
| CorresNeRF (with 50% corres)    | 18.986 | 0.67  | 0.266  | 1.085      |
| CorresNeRF (with 100% corres)   | 19.83  | 0.70  | 0.29   | 0.91       |

Notably, even with only 3.125% of the correspondences, CorresNeRF significantly outperforms the baseline NeRF model. The performance of CorresNeRF enhances as the correspondence quantity increases. When 100% of the correspondences are used, CorresNeRF achieves its peak performance. Thus, as long as quality correspondences are provided in adequate numbers, CorresNeRF can surpass the regular NeRF's performance.

## Performance of CorresNeRF with more input views (3-view and 6-view)

We tested the robustness of CorresNeRF with varying input view counts. Specifically, we doubled the number of input views from 3 to 6 and then evaluated CorresNeRF's performance on the LLFF dataset.

| Method               | PSNR↑ | SSIM↑ | LPIPS↓ | Depth MAE↓ |
| -------------------- | ----- | ----- | ------ | ---------- |
| NeRF (3 views)       | 16.79 | 0.56  | 0.37   | 1.66       |
| CorresNeRF (3 views) | 19.83 | 0.7   | 0.29   | 0.91       |
| NeRF (6 views)       | 20.15 | 0.69  | 0.22   | 1.08       |
| CorresNeRF (6 views) | 21.51 | 0.74  | 0.22   | 0.85       |

The table indicates that CorresNeRF consistently outshines the baseline NeRF model, regardless of whether 3 or 6 views are used. Given that CorresNeRF is a plug-and-play module that can be added to any NeRF, provided quality image correspondences are available, its addition can boost the performance of NeRF, even in dense-view configurations.

## Using Different Image Matchers for CorresNeRF

We assessed CorresNeRF's performance using correspondences derived from different image matching techniques, namely LoFTR [55] and DKMv3 [56]. The LLFF dataset served as our testing ground.

| Method                  | PSNR↑ | SSIM↑ | LPIPS↓ | Depth MAE↓ |
| ----------------------- | ----- | ----- | ------ | ---------- |
| NeRF                    | 16.79 | 0.56  | 0.37   | 1.66       |
| CorresNeRF (with LoFTR) | 18.13 | 0.64  | 0.30   | 1.10       |
| CorresNeRF (with DKMv3) | 19.83 | 0.70  | 0.29   | 0.91       |

The results show that CorresNeRF can outperform the vanilla NeRF, irrespective of the image matching technique employed, as long as quality correspondences are acquired. Additionally, CorresNeRF benefits from a "free" performance boost when a superior image matching method is employed, offering avenues for further enhancing CorresNeRF's performance.

[55] LoFTR: Detector-Free Local Feature Matching with Transformers, CVPR 2022
[56] DKM: Dense Kernelized Feature Matching for Geometry Estimation, CVPR 2023"


Meta-review: The paper has some good ideas on the use of correspondences as priors, and achieves good results. The majority of the reviewers were positiveThe AC believes that the authors successfully rebutted the two negative reviewers.

It would be beneficial to include the CVPR 23 paper "SPARF: Neural Radiance Fields from Sparse and Noisy Poses" in related work, as the ideas are very similar, even though it was published after the submission.
Decision: Accept (poster)