Learning Geometry Consistent Neural Radiance Fields from Sparse and Unposed Views

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 12:11:18

Abstract: The recent progress in novel view synthesis is attributed to the Neural Radiance Field (NeRF), which requires plenty of images with precise camera poses. However, collecting available dense input images with accurate camera poses is a formidable challenge in real-world scenarios. In this paper, we propose Learning Geometry Consistent Neural Radiance Field (GC-NeRF), to tackle this challenge by jointly optimizing a NeRF and camera poses under sparse (as low as 2) and unposed images. First, GC-NeRF establishes the geometric consistencies in the image-level, which produce photometric constraints from inter- and intra-views for updating NeRF and camera poses in a fine-grained manner. Second, we adopt geometry projection with camera extrinsic parameters to present the region-level consistency supervisions, which construct pseudo-pixel labels for capturing critical matching correlations. Moreover, GC-NeRF presents an adaptive high-frequency mapping function to augment the geometry and texture information of the 3D scene. We evaluate the effectiveness of GC-NeRF, which sets a new state-of-the-art in the sparse view jointly optimized regime on multiple challenge real-world datasets.

Main paper: 
 related work: 
 We mainly introduce two different aspects of researches for NeRF, i.e., the novel view images synthesis and the camera and extrinsic parameters estimation, which are relevant to our method. Novel View Synthesis for NeRF. In recent years, Neural Radiance Field (NeRF) [21] has become the favored image rendering because of its remarkable implicit scene representation. Numerous methods [2, 31, 32, 40, 45, 47] are presented to improve the performance of NeRF. Typically, a number of researchers have integrated depth or point cloud to supervise the training and rendering stage [7, 26] . Moreover, an additional regularisation is another effective strategy to augment the photometric quality [14, 43, 49] . Recently, many innovative constructs have accelerated training and rendering [9, 11, 22] . All of these models achieve fantastic results. However, most of them rely on sufficient images, which are indispensable to pre-computed camera extrinsic parameters.Different from the above approaches, we exploit the geometric consistency relations between input views, which enables the novel view synthesis for implicit scene representation with only two unposed images.Pose Estimation for NeRF. Weaning NeRF off camera parameter preprocessing is a significant attempt for optimizing NeRF nowadays [20, 44, 46] . Based on the SLAM technique, some researchers employ RGB-D data or exploit SLAM's tracking capabilities to pre-calculate the camera poses [27, 36, 53] . Moreover, the other methods focus on learning NeRF model and camera poses jointly [1, 42] . iNeRF [18] sets camera poses as learnable parameters, which can be optimized by minimizing the residual between the predicted and ground truth pixels. BARF [17] combines bundleadjustment strategy and NeRF to solve the joint problem of pose estimation and NeRF model. SCNeRF [13] jointly learns the scene and the camera parameters by self-calibration algorithms. GARF [4] employs Gaussian activation to improve the joint optimization of pose and scene. A mount of works [3, 10, 29, 48] adopt the generative network or pose/depth estimators to predict camera poses before learning the camera and NeRF model.Although these solutions show infusive performance, they still suffer from indispensable pose initialization or sufficient images. Toward the end, We jointly optimize camera parameters and NeRF by image-and region-level supervision in sparse views settings. 
preliminaries: 
 Neural radiance field (NeRF) [21] represents continuous scenes as regressing from a single 5D coordinate to view-dependent RGB color, which can be optimized by minimizing photometric supervision as follows:EQUATIONwhere Θ is a basic MLP network. Î is the predicted image.[R|t] = Γ ∈ SE(3), R ∈ SO(3), t ∈ R 3 , and K ∈ R 3×3 represented camera extrinsic and intrinsic parameters, respectively. Particularly, we can synthesize the color 𝐶 of pixel p in Î along the ray from 𝑡 𝑛 to 𝑡 𝑓 as follows:EQUATIONwhere 𝜎 (r p (𝑡)) ∈ R and c(r p (𝑡), d p ) ∈ R 3 denote volume density and RGB color of 3D location x = r p (𝑡), respectively. d p ∈ S 2 is the ray direction, which is calculated by camera parameters [R|t] and K, and pixel p.𝑇 (𝑡) = exp(-∫ 𝑡 𝑡 𝑛 𝜎 (r p (𝑠))𝑑𝑠) indicates the accumulated transmittance. Notably, we can predict a view-invariant volume density 𝜎 and a view-dependent color c according to 3D location x and view direction d by an MLP neural network 𝑓 Θ such that:EQUATIONwhere 𝛾 (•) : R 3 → R 2𝐿 denotes the positional encoding function under 𝐿 harmonics. Finally, a photometric rendering loss could optimize Θ by Eq. (1). 
method: 
 In this section, we first elucidate the motivation and method overview in Section 4.1. Then we report the framework of GC-NeRF in Section 4.2 -4.6.This work aims to improve novel view synthesis by NeRF from sparse and unposed camera images. Specifically, we address on the challenging task of synthesizing novel views using only sparse input images (as low as 2 views), without requiring accurate or specific initial camera poses. This results in two critical challenges. Firstly, only a few input images for NeRF [21] in training will overfit instantly, which leads to terrible rendering quality of meaningful 3D geometry, even with ground-truth camera parameters [39] . Secondly, current jointly optimizing for pose and NeRF approaches focus on dense views of 3D scenes [1] , along with relying on proper pose initialization. Especially, the above challenges will restrain each other, such as the existing photometric supervision is weak to perceive 3D geometric scenes from sparse views. Thus failure to get rid of poses under sparse input views.Towards this end, we propose a novel Learning Geometry Consistent Neural Radiance Field method, dubbed GC-NeRF, which combines image-and region-level consistencies to overcome the joint optimization problem. And an adaptive positional encoding strategy is designed for augmenting high-frequency perception.Specifically, given a pair of images 𝐼 𝑆 and 𝐼 𝑇 , along with the correspondences between matching points in two views, we establish geometric consistency at both the image-and region-level.For the image-level consistencies, we propose an adjacent photometric supervision that leverages photometric rendering in standard NeRF. Specifically, we employ epipolar geometry to provide pseudo color labels p 𝑆→𝑇 and p 𝑇 →𝑆 for the rendered images Î𝑇 and Î𝑆 , respectively, from the corresponding target and source views. The adjacent photometric supervision enables to optimize NeRF model and camera poses simultaneously.For the region-level consistencies, we consider the matching points relations of the input image pair 𝐼 𝑆 and 𝐼 𝑇 as re-projection labels for key pixels. As the Lambertian model, we aim to ensure that the re-projection of a pixel p 𝑆 onto the target image p𝑆→𝑇 ideally corresponds to the matched pixel with the same color. We establish this geometric relationship on the rendered images Î𝑆 and Î𝑇 of the image pair, facilitating the joint optimization of the NeRF model and camera poses.Finally, we introduce the adaptive fusion random Fourier features (AdaRFF) positional encoding module based on the random Fourier features (RFF) transformation, which adaptively augments high-frequency variations of the scene when mapping coordinates to a higher-dimensional space. This framework overcomes the limitations of previous sparse view setting approaches, improving not only the global geometry but also the high-frequency details and appearance of the rendered images.In the following, we elaborate the pipeline of GC-NeRF, including the Image Level Consistency, Region Level Consistency, Adaptive High Frequency Positional Encoding.To solve the joint optimization problem of NeRF model and camera poses, we first initialize camera pose parameters R ∈ SO(3) as the quaternions q(w, x, y, z) to represent a 3D rotation. Different from the previous works to initialize with the axis-angle [42] , the 6-vector representations [13, 39] or the Euler angles [1] , the quaternions strategy only has 4 parameters rather than 6 without Gimbal lock problem [34] , which alleviates the gradients backward to update the three degrees of freedom rotation effectively. The transformation from a quaternion q(w, x, y, z) to a rotation matrix R ∈ R 3×3 is as follows:EQUATIONThe image-level consistency consists of two aspects: the intra-view relations between rendering images and ground truth (GT) images at the same poses, as well as the inter-view consistency associations between rendered images and the GT from adjacent images. Image Photometric Supervision. The image photometric loss is a critically acclaimed strategy to bridge the gap between the rendered and the GT images at the same viewpoint, which adopts the squared error of intra-view pixel color to supervise NeRF model convergence [1] . The details are as follows:EQUATION)where R is the set of rays in each batch. 𝐶 (p) and Ĉ (p) denote the ground truth and predicted RGB colors of pixel p for ray r by Eq. ( 2), respectively. Adjacent Photometric Supervision. Directly employing sparse or unposed image settings will lead to corrupted NeRF [21] collapsing with overfitting. Because only the image photometric loss (Eq. ( 5)) is impossible to support the harsh demands with the limited ground-truth from the intra-view pixel color. We present a training supervision upon inter-view images, adjacent photometric loss, to borrow enough constrains from the adjacent view for augment perceiving multi-view consistency for 3D scenes.We get the inspiration from epipolar geometry to augment the geometric correspondences of input views. Given two images 𝐼 𝑆 and 𝐼 𝑇 and their rendered images Î𝑆 and Î𝑇 with source and target viewpoints Γ 𝑆 and Γ 𝑇 , respectively. We can warp Î𝑆 to pose [R 𝑇 |t 𝑇 ] by the transformation as follows: 𝑄 𝑆 = R 𝑆 t 𝑆 -1 (𝐷 𝑆 K -1 𝑆 p 𝑆 ), 𝑄 𝑆→𝑇 = R 𝑇 t 𝑇 𝑄 𝑆 , p 𝑆→𝑇 = Norm(K 𝑇 𝑄 𝑆→𝑇 ),NeRF Model Θ Γ ! Γ " 𝐼 ' ! 𝐼 ' " 𝑄 ) ! 𝑄 ) " 𝐼 ' !→" 𝐼 ' "→! 𝐼 ! 𝐼 " 𝐼 ' "Eq. (7) Eq. ( 8) Eq. ( 9) Eq. ( 5)Eq. ( 5)Eq. ( 7)[𝐑|𝐭] [𝐑|𝐭] $% Scene 𝐼 ' ! 𝐼 ' !→" 𝐼 ! 𝐼 " 𝐼 ! 𝐼 ' ! 𝐼 ' !→" 𝑄 ) Γ Camera pose Predicted 3D Point Rendered Image Ground Truth ImageRe-projected Image We first establish geometric relations between the image pair to construct photometric supervisions from intra-and inter-view in the image-level (Eq. ( 7)). For region-level consistencies, we present a novel photometric rendering loss (matching projected loss) between the projected pixels and the matching pixels in the rendered images (Eq. ( 8)). We also bridge the gap of 3D points for matching pixels to supervise the optimization of camera extrinsic parameters and the density field (Eq. ( 9)).viewpoint Γ 𝑇 . After that, we can obtain the warped image Î𝑆→𝑇 . To optimize pose and NeRF model jointly, we then assess the squared error between Ĉ (p 𝑆→𝑇 ) in Î𝑆→𝑇 and the pseudo color label 𝐶 (p 𝑇 ) in 𝐼 𝑇 as follows:EQUATION)where 𝑚 is the mask to filtrate the projected pixels out-of-bounds [5] .We also calculate the errors between Ĉ (p 𝑇 →𝑆 ) in Î𝑇 →𝑆 and the pseudo color label 𝐶 (p 𝑆 ) in 𝐼 𝑆 . The Eq. ( 6) is differentiable during the training stage. Thus the adjacent photometric supervision can pass the gradients to camera parameters [R|t] of image pairs, while refining rendering quality by the backward for NeRF model. Of utmost importance, the occlusion problem of multi-view images will transform distorted or wrong semantics images Î𝑆→𝑇 and Î𝑇 →𝑆 , which affect the accuracy of pseudo color labels sincerely. We present a technique to compare the projected depth 𝑑 𝑟 from p 𝑆 to p 𝑆→𝑇 and the rendered depth 𝑑 𝑝 of p 𝑆→𝑇 in 𝐼 𝑇 . Then we reserve the pixels with a threshold 𝛽 < 𝑑 𝑟 𝑑 𝑝 < 1 𝛽 (𝛽 ≤ 1). Till here, the unambiguous gradients that are calculated from the adjacent view by Eq. ( 7) can further facilitate the joint optimization task.The image-level consistency favors the global supervision for 3D scenario perception across input views. However, we assume that access to only sparse views (i.e., 2 views) with unposed camera pose predicts (i.e., initialized with identity). Thus the above consistency losses are not sufficient to fulfill the joint optimization completely. We present additional supervisions from the other aspect, regionlevel consistencies, to augment 3D geometric analysis robustly and consistently.Matching Association. The region-level consistency depicts the geometric affiliation of key pixels among multi-view images. We adopt the key points matching strategy, a pre-trained correspondence regression method (LoFTR [37] ), to extract a matching point p 𝑇 for each pixel p 𝑆 . Empirically, any learned based [8, 12] or classical [19, 28] matching solver can be qualified to establish matching associations of input views. Logically, the greater the number of matching points with high accuracy, the greater the facilitation. However, for augmenting NeRF joint optimization significantly, we only need 1% -2% of the total pixel count as matching points to be converged, rather than 10% or even more matching points, which will be illustrated in Section 5. 4 .Matching Projected Supervision. The core idea is to calculate whether the color of the re-projected pixel p𝑆 is consistent with the color of the matching point p 𝑇 in Î𝑇 . We sample an image pair 𝐼 𝑆 and 𝐼 𝑇 . Then we can obtain the matching sequence M. Specifically, for pixel p 𝑆 in the viewpoint Γ 𝑆 , we will get the corresponding projected pixel p𝑆 following Eq. ( 6) in the viewpoint Γ 𝑇 . In the rendering image Î𝑇 , the matching pixels p 𝑇 and p𝑆 manifest the same color while camera pose parameters and NeRF model are convergence. Therefore, we narrow the pose discrepancy by supervising the matching pixel colors' consistency. We formulate our matching projected loss as follows:EQUATION)where p𝑆 is the re-projected pixel of p 𝑆 in viewpoint Γ 𝑇 . p 𝑇 denotes the matching pixel of p 𝑆 in viewpoint Γ 𝑇 , and 𝑐 is the confidence of the matching points.For the multi-view occlusion problem, previous work [5] could alleviate it properly by the mask strategy. However, under the sparse and unposed images settings, the mentioned scheme can not distinguish the occlusion phenomenon such two aspects: firstly, it is a real occlusion circumstance in the real world scenario. Secondly, it is most likely that the pose is inaccurate because the NeRF model and camera parameters have not converged, thereby calculating "pseudo-occlusion". This greatly hinders NeRF's ability to perceive scene 3D geometric information. Specifically, GC-NeRF establishes matching projected consistency supervision by matching points can effectively alleviate this problem.Moreover, compared with the re-projected error [39] , the matching projected supervision has ability to update the fully connected layers for the density 𝜎 and color c of 3D locations, and camera parameters Γ simultaneously. However, the former only learned the density and camera parameters, which enjoys a weaken capacity for NeRF and pose joint optimization. We therefore present this matching projected supervision for region consistency.Space Similarity Supervision. Different from the matching projected supervision in Eq. ( 8), we also build the consistency loss function directly based on projected 3D coordinates to enhance NeRF for perceiving the region's geometric information. Benefiting from the matching sequence M, the 3D coordinates Q𝑆 and Q𝑇 can be predicted from the corresponding matching points under Γ 𝑆 and Γ 𝑇 . Before the model converges, there will be a spatial position deviation in the 3D coordinates calculated for the same pair of matching points p 𝑆 and q 𝑇 from two viewpoints. We present the 3D space similarity supervision to reduce this bias. The specific calculation is as follows:EQUATIONwhere Q ∈ R 3×1 is calculated by Eq. ( 6), which omits subscript index. Due to the space similarity supervision, We can make the two point clouds infinitely close until they are almost identical, that is the camera pose is fitted as accurately as possible.NeRF heavily relies on a positional encoding module to map the inputs into a high-dimensional space. However, recent work by Rahaman [24] has revealed that deep networks exhibit a bias toward learning lower-frequency functions. Thus a simple Fourier feature transformation that is commonly used in the standard NeRF [21] , which fails to capture high-frequency information effectively [38] .The failure of the positional encoding module drastically reduces the ability to represent the details of geometry and texture of the scene (e.g. the rendered appearance becomes over-smoothed). This limitation is particularly severe for GC-NeRF, because it stems from the photometric and geometric constraints of the quality of rendered pixels' color. This phenomenon hinders the effectiveness of the image-and region-level geometric consistency supervisions.Drawing inspiration from [25] , we propose a novel approach called adaptive fusion random Fourier features (AdaRFF), by combining RFF with the conventional Fourier feature (FF) mapping. The presented adaptive fusion RFF component aims to enhance the capacity to learn high-frequency geometry and texture, thus enhancing the quality of scene perception and rendering. The RFF mapping function is calculated as follows:EQUATIONwhere l ∈ R 3 denotes the probability space, which is equipped with the Gaussian measure and non-zero standard deviation. We follow [21] to set 𝐿 = 10. Then we combine the conventional Fourier feature with RFF to introduce adaptive fusion weight coefficients g, which can dynamically filtrate the desired frequency bands for optimal scene representation. The details are as follows:EQUATIONEQUATIONwhere g ∈ R 2L denotes the adaptive coefficients. 𝜎 (•) is the sigma function. W 1 , W 2 ∈ R 2L×2L depict learnable parameters. • is the Hadamard product. And 𝛽 (•) and 𝛾 (•) denote RFF and the conventional Fourier feature mapping functions. Thus, GC-NeRF can choose the proper frequency bands and scale adaptively to reconstruct a high-fidelity neural scene in novel view synthesis.Objective Function. Assembling all loss terms, our overall training objective loss function is formulated as follows:EQUATION)where 𝜆 1 , 𝜆 2 , and 𝜆 3 are predefined weighting coefficients for each loss terms. By minimising the Eq. ( 13), our approach returns optimised camera poses Γ and NeRF's MLP network Θ.Training Pipeline. For GC-NeRF, there is only one MLP network rather than a coarse and a fine networks. The training is split into two stages. First, we only use the image photometric supervision Eq. ( 5) to assist camera pose parameters representing discrepancies between each other. Then Eq. ( 5) -Eq. ( 9) supervise together to establish multi-view geometry consistency for optimizing pose and NeRF parameters jointly. Second, we freeze camera pose parameters and finetune the NeRF model with four supervisions until rendering high quality novel view images. Simultaneously analyzing image pairs does not result in excessive computation for the training pipeline, as there are shared operations in color and depth rendering. 
experimental settings: 
 Datasets. We evaluate all baselines and our presented GC-NeRF on the Tanks and Temples [15] , LLFF [33] , and NeRF real 360 [21] real-world scene datasets, where we report the results of unknown and known pose methods, respectively. Tanks and Temples includes both outdoor scenes and indoor environments. Following [1] , we use 8 scenes to evaluate all methods. We chose scenes captured at both indoor and outdoor scenarios, and 1/8 of the images in each sequence are used for novel view synthesis. LLFF contains 8 sets of different real-life scenes, which are primarily captured in the forward-facing direction. Consistent with [21] , we evenly select the test set by extracting every 8th image from the video list. NeRF Real 360 is composed of 2 different complex object-level scenes, which are captured by a set of inward-facing views. Each scene has been taken from 99 and 116 positions, corresponding to the number of RGB images. We follow the protocol of [21] and evaluate 1/8 of the views for novel view synthesis.Metrics. We report the performances of Novel View Synthesis, Camera Pose Estimation, and Depth Estimation in three benchmarks. For Novel View Synthesis, following [1, 13, 17] , we select Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM) [41] , and Learned Perceptual Image Patch Similarity (LPIPS) [51] standard evaluation metrics. For Camera Pose Estimation, we employ Relative Pose Error (RPE) to calculate the relative pose errors between image pairs, which includes rotation and translation errors [16, 35, 52] . Notably, rotation errors are in degree, and translation errors are multiplied by 100. For Depth Estimation, we follow [39] to adopt the Mean Depth Absolute Error (MDAE) for comparing the rendered and the ground-truth depth values.Implementation Details. We implement our framework based on previous work [1, 21] . For GC-NeRF, 1024 rays will be calculated in every batch. And we sample 128 coordinates along each ray with a single MLP network. We initialize camera extrinsic parameters by: the rotation matrix R ∈ R 3×3 as E + Δ 𝑟 , the translation vector t ∈ R 3 as 0 + Δ 𝑡 . Moreover, we follow [1] to align test camera poses before the evaluation. As for baselines, we defer to the paper and source codes to conduct experiments with the same settings, which are under 2 input views and 20K iterations with about 4-5 hours on a single RTX 3090 GPU.We evaluate GC-NeRF and the SOTA unknown pose methods with 2 input views in different metrics. Moreover, more details results are reported in the supplementary.Baselines. We compare to four SOTA unknown pose baselines. BARF [17] , an effective method with a simple strategy for training NeRF from imperfect camera poses. SCNeRF [13] is a selfcalibration algorithm, which learns the scene and camera parameters jointly. NoPeNeRF [1] integrates depth additionally to establish relations between adjacent views for unposed NeRF. SPARF [39] jointly optimizes camera poses and scenes with three input views, which rely on noised rather than initializing poses with identity. For a fair comparison, we eliminate the GT depth in NoPeNeRF [1] .Results of Novel View Synthesis. As illustrated in Table 1 , for the quantitative results, GC-NeRF outperforms all baselines in three datasets with different metrics by a large margin. Specifically, compared with SPARF in the Tanks and Temples dataset, GC-NeRF has increased by 2.05, 0.10, and 0.07 with respect to PSNR, SSIM, and LPIPS. This phenomenon demonstrates that the presented various consistency strategies can augment the ability of NeRF model from sparse and unposed images. For the qualitative analysis of the image rendering, GC-NeRF outperforms the SOTA unposed NeRFs significantly under two input views, which are shown in Fig. 2 . This further illustrates the superiority of the geometric consistency framework to augment NeRF model and pose joint optimization.Results of Camera Pose Estimation. The performances of pose estimation on three datasets are illustrated in Table 2 . Compared with the SOTA approaches, our method achieves the best performance consistently. Specifically, in the Tanks and Temples scene, the accuracy of camera pose metrics for GC-NeRF is an order of magnitude higher than SPARF.Results of Depth Estimation. We evaluate the rendered depth on Tanks and Temples, LLFF, and NeRF Real 360 datasets, respectively. As shown in the MDAE metric of Table 2 , GC-NeRF performs superior accuracy for depth rendering, which precedes other baselines markedly. In addition, the qualitative visualization results on three datasets are exhibited in Fig. 2 .We report the performance of GC-NeRF and the SOTA known pose methods with 2 input views in novel view synthesis quality under fixed ground-truth camera parameters.Baselines. We compare to four designed for tackling sparse image rendering methods, which are DSNeRF [6] , RegNeRF [23] , SPARF [39] , along with the standard NeRF [21] . Especially, all baselines employ ground-truth camera poses.Results of Novel View Synthesis. We exhibit the performance on three datasets in Table 3 and Fig. 3 . Consequently, our approach GC-NeRF achieves the best performance compared to other baselines on all datasets and three rendering quality metrics. This phenomenon demonstrates that: Firstly, adopting image-level consistency covers the shortage of the traditional rendering losses, which enhances novel view synthesis performance. Secondly, the incorporation of region-level consistency significantly promotes NeRF's ability for depth estimation, thereby directly advancing the perception of complex scenes. Lastly, the AdaRFF positional encoding boosts high-frequency information within the images, maximizing the benefits of region consistency supervision. As a result, GC-NeRF achieves SOTA results even in the absence of GT poses. In this section, we conduct a comprehensive analysis of the components and hyper-parameters that are essential to our approach. All methods are evaluated on the LLFF [33] dataset.Effect of Image and Region Level Consistency. We ablate the key supervisions of our approach with novel view synthesis (NVS) and pose estimation (PE), here eliminating image-and regionlevel consistencies from GC-NeRF, respectively. We also replace the region-level supervisions with the re-projection error (w/ ReProE) to test the performance. The results are illustrated in Table 4 . We observe that the first row achieves the best performance. Because GC-NeRF establishes robustness correspondence among multi-view images, which render higher quality images and accurate camera poses simultaneously. Comparing the rest the rows with the standard NeRF [21] in Table 3 , the formers achieve better results, which demonstrates that the image-and region-level geometric supervisions are effective for NeRF and pose joint optimization. Impact of Positional Encoding Strategy. We examine the impact of the adaptive random Fourier feature (w/ AdaRFF) positional encoding on the image rendering and PE performances by removing or changing it to other alternatives. The results are listed in Table 5 . Without AdaRFF, the alternative strategies are the coarse-to-fine 1 and Table 2 , all variants achieve better performance. It verifies the superiority of the image-and region-level geometric consistency supervisions for NeRF and poses joint optimization. Amount of Training Data. In Table 1 , we evaluate our presented approach GC-NeRF for joint NeRF-pose optimization, when considering only 2 input views. For completeness, we here report results on LLFF data when 5 and 8 input views are available. The experimental settings are the same as 2-view settings, except for the number of training views. As illustrated in Table 6 , the trend is similar for 2, 5, and 8 input images. Although SPARF [39] yields better performance as the images increase, it still struggles with camera poses. GC-NeRF outperforms SPARF consistently in three groups, and both the rendering quality and camera parameters are more satisfactory. Because the image-and region-level supervisions provide sufficient multi-view geometry consistencies, which facilitate the optimization of NeRF model and pose parameters simultaneously.Amount of Matching Points. The matching points are the basis of GC-NeRF to construct region-level consistency. We evaluate the effectiveness of matching points with different quantities on the LLFF scene. In Fig. 4 , all metrics get better as the number of matching points gets bigger. The phenomenon depicts that both methods can improve the rendering quality as the matching gets denser. However, GC-NeRF will oscillate at a high level after reaching the threshold (i.e. 5k), because of calculating gradients for both pose and NeRF model. Yet SPARF need more than 35k to converge just for density field. Notably, the number of reliable matching points is extremely limited with viewpoint variations in the real-world scenario. It illustrates the availability and robustness of GC-NeRF. 
conclusion: 
 Current methods for novel view synthesis struggle to render highquality images and preserve high-frequency details, when trained from sparse and unposed images by the joint optimization of NeRF models and camera poses. This paper proposes learning geometry consistent neural radiance field from sparse and unposed views (GC-NeRF) for novel view synthesis. The presented GC-NeRF establishes the consistency relations from image-and region-level. The former adopts the epipolar geometry for generating pseudo-color labels to construct adjacent photometric supervision, which integrates intraand inter-views photometric constraints to augment the gradients for both NeRF and camera poses. The latter employs matching points to capture the re-projected color labels for facilitating the joint optimization of the NeRF and poses. The proposed adaptive fusion random Fourier features positional encoding module maps the input coordinates to a high dimension space adaptively, which effectively fits the high frequency variation of the 3D scene. We evaluate GC-NeRF and several state-of-the-art methods in three challenging real-world scenarios. The experimental results verify the effectiveness and robustness of the GC-NeRF. We also plan to extend our strategy to other NeRF's hotspot applications (e.g. relighting with NeRF from sparse and unposed views).  
: 
 Nevertheless, the remarkable performance of NeRF is heavily contingent on two critical requirements: dense input views and precise camera poses. These requirements impose significant limitations on the practical applicability of NeRF in real-world scenarios. In many cases, such as VR/XR applications, the availability of input images becomes sparse, often leaving only a few RGB images captured from specific viewpoints [39] . Consequently, NeRF may suffer from overfitting and produce unsatisfactory or even erroneous synthesized images [14, 23] . Additionally, although mature and convenient community repositories like COLMAP [30] exist to assist with camera pose estimation, which is an essential preprocessing step before NeRF training, the non-differentiable nature of these methods presents challenges when integrating them with NeRF. This limitation hinders the seamless integration between pose estimation and NeRF, impeding the joint optimization of NeRF and camera poses for improved performance. Furthermore, it increases NeRF's reliance on third-party libraries, consequently hindering research progress and practical deployment, particularly in scenarios with sparse views where accurate camera pose estimation using COLMAP becomes challenging [50] .Numerous researchers have made efforts to reduce NeRF's dependence on accurate camera poses. These efforts can be broadly classified into two approaches: deep network-updated pose optimization and estimator-based pose prediction. The former treats camera poses as learnable parameters, formulating novel view synthesis as a joint optimization problem of NeRF and camera poses [1, 13, 17, 42] . The latter approach involves incorporating a separate pose estimator to predict camera poses corresponding to the input images before feeding them into NeRF [3, 10, 29, 48] . While both approaches have shown promising potential, they heavily rely on having a sufficient number of input views, and their performance significantly deteriorates when dealing with sparse view images. Because the under-constrained nature of the 3D space limits the convergence of the deep network module to an optimal state due to insufficient training data coverage.Recently, a growing number of researches dedicated to enhancing the performance of NeRF under sparse view settings. Some methods focus on training conditional neural field models on largescale datasets [2, 47] . Additionally, various regularization techniques have been proposed to address the challenges posed by sparse views in terms of the color and geometry for the scene [6, 14, 23] . Although these researches show impressive performances, they have not completely gotten rid of the precise camera poses. It is worth noting the closest to our work is SPARF [39] , which is also under sparse view conditions. However, the latter excessively relies on the initialization of camera poses, training NeRF under noisy pose settings. It does not fundamentally alleviate the dependence on camera poses to achieve joint optimization of NeRF and poses.In this paper, we propose the Learning Geometry Consistent Neural Radiance Field (GC-NeRF), to optimize NeRF and camera poses jointly from sparse input views. We illustrate the framework of the GC-NeRF in Figure 1 , where the whole pipeline consists of the Image Level Consistency, Region Level Consistency, and Adaptive High Frequency Positional Encoding. For the Image Level Consistency, we establish correlations from intra-and inter-view images for constructing different color labels. Specifically, we first supervise the quality of image synthesis with the standard photometric rendering. Till here, we establish the projection geometric relationship between the source and target viewpoints, and provide pseudolabels from the source (target) view to the rendered image of the target (source) view. This approach constructs an adjacent photometric supervision to enhance the learning capability of camera poses in the joint optimization problem. For the Region Level Consistency, GC-NeRF localizes key regions of the source and target views by the correspondences between matching points across image pairs. Then we re-project the matching points to capture the pixel position labels and 3D coordinates labels. Consequently, GC-NeRF establishes a novel photometric rendering loss, matching projected supervision, between the projected pixels and the labeled pixels in the rendered images. This effectively augments the joint optimization performance. For the Adaptive High Frequency Positional Encoding, the proposed adaptive fusion random Fourier features (AdaRFF) strategy can map the coordinates to a high-dimensional space. It will adaptively filter high-frequency features into MLP network so that benefiting the NeRF model to render high-quality novel view images even with sparse input views.We evaluate GC-NeRF and the state-of-the-art baselines, which effectively outperform various methods to achieve the SOTA performance on the Tanks and Temples [15] , LLFF [33] , and NeRF Real 360 [21] datasets for sparse and unposed images NeRF optimization. The main contributions are summarized as follows:• We advocate the idea of GC-NeRF to optimize NeRF model from sparse and unposed images. • We propose effective supervisions of GC-NeRF, the imageand region-level consistencies, for establishing geometric correspondences for input views and augmenting the joint optimization capabilities of NeRF model and camera poses. • We present a novel position encoding method, adaptive fusion random Fourier features, for passing input coordinates to a higher dimension space and performing satisfactorily at representing high-frequency variation in the scene. • We conduct comprehensive analyses on GC-NeRF, which achieves the state-of-the-art results in several real-world scenarios, demonstrating the effectiveness of our method. 
 
 Appendix:  


Decision: The submission received 4 final ratings, with only 1 being negative. Positive reviewers basically acknowledge the novelty. Reviewer 8BYU, who gave a negative rating, feels that this is a careful rebuttal and states that 'the reviewer's name in the rebuttal is wrong. Second, I don't understand what "They are comparable because RGB rendering is only 40K model parameters than depth."'

After carefully checking the rebuttal, the AC also found the mistake pointed out by Reviewer 8BYU. However, the AC can connect the rebuttal and the review even if the reviewer's names are incorrectly presented, and the concerns have been well addressed.

Considering that the contribution of this work outweighs such a mistake in the rebuttal, the AC decided to support this work on condition that the authors will follow the review to revise the paper in the camera ready version.

Reviews: review: paper_summary: paper_summary: This paper proposes GC-NeRF, a method for learning neural radiance fields from sparse and unposed views. It introduces image-level and region-level consistency losses to jointly optimize the NeRF model and camera poses. An adaptive positional encoding strategy is also presented to better capture high-frequency details. Through the introduction of multiple losses, GC-NeRF enables NeRF to learn and generate high-quality novel views solely from a sparse set of images.
strength_weakness: strength_weakness: - The paper is overall clear and well-motivated. The proposed method (GC-NeRF) employs multiple losses where the justification of using each of the losses is clearly written.
- GC-NeRF achieves state-of-the-art performance in multiple datasets, showing large performance improvements compared to prior works.
- The proposed adaptive positional encoding seems like a useful enhancement to capture high-frequency scene details.
limitations: limitations: Although the work shows promising performance in multiple datasets, I have a few concerns I would like the authors would resolve. With my questions resolved, I will change my rating to a higher rating.

- Although the introduction of the proposed adaptive position encoding is novel and convincing, I did not understand the big performance gap compared to using the frequency annealing strategy adapted in SPARF[1] and BARF[2]. The concern stems from the large performance gap shown in Table 3 of the paper, showing 3.1dB gap with the Sparf encoding. Specifically, I would like to see a theoretical comparison of the proposed AdaRFF vs Frequency Annealing.
- Similar to the previous concern, I would like to see the ablation of removing the AdaRFF in GC-NeRF’s architecture. Comparing Table 2 and Table 3, replacing AdaRFF seems to show similar results with SPARF. To understand the effectiveness of other components and AdaRFF, I would like to see additional experiments of using SPARF + AdaRFF.
- If my understanding is correct, $L_{aps}$ and $L_{mps}$ seem to be similar as the training converges. Specifically, when  $p_s$ has a similar depth with the $p_{s \rightarrow t}$, this indicates the high chance that the two pixels come from the surface of the scene. This implicitly confirms that the two pixels should correspond in the rendering as well. From this knowledge, is there any special reason for adopting both losses? It seems that using $L_{mps}$ is enough.

[1] Truong, Prune, et al. "Sparf: Neural radiance fields from sparse and noisy poses." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[2] Lin, Chen-Hsuan, et al. "Barf: Bundle-adjusting neural radiance fields." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.
paper_summary: paper_summary: This paper proposes GC-NeRF for jointly optimizing NeRF camera poses under sparse and unposed image sets. GC-NeRF use geometric consistencies between training views, i.e. warp one image to another camera pose according to depth. It also uses region-level consistency that enforce pixel-level consistency between keypoints at different viewpoints. Finally, GC-NeRF presents an adaptive high-frequency mapping as an alternative to original positional embedding in NeRF. Results show that the proposed method achieve superior performance on multiple challenging datasets.

image- and region-level consistencies
strength_weakness: strength_weakness: - The adaptive positional encoding strategy designed for augmenting high-frequency perception is novel and interesting
- The results look quite impressive for 2-view inputs
limitations: limitations: - To my knowledge, the geometric and matching projected supervision is not novel, similar ideas have already been used in many NeRF papers before [1][2] to deal with the sparse view setting, which this paper haven't cited.
- One question is that when the camera pose is not accurate at the begining of training, wouldn't enforcing these geometry constraints introduce a lot of errors?
- No comparisons against more recent methods like SparseNeRF [3]
- There are no enough comparisons on using other number of views, e.g. 3 view or 6 view with other baselines, making me unconfident about the efficacy of the method
- The proposed method seem to be a bit slow, needs 4 hours for each scene

[1] GeCoNeRF: Few-shot Neural Radiance Fields via Geometric Consistency
[2] StructNeRF: Neural Radiance Fields for Indoor Scenes with Structural Hints
[3] SparseNeRF: Distilling Depth Ranking for Few-shot Novel View Synthesis
paper_summary: paper_summary: The paper presents a method called Learning Geometry Consistent Neural Radiance Field (GC-NeRF) for synthesizing views from sparse and unposed images. This method optimizes both a neural radiance field and camera poses simultaneously by establishing image-level and region-level geometric consistencies. It introduces an adaptive high-frequency mapping function to improve the 3D scene's geometric and textural details.
strength_weakness: strength_weakness: The numerical results looks good under the sparse setting.
limitations: limitations: 1. Although mentioned in the limitation, I'm interested in a detailed comparison of computational costs and training times between the baseline methods and the proposed approach. This comparison would help understand the trade-offs in reconstruction quality against the computational cost required.
2. The authors stated that SPARF heavily depends on pose initialization. However, it's not entirely clear how the proposed method reduces this dependence with its new initialization strategy. I'm curious about how the proposed method performs under various initialization conditions, and how SPARF would perform if it used the same pose initialization as the proposed method.
3. The illustration appears somewhat cluttered, with numerous arrows pointing in different directions, suggesting the addition of many regularization terms to prevent overfitting rather than presenting a novel and elegant solution.
paper_summary: paper_summary: This paper proposes GC-NeRF for novel view synthesis from sparse input images without known poses. It jointly optimizes NeRF models and camera poses with 1) image warping and 2) matching point warping. An adaptive positional encoding method is also introduced to improve high-frequency details. Experiments with 2 input images show that GC-NeRF achieves state-of-the-art results in multiple situations with unposed and sparse views.
strength_weakness: strength_weakness: The proposed GC-NeRF is a method to solve novel view synthesis under sparse and unposed views, which is a challenging task. By combining image-based and point-based warping techniques to add photometric and geometric constraints, this method achieves state-of-the-art performance in settings with sparse and unposed views.  The effects of each contribution are also well-validated according to experiments with image and geometry metrics, and the results seem promising.  This paper is easy to follow.
limitations: limitations: - **More views.**  The comparisons in the paper are mainly focused on settings with 2 views. Despite that Table 6 has shown the method's availability for settings with more unposed views and meanwhile can perform better than SPARF, it's unclear that: How to extend the proposed GC-NeRF from an image pair to multiple views? Is it an overall joint optimization, or pair-by-pair with multiple stages? And are there any additional hypotheses, like the images are sorted as a sequence (used in NoPeNeRF)? As the 2-view setting is rare in both previous works and real-world applications, it's essential to add an explanation about the details for multiple views.

- **Detailed experiment settings should be declared.** In section 4.2 Unknown Pose setting, it's unclear whether all baselines are with the same unknown pose as in the proposed method, or keep their original settings (e.g. using noisy poses for SPARF instead of unknowns).  

- **Typos.** 
  1) text format for the same symbols in eq (10) are inconsistent.
  2) the expressions in line 338 and eq (6) are suggested to be unified whether $[R \quad t]$ or $[R|t]$ 
 

- Although the combination of the contributions has a great effect, some similar insights can be found in previous works (e.g. monodepth and SPARF), and the learnable positional encoding is somehow similar to current learning-based encoders like instant-NGP. Nevertheless, this combination is also valuable for research.

Will consider increasing the rating if my concerns are well solved