CRAYM: Neural Field Optimization via Camera RAY Matching

Authors: liqiang  lin
wenpeng  wu
chi-wing  fu
hao  zhang
hui  huang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-09 11:59:30

Abstract: We introduce camera ray matching (CRAYM) into the joint optimization of camera poses and neural fields from multi-view images. The optimized field, referred to as a feature volume, can be “probed” by the camera rays for novel view synthesis (NVS) and 3D geometry reconstruction. One key reason for matching camera rays, instead of pixels as in prior works, is that the camera rays can be parameterized by the feature volume to carry both geometric and photometric information. Multi-view consistencies involving the camera rays and scene rendering can be naturally integrated into the joint optimization and network training, to impose physically meaningful constraints to improve the final quality of both the geometric reconstruction and photorealistic rendering. We formulate our per-ray optimization and matched ray coherence by focusing on camera rays passing through keypoints in the input images to elevate both the efficiency and accuracy of scene correspondences. Accumulated ray features along the feature volume provide a means to discount the coherence constraint amid erroneous ray matching. We demonstrate the effectiveness of CRAYM for both NVS and geometry reconstruction, over dense- or sparse-view settings, with qualitative and quantitative comparisons to state-of-the-art alternatives.

Main paper: 
 introduction: 
 Recent advances on multi-view 3D reconstruction have been propelled by the emergence of neural fields [39] , including implicit functions [42, 36, 24, 33] and radiance fields (NeRF) [26, 44, 13, 1, 10, 27, 19, 7] . A critical component to all image-to-3D reconstruction methods, including traditional approaches such as multi-view stereo (MVS) [11] , is to obtain camera poses for the input images. In practice, the camera information may be available from the acquisition devices, e.g., through the GPS or inertial measurement unit (IMU), while in other cases, it is estimated, e.g., using structure-frommotion (SfM) [8, 29] . In both cases, these camera poses can be noisy, thus hindering the performance of the multi-view 3D reconstruction.In light of the importance of having accurate camera poses, various methods have been proposed to improve their estimations. One line of approaches, which can be referred to as bundle-adjusting neural fields, jointly optimize [38, 20, 12, 2, 5] camera poses along with results from rendering and Figure 1 : Our method, neural field optimization with camera ray matching (CRAYM), incorporates contextual information for per-ray processing and enforces color + geometric consistence between matched rays. Compared to SPARF [32] which utilizes dense pixel correspondences and the stateof-the-art, bundle-adjusting L2G-NeRF [5] , both aimed at handling noisy camera poses, CRAYM produces superior results especially over fine details; see the zoom-ins on the right. Results are shown the Drums model from NeRF-Synthetic [26] on dense views. geometry (e.g., depth) estimation, where the camera rays are considered independently for their roles in color and geometry prediction. By now, more works [3, 4, 32, 18, 16, 30, 6] realize the importance of exploiting correlations between input images, i.e., multi-view consistency, to impose additional constraints on the joint optimization. Along these lines, much effort has been invested into matching and optimization with respect to image features, whether convolutional or transformer-based.Motivated by multi-view spatial analysis, several works have proposed geometric constraints involving camera rays and projections [16, 32, 18] . Most recently, SPARF [32] defines a re-projection loss as a spatial distance between image pixels to enforce that matched pixels between NeRF training images be back-projected onto the same 3D point. However, the effectiveness of this loss depends critically on how reliable the pixel correspondences are. In their work, these correspondences and their confidence estimates were both obtained by a pre-trained network [31] , which is independent of the joint camera-scene optimization.In this paper, we introduce camera ray matching into the joint optimization of camera poses and a neural field, referred to as a feature volume, which can be "probed" by the camera rays for both rendering, e.g., novel view synthesis as in NeRF [26] , and 3D reconstruction, as in NeuS [34] .The key reasons for matching camera rays, instead of pixels [32, 18, 6] , are two-fold. First, these rays carry 3D spatial information than just 2D pixel values to facilitate formulating explicit geometric losses when optimizing camera poses [16] , as dictated by multi-view analysis. Second and more importantly, the camera rays can be parameterized by the feature volume -they carry both geometric and photometric information. Any constraint arising from camera ray matching can be passed onto the feature volume. Hence, both the matching itself and the associated matching confidence can be incorporated into the joint optimization and network training, to impose physically meaningful constraints to improve the final quality of both geometry reconstruction and rendering.Our network, coined CRAYM (for Camera RAY Matching), takes as input an uncalibrated set of images capturing a 3D object, and is trained to predict the feature volume along with all the camera rays subjected to a combination of photometric rendering losses and geometric losses dedicated to ensuring multi-view consistency between camera rays. We consider two types of rays. The first are called key rays, which pass through keypoints detected in the input images, typically spanning regions with sharp features and rich textures over the 3D object. The other rays are called auxiliary rays, which pass through points around keypoints to offer contextual and local structural information as we reason about the key rays in our optimization framework.As our main constraint for matched ray coherence, we enforce color consistency between renderings along two key rays whose corresponding keypoints from two different views are matched [9] . However, we must account for potential erroneous matches due to occlusion or unreliable local image features used by the matching network. To this end, we aggregate features along each key ray through the feature volume. The matchability between two rays is defined by a cosine similarity between the accumulated ray features and applied as a weight to either accentuate or discount the color consistency constraint, allowing our optimization model to naturally degenerate itself to handle unrelated rays separately. Also, to improve the robustness of feature learning, we enhance the feature along each key ray by integrating features from surrounding auxiliary rays.We evaluate our method on both the synthetic objects from NeRF-Synthetic [26] and the real scenes from UrbanScene3D [22] , for novel view synthesis and 3D geometry reconstruction, over dense-and sparse-view settings. Compared to state-of-the-art alternatives, CRAYM produces superior results especially over fine details. 
related works: 
 Neural Fields. As a pioneer work, NeRF [26] synthesizes novel views of static objects/scenes from a set of posed images by optimizing a coordinate-based neural network, which predicts the volume density and color for a sampled point in the 3D space. Since then, numerous methods have emerged to improve the rendering quality [44, 13, 1] and rendering efficiency [10, 27, 19, 7] . To extract high-quality surfaces from the learned implicit representation, NeuS [42] and VolSDF [42] propose to learn an implicit signed distance field (SDF) representation for the scenes. These methods can achieve impressive results on both novel view synthesis and 3D reconstruction, however, the requirement of precise camera pose limits their applicability in practice.Bundle-Adjusting Neural Fields. With the realization that positional encoding is susceptible to suboptimal registration, BARF [20] applies a smooth mask on the encoding at different frequency bands for a coarse-to-fine training, while [12] presents an adaptive positional encoding. L2G-NeRF [5] first learns the pixel-wise transformations for every pixel in a frame and then aligns the frame-wise transformation with the pixel-wise transformations. Common to all the above methods is that their joint optimization of pose and scene representation processes each image and each ray separately, without considering their multi-view correlations. As a result, the pose optimization may not be stable, thereby leading to floaters and blurriness in both novel view renderings and 3D reconstruction. Note that our method also involves per-ray processing, by combining information from auxiliary rays with that of a key ray. This is similar to the patch-level feature processing in CR-NeRF [41] , which considers multiple rays indiscriminately across the image, without the notion of key rays.Neural Fields with Image Matching. Image matching can help establish geometric priors to improve the generalizability of NeRF, to either novel scenes or the sparse-view setting. MVSNeRF [3] constructs a cost volume by warping the image features extracted with a 3D CNN onto a plane sweep, from which a generalizable radiance field is learned. SparseNeuS [24] constructs a 3D volume with the variance of all the projected features from multi-view images. DBARF [4] optimizes camera poses and depth with a cost map constructed by the differences of image features. CorresNeRF [18] proposes to regularize the NeRF training with a pixel re-projection loss for the associated pixels and a depth loss for the predicted depth. GPNR [30] aggregates features of the image patches along epipolar lines with several stacked transformers. MatchNeRF [6] learns a generalizable NeRF with the cosine similarity of image features for each image pair as the shape prior. All these methods integrate image features and utilize the matching within or between different views. With more emphasis placed on multi-view geometry reasoning, SCNeRF [16] learns a pinhole model for each camera under the supervision of a re-projected ray distance loss, while SPARF [32] optimizes its network with a re-projection loss, measuring spatial distances between pixels in the same view. In contrast, the matched ray coherence formulation in our optimization accounts for both photometric and geometry information as obtained from the feature volume; the coherence constraint is also explicitly integrated into the network instead of only serving to define a loss. 
method: 
 We are interested in neural networks that can reconstruct a 3D model, e.g., a radiance field [26] or an implicit field [34] , from a set of M images {I i } M i=1 capturing a 3D object from multiple views. Typically, each image is associated with a known or estimated camera poseT i = [R i |t i ],where R i ∈ SO(3) and t i ∈ R 3 . The network is trained by minimizing a photometric error L p between the Figure 2 : Overview of our CRAYM pipeline. After extracting keypoints (red dots) from input images and matching them using a pre-trained network, we train our CRAYM network to optimize a 3D feature volume V which encodes both geometric and photometric information about the target 3D object and can be queried by camera rays for both novel view synthesis (via the Texture Network) and 3D reconstruction (via the Geometry Network). The volume optimization is subject to photometric losses through rendering along camera rays passing through the keypoints (i.e., the key rays), which is enhanced (in the KRE) by integrating features from auxiliary rays, i.e., rays passing through nearby auxiliary points (yellow dots) in the images. Matched ray coherence (MRC) is imposed on matched key rays, in terms of color consistency, while potentially mismatched rays can be identified by comparing accumulated features along the key rays through V. On top of the standard photometric loss, we introduce two geometric losses, the epipolar loss and point-alignment loss, to explicitly optimize ray-to-ray coherency to maximize the reconstruction quality of the feature volume.input images and the multi-view renderings, { Îi } M i=1 , of the target 3D object from the camera views:min i x ||I i (x) -Îi (x)|| 2 2, where I i (x) is the color of image I i at pixel x. Each pixel is associated with a specific ray in 3D from the object/scene, through the pixel center, towards the camera: {r(t) = r o + tr d |t ≥ 0}, where r o is the camera center and r d is the normalized view direction of ray r. The rendered color of ray r, i.e., the pixel color I i (x), can be produced using volume rendering by accumulating the color and opacity σ along the ray r.Considering that the camera poses can be noisy, the reconstructed radiance field or implicit field may not produce clean and sharp renderings with details. At a high noise level, some methods may even fail to produce results; see examples shown in Sections 4.2 and 4.4. Beyond existing approaches that map r(t) to opaque density (or opacity) and color implicitly with a ray-wise network, we propose CRAYM to learn the implicit field by matching rays across different images and formulating geometric priors.Figure 2 overview our CRAYM pipeline. From the input images, our goal in the 3D neural field optimization is to construct a 3D feature volume V to faithfully represent the target object. In detail, we represent feature volume V using multi-resolution hash encoding [27] and end-to-end optimize it for the target object. The feature f (p) of point p in the 3D feature volume can be extracted byEQUATIONwhere M is the progressive feature mask [19] for filtering out fine-level features during early iterations of the coarse-to-fine training. Very importantly, to account for the noise in the camera poses, we parameterize the transformation matrices of the cameras as variables in the joint optimization of the pose and implicit field with the feature volume V.As mentioned in the introduction, we consider two types of rays to probe the feature volume, i.e., key rays and auxiliary rays. Both rays are issued from the cameras through the pixel centers. To obtain key rays {r k }, which typically associate to surface points with rich textures and sharp features, we detect keypoints on each input image using SuperPoint [9] and perform point-to-point matching between image pairs using SuperGlue [28] . Then, we can obtain a set of sparse ray-to-ray matchings between image pairs. Note that these results may not be accurate for various reasons such as occlusion and unreliable matching, but they provide useful information for our pipeline to start with. As for the auxiliary rays {r a }, they are sampled around the keypoints to provide contextual or local structural information when we reason about the key rays; see Section 3.2 for details.Once optimized, the feature volume can be used for novel view synthesis or for multi-view 3D reconstruction. The color prediction for novel view synthesis is accomplished by a texture network Φ t , as in a typical NeRF [26] setting, and the latter is accomplished by a geometry network Φ g , as in a typical NeuS [34] setting. Specifically, the geometry network takes a 3D point p sampled along r and the feature at p as input to produce an SDF value and then an opaque density σ to render the 3D object and extract the 3D reconstructions. Here, we propose the Key Rays Enrichment (KRE) module (Section 3.2) to improve the robustness in the process by enhancing the features along the key ray using the features sampled by the auxiliary rays.Subsequently, the texture network takes the output features from geometry network, ray directions, and normal at p as inputs to predict color c(p) at point p. Further, we design the Matched Rays Coherency (MRC) module (Section 3.3) to enhance the volume rendering quality by considering matchability between rays and learning to maintain coherency between ray matchings. Particularly, the MRC module can effectively reduce the influence of mismatched rays by disambiguating the camera ray matchings.A pair of the matched key rays, r k and r k ′ , are sampled with the corresponding auxiliary rays during each iteration. The geometry network, texture network, and feature volume optimization are jointly trained end-to-end. Besides the photometric loss, we formulate the epipolar loss and point-alignment loss (Section 3.4) to explicitly promote coherency among the ray matchings and boost performance.As the input images are captured through a perspective projection, all rays in 3D through the same image should converge at a common camera point. In previous works, for each iteration, rays are optimized separately, so the pose optimization may not be stable. As different rays may back propagate gradients in different directions, the optimized poses may oscillate during the training. Hence, we introduce the KRE module to stabilize the optimization by learning structural information around each key ray. This is done by sampling auxiliary rays around the key ray to enrich the feature of the key ray with more contextual information:EQUATIONwhere p k is a point along key ray r k ; {q j } are points around p k sampled along the j-th auxiliary ray around r k ; and function g fuses features f (p k ) and f (q i ). Then, we employ the geometry network Φ g to predict the SDF value at p k and feature vector f ′′ (p k ), from which we can further obtain the color of point p k with the texture network. Please refer to the supplemental materials for the details.Next, we propose to learn the coherency of features accumulated in the 3D feature volume between the matched key rays. The purpose is to enhance the camera ray matching and account for imprecise ray matchings, since keypoints matching is performed only on local image features.Similar to color accumulation in volume rendering, we calculate the aggregated feature along a key ray r k as the feature of r k :EQUATIONThe function T (r k (t)) = exp(-t 0 σ(s)ds) denotes the accumulated transmittance along key ray r k . Essentially, the accumulated ray feature f (r k ) is an integration of density-weighted features along a ray. When the network converges, the actual surface point at which ray r k intersects with the paired matched key ray should have the highest density. Hence, we consider coherency between the matched rays to optimize the learning of the opacity density and point color, such that we can enhance the coherency of features accumulated along the matched key rays. In return, this will help to optimize the parameters and the 3D feature volume, when training the pipeline. Therefore, we fuse the rendered color c(r k ) of the matched rays based on the cosine similarity between their accumulated features:EQUATIONwhere w is the matchability calculated as the cosine distance between the accumulated features of the matched rays. When two key rays are mismatched, e.g., due to occlusion or ambiguities of weak texture areas and similar structures, our formulation can learn to degenerate itself to a form that separately optimizes individual rays.Further, we introduce the following two geometric losses to more explicitly promote the coherency of the ray matchings:Epipolar loss. Given a pair of matched keypoints x k and x ′ k on two different input images, which associate with camera centers O and O ′ , respectively, (see the illustration in Figure 3 ), we can estimate the depths at x k and x ′ k by using a depth accumulation formulation similar to Equation 3, and then project points x k and x ′ k into the 3D object space to obtain 3D locations p k and p ′ k , respectively. If the camera poses, the matchings, and the depths are precise, the two rays through x k and x ′ k should precisely intersect at a common point, say P , on the target object surface, such that p k and p ′ k align with P . Also, we denote e and e ′ as the epipolar points on the two images; these points are the image-space locations at which the line OO ′ intersects the two image planes; see Figure 3(a) .During the training, the depth estimation of x k can vary, so p k may vary along ray r k . If the camera poses are precise, the projection of p k onto the image plane of the other camera should lie on the epipolar line e ′ O ′ . In case of noisy camera poses, the projection of p k may not lie exactly on e ′ O ′ , so we explicitly enforce the epipolarity during the training by minimizing the distance between p k 's projection and the epipolar line e ′ x ′ k usingEQUATIONSince the epipolar loss is not affected by depth, we decouple the unreliable depth estimation from the epipolar loss with ray marching to constrain the camera poses.Point-alignment loss. The epipolar loss focuses on enhancing the projection consistency for producing more precise camera poses. To complement it, we introduce the point-alignment loss to facilitate depth convergence for improving the reconstruction of fine details. In detail, we consider the triangle EQUATIONNote that depth estimation may be unreliable and likely unstable early in the training, so we use the point-alignment loss only after a certain number of training iterations.Overall loss. After constructing the epipolar loss L e and the point-alignment loss L a , we put them together with the photometric loss L p and SSIM loss L s to form the overall loss functionEQUATIONwhere L p is calculated between the input images and the rendered images and is modeled as an MSE loss. 
results: 
 We evaluate our method on the NeRF-Synthetic dataset [26] with eight synthetic objects (Section 4.2), the LLFF dataset [25] , and the real scenes from the UrbanScene3D dataset [22] (Section 4.3). We compare our method on both novel view synthesis and 3D reconstruction with NeRF [26] , NeuS [34] , BARF [21] , L2G-NeRF [5] , PET-NeuS [37] , SPARF [32] , and BAA-NGP [23] . Since NeRF [26] , NeuS [34] , and PET-NeuS [37] are designed for neural implicit field with fixed and precise poses, we set the camera transformations as variables to be optimized jointly with the neural field, as in our method.While other methods optimize the radiance field, in which the target values, radiances, of points are more independent of each other, the optimization of SDFs in NeuS and PET-NeuS poses a challenge to the requirements of non-local geometric constraints to correctly form the shape, making them more vulnerable to unstable pose optimization. As the camera rays are parameterized by our feature volume to carry both geometric and photometric information, our geometric constraints on the camera ray matching can effectively lead to better optimization of the geometry. The joint optimization of camera pose and implicit SDF may also fail to produce results for NeuS and PET-NeuS, when the camera poses are at a high noise level. With the assistance of camera ray matching, CRAYM outperforms other methods on both novel view synthesis and 3D reconstruction at varying noise levels. We report the PSNR, SSIM, and LPIPS [43] for quantitative comparisons on novel view synthesis and Chamfer distance (CD) for the 3D reconstruction. A test-time photometric pose optimization is performed to evaluate these metrics, following prior works [21, 32, 5] . The quantitative evaluations on the other metrics are provided in the supplementary materials.To evaluate the registration quality of the optimized training poses, we use Procrustes analysis [14] to find the 3D similarity transformation that aligns the optimized training poses with the calibrated camera poses, following BARF [21] . As Figure 4 shows, the optimized poses produced by CRAYM align well with the ground-truth poses with lower translation errors. The average translation errors and rotation errors are reported in Table 1 . With the contextual information and feature coherency of camera ray matching, the camera poses produced by CRAYM are better optimized for the construction of the implicit filed, so that CRAYM is able to produce high-quality rendered views, as well as more precise 3D reconstructions. For the evaluation on the NeRF-Synthetic dataset, we follow the same setting of noisy poses as L2G-NeRF [5] , which perturbs the ground-truth camera poses with additive noise as the initial poses. As NeuS [34] and PET-NeuS [37] fail to produce results at such a setting, we present only the results of NeRF [26] , BARF [21] , SPARF [32] , and L2G-NeRF [5] .The mean results of the eight objects and four of them are given in Table 2 . NeRF fails to extract meshes from the reconstructed radiance fields on Hotdog and Ship. Figure 5 shows the results of view synthesis and 3D reconstruction visually for BARF, SPARF, L2G-NeRF, and our method. SPARF produces over smoothing results with dense input, as shown in Figure 5 . As can be seen in Figure 5 , CRAYM is able to produce clean and complete renderings and reconstructions with fewer floaters and less blurriness.We first evaluate our method on the LLFF dataset [25] for high-fidelity view synthesis of the eight real scenes. Compared with BARF [21] , L2G-NeRF [5] , and BAA-NGP [23] , our method is able to produce high-quality results with fewer artifacts and better scores in terms of PSNR, SSIM, and LPIPS, as shown in the As the images in the LLFF dataset are captured with a restricted range of angles, we further assess our method on the real scenes PolyTech and ArtSci from the UrbanScene3D [22] dataset with two sets of drones captured images [45] associated with GPS information. The large-scale scenes are captured with hundreds of images, which share smaller overlaps than the images from the NeRF-Synthetic dataset. Considering the limitations of memory, we reduce the size of the original high-resolution images by using bicubic interpolation. As the GPS information may not be reliable and the positions in GPS may shift in meters, we preprocess the poses from GPS with COLMAP [29] and add a small noise to the calibrated poses, following L2G-NeRF [5] .The UrbanScene3D dataset contains high-precision LiDAR scans for the target buildings, PolyTech and ArtSCi. Therefore, we evaluate the reconstructions quality with the point cloud scans as the ground truths. We crop the reconstructed meshes according to the LiDAR scans and align them using ICP. NeRF, BARF, and L2G-NeRF produce blurry renderings and degenerated meshes for the real scenes, while SPARF may fail to process such data with dense correspondences. Therefore, we only provide the visual results of NeuS, PET-NeuS, and our method.The quantitative results of NeuS, PET-NeuS, and our method are provided in Table 4 . As Figure 6 shows, NeuS and PET-NeuS tend to produce over smoothing results, while our method is able to extract meshes with fine details. A mesh reconstructed directly from the original high-resolution images using ContextCapture2 , a commercial MVS solution, is provided as a reference.Comparison on varying noise levels. To evaluate model robustness, we evaluate on the LEGO data sample with poses at varying noise levels. The "high noise level" means we use the same noise setting as L2G-NeRF [5] ; the "low noise level" means we perturb the ground-truth poses with half of the noise as L2G-NeRF; and "w/o noise" means the poses are initialized as ground-truth poses without noise. The transformation matrices of the camera poses are all set as variables to be optimized. Table 5 summarizes the results. We can see that NeuS [34] and PET-NeuS [37] are more sensitive to noise and cannot effectively handle the high-noise setting, while the other methods can not produce accuracy reconstructions. With the contextual information learned in the camera ray matching and the explicit utilization of matched rays, CRAYM is able to obtain better results for all the noise settings. Ablation of major modules and losses. To demonstrate the efficiency of the proposed modules and geometric losses, we conduct an ablation study on the LEGO data sample. The results are reported in Table 6 . Similar with BARF, which applies a smooth mask on the encoding at different frequency bands for neural radiance field, we apply a progressive feature mask on the hash encoding with a coarse-to-fine training of the neural implicit field as our baseline, which combines BARF [21] and NeuS2 [35] . The KRE module improves the robustness to noisy poses in the training, whereas the MRC module effectively enhances the quality of the volume renderings with ray matching. In addition to that, the proposed geometric losses further help our framework to obtain better camera pose optimization. 
conclusion and discussion: 
 Our method, CRAYM, addresses the issue of noise camera poses for multi-view 3D reconstruction and view synthesis. The key idea is to jointly optimize a neural field and camera poses by incorporating contextual information (via KRE) and enforcing geometric and photometric consistency (via MRC and geometric losses) through camera ray matching.Experiments demonstrate that our method outperforms state-of-the-art alternatives under various settings: dense-vs. sparse-views, and different noise levels. However, the implicit field and optimizable pose transformations may not converge when the poses are randomly initialized or extremely noisy. A stronger pose regularization prior to the field optimization may resolve this problem. Furthermore, the meshes extracted from the constructed SDFs may still contain messy inner structures over invisible areas.A promising future work is to apply the ray matching to the 3D Gaussian splatting, which will greatly improve the rendering efficiency of CRAYM. However, extracting reconstructions with fine geometric structures from 3D Gaussians is still an open problem.Finally, CRAYM has been designed to rely on sparse key rays for dense-view reconstruction, while a dense counterpart may bring up extra overhead. In our Matched Ray Coherency formulation, we explicitly account for potentially erroneous (i.e., low-quality) 2D matches by using the matchability between two rays as a weight to either accentuate or discount the color consistency constraint. In terms of sensitivity with respect to the density of the 2D matches, in our experiments, we have observed that even with sparse input views and sparsely distributed matched rays, CRAYM can still notably improve the optimization convergence. An effective approach to utilize ray matching for both sparse and dense inputs may further boost the performance of CRAYM.A Appendix / supplemental material A.1 Implementation A.1.1 Camera Pose Parameterization.The camera poses {T i }, where 3), need to be parameterized and optimized in the training process. As a high frequency operation, pose parameterization needs to be simple and efficient, since the conversion between pose parameterization and transformation matrix is performed in every training iteration.T i = [R i |t i ] ∈ SE(The position component t of T is simply represented as a three-dimensional vector. Instead of parameterizing the rotation R as an exponential map exp(r) from the Lie algebra so(3) to the Lie group SO(3), we represent R as a six-dimensional vector composed byR = [v a |v b ], v a ∈ R 3 , v b ∈ R 3 . v a , v b, and v c = v a × v b span the three bases of the camera space. To obtain rotation matrix R from R, we perform a Gram-Schmidt orthonormalization on v a , v b , and v c , resulting in three orthonormal bases of the camera space, va , vb , and vc . Thus, the rotation matrix isEQUATIONA.1.2 Key Rays Enrichment Module Figure 7 shows an example of matched pixels. Due to the noise in the camera poses, the surface point captured by the associated keypoint on the input image may not intersect with the key ray r k exactly, the integration of features from the neighboring points {q i } sampled by auxiliary rays {r o } can greatly facilitate a more stable optimization associated with key ray r k ; see Figure 8 .Procedure-wise, we propose to first fuse the feature of p k with the features of the surrounding {q i } to produce the enriched feature f ′ (p k ), which can better describe the local radiance feature and geometry feature of the captured object with structure information.EQUATIONThe features {f (q i )} of the auxiliary rays {r a } remain untouched: f ′ (q i ) = f (q i ). Further, we adopt the geometry network Φ g to process the features f ′ (p) of both key rays and auxiliary rays to extract the SDF value at point p in the radiance field. From the extracted SDF values, we can then produce the 3D reconstruction of the target object:EQUATIONwhere f ′′ (p) is the output feature of point p. f ′′ (p) is one of the inputs to the texture network Φ t . The contextual information of ray matchings facilitates a more stable pose optimization and promotes the details of the geometry, i.e., the accuracy of the SDF values.The color of point p sampled on key rays or auxiliary rays can then be obtained with the texture network Φ t asEQUATIONwhere r d is the normalized direction of ray r and Normal(p) is the normal of the implicit surface at p computed as the gradient of SDF(p). We take the normal at p into account to boost the training of Φ t . The color c(r) of ray r is then rendered with the opaque densities and colors of all the points sampled along r, where the opaque densities are obtained with the SDF values, normal vectors, and ray directions [34] .  
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: The proposed method is not tied to particular applications.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: All the related papers are cited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: We use a 16-level hash grid [27] to encode the feature volume V. The feature length of each level is two. Therefore, the base resolution of V is 32. We apply a progressive feature mask [19] on the hash encoding, which starts at level 4 and is updated to the next level every 1,000 iterations. The geometry network ϕ g is implemented as a three-layer MLP with the ReLU activation for the input and hidden layers. The texture network ϕ t is implemented as a four-layer MLP with the ReLU activation for the input and hidden layers. The ray directions are encoded using the spherical harmonic representation [10] and fed into the texture network ϕ t together with the output features of ϕ g to predict the color of the sampled points on the ray. The whole network is optimized with the AdamW optimizer with a learning rate of 0.01, β = [0.9, 0.99], and ϵ = 1.0 -15 . The variance [34] of the geometry network ϕ g is initialized as 0.3 and is optimized with a learning rate of 0.001. We adopt a warm-up training for the first 500 iterations with the LinearLR scheduler. All the experiments are conducted with an Nvidia GV100. Next, we report the Hausdorff distance (HD), precision, recall, and F-score [17] for the reconstruction quality evaluated on the NeRF-Synthetic [26] dataset and the UrbanScene3D [22] dataset. The NeRF-Synthetic dataset contains eight synthetic objects, which are captured with 100 images. We evaluate our method on the two real scenes, PolyTech and ArtSci, from the UrbanScene3D [22] dataset, on which we measure the quality of the reconstructions with the high-resolution LiDAR scans as the ground truths.Tables 7 and 8 report the reconstruction quality, compared with NeRF [26] , BARF [21] , SPARF [32] , and L2G-NeRF [5] on the two datasets. A threshold of 0.01 is used to extract the inliers and outliers for the calculation of the precision, recall, and F1 score. The precision measures the reconstruction accuracy by calculating the distances from the reconstructed models to the ground truths. The recall measures the reconstruction completeness by determining the extent of the ground-truth points covered by the reconstructed models. A high F-score means both high accuracy and high completeness of the reconstructed models. As we can see from Tables 7 and 8 , our method is able to produce high-quality reconstructions with both high accuracy and high completeness. It is worth to note that compared with other methods, our method achieves the top performance on all metrics consistently for both datasets.Further, we evaluate our method on the real scene Bank from TwinTex [40] , which is a set of high-resolution drone-captured images. We preprocess the images with COLMAP [29] to obtain the calibrated poses and perturb the poses with additive noise ξ, where ξ ∈ se(3) and ξ ∈ N (0, nI), as the initial poses, following the procedure on real scenes PolyTech and ArtSci of UrbanScene3D [22] . For these real scenes, we set n as a small value 0.01.Since TwinTex does not provide a LiDAR scan for the scene Bank, we only report the evaluation results of novel view synthesis in Table 9 . Figure 10 shows the comparison on the real scene Bank visually with NeuS [34] and PET-NeuS [37] . While the other methods tend to generate renderings with blurriness, our method is able to produce sharper results with more fine details. Figure 9 further shows results of our method on another real scenes CSSE. The DTU [15] dataset is aimed at multi-view stereo (MVS) evaluation, containing image sets captured with structured light scanners mounted on an industrial robot arm. We evaluate our method on the first five image sets used in the NeuS [34] , comparing with NeuS [34] , PET-NeuS [37] , and SPARF [32] . Each image set contains 48 images. We use 90% of them as training set and the remaining 10% images as testing data. The quantitative results of novel view synthesis on these data are shown in the Table 10 . Figure 11 shows some of the novel view synthesis results visually. As we can see in Figure 11 , our method produces renderings with much more details.Since SPARF is originally designed for sparse input with re-projection loss of dense pixel correspondences, we further evaluate our method on the LEGO data with sparse input, which contains only three views as the training images. The comparison of SPARF [32] and our method is shown in Table 11 . Though CRAYM aims for bundle-adjusting neural implicit field with dense views as input, it still obtains a result comparable with SPARF, for sparse input views.• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: The research conforms with the code of ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).https://www.bentley.com/en/products/brands/contextcapture 


Decision: This paper proposes a new method to jointly optimize camera poses and neural fields from multi-view images. The method incorporates contextual information and enforces geometric and photometric constraints through camera ray matching. The paper received one strong accept and three borderline accepts after the rebuttal, with most of the concerns raised by reviewers have been addressed.

Most reviewers acknowledged that the joint optimization along matching rays within the feature volume, as well as the design of key rays and auxiliary rays, effectively enhances volume optimization even in the presence of errors in matching and camera poses. Therefore, the paper is recommended for acceptance.

Reviews: review: paper_summary: paper_summary: This paper introduces Camera Ray Matching for optimizing camera poses and neural fields from multi-view images. The optimized feature volume supports novel view synthesis and 3D geometry reconstruction by probing camera rays, which carry both geometric and photometric information. CRAYM claims to improves efficiency and accuracy by focusing on keypoints and integrating multi-view consistencies, enhancing both geometric reconstruction and photorealistic rendering. The method shows result in NVS and geometry reconstruction compared to baseline methods.
strength_weakness: strength_weakness: - Experiments were only conducted on NeRF-synthetic datasets and not on LLFF datasets.
- Comparison is made with older baseline methods (e.g., SPARF, BARF, L2G) which are more than 2 years old. It’s recommended to include more recent methods such as NoPe-NeRF and BAA-NGP.
- It is suggested that the authors perform Neural Image Alignment to enhance the evaluation.
questions: questions: Figure 5 and Table 3 appear to have a significant overlap in the information they present.
limitations: limitations: YES
paper_summary: paper_summary: This work suggests a novel neural representation and training scheme that jointly solves for the scene representation and the multi-view camera localization. It is done using several new ideas that generalize existing NeRF based methods. 
The representation itself is a combination of a geometry-network, which predicts a signed-distance-function (SDF) and a feature vector, that are fed into the texture-network that predicts the usual color and density values.
The main key novelty, is that the optimization is done over matching rays, obtained from matching keypoints using a pretrained network. The standard photometric loss function is extended to incorporate an epipolar loss (that constrains the camera positions) and a point-alignment loss that ensures the ray intersect at the predicted depth estimates along the rays. Another strong addition, is the use of 'auxiliary' rays around each matched pair of rays, from which features are fused to produce a more robust representation, that can aid the optimization under errors in matching and camera poses.
Extensive experiments demonstrate the importance of each component and the strong performance of their combination.
strength_weakness: strength_weakness: * Reproducibility - I believe that many details are missing (including from the appendix) for one to be able to implement the proposed method. For example:
   * What are the settings of the preprocessing SuperPoint and SuperGlue matching? What is the typical match density?
   * How are the auxiliary rays sampled? How many and under which distribution?
   * What is the function g in Eq 2 that fuses the key and auxiliary features?
   * What are the balance weights in the final loss (Eq. 7)?
   * How are poses initialized?
* Complexity - There is no discussion what so ever about the impact of the suggested changes on memory and runtime complexity, but at traning and in inference.
* Qualitative results are relatively limited. 
   * Synthesized images are all very small, so it is difficult to appreciate the fidelity.
   * No depth images are shown
   * No examples of key and auxiliary point matches are shown (over entire images)
questions: questions: * Is the training done only on matching key points? If so, what happens if correct matching keypoints do not 'cover' the space adequately?
* How sensitive is the method to the quality and density of the 2D matches? It reportedly worked well on sparse image sets, which is somewhat surprising.
* What are the runtimes (training and inference) compared to some baseline methods?
* How different is the use of auxiliary rays compared to sampling from conical frustums as in Mip-NeRF?
limitations: limitations: Adequately discussed.
paper_summary: paper_summary: This paper presents a new technique called camera ray matching, which is integrated into the joint optimization of camera poses and a neural field. The method utilizes an uncalibrated set of images as input, incorporating photometric and geometric constraints through key points and key rays matching, with the aim of enhancing the quality of novel view rendering and 3D surface reconstruction. The approach comprises two simple modules and is implemented using grid-based representation (iNGP). Photometric experiments were exclusively compared with MLP-based methods, specifically NeRF-like, on the synthetic dataset of the vanilla NeRF, while geometric experiments demonstrate some positive results. The authors provide additional results in the appendix.
strength_weakness: strength_weakness: Too many factors are taken into account in the writing simultaneously, which leads to a lack of clear theme or a clear academic or technical problem to be addressed in this paper. The work appears to build incrementally upon previous research and offers limited novelty. The so-called Epipolar loss and Point-alignment loss are actually based on Bundle Adjustment (BA), using key points matching, which has been previously applied in the optimization of neural reconstruction in works such as SCNeRF, BARF, L2G, and Level2sfm. The proposed two modules do not bring significant innovation. It is also confusing that this work is implemented using a grid-based representation (i.e., iNGP), while the compared methods are implemented using MLP-based representation, which does not allow for a precise and fair comparison. I suggest that the authors refer to ZipNeRF for guidance on how to formulate research problems and conduct appropriate comparisons.
questions: questions: Based on the mentioned shortcomings, I have several questions:

1. What is the main research problem addressed in this paper? It appears that the paper intends to address three problems simultaneously: camera pose, fine-detailed rendering, and accurate surface reconstruction. However, if the goal is to solve the camera pose problem, it would be beneficial to present more results related to the accurate regression of camera poses before discussing high-fidelity results. Could you provide additional results about the accurate regression of camera poses?

2. If the primary focus is on the issue of high-fidelity rendering under noisy camera poses, the experimental results do not offer strong conviction. It would be beneficial to showcase more experimental results on different types of datasets, similar to how BARF, SPARF, and L2G-NeRF were compared on the LLFF dataset, with an emphasis on trajectory of pose optimization and high-fidelity renderings.

3. Initialization of camera poses is a sensitive issue in joint optimization. Have you attempted to change the initial camera poses or initialize poses using COLMAP to test the robustness of pose regression?

4. This work is implemented using grid-based representation, while the compared methods are implemented using MLP-based representation, leading to an imprecise and unfair comparison. I am curious about the total number of iterations each method was trained for. BARF and L2G-NeRF trained all models for 200K iterations, and SPARF trained models for 100K iterations. It seems likely that this paper trained for significantly fewer iterations (perhaps 10k iterations) due to the inherent faster convergence of the grid-based representation. It would be fairer to re-implement this paper using MLP-based representation and train for a duration comparable to other methods, then make comparisons. Additional experiments are needed to support these claims.

5. The comparison under sparse-view conditions lacks precision and distinction. SPARF evaluated on the DTU dataset with only 3 views, while you used 48 images in your setting, although you have reported results for sparse input (3 views) on the LEGO data, it may not be sufficiently convincing.

6. Since Neus and PET-NeuS have compared the surface reconstructions on the DTU dataset, I would appreciate the inclusion of visual results. Additional ablation studies on 3D surface reconstruction using your modules on the DTU dataset would enhance the paper.

7. Considering the integration of multi-resolution hash encodings into a neural surface representation, I recommend that the authors compare with Neus2. Neus2 has compared their work with Instant-NGP and Neus, showcasing fast training and detailed surface reconstruction. If the goal is to demonstrate the superiority of 3D surface reconstruction, it is most appropriate to consider Neus2 in the comparison.
limitations: limitations: The authors have identified a limitation where the meshes extracted from the constructed SDFs may still contain messy inner structures over invisible areas. I recommend that the authors explore the possibility of finding the SDFs of surface points instead of the SDFs of each sampled point along rays. This would involve assessing whether the depth accumulated by all points along the rays is more accurate than the output surface generated by all sampled points.
paper_summary: paper_summary: The manuscript #3263 entitled "CRAYM: Neural Field Optimization via Camera RAY Matching" proposes a novel uncalibrated NeRF strategy based on prior keypoints matching across images. Specifically, the authors propose two novelties to improve the quality of the reconstruction and the pose estimation of the cameras: 1) Enriched ray features using surrounding rays sampled around the keypoints and 2) a ray matching index which can be used to re-weight the color regression part, leading to better robustness to occlusions.
The proposed technique has been evaluated across various standard datasets and against meaningful NeRF-like algorithms.
strength_weakness: strength_weakness: - The robustness of the approach against outlier matches is not evaluated. Introducing artificial outliers (wrongly matched keypoints) into the dataset to assess how well the technique can handle mismatches would be of some interest.
*Question*: Would the matched rays consistency and the epipolar geometry compensate for that? Or would the training diverge?

- As stated in the literature review of this manuscript, other approaches taking advantage of the epipolar geometry and prior matching have already been designed in this context. I have difficulty understanding what is significantly different with this work apart from the sampling of additional surrounding rays and the matching ray index used to weight color prediction using image pairs. These two novelties seem rather incremental, but they nonetheless lead to strongly improved results.
*Question*: I assume that the other keypoints-based approaches are not "self-calibrated". Is the proposed technique the first "keypoint-based" calibration-free NeRF? If it is not the case, it would be meaningful to compare against such techniques too.

- Adding surrounding rays around a key ray appears to be quite effective; however, the sampling of auxiliary rays is not well described in the paper.
*Question*: How are the rays sampled?

- The initialization of the pose lacks details.
 *Question*: What is the effect of the pose initialization on the result?

- The intrinsic parameters of the camera could additionally be optimized.
*Question*: Just out of curiosity, have you conducted such an experiment?

- In equation (4), it seems that the proposed solution considers only pairs of images. 
*Question*: How are those pairs selected?

The proposed approach is inspired by existing techniques integrating matched keypoints (like using the epipolar loss) and other techniques, such as NeuS.

- The loss function contains many regularization factors.
*Question*: Is the final loss hard to balance?


Overall, the paper is interesting and proposes a few contributions that seem to lead to strongly improved results. Moreover, the approach has been evaluated on various standard datasets and against representative methods. However, this novel approach remains relatively incremental, and many points remain to be clarified regarding the robustness of the technique. For all the above-mentioned reasons, I would like to issue a rather mixed opinion regarding the acceptance of this work for this conference.
questions: questions: The questions are integrated in the Weaknesses part.
limitations: limitations: The limitations of the paper may not have been entirely investigated, specifically in terms of robustness. For instance, the influence of the initial pose and outliers has not been demonstrated.

Comments: comment: Dear Reviewer 67PC,

Thank you very much for recognizing our work and for your positive rating. We appreciate your feedback and will continue to refine our manuscript. In the revision, we will include optimized trajectories, a discussion on grid-based methods, additional experimental details and results. 

Thanks again,
The authors
Thanks for the great efforts of the authors! After carefully reading the responses both from the authors and other reviewers, most of my concerns have been addressed. I tend to improve my rating to BA now and suggest the authors add more experimental details/results and discussions in the revision.
Dear Reviewer 67PC,

Thank you for your comprehensive reviews and constructive suggestions. We have evaluated our method on the LLFF dataset and compared it with NeuS2. The quality of pose regression by our method is presented in Table 9 of the supplemental material. We hope the rebuttal will help address your concerns, and we are eager to resolve any further issues to ensure our submission meets the high standards of the conference.
Dear Reviewer Bu9t,

Thank you for your thorough reviews and thoughtful evaluation of our paper. We greatly appreciate your positive rating. We will continue to refine our manuscript in light of your feedback and hope to meet your expectations in the final version.
Dear Reviewer GtQM,

We are pleased to learn that the majority of your concerns have been addressed. Thank you for your thoughtful and constructive feedback on our submission. We will include a discussion about NoPe-NeRF in the revised version. We are committed to addressing any additional concerns you may have to ensure our paper meets the high standards of the conference.
I would like to sincerely thank the authors for their responses, which have clarified many of my questions. However, in light of the other reviews, I agree that this paper appears to be rather incremental. For this reason, I would like to maintain my original rating of BA.
The author addresses most of my concerns. Please add a discussion about NoPe-NeRF, even if not making a direct comparison. I am raising the score to BA.
Thank you to all the reviewers for your detailed and constructive feedback on our submission. We have addressed all the questions raised and conducted the required experiments. These primarily involved comparative analyses on the LLFF dataset, and the comparisons with BAA-NGP and NeuS2. 

In our experiments, the proposed method demonstrated superior performance across various metrics compared to the alternatives. The uploaded PDF file includes figures illustrating the optimized trajectories of the different methods evaluated. 

We remain open to addressing any further questions or concerns.