CRAYM: Neural Field Optimization via Camera RAY Matching
Authors: Liqiang Lin
Wenpeng Wu
Chi-Wing Fu
Hao Zhang
Hui Huang
Venue: NeurIPS 2024 poster (2024)
Keywords: Neural Implicit Fields
Novel View Synthesis
3D Reconstruction
Primary area: optimization_for_deep_networks

Abstract: We introduce camera ray matching (CRAYM) into the joint optimization of camera poses and neural fields from multi-view images. The optimized field, referred to as a feature volume, can be “probed” by the camera rays for novel view synthesis (NVS) and 3D geometry reconstruction. One key reason for matching camera rays, instead of pixels as in prior works, is that the camera rays can be parameterized by the feature volume to carry both geometric and photometric information. Multi-view consistencies involving the camera rays and scene rendering can be naturally integrated into the joint optimization and network training, to impose physically meaningful constraints to improve the final quality of both the geometric reconstruction and photorealistic rendering. We formulate our per-ray optimization and matched ray coherence by focusing on camera rays passing through keypoints in the input images to elevate both the efficiency and accuracy of scene correspondences. Accumulated ray features along the feature volume provide a means to discount the coherence constraint amid erroneous ray matching. We demonstrate the effectiveness of CRAYM for both NVS and geometry reconstruction, over dense- or sparse-view settings, with qualitative and quantitative comparisons to state-of-the-art alternatives.

CRAYM: Neural Field Optimization via Camera RAY Matching
Liqiang Lin' Wenpeng Wu'
Chi-Wing Fu² Hao Zhang Hui Huang"*
*College of Computer Science and Software Engineering, Shenzhen University
2 Department of Computer Science and Engineering. The Chinese University of Hong Kong
SSchool Computing Science, Simon Fraser University linliqiaan manniiggnn2200mmmaa.. com wenpenggeg@gmail... com
cwfuêcse cuhk, edu. hk haoz@sfu. hhzhiyandgmail,co com
Abstract
We introduce camera ray matching (CRAYM) into the joint optimization of camera poses and neural fields from multi-view images. The optimized field, referred as a feature volume, can be "probed" by the camera rays for novel view synthesis (NVS) and 3D geometry reconstruction. One key reason for matching camera rays, instead of pixels as in prior works, that the camera rays can be parameter- ized by the feature volume to carry both geometric and photometric information. Multi-view consistencies involving the camera rays and scene rendering can be naturally integrated into the joint optimization and network training, to impose physically meaningful constraints to improve the final quality of both the geometric reconstruction and photorealistic rendering. We formulate our per-ray optimization and matched ray coherence by focusing on camera rays passing through keypoints in the input images elevate both the efficiency and accuracy of scene correspon- dences. Accumulated ray features along the feature volume provide means to discount the coherence constraint amid erroneous ray matching. We demonstrate the effectiveness of CRAYM for both NVS and geometry reconstruction, over dense- or sparse-view settings, with qualitative and quantitative comparisons to state-of-the-ard alternatives.
1 Introduction
Recent advances multi-view 3D reconstruction have been propelled by the emergence of neural fields [39], including implicit functions [42, 36, 24, 33] and radiance fields (NeRF) [26, 44, 13, 1, 10, 27, 19, 7]. A critical component to all image-to-31) reconstruction methods, including traditional approaches such as multi-view stereo (MVS) [11], to obtain camera poses for the input images. In practice, the camera information may be available from the acquisition devices, e.g., through the GPS or inertial measurement unit (IMU), while in other cases, it is estimated, e.g., using structure-froms motion (SfM) [8, 29]. In both cases, these camera poses can be noisy, thus hindering the performance of the multi-view 3D reconstruction.
In light of the importance of having accurate camera poses, various methods have been proposed to improve their estimations. One line of approaches, which can be referred to as bundle-adjusting neural fields, jointly optimize [38, 20, 12, 2, 5] camera poses along with results from rendering and geometry (e.g.. depth) estimation, where the camera rays are considered independently for their roles in color and geometry prediction. By now, more works [3, 4, 32, 18, 16, 30, 6] realize the importance of exploiting correlations between input images, i.e., multi-view consistency, to impose additional
"Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

SPARE
L2G-NeRR
Ours
Figure I: Our method, neural field optimization with camera ray matching (CRAYM), incorporates contextual information for per-ray processing and enforces color + geometric consistence between matched rays. Compared SPARF [32] which utilizes dense pixel correspondences and the state- of-the-art, bundle-adjusting L2G-NeRF [5], both aimed at handling noisy camera poses, CRAYM produces superior results especially over fine details; the zoom-ins on the right. Results shown the Drums model from NeRI-Synthetic [26] on dense views.
constraints on the joint optimization. Along these lines, much effort has been invested into matching and optimization with respect image features, whether convolutional or transformer-baat based.
Motivated by multi-vien spatial analysis, several works have proposed geometric constraints involv- ing camera rays and projections [16, 32, 18]. Most recently, SPARF [32] defines re-projection loss as a spatial distance between image pixels to enforce that matched pixels between NeRF training 3D point. However, the effectiveness of this loss depends
images be back-projected onto the critically on how reliable the pixel correspondences are. In their work, these correspondences and their confidence estimates were both obtained by pre-trained network [31], which independent of the joint camera-scene optimization.
In this paper, introduce camera ray matching into the joint optimization of camera poses and a neural field, referred to as a feature volume, which can be "probed" by the camera rays for both rendering, e.g., novel view synthesis in NeRF [26], and 3D reconstruction, as in NeuS [34].
The key reasons for matching camera rays, instead pixels [32, 18, 6], are two-fold. First, these rays carry 3D spatial information than just 2D pixel values to facilitate formulating explicit geometric losses when optimizing camera poses [16], as dictated by multi-view analysis. Second and more importantly, the camera rays can be parameterized by the feature volume - they carry both geometric and photometric information. Any constraint arising from camera ray matching can be passed onto the feature volume. Hence, both the matching itself and the associated matching confidence can be incorporated into the joint optimization and network training, to impose physically meaningful constraints improve the final quality of both geometry reconstruction and rendering.
Our network, coined CRAYM (for Camera RAY Matching), takes as input an uncalibrated set of images capturing a 3D object, and is trained to predict the feature volume along with all the camera rays subjected to a combination of photometric rendering losses and geometric losses dedicated to ensuring multi-view consistency between camera rays. We consider two types of rays. The first are called key rays, which pass through keypoints detected in the input images, typically spanning regions with sharp features and rich textures over the 3D object. The other rays called auxiliary rays, which pass through points around keypoints to offer contextual and local structural information as we reason about the key rays in our optimization framework.
As our main constraint for matched ray coherence, we enforce color consistency between renderings along two key rays whose corresponding keypoints from two different views are matched [9]. However, we must account for potential erroneous matches due to occlusion or unreliable local image features used by the matching network. To this end, we aggregate features along each key ray through the feature volume. The matchability between two rays is defined by cosine similarity between the accumulated ray features and applied as weight to either accentuate discount the

---

color consistency constraint, allowing our optimization model to naturally degenerate itself to handle unrelated rays separately. Also, to improve the robustness of feature learning, we enhance the feature along each key ray by integrating features from surrounding auxiliary rays.
We evaluate our method on both the synthetic objects from NeRF-Synthetic [26] and the real scenes from UrbanScene3D [22], for novel view synthesis and 3D geometry reconstruction, over dense- and sparse-view settings. Compared state-of-the-art alternatives, CRAYM produces superior results especially over fine details.
2 Related Works
Neural Fields. As a pioneer work, NeRF [26] synthesizes novel views of static objects/scenes from a set of posed images by optimizing a coordinate-based neural network, which predicts the volume density and color for sampled point in the 3D space. Since then, numerous methods have emerged to improve the rendering quality [44, 13. 1] and rendering efficiency [10, 27, 19, 7]. To extract high-quality surfaces from the learned implicit representation, NeuS [42] and VoISDF [42] propose to learn an implicit signed distance field (SDF) representation for the scenes. These methods can achieve impressive results both novel view synthesis and 3D reconstruction, however, the requirement of precise camera pose limits their applicability in practice.
Bundle-Adjusting Neural Fields. With the realization that positional encoding susceptible to suboptimal registration, BARF [20] applies a smooth mask on the encoding at different frequency bands for coarse-to-fine fine training, while [12] presents an adaptive positional encoding. L.2G-NeRF [5] first learns the pixel-wise transformations for every pixel in a frame and then aligns the frame-wiss transformation with the pixel-wise transformations. Common to all the above methods is that their joint optimization of pose and scene representation processes each image and each ray separately, without considering their multi-view correlations. As a result, the pose optimization may not be stable, thereby leading to floaters and blurriness in both novel view renderings and 3D reconstruction. Note that our method also involves per-ray processing, by combining information from auxiliary rays with that of key ray. This is similar to the patch-level feature processing in CR-NeRF [41], which considers multiple rays indiscriminately across the image, without the notion of key rays.
Neural Fields with Image Matching. Image matching can help establish geometric priors to improve the generalizability of NeRF, to either novel scenes or the sparse- view setting. MVSNeRF [3] constructs a cost volume by warping the image features extracted with 3D CNN onto a plane sweep, from which a generalizable radiance field is learned. SparseNeuS [24] constructs a 3D volume with the variance of all the projected features from multi-view images. DBARF [4] optimizes camera poses and depth with a cost map constructed by the differences of image features. CorresNeRF [18] proposes to regularize the NeRF training with a pixel re-projection loss for the associated pixels and depth loss for the predicted depth. GPNR [30] aggregates features of the image patches along epipolar lines with several stacked transformers. MatchNeRF [6] learns generalizable NeRF with the cosine similarity of image features for each image pair the shape prior. All these methods integrate image features and utilize the matching within or between different views. With more emphasis placed multi-view geometry reasoning, SCNeRF [16] learns a pinhole model for each camera under the supervision of a re-projected ray distance loss, while SPARF [32] optimizes its network with re-projection loss, measuring spatial distances between pixels in the same view. In contrast, the matched ray coherence formulation in our optimization accounts for both photometric and geometry information obtained from the feature volume; the coherence constraint is also explicitly integrated into the network instead of only serving to define a loss.
3 Method
We are interested in neural networks that can reconstruct a 3D model, e.g., radiance field [26] or an implicit field [34], from a set of M images ( capturing a 3D object from multiple views. Typically, each image is associated with known estimated camera pose T, = [R,]ts], where R, E S3( and t, E R The network is trained by minimizing a photometric error Lp between the input images and the multi-view renderings, {ī, MMM 1. of the target 3D object from the camera views: min E, Σx (x) - (x(x) 12, where I, [xx) is the color of image I, at pixel z.
3

---

Input Images
Rendered Images
Photometric
3D Feature Volume V
KRE Module
Loss
Fused Color
Epipolar
Loss
MRC Module
Point-alignment
Loss
Key Rays Enrichment Module (KRE)
Matched Rays Coherency Module (MRC)
SDF
Auxiliary
Key
Geometry Network
Texture Network
Ray Rendering
Figure 2: Overview of our CRAYM pipeline. After extracting keypoints (red dots) from input images and matching them using pre-trained network, we train our CRAYM network optimize a 3D feature volume V which encodes both geometric and photometric information about the target 3D object and can be queried by camera rays for both novel view synthesis (via the Texture Network) and 3D reconstruction (via the Geometry Network). The volume optimization is subject to photometric losses through rendering along camera rays passing through the keypoints (i.e., the key rays). which is enhanced (in the KRE) by integrating features from auxiliary rays, i.e., rays passing through nearby auxiliary points (yellow dots) in the images. Matched ray coherence (MRC) is imposed on matched key rays, in terms of color consistency, while potentially mismatched rays can be identified by comparing accumulated features along the key rays through V. On top of the standard photometric loss, we introduce two geometric losses, the epipolar loss and point-alignment loss, to explicitly optimize ray-to-ray coherency to maximize the reconstruction quality of the feature volume.
Each pixel is associated with specific ray in 3D from the object/scene, through the pixel center, towards the camera: (r(t) ro tralt 2 0). where r, is the camera center and r is the normalized view direction of ray The rendered color of ray r, i.e., the pixel color [x)), can be produced using volume rendering by accumulating the color and opacity a along the ray r.
Considering that the camera poses can be noisy, the reconstructed radiance field or implicit field may not produce clean and sharp renderings with details. At high noise level, some methods may even fail to produce results; see examples shown in Sections 4.2 and 4.4. Beyond existing approaches that map r(t) to opaque density (or opacity) and color implicitly with ray-wise network, we propose CRAYM to learn the implicit field by matching rays across different images and formulating geometric priors.
3.1 The CRAYM Pipeline
Figure 2 overview our CRAYM pipeline. From the input images, our goal in the 3D neural field optimization is to construct a 3D feature volume V to faithfully represent the target object. In detail, we represent feature volume using multi- resolution hash encoding [27] and end-to-end optimize for the target object. The feature f() of point in the 3D feature volume can be extracted by
f(p) = M(V(p)).
where M is the progressive feature mask [19] for filtering out fine-level features during early iterations of the coarse-to-fine training. Very importantly, to account for the noise in the camera poses, we parameterize the transformation matrices of the cameras as variables the joint optimization of the pose and implicit field with the feature volume V.

---

As mentioned in the introduction, we consider two types of rays to probe the feature volume. i.c. key rays and auxiliary rays. Both rays are issued from the cameras through the pixel centers. To obtain key rays (r&}. which typically associate to surface points with rich textures and sharp features, we detect keypoints on each input image using SuperPoint [9] and perform point-to-poing matching between image pairs using SuperGlue [28]. Then, we can obtain set of sparse ray-to-ray matchings between image pairs. Note that these results may not be accurate for various reasons such as occlusion and unreliable matching, but they provide useful information for our pipeline to start with. As for the auxiliary rays {r they are sampled around the keypoints to provide contextual or local structural information when we reason about the key rays; see Section 3.2 for details.
Once optimized, the feature volume can be used for novel view synthesis or for multi-view 3D reconstruction. The color prediction for novel view synthesis is accomplished by texture network 1, as in a typical NeRF [26] setting, and the latter is accomplished by geometry network gg. as in typical NeuS [34] setting. Specifically, the geometry network takes a 3D point p sampled along r and the feature at p as input to produce an SDF value and then an opaque density a to render the 3D object and extract the 3D reconstructions. Here, we propose the Key Rays Enrichment (KRE) module (Section 3.2) to improve the robustness in the process by enhancing the features along the key ray using the features sampled by the auxiliary rays.
Subsequently, the texture network takes the output features from geometry network, ray directions, and normal at as inputs to predict color c(p) at point Further, we design the Matched Rays Coherency (MRC) module (Section 3.3) to enhance the volume rendering quality by considering matchability between rays and learning to maintain coherency between ray matchings. Particularly, the MRC module can effectively reduce the influence of mismatched rays by disambiguating the camera ray matchings.
pair of the matched key rays, Fk and rk', are sampled with the corresponding auxiliary rays during each iteration. The geometry network, texture network, and feature volume optimization are jointly trained end-to-end. Besides the photometric loss, formulate the epipolar loss and point-alignment loss (Section 3.4) to explicitly promote coherency among the ray matchings and boost performance.
3.2
Key Rays Enrichment Module
As the input images are captured through perspective projection, all rays in 3D through the same image should converge at a common camera point. In previous works, for each iteration, rays optimized separately, the pose optimization may not be stable. As different rays may back propagate gradients in different directions, the optimized poses may oscillate during the training. Hence, we introduce the KRE module to stabilize the optimization by learning structural information around each key ray. This is done by sampling auxiliary rays around the key ray to enrich the feature of the key ray with more contextual information:
f'())) Σ E9(((n))
where pk is point along key ray rk; {ęr} are points around pa sampled along the j-th auxiliary ray around r&; and function g fuses features f(pk) and fq:)). Then, we employ the geometry network y to predict the SDF value at Pk and feature vector f" (Pc)), from which we can further obtain the color of point pi with the texture network. Please refer to the supplemental materials for the details.
3.3 Matched Rays Coherency Module
Next, we propose to learn the coherency of features accumulated in the 3D feature volume between the matched key rays. The purpose is to enhance the camera ray matching and account for imprecise ray matchings, since keypoints matching is performed only on local image features.
Similar to color accumulation in volume rendering, we calculate the aggregated feature along key ray rk as the feature of rk:
f(xx) = / (f
(3)
The function T(nx(1)) exp(- fn (s)ds) denotes the accumulated transmittance along key ray Tk.
5

---

Pk
Pk
Pk.
x
X
x
e
0"
O'
0"
(a) Epipolar Loss
(b) Paint-alignment Loss
Figure 3: Illustrating of geometric losses. The red lines in the left subfigure epipolar lines. The epipolar loss constrains the relative transformations between cameras, so that the projection of keypoint p& onto the image plane of the other camera should lie on the epipolar line e'zi. With the camera poses constrained by the epipolar loss, the voint-alignment loss further constrains the depth of 2g and oz aiming align pk and PA with P.
Essentially, the accumulated ray feature ((rk ) an integration of density-weighted features along ray. When the network converges, the actual surface point at which ray F intersects with the paired matched key ray should have the highest density. Hence, we consider coherency between the matched rays to optimize the learning of the opacity density and point color, such that we enhance the coherency of features accumulated along the matched key rays. In return, this will help to optimize the parameters and the 3D feature volume, when training the pipeline. Therefore, we fuse the rendered color c(rg) ) the matched rays based on the cosine similarity between their accumulated features:
c(rx) wc(r)) (1 w)((r=),
where w is the matchability calculated as the cosine distance between the accumulated features of the matched rays. When two key rays are mismatched, e.g., due to occlusion ambiguities of weak texture areas and similar structures, our formulation can learn to degenerate itself to form that separately optimizes individual rays.
3.4 Loss Function
Further, we introduce the following two geometric losses to more explicitly promote the coherency of the ray matchings:
Epipolar loss. Given pair of matched keypoints Iy and z/f two different input images, which associate with camera centers O and O', respectively, (see the illustration in Figure 3), we can estimate the depths 2g and 2/g by using depth accumulation formulation similar Equation 3, and then project points and Ty into the 3D object space to obtain 3D locations and ph. respectively.
If the camera poses, the matchings, and the depths precise, the two rays through Iı and z', should precisely intersect at a common point, say P. on the target object surface, such that pk and p/2 align with P. Also, we denote and c' as the epipolar points the two images; these points image-space locations at which the line 00' intersects the two image planes; see Figure 3(a).
the
During the training, the depth estimation of can vary, so may vary along ray If the camera poses are precise, the projection of pt: onto the image plane of the other camera should lie on the epipolar line e'O'. In case of noisy camera poses, the projection of Pı: may not lie exactly on e'O', so we explicitly enforce the epipolarity during the training by minimizing the distance between pk projection and the epipolar line 24. using
No NR.
L.
(5)
Since the epipolar loss is not affected by depth, we decouple the unreliable depth estimation from the epipolar loss with ray marching to constrain the camera poses.
Point-alignment loss. The epipolar loss focuses on enhancing the projection consistency for producing more precise camera poses. To complement we introduce the point-alignment loss to facilitate depth convergence for improving the reconstruction of fine details. In detail, we consider the triangle

---

Initial Poses
BARF
L2G-NeRF
Ours
Figure 4: Visualization the initial and optimized camera poses for the LEGO scene in the NeRF- Synthetic dataset [26]. (Purple: ground-truth poses; blue: initial or optimized poses; red lines: translation errors.)
formed by intersection point P. line segment PsO,, and line segment p/,0' (Figure 3(b)), and aim to minimize the distance between points pk and p and align them:
No
La
Dist
(6)
N
Note that depth estimation may be unreliable and likely unstable early in the training, so we use the point-alignment loss only after certain number of training iterations.
Overall loss. After constructing the epipolar loss Le and the point-aligmmen loss Las we put them together with the photometric loss Lp and SSIM loss L, to form the overall loss function
== A * Tep + X2 * Leg + X3 The + X4 T.a
where Lp is calculated between the input images and the rendered images and is modeled as an MSE loss.
4 Results
We evaluate our method on the NeRF-Synthetic dataset [26] with eight synthetic objects (Section 4.2),
the LLFF dataset [25], and the real scenes from the UrbanScene3D dataset [22] (Section 4.3), We
compare our method on both novel view synthesis and 3D reconstruction with NeRF [26], NeuS [34],
BARF [21], I.2G-NeRF [5], PET-NeuS [37], SPARF [32], and BAA-NGP [23]. Since NeRF [26),
NeuS [34], and PET-NeuS [37] are designed for neural implicit field with fixed and precise poses, we
the camera transformations as variables be optimized jointly with the neural field, as in our
method.
While other methods optimize the radiance field, in which the target values, radiances, of points more independent each other, the optimization of SDFs in NeuS and PET-NeuS poses a challenge to the requirements of non-local geometric constraints correctly form the shape, making them more vulnerable to unstable pose optimization. As the camera rays are parameterized by our feature volume earry both geometric and photometric information, our geometric constraints the camera ray matching can effectively lead to better optimization of the geometry. The joint optimization of camera pose and implicit SDF may also fail produce results for NeuS and PET-NeuS, when the camera poses are at a high noise level. With the assistance of camera ray matching, CRAYM outperforms other methods on both novel view synthesis and 3D reconstruction at varying noise levels. We report the PSNR, SSIM, and LPIPS [43] for quantitative comparisons on novel view synthesis and Chamfer distance (CD) for the 3D reconstruction. test-time photometric pose optimization is performed to evaluate these metrics, following prior works [21, 32, 5]. The quantitative evaluations on the other metrics are provided in the supplementary materials.
4.1 Pose Alignment
To evaluate the registration quality of the optimized training poses, we use Procrustes analysis [14] find the 3D similarity transformation that aligns the optimized training poses with the calibrated camera poses, following BARF [21]. As Figure 4 shows, the optimized poses produced by CRAYM align well with the ground-trui truth poses with lower translation errors.
7

---

SPART
L2G-NeFF
Outs
Figure 5: Qualitative comparison results of novel view synthesis and surfaces reconstruction on the synthetic objects.
The average translation errors and rotation er-
Table
Poses registration errors evaluated on the LEGO scene in the NeRF-Synthetic dataset [26].
are reported in Table With the contextual information and feature coherency of camera ray matching. the camera poses produced by CRAYM are better optimized for the construc- tion the implicit filed, so that CRAYM is able to produce high-quality rendered views, well as more precise 3D reconstructions.
Method
Rotation (*)
Translation
BARF [21]
ICCV'21
9.02
0.81
SPARF [32]
CVPR23/21
10.64
0.53
[5]
CVPR23/21
CRAYM (ours)
1.21
0.19
Table 2: Results on the NeRF-Synthetic dataset.
4.2 Evaluation on Synthetic Objects
Metries
PSNR1
IPIPS..
CD)
For the evaluation on the NeRF-Synthetic dataset, we follow the same setting of noisy poses L2G-NeRF [5], which perturbs the ground-truth camera poses with additive noise as the initial poses. As NeuS [34] and PET- NeuS [37] fail to produce results at such a set- ting, present only the results of NeRF [26], BARF [21], SPARF [32], and L2G-NeRF [5]. The mean results of the eight objects and four of them are given in Table 2. NeRF fails to extract meshes from the reconstructed radiance fields on Hotdog and Ship. Figure 5 shows the results of view synthesis and 3D reconstruction visually for BARF, SPARF, L2G-NeRF, and our method, SPARF produces over smoothing results with dense input, as shown in Figure 5. As can be seen in Figure 5, CRAYM is able produce clean and complete renderings and reconstruc- tions with fewer floaters and less blurriness.
NeRF
16.69
0.77
0.39
2.23
BARF [21]
28.55
0.93
0.06
0.09
Chair
SPARF [32]
23.80
0.87
0.19
0.14
L2G-NeRF [5]
30.99
0.95
0.05
0.10
(ours)
0.06
NeRF [26]
15.07
0.74
0.42
N/A
BARF [21]
30.12
0.95
0.04
0.38
Hotdog
SPARF [32]
29.10
0.13
0.07
L2G-NcRF [5]
34.56
0.97
0.03
0.38
CRAYM (ours)
36.42
0.02
0.05
NeRF [26]
11.11
0.60
0,58
0.58
BARF [21]
22.54
0.79
0.12
0.04
LEGO
SPARF [32]
22.47
0.80
0.25
0.09
L2G-NeRF [S]
27.71
0.91
0.06
0.12
CRAYM (purs)
31.60
0.96
0.03
0.04
NeRF [26]
13.08
0.73
0.53
0.48
[2.]]
30.37
Mic
SPARF [32]
28.36
0.91
0.17
0.24
L2G-NeRF [S]
30.91
0.97
0.05
0.17
CRAYM (ours)
31,02
0.97
0.05
0.04
NeRF [26]
13.29
0.68
0.49
0.64
[21]
Mean
SPARF [32]
23.90
0.84
0.23
0.18
L2G-NeRF [5]
28.62
0.07
0.17
CRAYM (ours)
30.34
0.95
0.05
0.06
4.3 Evaluation on Real Scenes
We first evaluate our method the LLFF
dataset [25] for high-fidelity view synthesis the eight real scenes. Compared with BARF [21],
L2G-NeRF [5], and BAA-NGP [23], our method is able to produce high-quality results with fewer artifacts and better scores in terms of PSNR, SSIM, and LPIPS, as shown in the Table 3.

---

Table 3: Qualitative comparison on novel view synthesis on the LLFF dataset [25].
PSNR 1
SSIM T
LPIPS
BARF [21]
L2G-NeRF
BAA-NGP
CRAYM
BARF [21]
L2G-NeRF
BAA-NGP
CRAYM
BARF [21]
L2G-NeRF
BAA-NGP
CRAYM
[5]
[23]
(ours)
[5]
[25]
(OWE5)
[5]
[23]
(ouES)
Pern
23.88
24.57
24.83
0.25
0.50
0.26
0.25
34.90
Horns
23.29
23.12
19.71
23,30
0.74
0.74
0.72
0.75
0.29
0.26
0.24
0.24
Leaves
18.91
19.02
19.96
19.57
0.55
0.56
0.68
0.60
0.35
0.35
0.23
0.29
Orchids
19.40
19:71
12.45
19.51
0.57
0.61
0.14
0.59
0.29
0.25
0.42
0.23
Roum
32,05
32.25
29.72
32,44
0.94
0,95
0.90
0.91
0.10
0
0.09
24.54
22.52
Reference
NeuS
Pet-NeuS
Ours
Figure 6: Qualitative results of novel view synthesis and surfaces reconstruction on real scenes captured by high-resolution cameras.
As the images in the LLFF dataset captured with a restricted range angles, we further as- sess method on the real scenes PolyTech and ArtSci from the UrbanScene3D [22] dataset with two sets of drones captured images [45] as- sociated with GPS information. The large-scale scenes are captured with hundreds of images, which share smaller overlaps than the images from the NeRF-Syntheti dataset. Considering the limitations of memory, we reduce the size of the original high-resolution images by using bicubic interpolation. As the GPS information may not reliable and the positions GPS may shift in meters, we preprocess the poses from GPS with COLMAP [29] and add a small noise to the calibrated poses, following 1.2G-NeRF [5].
Table 4: Results on the real scenes PolyTech and
ArtSci in the UrbanScene3D [22] dataset.
Scene
Method
PSNR
SSIM
LPIPS)
CD.
NeuS [34]
14.27
0.42
0.71
0.10
PolyTech
PET.Neu8 [37]
13.64
0.40
0.75
0.07
CRAYM (ours)
22.51
0.63
Neu' [34]
13.49
0.33
0.89
0.10
AnSci
PET NeuS [37]
15.18
0.36
0.88
0.08
CRAYM (ours)
19.37
0.42
0.63
0.02
The UrbanScene3D dataset contains high-precision LiDAR scans for the target buildings, PolyTeck and ArtSCi. Therefore, evaluate the reconstructions quality with the point cloud scans as the ground truths. We crop the reconstructed meshes according to the LiDAR scans and align them using ICP. NeRF, BARF, and L2G-NeRF produce blurry renderings and degenerated meshes for the real scenes, while SPARF may fail to process such data with dense correspondences. Therefore, we only provide the visual results of NeuS, PET-NeuS, and our method.
The quantitative results of NeuS, PET-NeuS, and our method are provided in Table 4. As Figure shows, NeuS and PET-NeuS tend to produce extract meshes with fine details. A mesh reconstructed directly from the original high-resolution images using ContextCapture" commercial MVS solution, is provided as reference.
smoothing results, while method is able to
4.4 Ablation Study
Comparison varying noise levels. To evaluate model robustness, we evaluate the LEGO data sample with poses at varying noise levels. The "high noise level" means we use the same noise setting L2G-NeRF [5]; the "low noise level" means perturb the ground-truth poses with half of the noise as L2G-NeRF; and "w/o noise" means the poses are initialized as ground-truth poses without noise. The transformation matrices of the camera poses all set variables to be optimized. Table 5 summarizes the results. We can see that NeuS [34] and PET-NeuS [37] are more sensitive to noise and cannot effectively handle the high-noise setting, while the other methods can not produce

---

accuracy reconstructions. With the contextual information learned in the camera ray matching and the explicit utilization of matched rays, CRAYM is able obtain better results for all the noise settings.
Ablation of major modules and losses. To
Table 5: Comparing different methods at varying noise levels.
demonstrate the efficiency of the proposed mod- ules and geometric losses, we conduct an abla- tion study on the LEGO data sample. The re- sults are reported in Table 6. Similar with BARF, which applies a smooth mask on the encoding different frequency bands for neural radiance field, we apply a progressive feature mask on the hash encoding with coarse-to fine training of the neural implicit field as our baseline, which combines BARF [21] and NeuS2[35]. The KRE module improves the robustness to noisy poses in the training, whereas the MRC module ef- fectively enhances the quality of the volume renderings with ray matching. In addition to that, the proposed geometric losses further help our framework obtain better camera pose op- timization.
Method
PSNR1
SSIMI
I.PISS
CDJ
w/o Noise
NeRF [26]
29.08
0.94
0.04
0.34
NeuS [34]
21.18
0.09
0.04
BARF [21]
28.33
0.05
0.36
SPARF [32]
22.73
0.25
0.09
PET-NeuS [37]
21.37
0.14
0.10
L2G-NeRF [5]
27.94
0.92
0.06
0.14
CRAYM (ours)
32.72
0.97
0.02
0.03
Low Noise Level
NeRF [26]
24.86
0.88
0.09
0.34
NeuS [34]
21.76
0.14
0.05
BARF [21]
28.32
0.05
0.36
SPARF [32]
22.55
0.25
0.09
PET-NeuS [37]
21.34
0.11
0.55
L2G-NeRF [5]
27.75
0.06
0.21
CRAYM (ours)
32.68
0.97
0.02
0.04
5 Conclusion and discussion
High Noise Level
NeRF [26]
11.36
0.81
0.56
0.57
Our method, CRAYM, addresses the issue of noise camera poses for multi-view 3D recon- struction and view synthesis. The key idea is to jointly optimize neural field and camera poses by incorporating contextual information (via KRE) and enforcing geometric and pho- tometric consistency (via MRC and geometric losses) through camera ray matching.
NeuS [34]
N/A
N/A
N/A
BARF [21]
14.48
0.29
0.04
SPARF [32]
22.47
0.25
0.09
PET-NeuS [37]
N/A
N/A
N/A
N/A
L2G-NeRF [5]
27.71
0.91
0.06
0.12
CRAYM (ours)
31.60
0.96
0.03
0.03
Table 6: Ablation of major modules and losses.
Experiments demonstrate that our method out- performs tfate-of-the-ar alternatives under vari- ous settings: dense- vs. sparse-views, and differ- ent noise levels. However, the implicit field and optimizable pose transformations may not con- verge when the poses are randomly initialized or extremely noisy. stronger pose regulariza- tion prior to the field optimization may resolve this problem. Furthermore, the meshes extracted from the constructed SDFs may still contain messy inner structures invisible areas.
Method
PSNR1
SSIM
LPIPS,
CD1
L2G-NeRF [5]
27.71
0.91
0.06
0.12
NeuS2 [35]
26.83
0.86
0.17
0.08
Baseline
27.30
0.91
0.10
0.06
KRE
28.64
0.93
0.07
0.05
KRE MRC
30.41
0.95
0.04
0.04
+ L.
29.43
0.92
0.07
0.05
+ L.
29.95
0.94
0.06
0.04
Our full pipeline
31.60
0.96
0.03
0.04
promising future work apply the ray matching the 3D Gaussian splatting, which will greatly improve the rendering efficiency of CRAYM. However, extracting reconstructions with fine geometric structures from 3D Gaussians still an open problem.
Finally, CRAYM has been designed to rely on sparse key rays for dense-view reconstruction, while dense counterpart may bring up extra overhead. In Matched Ray Coherency formulation, explicitly account for potentially erroneous (i.e., fow-quality) 2D matches by using the matchability between two rays as weight to either accentuate or discount the color consistency constraint. In terms of sensitivity with respect to the density of the 2D matches, in our experiments, we have observed that even with sparse input views and sparsely distributed matched rays. CRAYM can still notably improve the optimization convergence. An effective approach to utilize ray matching for both sparse and dense inputs may further boost the performance of CRAYM.
10

---

Acknowledgement
We thank the reviewers for their valuable comments. This work was supported parts by NSFC (U21B2023, U2001206), ICFCRT(W2441020) Guangdong Basic and Applied Basic Research Foundation (2023B 1515 121120),,) DEGP Innovation Team (2022KCTD0025, Shenzhen Science and Technology Program (KQTD202 108 10100040040 RCIC20200774114435000 and Scientific Development Funds from Shenzhen University.
References
[1] Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brually and Pratul P Srinivasan. Mip-NeRff: multiscale representation for anti-aliasing neural radiance fields. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition, pages 5855-5864, 2021.
[2] Wenjing Bian, Zirui Wang, Kejie Li, Jia- Vla-Wang Bian, and Victor Adrian Prisacariu. Nope-NeRF: Optimising neural radiance field with no pose prior. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition, pages 4160 4169, 2023.
[3] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. MVSNeRF: Fast generalizable radiance field reconstruction from multi-view stereo. In Proc. Int. Conf. Computer Vision, pages 14124-14133, 2021.
[4] Yu Chen and Gim Hee Lee. DBARF: Deep bundle-adjusting generalizable neural radiance fields. In Proc. IEEE Conf. Computer Vision & Pattern Recognition, pages 24-34, 2023.
[5] Yue Chen, Xingyu Chen, Xuan Wang, Qi Zhang, Yu Guo, Ying Shan, and Fei Wang. Local-to- global registration for bundle-adjusting neural radiance fields. In Proc. IEEE Conf. Computer Vision & Puttern Recognition, pages 8264-8273, 2023.
[6] Yuedong Chen, Haofei Xu, Qianyi Wu, Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai. Explicit correspondence matching for generalizable neural radiance fields, 2023.
[7] Zhiqin Chen, Thomas Funkhouser, Peter Hedman, and Andrea Tagliasacchi. MobileNeRF: Exploiting the polygon rasterization pipeline for efficient neural field rendering on mobile architectures. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition, pages 16569- 16578, 2023.
[8] David Crandall, Andrew Owens, Noah Snavely, and Dan Huttenlocher. Discrete-contingg optimization for large-scale structure from motion. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition, pages 3001-3008, 2011.
[9] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. SuperPoint: Self-supervised interest point detection and description. In Proc. IEEE Conf. on Computer Vision & Putlern Recognition, pages 224-236 2018.
[10] Sara Fridovich-Keil Alex Yu. Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition, pages 5501-5510, 2022.
[11] Yasutaka Furukawa and Jean Ponce. Accurate, dense, and robust multiview stereopsis. IEEE Trans. Pattern Analysis & Machine Intelligence, 32(8):1362- 1376. 2010.
[12] Zelin Gao. Weichen Dai, and Yu Zhang. Adaptive positional encoding for bundle-adjusting neural radiance fields. In Proc. Int. Conf. on Computer Vision, pages 3284-3294, 2023.
[13] Wenbo Hu, Yuling Wang, Lin Ma, Bangbang Yang, Lin Gao, Xiao Liu, and Yuewen Ma. Tri-MipRF: Tri-mip representation for efficient anti-aliasing neural radiance fields. Proc. IEEE Conf. on Computer Vision & Pattern Recognition, pages 19774-19783, 2023.
[14] John Hurley and Raymond B Cattell. The procrustes program: Producing direct rotation to test hypothesized factor structure. Behavioral science, 7(2):288, 1962.
11

---

[15] Rasmus Jensen, Anders Dahl, George Vogiatzis, Engin Tola, and Henrik Aanæs. Large scale multi-view stereopsis evaluation. In Proc. IEEE Conf. on Computer Vision & Pattern Recogni- lion, pages 406-413, 2014.
[16] Yoonwoo Jeong, Seokjun Ahn, Christopher Choy, Anima Anandkumar, Minsu Cho, and Jaesik Park. Self-calibrating neural radiance fields. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition, pages 5846-5854, 2021.
[17] Arno Knapitsch, Jaesik Park, Qian-Yi Yi Zhou, and Vladlen Koltun. Tanks and temples: Bench- marking large-scale scene reconstruction. ACM Trans. on Graphics, pages 1-13, 2017.
[18) Yixing Lao, Xiaogang Xu, Zhipeng Cai, Xihui Liu, and Hengshuang Zhao. CorresNeRF: Image correspondence priors for neural radiance fields. In Proc. Conf. on Neural Information Processing Systems, 2023.
[19] Zhaoshuo Li, Thomas Mülerr, Alex Evans, Russell H. Taylor, Mathias Unberath, Ming- Yu Liu, and Chen-Hsuan Lin. Neuralangelo: High-fidelity neural surface reconstruction. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition, pages 8456-8465, 2023.
[20] Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey, BARF: Bundle-adjussing neural radiance fields. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition, pages 5741-5751, 2021.
[21] Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey. BARF: Bundle-adjusting neural radiance fields. In Proc. Int. Conf. Computer Vision, 2021.
[22] Liqiang Lin, Yilin Liu, Yue Hu, Xingguang Yan, Ke Xie, and Hui Huang. Capturing, recon- structing, and simulating: the UrbanScene3D dataset. In Proc. Euro. Conf. Computer Vision, pages 93-109, 2022.
[23] Sainan Liu, Shan Lin, Jingpei Lu, Alexey Supikov, and Michael Yip. BAA-NGP: Bundle- adjusting accelerated neural graphics primitives. In Proc. IEEE Conf. on Computer Vision & Puttern Recognition, pages 850-857, 2024.
[24] Xiaoxiao Long, Cheng Lin, Peng Wang, Taku Komura, and Wenping Wang. SparseNeuS: Fast generalizable neural surface reconstruction from sparse views. In Proc. Euro. Conf. on Computer Vision, pages 210-227, 2022.
[25] Ben Mildenhall, Pratul Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Trans. on Graphics, 2019.
[26] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In Proc. Euro. Conf. on Computer Vision. 2020.
[27] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. ACM Trans. on Graphics, pages 1-15, 2022.
[28]
Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. SuperGlue: Learning feature matching with graph neural networks. In Proc. IEEE Conf. on Computer Vision & Puttern Recognition, pages 4938-4944 2020.
[29] Johannes I. Schonberger and JichMichael Frahm. Structure-from-motiie revisited. In Proc. IEFE Conf. on Computer Vision & Pattern Recognition, pages 4104-4113, 2016.
[30] Mohammed Suhail, Carlos Esteves, Leonid Sigal, and Ameesh Makadia. Generalizable patch- based neural rendering. In Proc. Euro. Conf. on Computer Vision, pages 156-174, 2022.
[31] Prune Truong, Martin Danelljan, Lue Van Gool, and Radu Timofte. Learning accurate dense correspondences and when trust them. In Proc. IEEE Conf. on Computer Vision & Puttern Recognition, 2021.
12

---

[32] Prune Truong, Marie-Julie Rakotosaona, Fabian Manhardt, and Federico Tombari. SPARF: Neural radiance fields from sparse and noisy poses. In Proc. IEEE Conf. on Computer Vision & Puttern Recognition, pages 4190-4200, 2023.
[33] Aditya Vora, Akshayand Gadi Patil, and Hao Zhang. DiViNeT: 3D reconstruction from disparate views via neural template regularization. In Proc. Euro. Conf. on Computer Vision, pages 210-227, 2022.
[34] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. NeuS: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. Proc. Conf. Neural Information Processing Systems, pages 27171-27183, 2021.
[35] Yiming Wang, Qin Han, Mare Habermann, Kostas Daniliddis, Christian Theobalt, and Lingjie Liu. NeuS2: Fast learning of neural implicit surfaces for multi-view reconstruction. In Proc.
Int. Conf. on Computer Vision, 2023.
[36] Yiqun Wang, Ivan Skorokhodov, and Peter Wonka. HF-NeuS: Improved surface reconstruction using high-frequency details. Proc. Conf. Neural Information Processing Systems, pages 1966-1978, 2022.
[37] Yiqun Wang, Ivan Skorokhodov, and Peter Wonka. PET-NeuS: Positional encoding tri-planes for neural surfaces. In Proc. IEEE Conf. Computer Vision & Pattern Recognition, pages 11998-12607,21 2023.
[38] Zirui Wang. Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. NeRF- Neural radiance fields without known camera parameters. arXiv preprint arXiv:2102.07064 2021.
[39] Yiheng Xie. Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. Neural fields in visual computing and beyond. Computer Graphics Forum, 2022.
[40] Weidan Xiong, Hongqian Zhang, Botao Peng, Ziyu Hu, Yongli Wu, Jianwei Guo, and Hui Huang. Twintex: Geometry-ave texture generation for abstracted 3D architectural models. ACM Trans. on Graphics (Proc. SIGGRAPH Asia), pages 227:1-227:11 2023.
[41] Yifan Yang, Shuhai Zhang, Zixiong Huang. Yubing Zhang, and Mingkui Tan. Cross-ray neural radiance fields for novel-view synthesis from unconstrained image collections. In Proc. Int. Conf. on Computer Vision, pages 15901-15911, 2023.
[42] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. Proc. Conf. on Neural Information Processing Systems, pages 4805 4815, 2021.
[43] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features perceptual metric. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition, pages 586-595, 2018.
[44] Kun Zhou, Wenbo Li, Yi Wang, Tao Hu, Nianjuan Jiang, Xiaoguang Han, and Jiangbo Lu. NeRFLiX: High-quality neural view synthesis by learning degradation-driven inter-viewpoinl mixer. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition, pages 12363-12374, 2023.
[45] Xiaohui Zhou, Ke Xie, Kai Huang, Yilin Liu, Yang Zhou, Minglun Gong, and Hui Huang. Offsite aerial path planning for efficient urban scene reconstruction. ACM Trans. on Graphics (Proc. SIGGRAPH Asia), pages 192:1-192:16, 2020.
13

---

A Appendix supplemental material
A.1
Implementation
A.1.1 Camera Pose Parameterization.
The camera poses [Ti). where T; = [Ra[t]] E SE(3), need to be parameterized and optimized in the training process. As a high frequency operation, pose parameterization needs to be simple and efficient, since the conversion between pose parameterization and transformation matrix is performed in every training iteration.
The position component of is simply represented as a three-dimensional vector. Instead of parameterizing the rotation R. as exponential map exp(r) from the Lie algebra (3) to the Lie group SO(3), we represent R as a six-dimensional vector composed by R = [ta [%)., 1/ E R V6 R Vay lb, and t.c 0a 2% span the three bases of the camera space. To obtain rotation matrix R from I, we perform a Fraam-Shhiie orthonormalization on ua Uh, and eg resulting in three orthonormal bases of the camera space, Va. Ub. and o. Thus, the rotation matrix
R= 10 10 1.
(8)
A.1.2 Key Rays Enrichment Module
Figure 7: Visualization of the matches between two images from the LEGO scene in the NeRF- Synthetic dataset [26].
Figure 7 shows an example of matched pixels. Due to the noise in the camera poses, the surface point captured by the associated keypoint on the input image may not intersect with the key ray rę exactly, the integration of features from the neighboring points {q:} sampled by auxiliary rays {r.) can greatly facilitate a more stable optimization associated with key ray rk; see Figure
Procedure-wise, we propose to first fuse the feature of pæ with the features of the surrounding {q:} to produce the enriched feature f'(ps), which can better describe the local radiance feature and geometry feature of the captured object with structure information.
No
S'(pk) = Σ Softmax( * f(g))) [g:)-
The features {f():)} of the auxiliary rays (ra) remain untouched: f'(g)) f(g.). Further, we adopt the geometry network 4, to process the features f'(p) of both key rays and auxiliary rays to extract the SDF value at point p in the radiance field. From the extracted SDF values, can then produce the 3D reconstruction the target object:
(SDF
(10)
where f"(p) is the output feature of point p. f") is one of the inputs to the texture network ot- The contextual information of ray matchings facilitates a more stable pose optimization and promotes the details of the geometry, i..., the accuracy of the SDF values.
The color of point p sampled on key rays or auxiliary rays can then be obtained with the texture network o as
c(p) 4 (f())))) ra, Normal(p)),
(11)
where is the normalized direction of ray and Normal(p) is the normal of the implicit surface at p computed as the gradient of SDF(p). We take the normal at p into account boost the training of
14

---

%
Figure 8: Illustrating the Key Ray Enrichment Module. The yellow ray Fş is the key ray and the gray rays (ra] are the neighboring auxiliary rays. The surface point (the blue point) captured by the associated keypoint (the red point) on the input image may not intersect with rş exactly due learn the contextual information of the surface point with the
to the unreliable pose, while neighboring rays of point pk sampled on rk-
Figure 9: View synthesis and 3D reconstruction results the real scenes CSSE produced by method.
The color c(r) of ray is then rendered with the opaque densities and colors of all the points sampled along r, where the opaque densities are obtained with the SDF values, normal vectors, and ray directions [34].
Truth
NeuS
Neu-
Figure 10: Comparison of view synthesis results on the real scenes Bank.
A.2 Training Details
We 16-1evel hash grid [27] to encode the feature volume V. The feature length of each level two. Therefore, the base resolution of V is 32. We apply a progressive feature mask [19] on the hash encoding, which starts level 4 and updated the next level every 1,000 iterations. The geometry network 6g is implemented as a three-layer MLP with the ReLU activation for the input and hidden layers. The texture network Φi implemented as a four-layer MLP with the ReLU activation for the input and hidden layers. The ray directions are encoded using the spherical harmonic representation [10] and fed into the texture network ái together with the output features of o to predict the color of the sampled points on the ray. The whole network is optimized with the AdamW W optimizer with learning rate of 0.01, 3 [0.9, 0.99), and e 1.0-15 The variance [34] of geometry network Φ is initialized as 0.3 and is optimized with a learning rate of 0.001. We adopt warm-up training for the first 500 iterations with the LinearL.R .R scheduler. All the experiments are conducted with Nvidia GV 100.
15

---

Table 7: Comparison of reconstruction quality the NeRF-Synthen [26] dataset.
Metries
Method
Mcan
Chair
Drums
Ficus
Hotdog
LEGO
Materials
Mic
Ship
NeRF [26]
ECCV120
2.19
3.22
2.05
1.63
N/A
2.56
1.78
1.89
N/A
BARF [21]
ICCV'21
1.52
0.38
1.42
0.28
1.13
0.44
1.90
1.26
5.34
HD
SPARF [32]
CVPR 23
1.11
0.80
1.42
1,09
0.54
0.92
1.35
1.64
1.10
L2G--NeF [5]
CVPR'23
1.57
0.43
1.57
0.48
1.14
0.99
2.15
1.09
4.69
CRAYM (ours)
0.98
0.38
0.73
0.23
0.59
0.37
0.35
0.57
4.62
NeRF
ECCV
1.17
N/A
1.99
0.71
N/A
3.72
0.81
NIA
BARF [21]
10CV"21
15:97
24.60
15.96
9.23
7.44
42.89
18.74
3.279
5.63
Precisionf
SPARF [32]
CVPR
15.12
28.40
10.14
0.22
30.18
28.06
10.85
5.30
7.83
1.2G-NeRFF [5]
CVPR'23
17.36
21.81
23.01
10.59
3.38
43.71
18.31
10.56
7.54
CRAYM (ours)
30.59
33.53
45.44
17.55
30.50
49.58
19.16
38.75
15.60
NeRF [26]
ECCV"20
3.18
N/A
1.69
11.82
N/A
3.36
5.75
2.80
N/A
BARF [21]
20.43
24.25
3.33
17.47
24.33
18.75
12.24
1.81
Recall"
SPARF [32]
CVPR
19.38
16.17
6.09
17.54
37.34
13.68
12.60
5.40
19.38
L2G-NeRF [5]
CVPR 23
26.98
19.97
48.06
21.10
10.99
68 08.00
19:51
19.34
8.84
CRAYM (ours)
42.29
43.65
50.11
45,95
31.17
81.63
21.67
40.29
29.77
NeRF [26]
N/A
1.34
N/A
3.53
1.25
NIA
BARF [2]]
ICCV'21
16.32
24.42
5,51
12.08
11.39
50.47
18.75
5.17
2.74
F-score
SPARF [32]
CVPR'23
16.03
35.19
12.46
0.42
22.18
32.04
12.10
7.46
6.39
L2G-NeRF [5]
CVPR'25
20.64
5.17
$3.22
13.66
8.14
CRAYM (ours)
34.76
37.93
47.66
25.40
30.83
61.69
20.34
39.50
20.48
Ground Truth
PET-NeuS
SPARF
Ours
Figure 11: View Synthesis on the DTU [15] dataset.
Table 8: Comparison of reconstruction quality on the real scenes PolyTech and ArtSci of the UrbanScene3D [22] dataset.
Scene
Method
HDL
Precision
Recallf
F-scoref
NeuS [34]
0.63
41.32
55.68
47.44
PolyTech
PET-NeuS [37]
0.46
43.29
46.83
44.99
CRAYM (ours)
0.04
81.59
91.21
88.13
NeuS [34]
0.67
33.73
31.89
30.24
ArtSci
PET-NeuS [37]
0.32
21.30
15.69
18.07
CRAYM (ours)
0.19
59.01
52.54
55,59
Table 9: Comparison of novel view synthesis on the real scene Bank from TwinTex [40].
Method
PSNRT
SSIM
LPIPSJ
NeuS [34] PET-NeuS [37]
NIPS'21 CVPR'23
11.69
0.25
0.91
13.22
0.27
0.93
CRAYM (ours)
18.68
0.46
0.59
16

---

Table 10: Comparison of novel view synthesis on the DTU [15] dataset.
Scene
Method
PSNR1
SSIM
LPIPS
NeuS [34]
NIPS'21
20.55
0.64
0,41
PET-NeuS [37]
CVPR'23 23
20.77
0.65
0.42
Scan24
SPARF [32]
CVPR'23
22.80
0.78
0.29
CRAYM (ours)
30.53
0.96
0.02
NeuS [34]
NIPS'21
19.27
0.72
0.30
PET-NeuS [37]
CVPR'23
18.68
0.71
0.30
Scan37
SPARF [32]
CVPR'23
21.99
0.72
0.31
CRAYM (ours)
28.64
0.95
0.03
NeuS [34]
NIPS'21
23.88
0.64
0.52
PET-NeuS [37]
CVPR'23
22.78
0.63
0.51
Scan40
SPARF [32]
CVPR'23
23.85
0.72
0,38
CRAYM (ours)
26.86
0.91
0.06
NeuS [34]
NIPS'21
16.83
0.66
0.39
PET-NeuS [37]
CVPR'23
23.10
0.73
0.38
Scan55
SPARF [32]
CVPR/23
19.80
0.63
0.49
CRAYM (ours)
30.96
0.98
0.02
NeuS [34]
NIPS'21
28.45
0.92
0.10
PET-NeuS [37]
CVPR'23
26.89
0.90
0.09
Scan63
SPARF [32]
CVPR'23
27.45
0.92
0.12
CRAYM (ours)
31.30
0.98
0.01
A.3
Reconstruction Quality
Next, we report the Hausdorff distance (HD), precision, recall, and F-score [17] for the reconstruction
quality evaluated on the beRF-Synthetti [26] dataset and the UrbanScene3D [22] dataset. The NeRF-Synthetic dataset contains eight synthetic objects, which captured with 100 images. We evaluate our method on the two real scenes, PolyTech and ArtSei, from the UrbanScene3D [22] dataset, on which measure the quality of the reconstructions with the high-resolution LiDAR scans as the ground truths.
Tables 7 and 8 report the reconstruction quality, compared with NeRF [26], BARF [21], SPARF [32], and L2G-NeRF [5] on the two datasets. A threshold of 0.01 used to extract the inliers and outliers for the calculation of the precision, recall, and Fl score. The precision measures the reconstruction accuracy by calculating the distances from the reconstructed models to the ground truths. The recall measures the reconstruction completeness by determining the extent of the ground-truth points covered by the reconstructed models. high F-score means both high accuracy and high completeness of the reconstructed models. As to produce high-quality reconstructions with both high accuracy and high completeness. It is worth note that compared with other methods, method achieves the top performance all metrics consistently for both datasets.
from Tables 7 and 8, our method is able
A.4 Results on Real Scenes
Further, we evaluate our method on the real scene Bank from TwinTex [40], which is a set of high-resolution drone-captured images. We preprocess the images with COLMAP [29] to obtain the
calibrated poses and perturb the poses with additive noise &, where se(3) and & E (0 nI), as
the initial poses, following the procedure real scenes PocyTech and ArtSci of UrbanScene3D [22]. For these real scenes, we set n as a small value 0.01.
Since TwinTex does not provide LiDAR scan for the scene Bank, we only report the evaluation results of novel view synthesis Table Figure 10 shows the comparison on the real scene Bank visually with NeuS [34] and PET-NeuS [37]. While the other methods tend to generate renderings with blurriness, our method able to produce sharper results with more fine details. Figure further shows results of our method on another real scenes CSSE.
17

---

Table 11: Results of sparse input (3 views) on the LEGO scene from the NeRF-Synthetic dataset [26].
Method
PSNR
SSIM
LPIPS
CD
SPARF [32]
CVPR'23
15.91
0.69
0.40
1.27
CRAYM (ours)
16.08
0.70
0.41
0.09
A.5 Results on the DTU Dataset
The DTU [15] dataset is aimed at multi-view stereo (MVS) evaluation, containing image sets captured with structured light scanners mounted an industrial robot arm. We evaluate our method on the first five image sets used in the NeuS [34], comparing with NeuS [34], PET--euS [37], and SPARF [32]. Each image set contains 48 images. We 90% of them training set and the remaining 10%
images as testing data. The quantitative results of novel view synthesis on these data are shown in the Table 10. Figure 11 shows some of the novel view synthesis results visually. As see in Figure 11 our method produces renderings with much more details.
A.6 Sparse Views
Since SPARF is originally designed for sparse input with re-projection loss of dense pixel corre- spondences, we further evaluate our method on the LEGO data with sparse input, which contains only three views the training images. The comparison of SPARF [32] and method is shown in Table 11. Though CRAYM aims for bundle-ade anduus-ajjstiig neural implicit field with dense views as input, it still obtains result comparable with SPARF, for sparse input views.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope. All the claims are demonstrated the experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of the proposed method is discussed in the Section 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The details of the CRAYM are given in the Section 3, Appendix A.1.I, and Appendix A.1.2.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer:
[No]
Justification:
All the experiments use open access dataset. The code is not released yet.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nip) cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /Codeeuomiisseeeeeeee
/nips. wwtpccccpppccceecccccccccccccclt Soneeigeiideeeeeeeuuiiiiiiii ssisssolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: The data splits are set the same as previous paper. The training details are given in the Appendix A.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Section and Appendix A.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
21

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix A.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http://rrrrs.ccpppppii////////// csQuidel ines?
Answer: [Yes]
Justification:
The research conforms with the code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
The proposed method is not tied to particular applications.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All the related papers are cited.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
23

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "UoW5siYRPY", "reviewer": "Reviewer_GtQM", "summary": "This paper introduces Camera Ray Matching for optimizing camera poses and neural fields from multi-view images. The optimized feature volume supports novel view synthesis and 3D geometry reconstruction by probing camera rays, which carry both geometric and photometric information. CRAYM claims to improves efficiency and accuracy by focusing on keypoints and integrating multi-view consistencies, enhancing both geometric reconstruction and photorealistic rendering. The method shows result in NVS and geometry reconstruction compared to baseline methods.", "questions": "Figure 5 and Table 3 appear to have a significant overlap in the information they present.", "limitations": "YES", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper is well-structured and easy to follow.", "weaknesses": "- Experiments were only conducted on NeRF-synthetic datasets and not on LLFF datasets.\n- Comparison is made with older baseline methods (e.g., SPARF, BARF, L2G) which are more than 2 years old. It’s recommended to include more recent methods such as NoPe-NeRF and BAA-NGP.\n- It is suggested that the authors perform Neural Image Alignment to enhance the evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the suggestions in the review and hope the rebuttal will help address the concerns raised.\n\nQ: Experiments were only conducted on NeRF-synthetic datasets and not on LLFF datasets. Also add Neural Image Alignment to enhance the evaluation.\n\nA: We have evaluated our method on the LLFF dataset and the results are listed in the four tables below. The first table shows the mean scores for PSNR, SSIM, and LPIPS, comparing our method (CRAYM) to three baselines including BAA-NGP, a recent method suggested by the reviewer. Our method outperforms all the baselines over all metrics. The remaining three tables show performance numbers for each of the metrics on various models.\n\nEvaluations on the Neural Image Alignment task will be added in the revision. \n\nTable 1: Mean scores over three metrics.\n\n|                        | BARF  | L2G-NeRF | BAA-NGP | CRAYM (ours)  |\n| ---------------- | ------- | ------------ | ------------ | ----------------- |\n| PSNR Mean    | 24.23 |         24.54 |         22.52 |           **24.76** |\n| SSIM Mean     |   0.73 |           0.75 |           0.68 |             **0.76** |\n| LPIPS Mean    |   0.23 |           0.20 |           0.22 |             **0.18** |\n\nTable 2: PSNR scores over various models.\n\n| PSNR     | BARF  | L2G-NeRF | BAA-NGP   | CRAYM (ours) |\n| ---------- | ------- | ------------ | ------------- | ----------------- |\n| Fern       |   23.88 |        24.57 |      19.37     |           **24.83** |\n| Flower   |    24.29 |       24.90 |      **25.16**     |           25.04 |\n| Fortress |    29.06 |      29.27 |      29.24      |           **29.39** |\n| Horns    |    23.29 |      23.12 |       19.71     |           **23.30** |\n| Leaves  |    18.91 |     19.02  |       **19.96**     |           19.57 |\n| Orchids |    19.46 |     19.71  |       12.45     |           **19.81** |\n| Room    |    32.05 |     32.25  |       29.72     |           **32.44** |\n| T-rex      |   22.92 |     23.49   |    **24.56**  |             23.68     |\n| Mean     |   24.23 |     24.54   |      22.52     |            **24.76** |\n\nTable 3: SSIM scores over various models.\n\n| SSIM      | BARF  | L2G-NeRF | BAA-NGP   | CRAYM (ours) |\n| ---------- | ------- | ------------ | ------------- | ----------------- |\n| Fern       |    0.71 |       0.75     |        0.50     |             **0.79** |\n| Flower    |   0.71 |        0.74     |       **0.81** |             0.76     |\n| Fortress  |   0.82 |       0.84     |        0.83     |             **0.85** |\n| Horns     |   0.74 |       0.74     |        0.72     |             **0.75** |\n| Leaves   |   0.55 |       0.56     |       **0.68** |              0.60     |\n| Orchids  |   0.57 |      **0.61** |        0.14     |              0.59     |\n| Room     |   0.94 |      **0.95** |        0.90     |              0.91     |\n| T-rex      |   0.78 |        0.80     |       **0.86** |              0.83     |\n| Mean     |   0.73 |        0.75     |        0.68     |             **0.76** |\n\nTable 4: LPIPS scores over various models.\n\n| LPIPS     | BARF  | L2G-NeRF | BAA-NGP   | CRAYM (ours) |\n| ---------- | ------- | ------------ | ------------- | ----------------- |\n| Fern       |     0.31 |      0.26     |        0.38     |             **0.25** |\n| Flower   |     0.20 |      0.17     |       **0.10** |               0.13     |\n| Fortress |     0.13 |     **0.11** |        0.14     |               0.12     |\n| Horns    |     0.29 |      0.26     |       **0.24** |              **0.24** |\n| Leaves  |     0.35 |      0.33     |       **0.23** |               0.29     |\n| Orchids |     0.29 |      0.25     |        0.42     |              **0.23** |\n| Room    |     0.10 |      **0.08**    |        0.12     |              0.09 |\n| T-rex      |     0.20 |      0.16     |        0.11     |              **0.10** |\n| Mean     |     0.23 |      0.20     |        0.22     |              **0.18** |\n\nQ: Include more recent methods such as NoPe-NeRF and BAA-NGP.\n\nA: The comparisons with BAA-NGP were included in the tables above, and as shown, our method outperforms all the baselines including BAA-NGP, over all three metrics PSNR, SSIM, and LPIPS, on the LLFF dataset. \n\nAs for NoPe-NeRF, we had tested the code released by the authors of that work. However, since NoPe-NeRF heavily relies on depth estimation results to adjust the camera poses, it cannot effectively generalize to other datasets, as in our setting.\n\nQ: Figure 5 and Table 3.\n\nA: Thanks! We will reduce the overlap in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j4mwzcBXTx", "reviewer": "Reviewer_LukG", "summary": "This work suggests a novel neural representation and training scheme that jointly solves for the scene representation and the multi-view camera localization. It is done using several new ideas that generalize existing NeRF based methods. \nThe representation itself is a combination of a geometry-network, which predicts a signed-distance-function (SDF) and a feature vector, that are fed into the texture-network that predicts the usual color and density values.\nThe main key novelty, is that the optimization is done over matching rays, obtained from matching keypoints using a pretrained network. The standard photometric loss function is extended to incorporate an epipolar loss (that constrains the camera positions) and a point-alignment loss that ensures the ray intersect at the predicted depth estimates along the rays. Another strong addition, is the use of 'auxiliary' rays around each matched pair of rays, from which features are fused to produce a more robust representation, that can aid the optimization under errors in matching and camera poses.\nExtensive experiments demonstrate the importance of each component and the strong performance of their combination.", "questions": "* Is the training done only on matching key points? If so, what happens if correct matching keypoints do not 'cover' the space adequately?\n* How sensitive is the method to the quality and density of the 2D matches? It reportedly worked well on sparse image sets, which is somewhat surprising.\n* What are the runtimes (training and inference) compared to some baseline methods?\n* How different is the use of auxiliary rays compared to sampling from conical frustums as in Mip-NeRF?", "limitations": "Adequately discussed.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* The paper presents an extension of the NeRF framework, based on several novel and interesting additions that are framed in a single pipeline. The experimental results show that these contributions work well together and yield new state-of-the-art results, across the board.\n* One promising idea, in my view, is the joint optimization of both geometry and texture networks, which clearly complement eachother and are helpful in obtaining stronger and more accurate constraints on the scene understanding (as opposed to most NeRF pipelines that focus on image reconstruction and are less accurate for 3D reconstruction).  \n* The other strong idea, is the joint optimization of matching rays, once again - imposing consistency contraints (on both camera and surface locations) that were not previously exploited to such an extent in prior work.\n* The paper is well written and the contributions are very clearly highlighted, while the understanding of the conventional parts is left for the reader (which is mostly fine).", "weaknesses": "* Reproducibility - I believe that many details are missing (including from the appendix) for one to be able to implement the proposed method. For example:\n   * What are the settings of the preprocessing SuperPoint and SuperGlue matching? What is the typical match density?\n   * How are the auxiliary rays sampled? How many and under which distribution?\n   * What is the function g in Eq 2 that fuses the key and auxiliary features?\n   * What are the balance weights in the final loss (Eq. 7)?\n   * How are poses initialized?\n* Complexity - There is no discussion what so ever about the impact of the suggested changes on memory and runtime complexity, but at traning and in inference.\n* Qualitative results are relatively limited. \n   * Synthesized images are all very small, so it is difficult to appreciate the fidelity.\n   * No depth images are shown\n   * No examples of key and auxiliary point matches are shown (over entire images)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your care and insights in the reviews and the encouraging remarks!\n\nQ: Reproducibility\n\nA: All the implementation details mentioned along with other useful information will be added to the supplemental material in the revision. The source code and any data used will surely be released upon paper publication.\n\nQ: Complexity\n\nA: When training CRAYM, each iteration consumes around twice the amount of resources vs. other low-resource alternatives, since our method optimizes two matched rays together in each iteration.  However, CRAYM converges much faster than these alternatives (10-20k iterations vs. 200k iterations) during training. During inference, the views are synthesized from the optimized field with the same rendering process and each ray is rendered separately, so the inference efficiency of CRAYM is the same as the other methods.\n\nQ: Qualitative results, synthesized images, depth images, example key, and auxiliary point.\n\nA: We will provide these results in the revised paper as space would allow. All other requested additions will be incorporated into the supplementary material.\n\nQ: Is the training done only on matching key points? \n\nA: No. Matched rays are optimized together, whereas rays without matched companions are optimized separately. Without using any matched rays, CRAYM will be downgraded to the baseline method. \n\nQ: How sensitive is the method to the quality and density of the 2D matches?\n\nA: In our Matched Ray Coherency formulation, we explicitly account for potentially erroneous (i.e., low-quality) 2D matches by using the matchability between two rays as a weight to either accentuate or discount the color consistency constraint; see line 66 in the paper. In terms of sensitivity with respect to the density of the 2D matches, in our experiments, we have observed that even with sparse input views and sparsely distributed matched rays, CRAYM can still notably improve the optimization convergence. We would be happy to provide such a sensitivity analysis in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KAfghjs2r1", "reviewer": "Reviewer_67PC", "summary": "This paper presents a new technique called camera ray matching, which is integrated into the joint optimization of camera poses and a neural field. The method utilizes an uncalibrated set of images as input, incorporating photometric and geometric constraints through key points and key rays matching, with the aim of enhancing the quality of novel view rendering and 3D surface reconstruction. The approach comprises two simple modules and is implemented using grid-based representation (iNGP). Photometric experiments were exclusively compared with MLP-based methods, specifically NeRF-like, on the synthetic dataset of the vanilla NeRF, while geometric experiments demonstrate some positive results. The authors provide additional results in the appendix.", "questions": "Based on the mentioned shortcomings, I have several questions:\n\n1. What is the main research problem addressed in this paper? It appears that the paper intends to address three problems simultaneously: camera pose, fine-detailed rendering, and accurate surface reconstruction. However, if the goal is to solve the camera pose problem, it would be beneficial to present more results related to the accurate regression of camera poses before discussing high-fidelity results. Could you provide additional results about the accurate regression of camera poses?\n\n2. If the primary focus is on the issue of high-fidelity rendering under noisy camera poses, the experimental results do not offer strong conviction. It would be beneficial to showcase more experimental results on different types of datasets, similar to how BARF, SPARF, and L2G-NeRF were compared on the LLFF dataset, with an emphasis on trajectory of pose optimization and high-fidelity renderings.\n\n3. Initialization of camera poses is a sensitive issue in joint optimization. Have you attempted to change the initial camera poses or initialize poses using COLMAP to test the robustness of pose regression?\n\n4. This work is implemented using grid-based representation, while the compared methods are implemented using MLP-based representation, leading to an imprecise and unfair comparison. I am curious about the total number of iterations each method was trained for. BARF and L2G-NeRF trained all models for 200K iterations, and SPARF trained models for 100K iterations. It seems likely that this paper trained for significantly fewer iterations (perhaps 10k iterations) due to the inherent faster convergence of the grid-based representation. It would be fairer to re-implement this paper using MLP-based representation and train for a duration comparable to other methods, then make comparisons. Additional experiments are needed to support these claims.\n\n5. The comparison under sparse-view conditions lacks precision and distinction. SPARF evaluated on the DTU dataset with only 3 views, while you used 48 images in your setting, although you have reported results for sparse input (3 views) on the LEGO data, it may not be sufficiently convincing.\n\n6. Since Neus and PET-NeuS have compared the surface reconstructions on the DTU dataset, I would appreciate the inclusion of visual results. Additional ablation studies on 3D surface reconstruction using your modules on the DTU dataset would enhance the paper.\n\n7. Considering the integration of multi-resolution hash encodings into a neural surface representation, I recommend that the authors compare with Neus2. Neus2 has compared their work with Instant-NGP and Neus, showcasing fast training and detailed surface reconstruction. If the goal is to demonstrate the superiority of 3D surface reconstruction, it is most appropriate to consider Neus2 in the comparison.", "limitations": "The authors have identified a limitation where the meshes extracted from the constructed SDFs may still contain messy inner structures over invisible areas. I recommend that the authors explore the possibility of finding the SDFs of surface points instead of the SDFs of each sampled point along rays. This would involve assessing whether the depth accumulated by all points along the rays is more accurate than the output surface generated by all sampled points.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "This work is a positive extension to the field of neural reconstruction (like NeRF and SDF) under the setting of images captured with noisy poses. Authors make efforts to simultaneously solve the problems involving the camera pose, detailed renderings, and accurate surface reconstruction. Experiments show good results.", "weaknesses": "Too many factors are taken into account in the writing simultaneously, which leads to a lack of clear theme or a clear academic or technical problem to be addressed in this paper. The work appears to build incrementally upon previous research and offers limited novelty. The so-called Epipolar loss and Point-alignment loss are actually based on Bundle Adjustment (BA), using key points matching, which has been previously applied in the optimization of neural reconstruction in works such as SCNeRF, BARF, L2G, and Level2sfm. The proposed two modules do not bring significant innovation. It is also confusing that this work is implemented using a grid-based representation (i.e., iNGP), while the compared methods are implemented using MLP-based representation, which does not allow for a precise and fair comparison. I suggest that the authors refer to ZipNeRF for guidance on how to formulate research problems and conduct appropriate comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the careful review and insightful questions. The reviewer's various suggestions are right on and we hope the rebuttal will alleviate the concerns raised.\n\nQ: Additional results on accurate regression of camera poses?\n\nA: Please refer to Table 9 in the supplemental material.  It shows *exactly* the quality of pose regression by our method. The results show that CRAYM yields better pose estimation than other methods. If requested and space permitting, we can move the relevant materials to Section 4 of the main paper.\n\nQ: It would be beneficial to showcase more experimental results on different types of datasets with an emphasis on trajectory of pose optimization and high-fidelity renderings.\n\nA: This request coincides with what Reviewer GtQM asked above. Hence, we copy the results on the LLFF dataset below. The trajectories of LEGO from NeRF-Synthetic (Blender) are shown in the uploaded PDF file. Other trajectories will be provided in the revised paper.\n\nTable 1: Mean scores over three metrics.\n\n|                        | BARF  | L2G-NeRF | BAA-NGP | CRAYM (ours)  |\n| ---------------- | ------- | ------------ | ------------ | ----------------- |\n| PSNR Mean    | 24.23 |         24.54 |         22.52 |           **24.76** |\n| SSIM Mean     |   0.73 |           0.75 |           0.68 |             **0.76** |\n| LPIPS Mean    |   0.23 |           0.20 |           0.22 |             **0.18** |\n\nTable 2: PSNR scores over various models.\n\n| PSNR     | BARF  | L2G-NeRF | BAA-NGP   | CRAYM     |\n| -------- | ----- | -------- | --------- | --------- |\n| Fern     | 23.88 | 24.57    | 19.37     | **24.83** |\n| Flower   | 24.29 | 24.90    | **25.16**     | 25.04 |\n| Fortress | 29.06 | 29.27    | 29.24     | **29.39** |\n| Horns    | 23.29 | 23.12    | 19.71     | **23.30** |\n| Leaves   | 18.91 | 19.02    | **19.96**     | 19.57 |\n| Orchids  | 19.46 | 19.71    | 12.45     | **19.81** |\n| Room     | 32.05 | 32.25    | 29.72     | **32.44** |\n| T-rex    | 22.92 | 23.49    | **24.56** | 23.68     |\n| Mean     | 24.23 | 24.54    | 22.52     | **24.76** |\n\nTable 3: SSIM scores over various models.\n\n| SSIM     | BARF | L2G-NeRF | BAA-NGP  | CRAYM    |\n| -------- | ---- | -------- | -------- | -------- |\n| Fern     | 0.71 | 0.75     | 0.50     | **0.79** |\n| Flower   | 0.71 | 0.74     | **0.81** | 0.76     |\n| Fortress | 0.82 | 0.84     | 0.83     | **0.85** |\n| Horns    | 0.74 | 0.74     | 0.72     | **0.75** |\n| Leaves   | 0.55 | 0.56     | **0.68** | 0.60     |\n| Orchids  | 0.57 | **0.61** | 0.14     | 0.59     |\n| Room     | 0.94 | **0.95** | 0.90     | 0.91     |\n| T-rex    | 0.78 | 0.80     | **0.86** | 0.83     |\n| Mean     | 0.73 | 0.75     | 0.68     | **0.76** |\n\nTable 4: LPIPS scores over various models.\n\n| LPIPS    | BARF | L2G-NeRF | BAA-NGP  | CRAYM    |\n| -------- | ---- | -------- | -------- | -------- |\n| Fern     | 0.31 | 0.26     | 0.38     | **0.25** |\n| Flower   | 0.20 | 0.17     | **0.10** | 0.13     |\n| Fortress | 0.13 | **0.11** | 0.14     | 0.12     |\n| Horns    | 0.29 | 0.26     | **0.24** | **0.24** |\n| Leaves   | 0.35 | 0.33     | **0.23** | 0.29     |\n| Orchids  | 0.29 | 0.25     | 0.42     | **0.23** |\n| Room     | 0.10 | **0.08**     | 0.12     | 0.09 |\n| T-rex    | 0.20 | 0.16     | 0.11     | **0.10** |\n| Mean     | 0.23 | 0.20     | 0.22     | **0.18** |\n\nQ: This work is implemented using grid-based representation, while the compared methods are implemented using MLP-based representation, leading to an imprecise and unfair comparison.\n\nA: This is a fair point when considering the *original* implementations. However, when making comparisons to CRAYM, we did implement BARF with *grid-based* representations as the baseline and the comparison results are shown in paper and also in the four tables above. The gap between the BARF baseline (27.30) and CRAYM (31.60) shows that the improvement comes mainly from exploiting the matched rays in our proposed formulation. The results of another grid-based representation method, namely NeuS2, are also provided in the table at the end of this rebuttal.\n\nQ: The comparison under sparse-view conditions lacks precision and distinction. SPARF was evaluated on the DTU dataset with only 3 views, while you used 48 images in your setting, although you have reported results for sparse input (3 views) on the LEGO data, it may not be sufficiently convincing.\n\nA: The overall synthesis quality with sparse views is demonstrated with the whole evaluation image set.  We randomly select three images from the evaluation set as the final results. The comparison with SPARF is shown below, with our method outperforming:\n\n|       | PSNR      | SSIM     | LPIPS    | CD        |\n| ----- | --------- | -------- | -------- | --------- |\n| SPARF | 15.91     | 0.69     | 0.40     | 1.270     |\n| CRAYM | **16.08** | **0.70** | **0.41** | **0.094** |\n\nQ: I recommend that the authors compare with Neus2.\n\nA: Good idea. Please first note that in Table 4 in the submitted paper, we applied a progressive feature mask on the multi-resolution hash encoding into a neural surface representation as our baseline, which combines BARF and NeuS2. The results of NeuS2, the BARF baseline, l2G-NeRF, and CRAYM are shown in the table below, with our method coming on top. Although NeuS2 produces better reconstructions than L2G-NeRF, L2G-NeRF outperforms NeuS2 in the quality of the rendered views.\n\n|          | PSNR      | SSIM     | LPIPS    | CD        |\n| -------- | --------- | -------- | -------- | --------- |\n| NeuS2    | 26.83     | 0.86     | 0.17     | 0.075     |\n| BARF Baseline | 27.30     | 0.91     | 0.10     | 0.063     |\n| L2G-NeRF | 27.71     | 0.91     | 0.06     | 0.115     |\n| CRAYM    | **31.60** | **0.96** | **0.03** | **0.039** |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qoKh6cLZa6", "reviewer": "Reviewer_Bu9t", "summary": "The manuscript #3263 entitled \"CRAYM: Neural Field Optimization via Camera RAY Matching\" proposes a novel uncalibrated NeRF strategy based on prior keypoints matching across images. Specifically, the authors propose two novelties to improve the quality of the reconstruction and the pose estimation of the cameras: 1) Enriched ray features using surrounding rays sampled around the keypoints and 2) a ray matching index which can be used to re-weight the color regression part, leading to better robustness to occlusions.\nThe proposed technique has been evaluated across various standard datasets and against meaningful NeRF-like algorithms.", "questions": "The questions are integrated in the Weaknesses part.", "limitations": "The limitations of the paper may not have been entirely investigated, specifically in terms of robustness. For instance, the influence of the initial pose and outliers has not been demonstrated.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The idea of separating key rays and auxiliary rays is interesting and meaningful.\n- Numerous and conclusive results.\n- Assessment on a large number of datasets.\n- Good ablation study underlying the benefit of each novelty.", "weaknesses": "- The robustness of the approach against outlier matches is not evaluated. Introducing artificial outliers (wrongly matched keypoints) into the dataset to assess how well the technique can handle mismatches would be of some interest.\n*Question*: Would the matched rays consistency and the epipolar geometry compensate for that? Or would the training diverge?\n\n- As stated in the literature review of this manuscript, other approaches taking advantage of the epipolar geometry and prior matching have already been designed in this context. I have difficulty understanding what is significantly different with this work apart from the sampling of additional surrounding rays and the matching ray index used to weight color prediction using image pairs. These two novelties seem rather incremental, but they nonetheless lead to strongly improved results.\n*Question*: I assume that the other keypoints-based approaches are not \"self-calibrated\". Is the proposed technique the first \"keypoint-based\" calibration-free NeRF? If it is not the case, it would be meaningful to compare against such techniques too.\n\n- Adding surrounding rays around a key ray appears to be quite effective; however, the sampling of auxiliary rays is not well described in the paper.\n*Question*: How are the rays sampled?\n\n- The initialization of the pose lacks details.\n *Question*: What is the effect of the pose initialization on the result?\n\n- The intrinsic parameters of the camera could additionally be optimized.\n*Question*: Just out of curiosity, have you conducted such an experiment?\n\n- In equation (4), it seems that the proposed solution considers only pairs of images. \n*Question*: How are those pairs selected?\n\nThe proposed approach is inspired by existing techniques integrating matched keypoints (like using the epipolar loss) and other techniques, such as NeuS.\n\n- The loss function contains many regularization factors.\n*Question*: Is the final loss hard to balance?\n\n\nOverall, the paper is interesting and proposes a few contributions that seem to lead to strongly improved results. Moreover, the approach has been evaluated on various standard datasets and against representative methods. However, this novel approach remains relatively incremental, and many points remain to be clarified regarding the robustness of the technique. For all the above-mentioned reasons, I would like to issue a rather mixed opinion regarding the acceptance of this work for this conference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful comments and suggestions, especially the series of detailed questions, which we will all answer below. We hope the answers will address your concerns.\n\nQ: The robustness of the approach against outlier matches is not evaluated. Introducing artificial outliers (wrongly matched key points) into the dataset to assess how well the technique can handle mismatches would be of some interest. Would the matched rays' consistency and the epipolar geometry compensate for that? Or would the training diverge?\n\nA: Yes, our Matched Ray Coherency formulation is precisely designed to compensate for wrongly-matched rays. The use of auxiliary rays serves to improve the robustness of ray matching. In practice, erroneous keypoint/ray matches do happen and they probably do not need to be \"artificially introduced.\" For example, when the noise level increases, wrongly-matched rays will tend to happen more frequently. To this end, our robustness experiments against different noise levels (see Table 2 in the paper) can be regarded also as a robustness test on our method's ability to handle wrongly-matched rays.\n\nQ: As stated in the literature review of this manuscript, other approaches taking advantage of the epipolar geometry and prior matching have already been designed in this context. I have difficulty understanding what is significantly different with this work apart from the sampling of additional surrounding rays and the matching ray index used to weight color prediction using image pairs. These two novelties seem rather incremental, but they nonetheless lead to strongly improved results.  _Question_: I assume that the other keypoints-based approaches are not \"self-calibrated\". Is the proposed technique the first \"keypoint-based\" calibration-free NeRF? If it is not the case, it would be meaningful to compare against such techniques too.\n\nA: Yes, CRAYM is the first \"keypoint-based\" calibration-free method, as far as we are aware of.  Other approaches usually compute the reprojection distance of the matched pixels, while we compute the distance between the reprojected pixel and the epipolar line as the epipolar loss, thereby boosting the convergence of the camera poses.\n\nQ: Adding surrounding rays around a key ray appears to be quite effective; however, the sampling of auxiliary rays is not well described in the paper.  _Question_: How are the rays sampled?\n\nA: Currently, we randomly sample n auxiliary rays from the k nearest rays of key rays. More details of auxiliary rays will be given in the revision.\n\nQ: The initialization of the pose lacks details.  _Question_: What is the effect of the pose initialization on the result?\n\nA: As stated in the paper, we follow L2G-NeRF to perturb the ground-truth camera poses with additive noise as the initial poses.  Table 2 shows the results of using different noise levels on the initial poses. In general, well-initialized poses produce robust and fast convergence.\n\nQ: The intrinsic parameters of the camera could additionally be optimized.  _Question_: Just out of curiosity, have you conducted such an experiment?\n\nA: All the evaluations assume known camera intrinsics, as most of the other methods do. We have performed experiments to additionally optimize the intrinsics and they succeed in learning the intrinsics with some more interactions.\n\nQ: In equation (4), it seems that the proposed solution considers only pairs of images.  _Question_: How are those pairs selected?\n\nA: We do not select image pairs for training. Instead, we randomly select a ray from the ray set generated with all the training images and their poses. If a matched ray is found for it, the matched ray pair is optimized. Otherwise, the selected ray is optimized on its own.\n\nQ: The loss function contains many regularization factors.  _Question_: Is the final loss hard to balance?\n\nA: Empirically, we found that these items in the loss function promote one another rather than weakening/antagonizing one another and we do not need to tune them for an optimal balance in producing our results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We want to thank all the reviewers for their comprehensive reviews of our paper. The insightful questions and various suggestions for additional experiments and clarifications will surely strengthen this work.

Here, let us first start with some quick remarks on common reviewer comments. The individual rebuttals will fully address all the reviewer questions and concerns, with additional experiment results provided as requested.

Two reviewers raised questions about the robustness/sensitivity of our method against wrongly-matched keypoints/rays. To this, let us reiterate that our Matched Ray Coherence formulation is designed to tackle this exactly; see lines 62-70 in the main paper. The robustness tests against different noise levels, shown in Table 2 of the main paper, can be regarded also as a test on how well our method handles potentially-mismatched keypoints/rays, since higher-level noise tends to introduce more mismatches.

Two reviewers requested experiments and comparisons on the LLFF dataset, as well as comparisons to more recent methods such as NoPe-NeRF, BAA-NGP, and NeuS2. To this end, we conducted these experiments and reported results in several new tables. The key conclusion is that our method, CRAYM, outperforms all the baselines and over all three metrics (PSNR, SSIM, and LPIPS). Please refer to the details in the individual rebuttals.

Other requested experiments can either be found in the supplementary material (e.g., on camera pose regression) or reported in the individual rebuttal (e.g., on sparse views). Again, our method comes on top.

All in all, we hope that the rebuttal will alleviate the concerns that the reviewers raised. We are happy to include as many new experimental results as space would allow in the revision. Code and any data used will be released upon paper publication.

-- The authors.

Meta-review: This paper proposes a new method to jointly optimize camera poses and neural fields from multi-view images. The method incorporates contextual information and enforces geometric and photometric constraints through camera ray matching. The paper received one strong accept and three borderline accepts after the rebuttal, with most of the concerns raised by reviewers have been addressed.

Most reviewers acknowledged that the joint optimization along matching rays within the feature volume, as well as the design of key rays and auxiliary rays, effectively enhances volume optimization even in the presence of errors in matching and camera poses. Therefore, the paper is recommended for acceptance.
Decision: Accept (poster)