GeoNLF: Geometry guided Pose-Free Neural LiDAR Fields

Authors: weiyi  xue
zehan  zheng
fan  lu
haiyun  wei
guang  chen
changjun  jiang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 13:16:19

Abstract: Although recent efforts have extended Neural Radiance Field (NeRF) into LiDAR point cloud synthesis, the majority of existing works exhibit a strong dependence on precomputed poses. However, point cloud registration methods struggle to achieve precise global pose estimation, whereas previous pose-free NeRFs overlook geometric consistency in global reconstruction. In light of this, we explore the geometric insights of point clouds, which provide explicit registration priors for reconstruction. Based on this, we propose Geometry guided Neural LiDAR Fields (GeoNLF), a hybrid framework performing alternately global neural reconstruction and pure geometric pose optimization. Furthermore, NeRFs tend to overfit individual frames and easily get stuck in local minima under sparse-view inputs. To tackle this issue, we develop a selective-reweighting strategy and introduce geometric constraints for robust optimization. Extensive experiments on NuScenes and KITTI-360 datasets demonstrate the superiority of GeoNLF in both novel view synthesis and multi-view registration of low-frequency large-scale point clouds.

Main paper: 
 introduction: 
 Neural Radiance Fields (NeRFs) [37] have achieved tremendous achievements in image novel view synthesis (NVS). Recent studies have extended it to LiDAR point cloud synthesis [23, 51, 67, 70] , mitigating the domain gap to real data and far surpassing traditional methods. Nevertheless, the majority of existing works exhibit a strong dependence on known precise poses. In the domain of images, conventional approaches rely on Structure-from-Motion algorithms like COLMAP [48] to estimate poses, which are prone to failure with sparse or textureless views. As an alternative, recent works [6, 21, 31, 41] such as BARF [31] employ bundle-adjusting techniques to achieve high-quality NVS while simultaneously enhancing the precision of pose estimation.However, the sparse nature of LiDAR point clouds and their inherent absence of texture information distinguish them significantly from images. Trivial bundle-adjusting techniques from the image domain become less applicable in this context, encountering the following challenges: (1) Outdoor LiDAR point clouds (e.g., 2Hz, 32-beam LiDAR keyframes in Nuscenes [9] ) exhibit temporal and spatial sparsity. NeRF easily overfits the input views without addressing the geometric inconsistencies caused by inaccurate poses. Consequently, it fails to propagate sufficient gradients for effective pose optimization. (2) Point clouds lack texture and color information but contain explicit geometric features. However, the photometric-based optimization scheme of NeRFs overlooks these abundant geometric cues within the point cloud, which hinders geometric-based registration.Figure 1 : Registration results. Pairwise algorithms such as GeoTrans [44] and ICP [5] suffer from error accumulation and local mismatches. Multi-view methods like SGHR [54] and MICP [13] still manifest outlier poses. Previous gradient-based approaches LiDARNeRF-HASH [21] lack geometric consistency. Our method effectively avoids outlier frames and achieves superior registration accuracy.An alternative to achieving pose-free LiDAR-NeRF is to employ point cloud registration (PCR) methods. Nonetheless, as the frequency of point cloud sequences decreases, the inter-frame motion escalates with a reduction in overlap. As presented in Fig. 1 , pairwise and multi-view registration approaches may all trap in local optima and suffer from error accumulation, making it challenging to attain globally accurate poses. Hence, integrating local point cloud geometric features for registration with the global optimization of NeRF would be a better synergistic approach.Furthermore, as demonstrated in [6, 53] , the incorporation of geometric constraints significantly enhances the optimization of both pose and radiance fields. In the image domain, this process involves introducing additional correspondences or depth priors. However, most methods treat them solely as loss terms without fully exploiting them. In contrast, point clouds provide interframe correlations (e.g., the closest point) for registration and explicit geometric information for reconstruction, presenting substantial advantages over images.To this end, we propose GeoNLF, integrating LiDAR NVS with multi-view PCR for large-scale and low-frequency point clouds. Specifically, to address the suboptimality of global optimization and guide NeRF in the early pose optimization stage to avoid local minima, we regulate NeRF with a pure geometric optimizer. This module constructs a graph for multi-view point clouds and optimizes poses through graph-based loss. To reduce overfitting, we devised a selective-reweighting technique involving filtering out frames with outlier poses, thereby lessening their deleterious impacts throughout the optimization process. Additionally, to fully leverage the geometric attributes of point clouds, we introduced geometric constraints for point cloud modality rather than relying solely on the range map for supervision. Furthermore, our approach has demonstrated excellent performance in large-scale scenarios with sparse point cloud sequences at 2Hz, spanning hundreds of meters. To summarize, our main contributions are as follows:(1) We propose GeoNLF, a novel framework for simultaneous large-scale multi-view PCR and LiDAR NVS. By exploiting geometric clues inside point clouds, GeoNLF couples geometric optimizer with neural reconstruction in the pose-free paradigm. (2) We introduce a selective-reweighting method to effectively alleviate overfitting, which presents excellent robustness across various scenarios. (3) Comprehensive experiments demonstrate GeoNLF outperforms state-of-the-art methods by a large margin on challenging large-scale and low-frequency point cloud sequences.Novel View Synthesis for LiDAR. Traditional simulators [17, 27, 49] and explicit reconstruct-thensimulate [20, 28, 35] method exhibit large domain gap compared to real-world data. Very recently, a few studies have pioneered in NVS of LiDAR point clouds based on NeRF, surpassing traditional simulation methods. Among them, NeRF-LiDAR [68] and UniSim [62] require both RGB images as inputs. LiDAR-NeRF [51] and NFL [23] firstly proposed the differentiable LiDAR NVS framework, and LiDAR4D [70] further extended to dynamic scenes. However, most of these approaches still require a pre-computed pose of each point cloud frame and lack attention to geometric properties.Point Cloud Registration. ICP [5] and its variants [45, 47, 43] are the most classic methods for registration, which rely on good initial conditions but are prone to falling into local optima. Learningbased method can be categorized into two schemes, i.e., end-to-end registration [65, 29, 24, 56, 1] and feature matching-based registration such as FCGF [14] . Recently, the specialized outdoor point cloud registration methods HRegNet [34] and HDMNet [61] have achieved excellent results. GeoTransformer [44] has achieved state-of-the-art in both indoor and outdoor point cloud registration. However, learning-based methods are data-driven and limited to specific datasets with ground truth poses, which requires costly pretraining and suffers from poor generalization.Multiview methods are mostly designed for indoor scenes. Apart from Multiview-ICP [13, 7, 36] , modern methods [2, 8, 52, 25] take global cycle consistency to optimize poses starting from an initial set of pairwise maps. Recent developments [19, 54, 3] such as SGHR [54] employ an iteratively reweighted least-squares (IRLS) scheme to adaptively downweight noisy pairwise estimates. However, their registration accuracy fundamentally depends on pairwise registration. The issues of pairwise methods for NVS still persist.Bundle-Adjusting NeRF. iNeRF [63] and subsequent works [32, 15] demonstrated the ability of a trained NeRF to estimate novel view image poses through gradient descent. NeRFmm [58] and SCNeRF [50] extend the method to intrinsic parameter estimation. BARF [31] uses a coarse-to-fine reconstruction scheme in gradually learning positional encodings, demonstrating notable efficacy. Subsequent work HASH [21] adapts this approach on iNGP [38] through a weighted schedule of different resolution levels, further boosting performance. Besides, some studies have extended BARF to address more challenging scenarios, such as sparse input [53] , dynamic scenes [33] and generalizable NeRF [12] . And [6, 53] uses monocular depth or correspondences priors for scene constraints, significantly enhancing the optimization of both pose and radiance fields. However, the aforementioned methods cannot be directly applied to point clouds or experience dramatic performance degradation when transferring. In contrast, our work is the first to introduce bundleadjusting NeRF into LiDAR NVS task and achieve excellent results in challenging outdoor scenarios. 
limitations: 
 Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: This paper does discuss the limitations of the work performed by the authors. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
methodology: 
 We firstly introduce the pose-free Neural LiDAR Fields and the problem formulation of pose-free LiDAR-NVS. Following this, a detailed description of our proposed GeoNLF framework is provided.Pose-Free NeRF and Neural LiDAR Fields. NeRF represents a 3D scene implicitly by encoding the density σ and color c of the scene using an implicit neural function F Θ (x, d), where x is the 3D coordinates and d is the view direction. When synthesizing novel views, NeRF employs volume rendering techniques to accumulate densities and colors along sampled rays. While NeRF requires precise camera parameters, pose-free NeRF only uses images I = {I i |i = 0, 1..., N -1} and treats camera parameters E = {E s |s = 0, 1...N -1} as learnable parameters similar to Θ. Hence, the simultaneous update via gradient descent of E and Θ can be achieved by minimizing the errorL = N i=0 ∥ Îi -I i ∥ 22 between the rendered and ground truth image Î, I:EQUATIONFollowing [70, 51] , we project the LiDAR point clouds into range image, then cast a ray with a direction d determined by the azimuth angle θ and elevation angle ϕ under the polar coordinate system: d = (cos θ cos ϕ, sin θ sin ϕ, cos ϕ) T . Like pose-free NeRF, our pose-free Neural LiDAR Fields treats LiDAR poses as learnable parameters and applies neural function F Θ to obtain a radiance depth z and a volume density value σ. Subsequently, volume rendering techniques are employed to derive the pixel depth value D: where δ refers to the distance between samples. We predict the intensity S and ray-drop probability R separately in the same way. Besides, our pose-free Neural LiDAR Fields adopted the Hybrid Planar-Grid representation from [70] for positional encoding γ(x, y, z) = f planar ⊕ f hash .EQUATIONEQUATIONwhere x is the 3D point, V, H store the grid features with M spatial resolution and C feature channels. This encoding method benefits the representation of large-scale scenes [70] .Problem Formulation. In the context of large-scale outdoor driving scenarios, the collected LiDAR point cloud sequence P = {P s |s = 0, 1, ..., N -1} serves as inputs with a low sampling frequency.The goal of GeoNLF is to reconstruct this scene as a continuous implicit representation based on neural fields, jointly recovering the LiDAR poses E = {E s |s = 0, 1, ..., N -1} which can align all point clouds P globally.In contrast to prior pose-free NeRF methods, our pipeline employs a hybrid approach to optimize poses. As shown in Fig. 2 , the framework can be divided into two alternately executed parts: global optimization of bundle-adjusting neural LiDAR fields (Sec. 3.2) and graph-based pure geometric optimization (Sec. 3.3) with the proposed Geo-optimizer. In the first part, we adopt a coarse-to-fine training strategy [31] and extend it to the Hybrid Planar-Grid encoding [70] . In the second part, inspired by multi-view point cloud registration, we construct a graph between multiple frame point clouds and propose a graph-based loss. The graph enables us to achieve pure geometric optimization, which encompasses both inter-frame and global optimization. Furthermore, we integrate the selectivereweighting strategy (Sec. 3.4) into the global optimization. This encourages the gradient of outliers to propagate towards pose correction while lowering the magnitude transmitted to the radiance fields, thus mitigating the adverse effects of outliers during reconstruction. To ensure geometry-aware results, we additionally incorporate explicit geometric constraints derived from point clouds in Sec. 3.5.In the stage of global optimization, we optimize Neural LiDAR Fields while simultaneously backpropagating gradients to the pose of each frame. By optimizing our geometry-constrained loss, which will be detailed in Sec. ) = {ξ = ρ ϕ , ρ ∈ R 3 , ϕ ∈ so(3)} byξ ′ = ξ + ∆ξ, followed by the exponential map to obtain the transformation matrix T :EQUATIONwhere ξ ∧ = ϕ ∧ ρ 0 T 0 and ϕ ∧ is the antisymmetric matrix of ϕ. Given a rotation vector ϕ ∈ so(3),rotation matrix R can be obtained through the exponential map R = exp(ϕ ∧ ) = ∞ n=0 1 n! (ϕ ∧ ) n .Simultaneously, we denote∞ n=0 1 (n+1)! (ϕ ∧) n as J . Then Eq. ( 4) can be rewritten as:EQUATIONConsequently, due to the coupling between R = ∞ n=01 n! (ϕ ∧ ) n and J = ∞ n=0 1 (n+1)! (ϕ ∧ )n , the translation updates are influenced by rotation. Incorporating momentum may lead to non-intuitive optimization trajectories [32] . Therefore, we omit the coefficient J from the translation term. This approach enables updating the translation of the the center of mass and the rotation around the center of mass independently.Coarse-to-Fine Positional Encoding. BARF [31] /HASH [21] propose to gradually activate highfrequency/high-resolution components within positional encoding. We further apply this approach to multi-scale planar and hash encoding [70] and found it also yields benefits in our large-scale scenarios. For the detailed formulation, we direct readers to reference [21] .ICP [5] is a classic method for registration based on inter-frame geometric correlations. The essence of ICP lies in searching for the closest point as correspondence in another frame's point at each iteration, followed by using Singular Value Decomposition (SVD) to solve Eq. ( 6), then iteratively refining the solution. Nonetheless, ICP frequently converges to local optima (Fig. 1 ). In contrast, NeRF optimizes pose globally through the implicit radiance fields. However, it lacks geometric constraints and overlooks the strong geometric information inherent in the point cloud, leading to poor geometric consistency. As a consequence, both ICP and NeRF acting individually tend to converge to local optima. Our goal is to employ a hybrid method, utilizing NeRF for global pose optimization and integrating geometric information as an auxiliary support.Drawing inspiration from ICP [5] , we recognize that minimizing the Chamfer Distance (CD) is in line with the optimization objective of each step in ICP algorithm, as demonstrated in Eq. ( 7):EQUATIONEQUATION)where q, p in point cloud Q, P are homogeneous coordinates. T P , T Q represent the transformation matrix to the world coordinate system. However, minimizing the original CD does not necessarily indicate improved accuracy due to the non-overlapping regions between point clouds. To alleviate this negative impact, we weight each correspondence based on Eq. ( 8), whereas w i in the original CD is normalized by the 1 N , N is the number of points.EQUATIONwhere d i denotes the distance between a pair of matching nearest neighbor points, t is the temperature to sharpen the distribution of the d i clipped . The distance d i is clipped to the size of the downsampled voxel grid. This soft assignment can be considered as an approximately derivable version of weighted averaging. Eq. ( 7) will degenerate to the original CD when t → 0, degrade to considering only correspondences with the minimum distance when t → ∞. Considering the distance lacks practical significance in initial optimization, the scheduler is set as a linear or exponential function to vary t from 0 to 0.5 as the optimization progresses. Building upon the above, as shown in Fig. 3 , we approximate the registration objective by optimizing the Graph-based Robust Chamfer distance (G-RCD). Specifically, we construct a graph (W, Y), where each vertex W represents a set of points and each edge Y corresponds to proposed RCD via Eq. ( 7). We connect each frame with its temporally preceding n frames to mitigate error accumulation in ICP [5] . Then RCD is calculated for all edges as Eq. ( 9), and M denotes the number of frames in the sequence. Notably, in Eq. ( 7), G-RCD is computed using the global transform matrix, enabling direct gradient propagation of the Graph-based loss to the global transformation matrix of each frame.L graph = 1 (nM -n(n+1)2EQUATIONDiscussion. As illustrated in Fig. 3 (b), insufficient geometric guidance leads to certain frame poses being optimized in the wrong direction. Geometric optimizer can address this issue by preventing pose updates strictly following NeRF and correcting wrong optimization directions that do not conform to global geometric consistency. This method involves externally modifying pose parameters and providing effective geometric guidance early in the ill-conditioned optimization process. Consequently, few iterations of graph-based RCD computation suffice to offer ample guidance for NeRF.In bundle-adjusting optimization, as shown in Fig. 4 (a), we observed that frames with outlier poses present significantly higher rendering losses during the early stages of training. However, low frequency and sparsity of point clouds result in quick overfitting of individual frames including outliers (cf. Fig. 4 (a)(b)). This leads to minimal pose updates when the overall loss decreases, resulting in incorrect poses and inferior reconstruction. Inspired by the capabilities of NeRF in pose inference [63] , we decrease the learning rate (lr) of neural fields for the top k frames with the highest rendering losses as Eq. ( 10), while keeping lr of poses unchanged. The strategy facilitates gradient propagation towards outlier poses, while the gradient flow to the radiance fields is concurrently diminished. Consequently, it's analogous to leveraging a pre-trained NeRF for outlier pose correction and lessens the adverse effects caused by outliers during the optimization process.EQUATIONWhere l ∈ [0, 1] denotes training progress. Akin to leaky ReLU [60] , we set the reweighting factor w 0 to a relatively small value. w 0 increases as the process progresses, which ensures the network's ongoing learning from these frames and avoids stagnation.Point clouds encapsulate rich geometric features. However, solely supervising NeRF training via range images pixel-wise fails to fully exploit their potential, e.g., normal information. Furthermore, the Chamfer distance can directly supervise the synthesized point clouds from a 3D perspective. Therefore, in addition to supervising via 2D range map, we propose directly constructing a threedimensional geometric loss function between the generated point cloud and the ground truth point cloud. Unlike our Geo-optimizer, Eq. ( 11) imposes constraints between synthetic point clouds P and ground truth point clouds P :EQUATIONBased on the point correspondences established between P and P as derived in Eq. ( 11), the constraint of normal can be formulated as minimizing:EQUATIONThus, the normal loss is calculated between the synthetic point cloud and the ground truth point cloud to ensure more accurate normal vectors of the point cloud synthesized from NeRF. Moreover, we also employ 2D loss function to supervise NeRF as Eq. (13) .EQUATIONwhere D represents depth and S, R represents intensity and ray-drop probabilities. Consequently, the loss for Neural LiDAR fields is weighted combination of the depth, intensity, ray-drop loss and 3D geometry constraints, which can be formalized asL = L r + λ n L normal + λ c L CD .4 ExperimentQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA]Answer: [Yes] Justification: This paper provide open access to the data and code.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setup: 
 Datasets and Experimental Settings. We conducted experiments on two public autonomous driving datasets: NuScenes [9] and KITTI-360 [30] dataset, each with five representative LiDAR point cloud sequences. We selected 36 consecutive frames at 2Hz from keyframes as a single scene for NuScenes, holding out 4 samples at 9-frame intervals for NVS evaluation. KITTI-360 has an acquisition frequency of 10Hz. We used 24 consecutive frames sampled every 5th frame to match scene sizes of Nuscenes, holding out 3 samples at 8-frame intervals for evaluation. We perturbed LiDAR poses with additive noise corresponding to a standard deviation of 20 deg in rotation and 3m in translation.Metrics. We evaluate our method from two perspectives: pose estimation and novel view synthesis.For pose evaluation, we use standard odometry metrics, including Absolute Trajectory Error (ATE) and Relative Pose Error (RPE r in rotation and RPE t in translation). Following LiDAR4D [70] for NVS evaluation, we employ CD to assess the 3D geometric error and the F-score with 5cm error threshold. Additionally, we use RMSE and MedAE to compute depth and intensity errors in projected range images, along with LPIPS [69] , SSIM [57] , and PSNR to measure overall variance.Implementation Details. The entire point cloud scene is scaled within the unit cube space. The optimization of GeoNLF is implemented on Pytorch [42] with Adam [26] optimizer. All the sequences are trained for 60K iterations. Our Geometry optimizer's lr for translation and rotation is the same as the lr for pose in NeRF with synchronized decay. We use the coarse-to-fine strategy [31, 21] , which starts from training progress 0.1 to 0. We compare our model with BARF [31] and HASH [21] , both of which use LiDAR-NeRF [51] as backbone. For PCR-assisted NeRF, we opt to initially estimate pose utilizing pose derived from GeoTrans [44] , which is the most robust and accurate algorithm among other PCR methods in our experiments. And subsequently we leverage LiDAR-NeRF [51] for reconstruction. For all Pose-free methods, we follow NeRFmm [58] to obtain the pose of test views for rendering. The quantitative and qualitative results are in Tab. 1 and Fig. 5 . Our method achieves high-precision registration and high-quality reconstruction across all sequences. However, baseline methods fail completely on certain sequences due to their lack of robustness. Please refer to Fig. 7 for details. Ultimately, our method excels in the reconstruction of depth and intensity, as evidenced by 7.9% increase in F-score on Nuscenes and 13.1% on KITTI-360 compared to the second best result. We conduct comprehensive comparisons of GeoNLF with pairwise baselines, including traditional method ICP [5] , learning-based GeoTrans [44] and outdoor-specific HRegNet [34] , as well as multiview baselines MICP [13] and learning-based SGHR [54] . For pairwise methods, we perform registration between adjacent frames in an Odometry-like way. For SGHR, we utilize FCGF [14] descriptors followed by RANSAC [18] for pairwise registration. The estimated trajectory is aligned with the ground truth using Sim(3) with known scale.GeoNLF outperforms both the registration and pose-free NeRF baselines. Quantitative and Qualitative results are illustrated in Tab. 2 and Fig. 1 . As depicted in Fig. 1 , most registration methods fail to achieve globally accurate poses and completely fail in some scenarios, leading to massive errors in average results. Significant generalization issues arise for learning-based registration methods due to potential disparities between testing scenarios and training data, including differences in initial pose distributions. This challenge is particularly pronounced in HRegNet [34] . While the transformer model GeoTrans [44] with its higher capacity offers some alleviation to the issue, it remains not fully resolved.In Despite the fact that GeoNLF has exhibited exceptional performance in PCR and LiDAR-NVS on challenging scenes, it is not designed for dynamic scenes, which is non-negligible in autonomous  
conclusion: 
 We introduce GeoNLF for multi-view registration and novel view synthesis from a sequence of sparsely sampled point clouds. We demonstrate the challenges encountered by previous pairwise and multi-view registration methods, as well as the difficulties faced by previous pose-free methods. Through the utilization of our Geo-Optimizer, Graph-based Robust CD, selective-reweighting strategy and geometric constraints from 3D perspective, our outlier-aware and geometry-aware GeoNLF demonstrate the promising performance in both multi-view registration and NVS tasks. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: This paper specify all the training and test details.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: This paper report appropriate information about the statistical significance of the experiments.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: This paper provide sufficient information on the computer resources.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We mentioned creators or original owners of assets and properly respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: 1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: This research conducted in the paper conform, in evrery respect, with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Justification: There is no societal impact of the work performed. No harm technical paper.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: The paper proposes a method for novel view synthesis from Lidar point clouds. It initially received highly discordant ratings (R, WA, A). The rebuttal and subsequent discussion appeared to have addressed the main technical concerns from the more negative reviewer, who decided to lean on a BA. The other two reviewers confirmed their initial ratings. 
Based on this, the AC has deemed that a consensus has been found towards acceptance. The AC warmly recommends the authors to incorporate the comments from the reviewers in the camera ready, including those around the clarity/presentation of the method.

Reviews: review: paper_summary: paper_summary: The paper proposes GeoNLF, a method for pose-free Lidar point cloud novel view synthesis. They propose optimizing point cloud poses simultaneously through a bundle-adjusting neural lidar field as well as through ICP-inspired geometric optimization. The neural lidar field is supervised with ground truth range images, intensity images, and ray-drop probability images as well as ground truth point clouds. The geometric optimization optimizes poses to minimize a weighted average of the chamfer distances between neighboring lidar frames. The proposed method is validated for lidar novel view synthesis and pose estimation using the Nuscenes and KITTI-360 datasets.
strength_weakness: strength_weakness: The methodology section reads like a work in progress, making it hard to understand the details of the proposed method.
1) There is no intuitive explanation as to why the "J" term in Eq. 5 can simply be dropped. While the main text states that dropping this term allows independent updates of rotation and translation, the supplementary material then states that simultaneous optimization yielded satisfactory results in L527.
2) P is used to denote poses in Eq. 1 as well as point clouds in Eq. 11 and ray-drop loss probabilities in Eq. 13. L173. Similarly, I is used to refer to NeRF images in L112 and then intensity images in Eq. 13. Intensities and ray-drop probabilities are referred to as S and R in L119.
3) L173 states that Eq. 7 shows how CD is in line with each step of ICP, but the optimization steps of ICP are never introduced.
4) Eq. 8 introduces d_clipped which is never used. It takes as parameter a distance "d" that is never introduced.
5) There is no explanation as to how the normals used in Eq. 12 are obtained. Are they obtained from the implicit density field for both predicted and pseudo-groundtruth pointclouds?
6) There is a lot of information in Fig. 3, but the caption does not explain what the figure is trying to show.
7) Sections 3.2 and 3.5 both discuss the nerf optimization, but are separated by sections 3.3 and 3.4 which discuss geometric optimization.
questions: questions: Please refer to the weakness section.
limitations: limitations: There is no discussion about the optimization time, which might be significantly increased due to the geometric optimization requiring computing the chamfer distance between all combinations of the 4 adjacent frames.
paper_summary: paper_summary: In this submission the authors propose a novel strategy to jointly optimize a Neural LIDAR Field and the corresponding LIDAR poses. They achieve this by utilizing a hybrid strategy of alternating pure graph-based geometric optimization and global optimization of the Neural Field. Additionally they introduce a new selective reweighting strategy that steers gradients in the Neural Field towards outlier poses and new geometric losses that improve geometric reconstruction. They perform experiments on two datasets and achieve better results than relevant baselines in both reconstruction and registration.
strength_weakness: strength_weakness: In order of severity: 

1. Is it truly Pose-Free? The starting point for the LIDAR pose optimization is not completely random but they are perturbed from Ground truth by noise with a standard deviation of 20deg in rotation and 3m in translation (line 235). This means that some form of reasonable initialization is necessary for the method, which would have to be obtained from somewhere. How to obtain this initialization is not discussed in the paper. 
2. The motivation for why this approach is necessary is unclear. The setting of obtaining sparse LIDAR scans without at least poses derived from GPS or vehicle odometry seems unrealistic, as well as the assumptions that cameras would not be available given that they are significantly cheaper than LIDAR Scanners. This subtracts from the otherwise strong paper because the setting and significance of the results cannot be properly contextualized. It is also not clear from the Introduction what motivation the simulation of LIDAR data in a Neural Field has and what potential applications are enabled by the proposed method. 
3. While the authors report standard pose registration metrics (RPE & ATE) they do not consider Inlier Ratio, Feature Matching Recall or Registration Recall as metrics. This makes a comparison to prior work such as GeoTransformer [45] more complicated since these are the main metrics which they evaluate on in their paper. ([45] Qin, Z., Yu, H., Wang, C., Guo, Y., Peng, Y., & Xu, K. (2022). Geometric transformer for fast and robust point cloud registration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 11143-11152)) 
4. Outlier/Inlier poses and frames are discussed in Figure 3 and for example line 208 but it is never clarified how their inlier/outlier status is determined. This makes reproducibility harder and should be clarified either in the paper or supplementary. 
5. In section 3.5 the authors introduce a new loss between generated and ground truth point clouds. It is not mentioned how the generated point cloud is obtained from the Neural Field, which is important to know since this determines how the gradient from the loss is backpropagated through this mechanism. This should also be clarified in the final version or supplementary. 
6. In Equation 13 the variable P is very confusing since in the equations above it represents a point cloud and then here probably refers to the ray-drop loss. This is in contrast to line 119 & 120 where intensity and ray-drop probability are introduced as $S$ and $R$ respectively. This should be made coherent with each other or at least clarified. 
7. In line 113-115 the authors explain how the direction of rays is determined but do not explain how they represent translation in their polar coordinate system. This could also be clarified in the supplementary. 


Minor comments: 
 - Line 16 NeRFs has achieved → NeRFs have achieved 
 - Line 49 In furtherance of mitigating overfitting → To reduce overfitting. Clarity > Complicated Words. 
 - Line 301 demonstrate the promising performance → demonstrate promising performance.
questions: questions: To focus discussion about the weaknesses raised above here are some detailed questions: 
1. How can the initial (bad) registration of the point clouds be obtained or is there a way this step can be removed ? Can the point cloud poses be initialized completely randomly or all as identity? 
2. Is there some reason (e.g. Privacy concerns) that prohibit the joint capture of Image data via cameras and LIDAR scans? This would enable standard Structure from Motion approaches for pose estimation. 
3. What is the overall motivation of representing LIDAR in a Neural Field and what benefits are gained by this? Are there downstream applications enabled by the proposed method? 
4. Can you report the Inlier Ratio, Feature Matching Recall or Registration Recall ? If not, why can this not be reported on this data?
limitations: limitations: The authors have addressed the limitations of their work with regards to dynamic scenes, since they only consider static scenes. 
Societal impact is not discussed.
paper_summary: paper_summary: This paper presents GeoNLF, a new approach to pose-free large-scale point-cloud registration (PCR) and LiDAR novel view synthesis (NVS). To address the limitation of both ICP-based and NeRF-based methods, the authors proposed a hybrid approach involving both robust ICP and global optimisation of NeRF. Experiments show the proposed method achieved better pose estimation and NVS results compared to previous SOTA methods.
strength_weakness: strength_weakness: 1. The proposed approach consists of so many building blocks which makes it a complicated system to work. Besides, from Tab. 3, it seems that except the geometric optimizer which contributes a lot to the final result, all other components seem to only produce very small improvement. 

2. How long does the optimisation take? It would be good to show some run-time comparison and analysis.
questions: questions: Please see the Weakness.
limitations: limitations: Yes.

Comments: title: Thanks for your comments!
Computation of normals
Comments on Rebuttal
comment: We greatly appreciate your suggestions, which have enriched the completeness of our work and provided valuable insights. In the final paper, we will revise the introduction based on your recommendations, particularly regarding the selection of sensors and the background on LiDARNeRF, and we will include the experiments in the supplementary material. Thank you again for your recognition.
We greatly appreciate your suggestions and the insightful comments you provided on the completeness of our paper. 

Here, we first outline the steps for computing normal vectors: 
1. For a point $A$ in the point cloud, we search for its $N$ neighbors (with $N = 50$ in our implementation) and center these points by subtracting the coordinates of $A$, resulting in a matrix $X\in \mathbb{R}^{N\times 3}$. 
2. We compute the covariance matrix of this neighborhood ($X^T X$) and perform a differentiable Singular Value Decomposition (SVD) to obtain the eigenvectors and eigenvalues. 
3. The normal vector typically corresponds to the eigenvector associated with the smallest eigenvalue of the covariance matrix, representing the local surface orientation. 

We use the differentiable point cloud library provided by PyTorch3D [1] for normal vector computation, ensuring that the entire process is fully differentiable. We will summarize the normal vector computation process in the main text and provide a detailed discussion in the appendix, with the code to be made publicly available thereafter.

[1] Ravi, Nikhila, et al. "Accelerating 3d deep learning with pytorch3d." arXiv preprint arXiv:2007.08501 (2020).
Could you please detail how normals are being computed? I believe this is an important detail to include in the manuscript since the proposed normal loss requires gradients to be backpropagated through the normal estimation function.
Dear Authors, 

thank you for the comprehensive rebuttal. 
I read it and my questions were addressed well. 

I would strongly encourage the authors to include the additional explanations provided here for Q2, Q3 in the introduction section of the final version of the paper, since this motivation was (in my opinion) missing in the initial submission. 

The additional experiments and clarifications provided in the rebuttal would also be nice to see in the final paper.

I will raise my score to Accept.