GeoNLF: Geometry guided Pose-Free Neural LiDAR Fields
Authors: Weiyi Xue
Zehan Zheng
Fan Lu
Haiyun Wei
Guang Chen
changjun jiang
Venue: NeurIPS 2024 poster (2024)
Keywords: Neural LiDAR Field
Pose-free
Geometry
Point Cloud
Registration
Primary area: machine_vision

Abstract: Although recent efforts have extended Neural Radiance Field (NeRF) into LiDAR point cloud synthesis, the majority of existing works exhibit a strong dependence on precomputed poses. However, point cloud registration methods struggle to achieve precise global pose estimation, whereas previous pose-free NeRFs overlook geometric consistency in global reconstruction. In light of this, we explore the geometric insights of point clouds, which provide explicit registration priors for reconstruction. Based on this, we propose Geometry guided Neural LiDAR Fields (GeoNLF), a hybrid framework performing alternately global neural reconstruction and pure geometric pose optimization. Furthermore, NeRFs tend to overfit individual frames and easily get stuck in local minima under sparse-view inputs. To tackle this issue, we develop a selective-reweighting strategy and introduce geometric constraints for robust optimization. Extensive experiments on NuScenes and KITTI-360 datasets demonstrate the superiority of GeoNLF in both novel view synthesis and multi-view registration of low-frequency large-scale point clouds.

GeoNLF: Geometry guided Pose-Free Neural LiDAR Fields
Weiyi Xue Tongji University xwy@tongji. edu. en
Zehan Zheng" Tongji University
Fan Lu Tongji University
zhengzehan@tongji. edu. en
1uf lufanOtongji edu. on
Haiyun Wei Tongji University 231 2311399ttongji.. www.:nnnnnni.uunnnoooooo..ee.eoa
Guang Chen
Changjun Jiang Tongji University cj cjjiang@tongj eergggjjeedduu cn
Tongji University guangchen@tongji. edu. en
Abstract
Although recent efforts have extended Neural Radiance Fields (NeRF) into LiDAR point cloud synthesis, the majority of existing works exhibit a strong dependence precomputed poses. However, point cloud registration methods struggle to achieve precise global pose estimation, whereas previous pose-free NeRFs over- look geometric consistency in global reconstruction. In light of this, we explore the geometric insights point clouds, which provide explicit registration priors for reconstruction. Based on this, propose Geometry guided Neural LiDAR Fields (GeoNLF), a hybrid framework performing alternately global neural reconstruc- tion and pure geometric pose optimization. Furthermore, NeRFs tend to overfit individual frames and easily get stuck in local minima under sparse-view, inputs. To tackle this issue, develop selective-reweightin geometric constraints for robust optimization. Extensive experiments on NuScenes and KITTI-360 datasets demonstrate the superiority of GeoNLF in both novel view synthesis and multi-view registration of lo-frequency large-scale point clouds.
strategy and introduce
Introduction
Neural Radiance Fields (NeRFs) 377 have achieved tremendous achievements in image novel view synthesis (NVS). Recent studies have extended it to LiDAR point cloud synthesis [23 67 70), mitigating the domain gap to real data and far surpassing traditional methods. Nevertheless, the majority of existing works exhibit a strong dependence on known precise poses. In the domain of images, conventional approaches rely Structure-from-MMonn from-Motion algorithms like COLMAP 48 to estimate poses, which are prone to failure with sparse or textureless views. As an alternative, recent works 6 21 31 41 such BARF (31) employ bundle-adjusting techniques to achieve high-quality NVS while simultaneously enhancing the precision of pose estimation.
However, the sparse nature of LiDAR point clouds and their inherent absence of texture information distinguish them significantly from images. Trivial bundle-adjusting techniques from the image domain become less applicable in this context, encountering the following challenges: (1) Outdoor LiDAR point clouds (e.g.. 2Hz, 32-beam LiDAR keyframes Nuscenes [9] exhibit temporal and spatial sparsity. NeRF easily overfits the input views without addressing the geometric inconsistencies caused by inaceurate poses. Consequently, fails to propagate sufficient gradients for effective pose optimization. (2) Point clouds lack texture and color information but contain explicit geometric features. However, the photometric-han based optimization scheme of NeRFs overlooks these abundant geometric cues within the point cloud, which hinders geometric-based registration.
Equal contribution. Corresponding author. Our code is availiable wttp:://hhhabcoomiisllaaaaGiCC/
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Input Point Clouds
Gradient-basse Nethed
Learning based
Traditional
Initial
LEowmenr HHASH
Geo Trans
ICP
rugh
RPE; 1.155m
RPE, 0.075as
RPF, 0.66m
RPS; 1664
RPE,
RPE, 1.127"
(.687m
ATE 1.8122m
reight
Ground Truth
Ours 0.01mm
FCG +SGHR
RPG, 1-737m
Multiviow ICP
0.007m
0.179
6.900
1.950"
ATE 3.440
2.640m
Figure/ I: Registration results. Pairwise algorithms such GeoTrans 44 and ICP (5] suffer from error accumulation and local mismatches. Multi-view methods like SGHR [54] and MICP [3] still manifest outlier poses. Previous gradient-based approaches LiDARNeRF-HASH 21 lack geometric consistency. Our method effectively avoids outlier frames and achieves superior registration accuracy.
An alternative to achieving pose-free LiDAR-NeRF is to employ point cloud registration (PCR) methods. Nonetheless, as the frequency of point cloud sequences decreases, the inter-frame motion escalates with reduction in overlap. As presented in Fig. | pairwise and multi-view registration approaches may all trap in local optima and suffer from error accumulation, making challenging to attain globally accurate poses. Hence, integrating local point cloud geometric features for registration with the global optimization of NeRF would be a better synergistic approach.
Furthermore, as demonstrated in 6 53], the incorporation of geometric constraints significantly enhances the optimization of both pose and radiance fields. In the image domain, this process depth priors. However, most methods treat
involves introducing additional correspondences them solely as loss terms without fully exploiting them. contrast, point clouds provide inter- frame correlations (e.g., the closest point) for registration and explicit geometric information for reconstruction, presenting substantial advantages over images.
To this end, we propose GeoNLF, integrating LiDAR NVS with multi-view PCR for large-scale and low-frequency point clouds. Specifically, address the suboptimality of global optimization and guide NeRF in the early pose optimization stage to avoid local minima, we regulate NeRF with a pure geometric optimizer. This module constructs a graph for multi-view point clouds and optimizes poses through graph-based loss. To reduce overfitting, we devised selective-reweighhnn technique involving filtering out frames with outlier poses, thereby lessening their deleterious impacts throughout the optimization process. Additionally, to fully leverage the geometric attributes of point clouds, we introduced geometric constraints for point cloud modality rather than relying solely on the range map for supervision. Furthermore, our approach has demonstrated excellent performance in scage scenarios with sparse point cloud sequences at 2Hz, spanning hundreds meters. To summarize, our main contributions are follows:
(1) We propose GeoNLF, a novel framework for simultaneous large-scale multi-view PCR and LiDAR NVS. By exploiting geometric clues inside point clouds, GeoNLF couples geometric optimizer with neural reconstruction in the pose-free paradigm. (2) We introduce a selective-rewefe renetihiie method to effectively alleviate overfitting, which presents excellent robustness across various scenarios. (3) Comprehensive experiments demonstrate GeoNLF outperforms state-of-the-art methods by large margin on challenging large-scale and low-frequency point cloud sequences.
2 Background and Related Work
Neural Radiance Fields. NeRF 337 and related works have achieved remarkable achievements in
NVS. Various neural representations
11 22 38), such as hash grids [38], triplanes [10] 22 55 66 have been proposed to enhance NeRF's performance. Due to introduce depth prior or
4
10
and diverse techniques [39
40
the lack of geometric information in images, some methods [16 46 59 64
point clouds as auxiliary data to ensure multi-view geometric consistency. However, the geometric
information and consistency encapsulated in point clouds are still not fully explored and utilized.

---

Novel View Synthesis for LiDAR. Traditional simulators 17 27 49
and explicit reconstruct then
simulate [20 28 351 method exhibit large domain gap compared to real-world data. Very recently, a few studies have pioneered in NVS of LiDAR point clouds based on NeRF, surpassing traditional simulation methods. Among them, NeRF-LiDAR (68 and UniSim [2] require both RGB images as inputs. LiDAR-NeRF [51] and NFL 23 firstly proposed the differentiable LiDAR NVS framework, and LiDAR4D 70 further extended to dynamic scenes. However, most of these approaches still require a pre-computed pose of each point cloud frame and lack attention to geometric properties.
Point Cloud Registration. ICP [5] and its variants 45
are the most classic methods for
registration, which rely on good initial conditions but are prone to falling into local optima. Learning-
based method can be categorized into two schemes. i.c., end-to-end registration [65] [2 [2 [56
and feature matching- based registration such as FCGF [4 Recently, the specialized outdoor
point cloud registration methods HRegNet (34] and HDMNet 661 have achieved excellent results. Geo' ranosfooorrrrr 44] has achieved
in both indoor and outdoor point cloud registration. However, learning-base based methods are data-driven and limited to specific datasets with ground truth poses, which requires costly pretraining and suffers from poor generalization.
Multiview methods are mostly designed for indoor scenes. Apart from Multiview-ICP [
modern methods 2 8 52 25 take global cycle consistency to optimize poses starting from an initial
set of pairwise maps. Recent developments 19
54 3 such as SGHR [44] employ an iteratively
reweighted seustes (IRLS) scheme to adaptively downweight noisy pairwise estimates. However, their registration accuracy fundamentally depends on pairwise registration. The issues of pairwise methods for NVS still persist.
Bundle-Adjustinn NeRF. iNeRF (63 and subsequent works 32 15 demonstrated the ability of a trained NeRF to estimate novel view image poses through gradient descent. NeRFmm [58] and SCNERF 50) extend the method to intrinsic parameter estimation. BARF [31] uses coarse-to-fine reconstruction scheme in gradually learning positional encodings, demonstrating notable efficacy. Subsequent work HASH (1 adapts this approach on iNGP (38) through a weighted schedule of different resolution levels, further boosting performance. Besides, some studies have extended BARF to address more challenging scenarios, such as sparse input (53), dynamic scenes [33] and generalizable NeRF [2). And [6, uses monocular depth or correspondences priors for scene constraints, significantly enhancing the optimization of both pose and radiance fields. However, the aforementioned methods cannot be directly applied to point clouds or experience dramatic performance degradation when transferring. In contrast, our work is the first introduce bundle- adjusting NeRF into LiDAR NVS task and achieve excellent results in challenging outdoor scenarios.
3 Methodology
We firstly introduce the pose-free free Neural LiDAR Fields and the problem formulation of pose-free LiDAR-NVS. Following this, a detailed description of our proposed GeoNLF framework is provided.
Pose-Free NeRF and Neural LiDAR Fields. NeRF represents 3D scene implicitly by encoding the density a and color e of the scene using an implicit neural function Fe (x. d), where æ is the
3D coordinates and d is the view direction. When synthesizing novel views, NeRF employs volume rendering techniques to accumulate densities and colors along sampled rays. While NeRF requires
precise camera parameters, pose-free NeRF only uses images I = { 0, 1..., N - 1} and treats camera parameters & = [E, = 0, 1.... - 1} as learnable parameters similar to O. Hence, the simultaneous update via gradient descent of E and O can be achieved by minimizing the error t | 1.11 between the rendered and ground truth image î, I:
L = E
θ arg min C(i, & II)
E,
we project the LiDAR point clouds into range image, then cast ray with direction d determined by the azimuth angle 0 and elevation angle ó under the polar coordinate system: d = (cos O cos o, sin @ sin o, cos o)" Like prse-free NeRF, our pose-free Neural LiDAR Fields treats LiDAR poses as learnable parameters and applies neural function Fe to obtain radiance depth 2 and a volume density value o. Subsequently, volume rendering techniques are employed to derive the pixel depth value D:
Following 70 51
N
D(r) = Σ T. (1-1-0 -0.h)) 2, T. = ex=exp(- Σa,0,l
(2)
3

---

Reweighting
Depth Supers Iaieo
Pose Gradient Flow
View
BA-Neural
LiDAR Fields
Readicteg Curppn
Sampling
Back Project
World
Rendering
Cavedinate
Sy stemm
Intuasity Ray-drop
P2
Range Imagee
Density
Prediction
Geounctric Supcrvisin
Pose
Project
Alteriatelyy
Robust Chamier
Vertex: Point
Geometry Optimizer
Overlapping Awars
Edge: Robast Chamfer Distancs
ss
Multi-frame Point Clouds
Figure 2: Overview of our proposed GeoNLE. We alternatively execute global optimization of bundle-adjusting neural LiDAR fields and graph-based pure geometric optimization. By integrating selective-reweightie strategy and explicit geometric constraints derived from point clouds, GeoNLF implements outlier-aware and geometry- mechanisms.
where 8 refers to the distance between samples. We predict the intensity S and ray-drop probability R separately in the same way. Besides, our pose-free Neural LiDAR Fields adopted the Hybrid Planar- Grid representation from [70] for positional encoding (zz y, 2) = fplanar @ frash.
3
Filanar II Bilinear(V, x)) R.XMMMMC fnash TriLincar(HH, x), MXMANIC
(3)
where is the 3D point, V. H. store the grid features with M spatial resolution and C' feature channels. This encoding method benefits the representation of large-scale scenes (70].
Problem Formulation. In the context of large scale outdoor driving scenarios, the collected LiDAR point cloud sequence P {Pa|s
..., N 1} serves as inputs with low sampling frequency. The goal of GeoNLF is to reconstruct this scene as continuous implicit representation based on neural fields, jointly recovering the LiDAR poses E = {E 0, 1,..,, N 111 which can align all point clouds P globally.
3.1 Overview of GeoNLF Framework
In contrast prior pose-free free NeRF methods, pipeline employs hybrid approach to optimize poses. As shown in Fig. 2 the framework can divided into two alternately executed parts: global optimization of bundle-adjuse adjustiiin neural LiDAR fields (Sec. 3.2 and graph-based pure geometric optimization (Sec. 3.3 with the proposed Geo-optimizer.
the first part, we adopt a coarse-to-fine training strategy [1 and extend it to the Hybrid Planar Grid encoding [70]. In the second part, inspired by multi view point cloud registration, we construct graph between multiple frame point achieve pure geometric optimization,
clouds and propose graph-based loss. The graph enables which encompasses both inter-frame and global optimization. Furthermore, we integrate the selective- reweighting strategy (Sec. 3.4) into the global optimization. This encourages the gradient of outliers to propagate towards pose correction while lowering the magnitude transmitted to the radiance fields, thus mitigating the adverse effects of outliers during reconstruction. To ensure geometry-aware results, we additionally incorporate explicit geometric constraints derived from point clouds in Sec. 3.5 3.5.
3.2 Bundle-Adjusting Neural LiDAR Fields for Global Optimization
In the stage of global optimization, we optimize Neural LiDAR Fields while simultaneously back- propagating gradients to the pose of each frame. By optimizing our geometry-come constrained loss, which will be detailed in Sec. 3.5 the pose is individually optimized to achieve global alignment.
LiDAR Pose Representation. In previous pose-free NeRF methods, poses are often modeled by T
R 1 E SE(() with a rotation R E 50(3) and a translation t E RR3 Pose updates
p
are computed in the special Euclidean Lie algebra se(3) = {E
E R3 o E so(3)) by

---

E' = -++ + Ag. followed by the exponential map to obtain the transformation matrix T:
8
Σx= ni (Ø^)"
E
(n+1)) (6")"p
= expp(E
Σ
(5
(4)
n
0"
Φ p and Φ^ is the antisymmetric matrix of Φ. Given a rotation vector Φ E so(3).
where 5^
0- 0
rotation matrix R can be obtained through the exponential map R = exp(ø") = 1
nt (d^)².
Simultaneously, we denote En [m+1j] (Ø^)" as J. Then Eq. 4 can be rewritten as:
R 0
Jp
(5)
Consequently, due to the coupling between R En= n! (6^)² and J En=o [n+1y (o*)", the translation updates are influenced by rotation. Incorporating momentum may lead non-intuitive optimization trajectories [32]. Therefore, omit the coefficient J from the translation term. This approach enables updating the translation of the the center mass and the rotation around the center of mass independently.
Coarse-to-Fiie Fine Positional Encoding. BARF (11 propose gradually activate high- brequeney/high-resoluee
components within positional encoding. We further apply this approach to multi-scale planar and hash encoding [70] and found it also yields benefits our large-scale scenarios. For the detailed formulation, we direct readers to reference (21]
3.3 Graph-based Pure Geometric Optimization
ICP [5] is classic method for registration based on inter-frame geometric correlations. The essence of ICP lies in searching for the closest point as correspondence in another frame's point at each iteration, followed by using Singular Value Decomposition (SVD) solve Eq. (6), then iteratively refining the solution. Nonetheless, ICP frequently converges to local optima (Fig D. In contrast, NeRF optimizes pose globally through the implicit radiance fields. However, lacks geometric constraints and overlooks the strong geometric information inherent in the point cloud, leading to poor geometric consistency. As a consequence, both ICP and NeRF acting individually tend to converge to local optima. Our goal is to employ hybrid method, utilizing NeRF for global pose optimization and integrating geometric information as an auxiliary support.
Drawing inspiration from ICP (5), we recognize that minimizing the Chamfer Distance (CD) is in line with the optimization objective of each step in ICP algorithm, demonstrated in Eq. 7
T = min Σ
min | Q
P.p. -qil2
(6)
P.EP
((P = Σ we min Tpp- 9.CQ
TQq
Σ
w. min TQq: Tpp.11
P.EP
(7)
D.EP
q:EQ
where q, p point cloud Q, P are homogeneous coordinates. Tp, TQ represent the transformation matrix to the world coordinate system. However, minimizing the original CD does not necessarily indicate improved accuracy due to the non-overlapping regions between point clouds. To alleviate this negative impact, we weight each correspondence based on Eq. (8), whereas w; in the original CD normalized by the N.N is the number of points.
W/: exp(t/ d' (diipped)
== scheduler((o),
& detipped = max(voxelsize,, d:)
(8)
exp(t/ ldspped))
where d, denotes the distance between a pair of matching nearest neighbor points, is the temperature to sharpen the distribution of the datspped' The distance d, is clipped to the size of the downsampled voxel grid. This soft assignment can be considered as an approximately derivable version of weighted
averaging. Eq. (7) will degenerate to the original CD when - O, degrade to considering only correspondences with the minimum distance when /. + co. Considering the distance lacks practical significance in initial optimization, the scheduler is set as linear or exponential function vary t from 0 to 0.5 the optimization progresses. Building upon the above, shown in Fig. 3
5

---

Graph-based RCD
RRE(rad) compared with GT Pase (RRE(y)-epocheee -xpoch(x))
Registration Result
(a) Geometric Optimizer
RCD with
Poses uptimized the wrong direction
Reduae the weigght
region
100g
Geo- Apo-Optinzze
1000
(b)w/ Geometric Optimizer
Non-Overbap
0d
Correct direction
00
Geo- Optimizer
1.00
2.0
400
70eam R1D0 Jull Iniler Sturmss
RAE Ronge of Inteer Prs 1000 IE single ostlers
ARE Range of outier funca
Figure 3: Graph-based RCD (left). We introduce control factor CD to diminish the weighting of non-overlapping regions between point clouds. Geo-optimizer and its impact on pose optimization (right). Pose errors are reduced after each increase caused by NeRF's incorrect optimization direction. Comparison of (a) and (b) shows Geo-optimizer prevents incorrect pose optimization of NeRF.
(o)Training Loss (Loss(y) (Loss(y)-spoccco)) epoch(x))
(b) (rad) compared with GT Pose (epoc(()))))( (c)Ropistration Result
100
mm
rewwighttgg Optinbeee
w/o selective- rraeeighting
w/d Gibb Optimizzer
solective roweightigg N/O Gev Optiizerr
750
-neelightng Optimizer
I// selective receeghhinn Geo- Optiniit
Pre trainerined NeRF provide
globally consistent geometric optinization.
Opt-Oppi/bber
Figure 4: Impact of releetive-reweighting training strategy pose optimization. (a) Frames with outlier poses exhibit significantly higher losses. With selective-rewcightingg outlier frames maintain relatively higher loss without overfitting. (b) After several training iterations, the pre-trained outlier-aware NeRF can provide globally consistent geometric optimization for outlier frames.
approximate the registration objective by optimizing the Graph-based Robust Chamfer distance (G- RCD). Specifically, we construct graph (W, ))) where each vertex W represents of points and each edge Y' corresponds to proposed RCD via Eq. (7). We connect each frame with its temporally preceding n frames mitigate error accumulation ICP [5]. Then RCD is calculated for all edges as Eq. (9), and M denotes the number of frames in the sequence. Notably, in Eq. 7 G-RCD is computed using the global transform matrix, enabling direct gradient propagation of the Graph- based loss to the global transformation matrix of each frame.
ggraph
Σ L(.))) (K.))EE
(nM m(++l)
Discussion. As illustrated in Fig. 3(b), insufficient geometric guidance leads to certain frame poses being optimized in the wrong direction. Geometric optimizer can address this issue by preventing pose updates strictly following NeRF and correcting wrong optimization directions that do not conform to global geometric consistency. This method involves externally modifying pose parameters and providing effective geometric guidance early in the ill-conditioned optimization process. Consequently, few iterations of graph-based RCD computation suffice to offer ample guidance for NeRF
3.4 Selective- Strategy for Outlier Filtering
In bundle-adjusting optimization, as shown in Fig. (a), we observed that frames with outlier poses present significantly higher rendering losses during the early stages of training. However, low

---

frequency and sparsity of point clouds result in quick overfitting of individual frames including outliers (cf. Fig. I (b)). This leads to minimal pose updates when the overall loss decreases, resulting in incorrect poses and inferior reconstruction. Inspired by the capabilities of NeRF in pose inference 633. we decrease the learning rate (1r) of neural fields for the top frames with the highest rendering losses as Eq. (TO), while keeping 1r of poses unchanged. The strategy facilitates gradient propagation towards outlier poses, while the gradient flow to the radiance fields is concurrently diminished. Consequently, it's analogous to leveraging pre-trained NeRF for outlier pose correction and lessens the adverse effects caused by outliers during the optimization process.
Prouttiers = (wo + 1(1 wa))lrintiers
(wo 0)
(10)
Where E [0, 1] denotes training progress, Akin to leaky ReLU 660, we set the reweighting factor Wp to a relatively small value. wy increases as the process progresses, which ensures the network's ongoing learning from these frames and avoids stagnation.
3.5 Improving Geometry Constraints for NeRF
Point clouds encapsulate rich geometric features. However, solely supervising NeRF training via range images pixel-wise fails to fully exploit their potential, e.g., normal information. Furthermore, the Chamfer distance can directly supervise the synthesized point clouds from 3D perspective. Therefore, in addition to supervising via 2D range map, we propose directly constructing a three- dimensional geometric loss function between the generated point cloud and the ground truth point cloud. Unlike our Geo-optimizer, Eq. imposes constraints between synthetic point clouds P and ground truth point clouds P:
Len Σ min lp: pill + NP PIEP
Σ min lip - p:l12
Np
AED MEE
(11)
Based on the point correspondences established between P and P as derived in Eq. the constraint of normal can be formulated minimizing:
Lnormal
1 Np
Σ A.E
min (W - N(x)
1 NP
Σ min (W()))) (1) P.EP P.EP
(12)
PLEP
Thus, the normal loss is calculated between the synthetic point cloud and the ground truth point cloud ensure more accurate normal vectors of the point cloud synthesized from NeRF. Moreover, we also employ 2D loss function to supervise NeRF as Eq. (3))
(r(r)=
Σ R
da Dell - D(r)l,, + A. [|s(r) s(r)ll + A. [(((x)) R(F))l²
(13)
where D represents depth and S, represents intensity and ray-drop probabilities. Consequently. the loss for Neural LiDAR fields is weighted combination of the depth, intensity, ray-drop loss and 3D geometry constraints, which can be formalized as L = L.r + Xn + AcLeD-
4 Experiment
4.1 Experimental Setup
Datasets and Experimental Settings. We conducted experiments on two public autonomous driving datasets: NuScenes [9] and KITTI-360 30 dataset, each with five representative LiDAR point cloud sequences. We selected 36 consecutive frames at 2Hz from keyframes as a single scene for NuScenes, holding out 4 samples at 9-frame intervals for NVS evaluation. KITTI-360 has an acquisition frequency of 10Hz. We used 24 consecutive frames sampled every 5th frame to match scene sizes of Nuscenes, holding out 3 samples at 8-frame intervals for evaluation. We perturbed LiDAR poses with additive noise corresponding to a standard deviation of 20 deg in rotation and 3m in translation.
Metries. We evaluate our method from two perspectives: pose estimation and novel view synthesis. For pose evaluation, we use standard odometry metrics, including Absolute Trajectory Error (ATE) and Relative Pose Error (RPE, in rotation and RPF, translation). Following LiDAR4D 770 for NVS evaluation, we employ CD to assess the 3D geometric error and the F-score with 5cm error
7

---

Method
Duraset
Point Cloud
Depth
Ineesity
CDI
F-score"
RMSEL
ModAE]
LPIPSL
SSIM?
PSNR1
RMSEL
ModAE]
UPISS
SSIM
PSNR1
BARF-LN LN
51
1,2695
0.7589
8.2414
0.1125
0.1432
0.5856
20.89
0.392
00144
0.1023
0,6119
26.2330
7,5353
20.6244
0,6299
Nuscemes
GeoTrans
4,1587
0.2993
10,789
2.1529
0,1455
0.3671
17.5885
0,0679
0.056
0.1149
0.3476
23.3111
GeoNLF (Ours)
0.2408
0.3647
5,8208
0.0811
0,0727
0.7746
22.9472
0,0378
0.0100
0.0774
0,7368
28,6778
BAR 51
3,1001
0.6156
7.5767
20083
0,577
0.2834
22.5759
0,2121
0.1575
0,7121
0.1468
11.9778
KITTI-3611
6.3005
2.1686
0,5176
0.3752
22.6196
0.1902
12.9266
25.6137
12.9707
GooNTF (Ours)
6.2363
0.9178
4,6293
0.1009
0.3000
0.6272
25:2756
0.1495
0.1525
0.5399
0,3165
16.5813
Table 1: NVS Quantitative Comparison on Nuscenes and KITTI-360. We compare our method
to different types of approaches and color the top results as best and second best averaged over the 5 sequences.
All results are
Nuscenes
KITTI-3G 360
GeONLF (Ours)
Geotrans Geotrans-assisted
HASH-LN
BARF-LN
Figure 5: Qualitative comparison of NVS. We compared GeoNLF with other pose-free methods and asssieeee NeRF. Especially, GeoTrans fails Nuscenes due the inaccurate poses.
threshold. Additionally, we use RMSE and MedAE to compute depth and intensity errors in projected range images, along with LPIPS (9), SSIM [57] and PSNR to measure overall variance.
Implementation Details. The entire point cloud scene is scaled within the unit cube space. The optimization of GeoNLF is implemented Pytorch 442 with Adam (26) optimizer. All the sequences are trained for 60K iterations. Our Geometry optimizer's 1r for translation and rotation is the same as the Ir for pose NeRF with synchronized decay. We use the coarse-to- -o-ffee strategy 21 which starts from training progress 0. to 0.8. The reweight coefficient for the top-5 frames linearly increases from 0.15 during training. After every mi epoch of bundle adjusting global optimization, we proceed with m2 epoch of pure geometric optimization, where m2/m1 decrease from 10 to 1.
4.2 Comparison in LiDAR NVS
We compare our model with BARF 31 and HASH 21], both of which use LiDAR-NeRF[S 51] as backbone. For PCR-assisted NeRF, we opt to initially estimate pose utilizing pose derived from GeoTrans 44], which is the most robust and accurate algorithm among other PCR methods in our experiments. And subsequently we leverage LiDAR-NeRP [5I] for reconstruction. For all Pose-free free methods, we follow NeRFmm (58) obtain the pose of test views for rendering. The quantitative and qualitative results are in Tab. and Fig. 5 Our method achieves eigh-precision registration and high-quality reconstruction across all sequences. However, baseline methods fail completely on certain sequences due to their lack of robustness. Please refer to Fig. for details. Ultimately, our method excels in the reconstruction of depth and intensity, as evidenced by 7.9% increase F-score on Nuscenes and 13. 13.1% 1% on KITTI-360 compared to the second best result.

---

Midied
Nakcns
Mehhol
Closa
Deth
lniaaity
Posc
1331
30383
1A09
CD)
PSNIR 1
1533
MIC2
27
58.54
1.100
2319
35.84
1418
1.43
Greptirr
SCREA
100.55
2,65
1351
12.57
0/06
2339
wM RCD
02711
21.1323
26,7232
8,476
0.163
0.352
GenTra
16,077
D300
0.88
0.33
0316
0.0544
21 109
25,3289
K.126
01156
BARFUU
195.74
1230
wit 4a
0.2977
21.7125
18.2200
7.233
DASIEUN
180.30
0333
1151
145
3,171
3665
GeNNLF
5.49
6.36
GeNNFF
0.2363
22,0472
28
7
0.183
0.338
Table Pose estimation aceuracy comparison.
Table 3: Ablation study on Nuscenes.
(a) w/o SR w/o Geo-optimizer
(b) w/ SR
w/o Geo-optimizer
(c) w/ SR
w/ Geo-optimizer
Figure 6: Qualitative results of ablation study. We present the NVS and Registration results in the first and second rows. Outlier frames emerged w/o SR or w/o G-optim.
4.3 Comparison in Pose Estimation
We conduct comprehensive comparisons of GeoNLF with pairwise baselines, including traditional
method ICP [5). learning-based Geo'Trans [4] and outdoor-specifie HRegNet (34), as well as multi- view baselines MICP [13] and learning-basee SGHR (54). For pairwise methods, we perform registration between adjacent frames in Odometry-like way. For SGHR, we utilize FCGF [14]
descriptors followed by RANSAC (88 for pairwise registration. The estimated trajectory is aligned with the ground truth using Sim(3) with known scale.
GeoNLF outperforms both the registration and pose-free NeRF baselines. Quantitative and Qualita- tive results are illustrated in Tab. and and Fig. [ As depicted in Fig. [ most registration methods fail to achieve globally accurate poses and completely fail in some scenarios, leading to massive errors in average results. Significant generalization issues arise for learning-based registration methods due to potential disparities between testing scenarios and training data, including differences initial pose distributions. This challenge is particularly pronounced in HRegNet [34]. While the transformer model GeoTrans [44] with its higher capacity offers some alleviation to the issue, it remains not fully resolved.
4.4 Ablation Study
In this Section, we analyze the effectiveness of each component of GeoNLE. The results of ablation studies are shown in Tab. 3 (1) Geo-optimizer, When training GeoNLF w/o geo-optimizer (w/ G-optim), pose optimization may initially converge towards incorrect directions. Excluding geo- optimizer in GeoNLF results in decreased pose accuracy and reconstruction quality. (2) Control factor of graph-based RCD. Although geo-optimizer crucial in the early stages of optimization. we find that using the original CD limits the accuracy of pose estimation. Removing the control factor (w/o RCD) leads to decreased pose estimation accuracy due to the presence of non-overlapping regions. (3) Selective-reweighting (SR) strategy. As presented in Figs. 4 and ol Tab. 3 outlier frames cause GeoNLE w/o SR strategy to overlook multi-view consistency, adversely affecting reconstruction quality. (4) Geometric constraints. Removing the 3D constraints (wo L3a) results in a decline in CD due to the photometric loss's inability to adequately capture geometric information.
4.5 Limination
Despite the fact that GeoNLF has exhibited exceptional performance in PCR and LiDAR-NVS on challenging scenes, it is not designed for dynamic scenes, which is non-nggliiible in autonomous

---

(a) Nuscenes
(b)KITTI-369
Figure 7: Qualitative registration results of HASH-LN and GeoNLF on Nuscenes and KITTI- 360 dataset. The first row contains original inputs, the second row shows the results of HASH-LN, and the third row displays the results of GeoNLE.
driving scenarios. Additionally, GeoNLF targets point clouds within a sequence, relying on the temporal prior of the point clouds.
5
Conclusion
We introduce GeoNLF for multi-view registration and novel view synthesis from a sequence of sparsely sampled point clouds. We demonstrate the challenges encountered by previous pairwise and multi-view registration methods, as well as the difficulties faced by previous pose-free methods. Through the utilization of our Geo-Optimizer, Graph-based Robust CD, selective-reweightnng strategy and geometric constraints from 3D perspective, our outlier-aware and geometry-aware GeoNLF demonstrate the promising performance in both multi-view registration and NVS tasks.
Acknowledgments
This work was supported by the National Key Research and Development Program of China 202420241 1000), in part by the National Natural Science Foundation China (No. 62372329), in part by Shanghai Scientific Innovation Foundation No.23DZZ10034000 in part by Tongji-Qomolo Autonomous Driving Commercial Vehicle Joint Lab Project, and in part by Xiaomi Young Talents Program.
10

---

References
[1] Yasuhiro Aoki, Hunter Goforth, Rangaprasad Arun Srivatsan, and Simon Lucey. Pointnetlk: Robust & efficient point cloud registration using pointnet. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7163-7172, 2019.
[2] Mica Arie-Nachimson, Shahar Z Kovalsky, Ira Senelmmacher-Shiizemmaa Amit Singer, and Ronen Basri. Global motion estimation from point matches. In 2012 Second international conference on 3D imaging, modeling, processing, visualization transmission, pages 81-88.
IEEE, 2012.
[3] Federica Arrigoni, Beatrice Rossi, and Andrea Fusiello. Spectral synchronization multiple views in se (3). SIAM Journal on Imaging Sciences, 9(4):1963-199)) 2016.
[4] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin -rrtil-ruaalla,, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1855-5864, 2021.
[5] Paul Best and Neil McKay. Method for registration of 3-d shapes. In Sensor fusion IV: control paradigms and data structures, volume 1611, pages 586-606. Spie, 1992.
[6] Wenjing Bian, Zirui Wang, Kejie Li, Jia-Wang Bian, and Victor Adrian Prisacariu. Nope- nerf: Optimising neural radiance field with no pose prior. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 4160-4169, 2023.
[7] Tolga Birdal and Slobodan Ilic. Cad priors for accurate and flexible instance reconstruction. In Proceedings of the IEEE international conference on computer vision, pages 133-142, 2017.
[8] Tolga Birdal and Umut Simsekli. Probabilistic permutation synchronization using the rie- mannian structure of the birkhoff polytope. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11105-11116 2019.
[9] Holger Caesar, Varun Bankiti, Alex Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621-11631 2020.
[10] Eric R Chan, Connor Z. Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry- aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16123-16133, 2022.
[11] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European Conference on Computer Vision, pages 333-350. Springer, 2022.
[12] Yu Chen and Gim Hee Lee. Dbarf: Deep bundle-adjusting generalizable neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24 34, 2023.
[13] Sungjoon Choi, Qian- Yi Zhou, and Vladlen Koltun. Robust reconstruction of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5556 5565, 2015.
[14] Christopher Choy, Jaesik Park, and Vladlen Koltun. Fully convolutional geometric features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2019.
[15] Junyuan Deng, Qi Wu, Xieyuanli Chen, Songpengcheng Xia, Zhen Sun, Guoqing Liu, Wenxian Yu, and Ling Pei. Nerf-loam: Neural implicit representation for large-scale incremennal lidar odometry and mapping. Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 8218 8218-8227, 2023.

---

[16] Kangle Deng, Andrew Liu, Jun Yan Zhu. and Deva Ramanan. Depth-supervised nerf: Fewer views and faster training for free. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pages 12882-1 12882-12911, 2022.
[17] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference on robol learning, pages 1-16. PMLR, 2017.
[18] Martin Fischler and Robert C Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6):381-385 1981.
[19] Zan Gojeic, Caifa Zhou, Jan D Wegner, Leonidas J Guibas, and Tolga Birdal. Learning multiview 3d point cloud registration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1759-1769, 2020.
[20] Benoît Guillard, Sai Vemprala, Jayesh K Gupta, Ondrej Miksik, Vibhav Vineet, Pascal Fua, and Ashish Kapoor. Learning to simulate realistic lidars. In 2022 TEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 8173-8180 IEEE, 2022.
[21] Hwan Heo, Taekyung Kim, Jiyoung Lee, Jaewon Lee, Soohyun Kim, Hyunwoo J Kim, and Jin- Hwa Kim. Robust camera pose refinement for multi-resolution hash encoding. International Conference on Machine Learning, pages 13000-13011 PMLR, 2023.
[22] Wenbo Hu, Yuling Wang. Lin Ma, Bangbang Yang, Lin Gao, Xiao Liu, and Yuewen Ma. Tri- miprf: Tri-mip representation for efficient anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19774-19783, 2023.
[23] Shengyu Huang, Zan Gojcic, Zian Wang, Francis Williams, Yoni Kasten, Sanja Fidler, Kon- rad Schindler, and Or Litany. Neural lidar fields for novel view synthesis. arXiv preprint arXiv:2305.0164. 2023.
[24] Xiaoshui Huang, Guofeng Mei, and Jian Zhang. Feature-mette registration: fast semi- supervised approach for robust point cloud registration without correspondences. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1336-11374, 2020.
[25] Shengze Jin, Iro Armeni, Mare Pollefeys, and Daniel Barath. Multiway point cloud mosaicking with diffusion and global optimization. arXiv preprint (xXiv:240400042) 2024.
[26] Diederík P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.69800 2014.
[27] Nathan Koenig and Andrew Howard. Design and use paradigms for gazebo, an open-source multi-robot simulator. In 2004 IEEE/RSJ international conference on intelligent robots and systems (IROS)IEEE Cat. No. 04CH37566), volume 3, pages 2149-2154. leee, 2004.
[28] Chenqi Li, Yuan Ren, and Bingbing Liu. Pegen: Point cloud generator for lidar simulation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 11676-11682. IEEE, 2023.
[29] Jiahao Li, Changhao Zhang, Ziyao Xu, Hangning Zhou, and Chi Zhang. Iterative distance-awan similarity matrix convolution with nutual-supervisee point elimination for efficient point cloud registration. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28; 2020, Proceedings, Part XXIV 16, pages 378-394- Springer, 2020.
[30] Yiyi Liao, Jun Xie, and Andreas Geiger. Kitti-360: novel dataset and benchmarks for urban scene understanding in 2d and 3d. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):3292-3310, 2022.
[31] Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey. Barf: Bundle-adjusting neural radiance fields. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 5741-5751, 2021.
12

---

[32] Yunzhi Lin, Thomas Müller, Jonathan Tremblay, Bowen Wen, Stephen Tyree, Alex Evans, Patricio A Vela, and Stan Birchfield. Parallel inversion of neural radiance fields for robust pose estimation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9377-9384. IEFE, 2023.
[33] Yu-Lun Liu, Chen Gao, Andreas Meuleman, Hung- Yu Tseng, Ayush Saraf, Changil Kim, Yung- Yu Chuang, Johannes Kopf, and Jia-Bin Huang. Robust dynamic radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13-23, 2023.
[34] Fan Lu, Guang Chen, Yinlong Liu, Lijun Zhang, Sanqing Qu, Shu Liu, and Ronggi Gu. Hregnet: A hierarchical network for large-scale outdoor lidar point cloud registration. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16014-16023, 2021.
[35] Sivabalan Manivasagam, Shenlong Wang, Kelvin Wong, Wenyuan Zeng, Mikita Sazanovich, Shuhan Tan, Bin Yang, Wei-Chiu Ma, and Raquel Urtasun. Lidarsim: Realistic lidar simulation by leveraging the real world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11167-11175 2020.
[36] Eleonora Maset, Federica Arrigoni, and Andrea Fusiello. Practical and efficient multi-view matching. In Proceedings of the IEEE International Conference on Computer Vision, pages 4568 4576, 2017.
[37] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
[38] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. ACM transactions on graphics (TOG), 41()): 1-15, 2022.
[39] Michael Niemeyer, Jonathan T Barron, Ben Mildenhall, Mehdi SM Sajjadi, Andreas Geiger, and Noha Radwan. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 5480-5490, 2022.
[40] Michael Oechsle, Songyou Peng, and Andreas Geiger. Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5589-5599, 2021.
[41] Keunhong Park, Philipp Henzler, Ben Mildenhall, Jonathan T Barron, and Ricardo Martin- Brualla. Camp: Camera preconditioning for neural radiance fields. ACM Transactions on Graphics (TOG), 42(6):1-11, 2023.
[42] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, reg-hperformanc deep learning library. Advances in neural information processing systems, 32, 2019.
[43] François Pomerleau, Francis Colas, Roland Siegwart, and Stéphane Magnenat. Comparing icp variants on real-world data sets: Open-source library and experimental protocol. Autonomous robots, 34:133 14:133-148, 2013.
[44] Zheng Qin, Hao Yu, Changjian Wang. Yulan Guo, Yuxing Peng, and Kai Xu. Geometric transformer for fast and robust point cloud registration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11143-11155 2022.
[45] Srikumar Ramalingam and Yuichi Taguchi. A theory of minimal 3d point to 3d plane registration and its generalization. International journal of computer vision, 102:73-90, 2013.
[46] Barbara Roessle, Jonathan T Barron, Ben Mildenhall, Pratul P Srinivasan, and Matthias NieBner. Dense depth priors for neural radiance fields from sparse input views. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 12892-12901, 2022.
13

---

[47] Szymon Rusinkiewicz and Mare Levoy. Efficient variants of the icp algorithm. In Proceedings third international conference on 3-D digital imaging and modeling, pages 145-152. IEEE, 2001.
[48] Johannes L Schonberger and Jan-Michael Frahm. Structure-frosd wareee-fooo-mmomoo revisited. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 2016.
[49] Shital Shah, Debadeepta Dey, Chris Lovett, and Ashish Kapoor. Airsim: High-fidelity visual and physical simulation for autonomous vehicles. In Field and Service Robotics: Results of the I Iith International Conference, pages 621-635. Springer, 2018.
[50] Liang Song, Guangming Wang, Jiuming Liu, Zhenyang Fu, Yanzi Miao, et al. Sc-nerf: Self- correcting neural radiance field with sparse views. arXiv preprint arXiv:2309.000 0.000008,, 2023.
[51] Tang Tao, Longfei Gao, Guangrun Wang, Peng Chen, Dayang Hao, Xiaodan Liang, Mathieu Salzmann, and Kaicheng Yu. Lidar-nerf: Novel lidar view synthesis via neural radiance fields. arXiv preprint arXiv:2304. 10406, 2023.
[52] GK Tejus, Giacomo Zara, Paolo Rota, Andrea Fusiello, Elisa Ricci, and Federica Arrigoni. Rotation synchronization via deep matrix factorization. 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 2113-2111111 IEEE, 2023.
[53] Prune Truong, Marie-Julie Rakotosaona, Fabian Manhardt, and Federico Tombari. Sparf: Neural radiance fields from sparse and noisy poses. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4190 42004 2023.
[54] Haiping Wang, Yuan Liu, Zhen Dong, Yulan Guo, Yu-Shen Liu, Wenping Wang, and Bisheng Yang. Robust multiview point cloud registration with reliable pose graph initialization and history reweighting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9506-9515, 2023.
[55] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprini arXiv:2106. 10689, 2021.
[56] Yue Wang and Justin M Solomon. Deep closest point: Learning representations for point cloud registration. In Proceedings of the IEEE/CVF international conference computer vision, pages 3523-3532, 2019.
[57] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4): 600-612, 2004.
[58] Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. Nerf-: Neural radiance fields without known camera parameters. arXiv preprint 2xXiv:2022070000 2021.
[59] Yi Wei, Shaohui Liu, Yongming Rao, Wang Zhao, Jiwen Lu, and Jie Zhou. Nerfingmvs: Guided optimization of neural radiance fields for indoor multi-view stereo. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5610-5619, 2021.
[60] Jin Xu, Zishan Li, Bowen Du, Miaomiao Zhang, and Jing Liu. Reluplex made more practical: Leaky relu. In 2020 IEEE Symposium on Computers and communications (ISCC), pages 1-7. IEEF, 2020.
[61] Weiyi Xue, Fan Lu, and Guang Chen. Hdmnet: A hierarchical matching network with double attention for large-scale outdoor lidar point cloud registration. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pages 3393-3403, 2024.
[62] Ze Yang, Yun Chen, Jingkang Wang. Sivabalan Manivasagam, Wei-Chiu Ma, Anqi Joyce Yang. and Raquel Urtasun. Unisim: A neural closed-loop sensor simulator. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1 1389-1399, 2023.
14

---

[63] Lin Yen-Chen, Pete Florence, Jonathan T Barron, Alberto Rodriguez, Phillip Isola, and Tsung- Yi Lin. inerf: Inverting neural radiance fields for pose estimation. In 2021 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 1323-1330. IEFE, 2021.
[64] Zehao Yu, Songyou Peng. Michael Niemeyer, Torsten Sattler, and Andreas Geiger. Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. Advances in neural information processing systems, 0052201882503333 2022.
[65] Wentao Yuan, Benjamin Eckart, Kihwan Kim, Varun Jampani, Dieter Fox, and Jan Kautz. Deepgmr: Learning latent gaussian mixture models for registration. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V 16, pages 733-750. Springer, 2020.
[66] Jian Zhang, Yuanqing Zhang. Huan Fu, Xiaowei Zhou, Bowen Cai, Jinchi Huang, Rongfei Jia, Binqiang Zhao, and Xing Tang. Ray priors through reprojection: Improving neural radiance fields for novel view extrapolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 18376-18386, 2022.
[67] Junge Zhang, Feihu Zhang, Shaochen Kuang, and Li Zhang. Nerf-lidar: Generating realistic lidar point clouds with neural radiance fields. arXiv preprint arXiv:2304.1 2023.
[68] Junge Zhang, Feihu Zhang, Shaochen Kuang, and Li Zhang. Nerf-lidar: Generating realistic lidar point clouds with neural radiance fields. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 11778-186, 2024.
[69] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[70] Zehan Zheng, Fan Lu, Weiyi Xue, Guang Chen, and Changjun Jiang. Lidar4d: Dynamic neural fields for novel space-time view lidar synthesis. arXiv preprint axXiv:240.002742, 2024.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: This paper does discuss the limitations of the work performed by the authors.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: This paper fully discloses all the information needed to reproduce the main experimental results of the paper.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification:
This paper provide open access to the data and code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/dde/eeee Cooeeeoodeesssseeeeee //nips.com /ttp://bbb///iiiiee/eeeeeeaaaaaa cassiicyncccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: This paper specify all the training and test details.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: This paper report appropriate information about the statistical significance of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
18

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: This paper provide sufficient information on the computer resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http::
Answer: [Yes]
Justification: This research conducted in the paper conform, in evrery respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
There is no societal impact of the work performed. No harm technical paper.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We mentioned creators original owners of assets and properly respected.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: This paper introduces assets well documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "ySLakXRwHX", "reviewer": "Reviewer_PTin", "summary": "The paper proposes GeoNLF, a method for pose-free Lidar point cloud novel view synthesis. They propose optimizing point cloud poses simultaneously through a bundle-adjusting neural lidar field as well as through ICP-inspired geometric optimization. The neural lidar field is supervised with ground truth range images, intensity images, and ray-drop probability images as well as ground truth point clouds. The geometric optimization optimizes poses to minimize a weighted average of the chamfer distances between neighboring lidar frames. The proposed method is validated for lidar novel view synthesis and pose estimation using the Nuscenes and KITTI-360 datasets.", "questions": "Please refer to the weakness section.", "limitations": "There is no discussion about the optimization time, which might be significantly increased due to the geometric optimization requiring computing the chamfer distance between all combinations of the 4 adjacent frames.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "1) The authors apply knowledge from nerf pose estimation to the problem of lidar novel view synthesis through the bundle-adjusting lidar nerf.\n2) They propose a novel alternating pose optimization between the implicit nerf representation and the explicit point clouds.\n3) The proposed method is supported by experiments on two real-world datasets which verify its validity.\n4) The paper proposes a new task of jointly recovering pose and lidar point cloud novel views, which is significant for lowering the impact of noisy estimated lidar poses.\n5) They achieve state-of-the-art results for lidar novel view synthesis with the proposed method, as well as better point cloud registration.", "weaknesses": "The methodology section reads like a work in progress, making it hard to understand the details of the proposed method.\n1) There is no intuitive explanation as to why the \"J\" term in Eq. 5 can simply be dropped. While the main text states that dropping this term allows independent updates of rotation and translation, the supplementary material then states that simultaneous optimization yielded satisfactory results in L527.\n2) P is used to denote poses in Eq. 1 as well as point clouds in Eq. 11 and ray-drop loss probabilities in Eq. 13. L173. Similarly, I is used to refer to NeRF images in L112 and then intensity images in Eq. 13. Intensities and ray-drop probabilities are referred to as S and R in L119.\n3) L173 states that Eq. 7 shows how CD is in line with each step of ICP, but the optimization steps of ICP are never introduced.\n4) Eq. 8 introduces d_clipped which is never used. It takes as parameter a distance \"d\" that is never introduced.\n5) There is no explanation as to how the normals used in Eq. 12 are obtained. Are they obtained from the implicit density field for both predicted and pseudo-groundtruth pointclouds?\n6) There is a lot of information in Fig. 3, but the caption does not explain what the figure is trying to show.\n7) Sections 3.2 and 3.5 both discuss the nerf optimization, but are separated by sections 3.3 and 3.4 which discuss geometric optimization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your positive feedback on our method's novelty and the significance of our new task to reduce the impact of noisy estimated LiDAR poses during LiDAR NVS. The issues you raised have been effectively addressed as follows. Regarding weaknesses 2,5 and your concern about the time-consuming limitation, please kindly refer to the global rebuttal.\n\n**[Q1] The Issue of Term J in Pose Representation Formula.** Commonly used $\\xi \\in \\mathfrak{se}(3)$ represents the transformation in a coupled manner, the translation component of $T$ will be influenced by the rotation parameter $\\phi \\in \\mathfrak{so}(3)$, as shown in Eq. (5) and L153-154. This means the translation of the rigid body's centroid is affected by the rotation. \nLet $B$ and $A$ denote the transformed and original point clouds, where $\\overline{B}$ and $\\overline{A}$ represent the homogeneous coordinates. Then, $\\overline{B} = T\\overline{A}$. By omitting $J$ in Eq. (5), according to the principles of matrix multiplication, our method for rigid body pose transformation $\\overline{B} = T_{\\textbf{w/o} J}\\overline{A}$  equates to $B = \\boldsymbol{R}A + \\rho = \\sum_{n=0}^{\\infty} \\frac{1}{n!}\\left(\\phi^{\\wedge}\\right)^n A+\\rho$, $\\phi \\in \\mathfrak{so}(3)$ and $\\rho \\in \\mathbb{R}^3$ are the pose parameters to be updated, and the translation is represented solely by $\\rho \\in \\mathbb{R}^3$. **In this way, the rotation parameter $\\phi$ and the translation parameter $\\rho $ represent the rigid body motion as rotation about the center of mass and translation of the center of mass.** This method represents rotation and translation separately and independently, which is reasonable and physically meaningful. \n\nBesides, in the context of “independently” in L157 of the main text, we refer to the fact that our approach optimizes rotation and translation simultaneously, but their updates are decoupled and do not interfere with each other. Thus, there is no conflict between the Appendix and the main text. We fully understand your concern and believe that changing “independently” to “in a decoupled manner” would be more appropriate.\n\n**[Q3] The Issue of Missing Descriptions in ICP Algorithms.** Thanks for your comments. \nBy describing “in line with the optimization objective of each step in the ICP algorithm\"(L173), we intend to state that our CD-based optimization is inspired by the well-known classic ICP algorithm. We do not use ICP in our method, and the core optimization step is clearly explained in Eq. (7) and L190-192 (through gradient descent). Given the fluency of description, **we briefly summarize the optimization steps of ICP in L163-L166 and Eq. (6) in our main text.** We will include the detailed optimization steps of ICP in the supplementary material.\n\n**[Q4]: The Issue of Distance Representation ($d$) in Eq. (8).** We apologize for not having explained our method more clearly. $d$ denotes the distance between a pair of matching nearest neighbor points. We intend to use the subscript “clipped\" to indicate that this distance has been truncated to be greater than a predefined threshold. Therefore, $d_{clipped}$ represents the $d_i$ we actually used in the first formula. We will update $d_{clipped}$ to $d_i$. The revised formula is as follows:\n\n$w_{i} = \\frac{\\mathrm{exp}(t/d_i)}{\\sum_{i=1}^N \\mathrm{exp}(t/d_i)}, t=\\mathrm{scheduler}(t_0),d_i=\\mathrm{max}(d_0,d)$\n    \nwhere $d$ denotes the distance between a pair of matching nearest neighbor points, $t$ is the temperature to sharpen the distribution of the $d_i$, and $d_0$ is a threshold we set. \n\n**[Q6]: The Issue of Caption Content of Fig. (3).** Based on your suggestion, we will further revise the caption for better readability. The revised caption of Fig. (3) is “**Graph-based RCD (left).** We introduce control factor $t$ in CD to diminish the weighting of non-overlapping regions between point clouds. **Geo-optimizer and its impact on pose optimization (right).** Pose errors are rapidly reduced after each increase caused by NeRF's incorrect optimization direction. Comparison of (a) and (b) shows Geo-optimizer prevents incorrect pose optimization of NeRF.\".\n\nThe content shown on the left side of Fig. (3) is explained in L185-190. As for the right side of Fig. (3), our primary intention is to provide readers with a clear and intuitive comparison through the figures. To facilitate this, we have carefully annotated the figures with arrows and text for better guidance. Reviewer fgnm and Reviewer TE2m also point out that this figure is informative and accurately demonstrates the effectiveness of our method. We hope the revised title will better explain the information in the figure.\n\n**[Q7]: The Issue of Organizational Structure in the Main Text.** We fully understand your perspective on creating a more cohesive structure. However, within the domain of NeRF / pose-free NeRF, it is common practice to place the NeRF optimization loss (Sec 3.5) at the end to better present the overall pipeline, such as Nope-NeRF[6], NFL[24], LiDARNeRF[52]. \n\nTherefore, we adopt the following structure: Sec 3.2 discusses two main aspects of pose-free NeRF, $\\textit{i.e.}$, the representation of pose and the encoding method of NeRF. This serves as a foundational premise for the subsequent discussions. Specifically, the representation of pose is essential for pose optimization, as the $T \\in \\text{SE}(3)$ in original NeRF lacks gradient propagation capabilities ($T + \\delta T \\notin \\text{SE}(3)$). And the encoding method is fundamental across all NeRF works. Then we proceed to discuss the optimization of pose (Sec. 3.3) and NeRF (Sec 3.4-3.5). These three sections (3.3-3.5) constitute our primary contributions. Based on your suggestion, we will integrate Sec 3.5 with the NeRF encoding content from Sec 3.2, thereby encapsulating the entirety of the NeRF optimization process within Sec 3.5, which will help to make the paper more cohesive.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RFiaN7fuLj", "reviewer": "Reviewer_fgnm", "summary": "In this submission the authors propose a novel strategy to jointly optimize a Neural LIDAR Field and the corresponding LIDAR poses. They achieve this by utilizing a hybrid strategy of alternating pure graph-based geometric optimization and global optimization of the Neural Field. Additionally they introduce a new selective reweighting strategy that steers gradients in the Neural Field towards outlier poses and new geometric losses that improve geometric reconstruction. They perform experiments on two datasets and achieve better results than relevant baselines in both reconstruction and registration.", "questions": "To focus discussion about the weaknesses raised above here are some detailed questions: \n1. How can the initial (bad) registration of the point clouds be obtained or is there a way this step can be removed ? Can the point cloud poses be initialized completely randomly or all as identity? \n2. Is there some reason (e.g. Privacy concerns) that prohibit the joint capture of Image data via cameras and LIDAR scans? This would enable standard Structure from Motion approaches for pose estimation. \n3. What is the overall motivation of representing LIDAR in a Neural Field and what benefits are gained by this? Are there downstream applications enabled by the proposed method? \n4. Can you report the Inlier Ratio, Feature Matching Recall or Registration Recall ? If not, why can this not be reported on this data?", "limitations": "The authors have addressed the limitations of their work with regards to dynamic scenes, since they only consider static scenes. \nSocietal impact is not discussed.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The proposed method with its hybrid optimisation and other components is novel and very interesting and achieves impressive results relevant to the field of LIDAR simulation. \n2. The paper is well written and gives deep insights into the optimisation procedure with well designed figures (e.g. Figure 3&4). The concepts are very clearly laid out and supported by experimental evidence and ablations. Some sentences could be improved but this is very minor and does not detract from the overall paper. \n3. It is very hard to find technical faults in this paper. The experiments are comprehensive and consider relevant baselines and the introduced components are backed up with mathematical foundations quite well.", "weaknesses": "In order of severity: \n\n1. Is it truly Pose-Free? The starting point for the LIDAR pose optimization is not completely random but they are perturbed from Ground truth by noise with a standard deviation of 20deg in rotation and 3m in translation (line 235). This means that some form of reasonable initialization is necessary for the method, which would have to be obtained from somewhere. How to obtain this initialization is not discussed in the paper. \n2. The motivation for why this approach is necessary is unclear. The setting of obtaining sparse LIDAR scans without at least poses derived from GPS or vehicle odometry seems unrealistic, as well as the assumptions that cameras would not be available given that they are significantly cheaper than LIDAR Scanners. This subtracts from the otherwise strong paper because the setting and significance of the results cannot be properly contextualized. It is also not clear from the Introduction what motivation the simulation of LIDAR data in a Neural Field has and what potential applications are enabled by the proposed method. \n3. While the authors report standard pose registration metrics (RPE & ATE) they do not consider Inlier Ratio, Feature Matching Recall or Registration Recall as metrics. This makes a comparison to prior work such as GeoTransformer [45] more complicated since these are the main metrics which they evaluate on in their paper. ([45] Qin, Z., Yu, H., Wang, C., Guo, Y., Peng, Y., & Xu, K. (2022). Geometric transformer for fast and robust point cloud registration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 11143-11152)) \n4. Outlier/Inlier poses and frames are discussed in Figure 3 and for example line 208 but it is never clarified how their inlier/outlier status is determined. This makes reproducibility harder and should be clarified either in the paper or supplementary. \n5. In section 3.5 the authors introduce a new loss between generated and ground truth point clouds. It is not mentioned how the generated point cloud is obtained from the Neural Field, which is important to know since this determines how the gradient from the loss is backpropagated through this mechanism. This should also be clarified in the final version or supplementary. \n6. In Equation 13 the variable P is very confusing since in the equations above it represents a point cloud and then here probably refers to the ray-drop loss. This is in contrast to line 119 & 120 where intensity and ray-drop probability are introduced as $S$ and $R$ respectively. This should be made coherent with each other or at least clarified. \n7. In line 113-115 the authors explain how the direction of rays is determined but do not explain how they represent translation in their polar coordinate system. This could also be clarified in the supplementary. \n\n\nMinor comments: \n - Line 16 NeRFs has achieved → NeRFs have achieved \n - Line 49 In furtherance of mitigating overfitting → To reduce overfitting. Clarity > Complicated Words. \n - Line 301 demonstrate the promising performance → demonstrate promising performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your careful reading and your questions provide valuable insights. We are pleased to address your concerns and engage in further discussion with you.\n\n**[Q1]: Initialization of Poses.** We follow previous works BARF[32] and NeRFmm[59], which are classical in the pose-free NeRF domain, to set up experiments. This setting is practical in autonomous driving scenarios, where pose estimations are not always accurate. Existing datasets with pose sensors(GPS, IMU), require calibration via time-consuming registration methods. Without initial pose information or with sparse inputs, SLAM or registration errors are more significant. Thus, adding random perturbations to each frame is a reasonable test of algorithm performance. \n\nNevertheless, we also perform experiments on Nuscenes with all initial poses set to identity, and our algorithm remains effective, as shown in Fig. (1) and Tab. (1) of the attached PDF. Additionally, you may run the demo we provide by adding the flag '--no\\_gt\\_pose', which implies that all poses are initialized to identity. For practical applications, initialization can use either the identity matrix, coarse point cloud registration results, or GPS/IMU information, especially in outdoor scenes.\n\n**[Q2]: Reasons for Prohibiting the Joint Capture of Image and LiDAR Scans.** Firstly, as noted in L21-23, pose-free NeRF in camera domain approaches aim to eliminate NeRF's reliance on SfM algorithms, which are time-consuming and often fail in homogeneous regions and fast-changing view-dependent appearances. Thus, the primary motivation for pose-free NeRFs is to enable reconstruction without relying on time-consuming SfM computations and avoiding the instability of SfM failures. Additionally, directly using point cloud registration algorithms might be a better choice. This is because 3D point cloud registration, in most cases, achieves more accurate poses compared to 2D-based registration. However, as shown in Fig.(1) of the main text and L33-38, pairwise and multiview point cloud registration may still be prone to local optima and error accumulation, making it difficult to achieve globally accurate poses.\n\nIn summary, poses obtained from GPS/IMU are often not sufficiently accurate(but can serve as a \"bad initialization\"), both image-based SfM and point cloud registration algorithms suffer from error accumulation and instability due to registration failures, and introduce extra computational overhead in reconstruction. Therefore, to achieve high-quality and efficient reconstruction, pose-free NeRFs are of practical significance.\n\nBesides, We commit to a LiDAR-only framework, aiming at achieving efficient reconstruction and NVS of point clouds without relying on images and accurate poses. Therefore, it has stronger applicability in LiDAR NVS.\n\n**[Q3]: Motivation and Benefit of Using NeRF and Downstream Application.** Traditional simulators and explicit reconstruct-then-simulate methods exhibit a large domain gap compared to real-world data. Moreover, traditional simulators like CARLA simulate\nwithin virtual environments, exhibiting diversity limitations and a reliance on costly 3D assets. Explicit reconstruction methods like LiDARsim[36] need to reconstruct mesh surfel representations, which are challenging for recovering precise surfaces in large-scale complex scenes. Additionally, NeRF inherently applies to the modeling of intensity and ray drop of LiDAR and does not require an intermediate modality for reconstruction, thereby eliminating the domain gap. Thus, some methods such as NFL[24] and LiDAR-NeRF[52], utilize NeRF to represent LiDAR data but need accurate poses.\n\nBesides, we appreciate your attention to the applications of our method. \n(1) Autonomous Driving Simulation: As shown in Fig. (3) of the attached PDF, we can conduct autonomous driving simulations in real-world scenarios rather than in simulators like CARLA, which is a prominent research field in autonomous driving. (2) Simulating point clouds from different LiDAR sensors: Our method allows for the adjustment of LiDAR sensors during NVS process. This flexibility facilitates simulations with various LiDAR configurations.\n\nAdditionally, unlike previous methods that require accurate poses from multiple sensors, our approach circumvents this complex and time-intensive process, offering substantial practical benefits. Further applications can be referenced in LiDAR4D[71] and we look forward to discussing them with you in the future discussion.\n\n**[Q4]: Pose Registration metrics.**\nRegistration Recall(RR) can be computed on this data. However, inlier Ratio and Feature Matching Recall cannot be reported on this data because these metrics are only applicable to feature-based point cloud registration methods, and our method does not require the computation of point features. Moreover, ATE and RPE are widely adopted in many pose-free NeRF works, such as Nope-NeRF[6] for evaluating the registration accuracy of a sequence of image/point clouds. However, RR is typically applied in pairwise registration, and is challenging to measure the registration performance of a sequence, because error accumulation in a sequence can prevent it from accurately reflecting registration performance. We realized that it is meaningful to compute the RR of adjacent frames by comparing  **gt relative poses** with predicted **relative poses**. The results are shown in Tab. (2) of the attached PDF.\n\n**[Weekness1(W1)]/ W[2]/ W[3].** Please refer to Q1/ Q2,Q3/ Q4.\n\n**[W4.]: Determining Outliers.** We directly select the top k frames with the highest training loss in each epoch as outliers without any priors (L206). The reason and effectiveness can be referenced in L201.\n\n**[W5, W6].** Please refer to the global rebuttal.\n\n**[W7]: Translation Representation.** We project the point cloud into a range image and each pixel on the image is represented as a ray, where the translation is the ray origin in the world coordinate system.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LmSGkf8DGV", "reviewer": "Reviewer_TE2m", "summary": "This paper presents GeoNLF, a new approach to pose-free large-scale point-cloud registration (PCR) and LiDAR novel view synthesis (NVS). To address the limitation of both ICP-based and NeRF-based methods, the authors proposed a hybrid approach involving both robust ICP and global optimisation of NeRF. Experiments show the proposed method achieved better pose estimation and NVS results compared to previous SOTA methods.", "questions": "Please see the Weakness.", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow.\n2. The combination of geometric-based ICP and direct NeRF optimisation is novel and the explanation is very clear.\n3. Fig. 3 and Fig. 4 are very informative in showing how the proposed geometric optimisation and selective re-weighting helps the optimisation process.\n4. The experimental results are promising.", "weaknesses": "1. The proposed approach consists of so many building blocks which makes it a complicated system to work. Besides, from Tab. 3, it seems that except the geometric optimizer which contributes a lot to the final result, all other components seem to only produce very small improvement. \n\n2. How long does the optimisation take? It would be good to show some run-time comparison and analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your positive feedback on our paper's presentation and the novelty of our method. Below are the responses to your concerns.\n\n**[Q1]: The issue of the algorithm's complexity and the role of the components(SR and geometric constraints).**\n\n**1. Complexity of Our Algorithm.** Our method combines LiDAR-NeRF with a Geo-optimizer, which constitutes the two main modules of our model. The Selective-Reweighting (SR) strategy (Sec 3.4) and the 3D geometric constraints (Sec 3.5) do not actually complicate the model structure and almost do not increase the optimization time (please kindly refer to global rebuttal for more details). \nSpecifically, the SR strategy adjusts the learning rate for different frames during optimization to avoid overfitting frames with high loss during training. The geometric constraints introduce a geometry-based loss function to ensure that the synthetic point cloud possesses more accurate geometric features. \n\n**2. The Role of the Components.** We appreciate your inquiry regarding the roles of these two strategies(SR and geometric constraints). Given that the Geo-optimizer is crucial for the success of registration, it significantly enhances performance. Without this core module, the remaining two optimization strategies struggle to exert their influence effectively. However, these strategies play pivotal roles in specific aspects and metrics. We will now provide detailed explanations:\n\n**2.1. SR Strategy.** SR strategy aims to prevent overfitting of NeRF to individual frames and plays a crucial role in specific sequences. It primarily addresses extreme cases where most frame poses are optimized well, except for a few overfitted outliers. However, in most sequences, the significant improvement in registration success with NeRF+Geo-optimizer reduces extreme cases. Another factor to consider is that our ablation experiments are averaged over five Nuscenes sequences, which may average out its effects on certain frames. Considering its effectiveness becomes particularly evident in specific sequences, especially when outlier frames appear during the optimization process, we conducted an ablation study on the specific sequence (KITTI-360, Seq-3) to demonstrate its effectiveness, as shown in Fig. (2)(a) in the attached PDF and table below. \n\n| | CD$\\downarrow$     | $\\mathrm{PSNR_{depth}}\\uparrow$ | $\\mathrm{PSNR_{intensity}}\\uparrow$ | $\\mathrm{RPE_t}\\downarrow$(cm) | $\\mathrm{RPE_r}\\downarrow$(deg) | $\\mathrm{ATE}\\downarrow$(m) |\n|-|-|-|-|-|-|-|\n| w/ SR  | 0.0510 | 28.5674|18.0307| **7.333** | **0.118** | **0.277** |\n| w/o SR | 0.0607 | 27.7158|17.2744| **69.570** | **0.667** | **1.421** |\n\nAs shown in the table, in the absence of the SR strategy, the pose metrics exhibit a sharp decline due to the presence of outlier frames. Additionally, the negative impact of the outlier frame on NeRF optimization significantly contributes to a comprehensive degradation in NVS performance.\n\n**In summary, the SR strategy can enhance the overall performance of the model by preventing NeRF from overfitting high-loss frames in the early stages and enables our algorithm to be more robust across various sequences. Importantly, this strategy introduces no additional computational burden in optimization time and does not increase system complexity, thereby demonstrating practical utility. Furthermore, we believe that this simple yet effective strategy will also be beneficial for other pose-free NeRF works.**\n\n**2.2. 3D Geometric Constraints.** Regarding 3D geometric constraints, we acknowledge their marginal improvement in pose optimization accuracy and the 2D evaluation metrics of NVS. Their impact primarily manifests in optimizing the 3D geometric features of the synthesized point cloud, offering specific practical significance. Primarily, our geometric constraints supervise the 3D aspects of the synthesized point cloud through a dedicated loss function, aiming for enhanced geometric fidelity. Therefore, the improvements from geometric constraints are primarily reflected in 3D point cloud error metrics, **notably reducing CD (a 17.8\\% reduction).** This indicates that geometric constraints help align the distribution of our model-synthesized point clouds closer to the ground truth point clouds in 3D space.\n\nAdditionally, since our geometric constraints primarily affect the 3D domain, their impact on the evaluation metrics focusing on 2D range maps in NVS is not significantly pronounced. Hence, we illustrate the effectiveness of our geometric constraints from a 3D perspective. \n\nAs shown in Fig. (2)(b) in the attached PDF, since previous LiDARNeRF[52] is optimized based on 2D range images, it effectively performs optimization from the front view perspective. Using 2D-based loss without our geometric constraints can still yield satisfactory results from the front view (compare Fig. (2) b.1 and b.2). **However, in the bird's-eye view, the reconstruction quality significantly deteriorates without our geometric constraints(compare Fig. (2) b.3 and b.4). This is because our geometric constraints provide supervision from a 3D perspective, addressing the limitations of 2D perspective supervision. As a result, our synthesized point cloud exhibits more realistic geometric features, such as smoother and more realistic surfaces.**\n\n**[Q2]: Run-time Comparison and Analysis.** Please refer to global rebuttal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all reviewers for their constructive comments on our work. We identified several comments that are common across more than one reviewer. Thus, we highlight them here.

**[Q1]: The issue of variable reuse in Eq.(13).** Sorry for the confusion. Since we introduce geometry-based losses as a constraint on the original Lidar-NeRF 2D-based loss, we focused primarily on the geometric component. This led to an oversight in the 2D-based loss (Eq.(13)), where we neglected to reintroduce previously defined variables. 

Based on your suggestions, throughout the text, we will consistently use $\mathcal{S}$ to represent intensity and $\mathcal{R}$ to represent ray-drop probabilities. Therefore, Eq.(13) can be rewritten as:

$\mathcal{L}_{r}(\mathbf{r})=
  \sum _{\mathbf{r} \in \mathbf{R}} (\lambda_d \begin{Vmatrix} \hat{D}(\mathbf{r}) - D(\mathbf{r}) \end{Vmatrix}_1 +
  \lambda_i \begin{Vmatrix} \mathcal{\hat{S}}(\mathbf{r}) - \mathcal{S}(\mathbf{r}) \end{Vmatrix}_2^2 + 
  \lambda_p \begin{Vmatrix} \mathcal{\hat{R}}(\mathbf{r}) - \mathcal{R}(\mathbf{r}) \end{Vmatrix}_2^2)$

Additionally, there is also a minor error of variable reuse in Eq.(1), we will use $K$ to represent the pose in Eq.(1). In this way, our variables are now consistent.

**[Q2]: The issue of calculating the normal in Sec 3.5, L222-L223.** We deeply regret not having provided a clearer explanation. As shown in Eq. (12), $\hat{P}$ represents the synthetic point cloud derived from the implicit density field, while $P$ denotes the ground truth point cloud, as mentioned in lines 220-221. Thus, the normal loss is calculated between the synthetic point cloud $\hat{P}$ and the ground truth point cloud $P$ to ensure more accurate normal vectors of the point cloud synthesized from NeRF. We apologize for the incorrect citation of Eq.(9), the definitions of $\hat{P}$ and $P$ should correspond to those in Eq.(11). 

Besides, the calculation of normal vectors involves searching for neighboring points around a point and using Principal Component Analysis to perform plane fitting and determine the normal vector. This process is implemented in many point cloud processing libraries, such as PCL and PyTorch 3D.

**[Q3]: The issue of run-time comparison and analysis.**
1. Run-time comparison. We provide the comparison in the table below.
||| GeoNLF | HASH-LN | BARF-LN | GeoTrans-assisted |
|-|-|-|-|-|-|
|Opt.| sec/epoch |4.06|3.46|10.69|3.45|
|Opt.| Total|~2.0h|~1.7h|~5.0h|~1.2h|
|Infer|sec/frame|0.24|0.24|0.67|0.23|

It is important to note that the other two pose-free baselines (HASH-LN and BARF-LN) completely fail on many sequences, unable to obtain accurate poses and usable reconstruction results. Regarding the GeoTrans-assisted LiDAR-NeRF, despite its short processing time(It uses the same NeRF as HASH-LN, but converges faster because it does not require pose updates), it requires a pre-trained model, which takes days to train and is limited by the training dataset, and still fails on some sequences. Although GeoNLF typically requires a longer time, our method achieves a better balance between accuracy and efficiency. GeoNLF demonstrates greater robustness and has practical value, successfully aligning and reconstructing all sequences.

2. Run-time analysis.

(1) Selective-Reweighting(SR) strategy: Our SR strategy does not introduce any additional time burden. It simply involves selecting the top-k frames based on the training loss from the previous epoch and adjusting their learning rates according to Eq. (11). 

(2) Geometric constraints: Normal-based loss and CD are computed between the nearest points of two given point clouds. Since Normal uses the correspondences from CD, we only perform the nearest neighbor search once. Consequently, this approach is time-efficient with 0.026s/frame and 0.8s/epoch. We optimize every 5 epochs, taking only 5 minutes (within 4.2\% of the total time).

(3) Geo-optimizer: We construct a graph where each frame is connected to the two preceding and two following frames. Given that the computation of CD is symmetric, meaning the CD between point cloud A and point cloud B is the same as that between B and A, each graph consisting of 32 frames will have 61 edges for CD computation, with each frame connected to its four neighboring frames. To leverage parallel computation, we downsample each point cloud to the same number of points $N=24000$. We then transform the CD computation into a calculation between two tensors of size $61 \times N \times 3$ based on the graph, allowing us to obtain the entire result in parallel. Consequently, the Geo-optimizer takes only 0.23s/iteration and 16 minutes in total(within 14\% of the total time).

Meta-review: The paper proposes a method for novel view synthesis from Lidar point clouds. It initially received highly discordant ratings (R, WA, A). The rebuttal and subsequent discussion appeared to have addressed the main technical concerns from the more negative reviewer, who decided to lean on a BA. The other two reviewers confirmed their initial ratings. 
Based on this, the AC has deemed that a consensus has been found towards acceptance. The AC warmly recommends the authors to incorporate the comments from the reviewers in the camera ready, including those around the clarity/presentation of the method.
Decision: Accept (poster)