Multimodal LiDAR-Camera Novel View Synthesis with Unified  Pose-free  Neural  Fields
Authors: Weiyi Xue
Fan Lu
Yunwei Zhu
Zehan Zheng
Sanqing Qu
Jiangtong Li
Ya Wu
Haiyun Wei
Guang Chen
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal
Pose-free
Novel View Synthesis
Primary area: applications

Abstract: Pose-free Neural Radiance Field (NeRF) aims at novel view synthesis (NVS) without relying on accurate poses, exhibiting significant practical value. Image and LiDAR point cloud are two pivotal modalities in autonomous driving scenarios. While demonstrating impressive performance, single-modality pose-free NeRFs often suffer from local optima due to the limited geometric information provided by dense image textures or the sparse, textureless nature of point clouds. Although prior methods have explored the complementary strengths of both modalities, they have only leveraged inherently sparse point clouds for discrete, non-pixel-wise depth supervision, and are limited to NVS of images. As a result, a Multimodal Unified Pose-free framework remains notably absent. In light of this, we propose MUP, a pose-free framework for LiDAR-Camera joint NVS in large-scale scenes. This unified framework enables continuous depth supervision for image reconstruction using LiDAR-Fields rather than discrete point clouds. By leveraging multimodal inputs, pose optimization receives gradients from the rendering loss of point cloud geometry and image texture, thereby alleviating the issue of local optima commonly encountered in single-modality pose-free tasks. Moreover, to further guide pose optimization of NeRF, we propose a multimodal geometric optimizer that leverages geometric relations from point clouds and photometric regularization from adjacent image frames. Besides, to alleviate the domain gap between modalities, we propose a multimodal-specific coarse-to-fine training approach for unified, compact reconstruction. Extensive experiments on KITTI-360 and NuScenes datasets demonstrate MUP's superiority in accomplishing geometry-aware, modality-consistent, and pose-free 3D reconstruction.

Multimodal LiDAR-Camera Novel View Synthesis with Unified Pose-free Neural Fields
Weiyi Xue Fan Lu Yunwei Zhu', Zehan Zheng', Haiyun Wei Sanqing Qu', Jiangtong Li', Ya Wu", Guang Chen
[xwy, lufan, 2432040, zhengzehan, 231 1399, sanqingqu, jiangtongli, guangchen @tongji.edu.cn, qpo mo0144 444 @ 163.com,
Tongji University, Shanghai Innovation Institute,
3 CNNC Equipment Technology Research (Shanghai) Co., Ltd. Corresponding author
Abstract
Pose-free Neural Radiance Field (NeRF) aims at novel view synthesis (NVS) with- out relying on accurate poses, exhibiting significant practical value. Image and LiDAR point cloud are two pivotal modalities in autonomous driving scenarios. While demonstrating impressive performance, single-modality pose-free NeRFs often suffer from local optima due to the limited geometrie information provided by dense image textures or the sparse, textureless nature of point clouds. Al- though prior methods have explored the complementary strengths of both modal- ities, they have only leveraged inherently sparse point clouds for discrete, non- pixel-wise depth supervision, and are limited to NVS of images. As result, Multimodal Unified Pose-free framework remains notably absent. In light of this, propose MUP, pose-free framework for LiDAR AACaeraa joint NVS in large- scale scenes. This unified framework enables continuous depth supervision for image reconstruction using LiDAR-Fields rather than discrete point clouds. By leveraging multimodal inputs, pose optimization receives gradients from the ren- dering loss of point cloud geometry and image texture, thereby alleviating the issue of local optima commonly encountered in single-modality pose-free tasks. Moreover, further guide pose optimization of NeRF, we propose multimodal geometric optimizer that leverages geometric relations from point clouds and pho- tometric regularization from adjacent image frames. Besides, to alleviate the do- main gap between modalities, we propose a multimodal-spedi training approach for unified, compact reconstruction. Extensive experiments on KITTI-360 and NuScenes datasets demonstrate MUP's superiority in accomplish- ing geometry-aware
coarse-to-fine
and pose-free 3D reconstruction.
Introduction
Neural Radiance Fields (NeRFs) | have made substantial strides in novel view synthesis (NVS) for images and LiDAR point clouds [36, 53 53 [2], with promising applications au- tonomous driving scenarios ES
39, 45. 46]. Recent developments have transitioned towards
pose-free paradigm [IG
C), facilitating reconstruction while accurately estimating sensor time-consuming cruccturefrrm-mmotioo algorithms like COLMAP [52] and on unreliable point cloud registration methods such as ICP
III
PA..
poses. This approach reduces dependence
[D]
BT. 2x]
both of which are susceptible failure in wide-baseline scenarios.
However, existing pose-free NeRFs have largely concentrated on single modalities, particularly on images. Nevertheless, due to the lack of geometric consistency, relying solely rich texture with-
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

CT POsE RUG
OU NUS 4wu.ks
Fore--Aee Registidas KMVG)
Figure 1: NVS results w/ and w/o accurate poses. Compared to continuous LiDAR- Came-a Fields, projecting LiDAR point clouds onto images as discrete depth priors fails to provide continuous, pixel-wise supervision. Multimodal NeRFs (without MMG) leverage continuous LiDAR-Fiele to constrain geometric consistency and optimize pose, aiding both reconstruction and pose optimiza- tion.
out geometry often leads to suboptimal results in large-scale scenes [19]] The assistance of Pseudo or real point clouds [3] can help address this issue. For instance, directly propagating depth loss to optimize geometric consistency [97]] and leveraging depth for correspondence establishment and reprojection refinement [[88, Gi] can both contribute to pose optimization. Nonetheless, when per- forming pose-free reconstruction and projecting discrete point clouds onto images for depth super- vision, only sparse set of pixels contains depth information, underutilizing point cloud geometry. As shown in Fig. M. LiDAR point clouds fail to provide pixel-wise supervision due to their inher- ent sparsity and discrete sampling. Comparing Fig. M reveals that supervision with discrete point clouds limits registration accuracy(c) and reconstruction quality(a). whereas continuous LiDAR- Field supervision excels in both. Moreover, Nope-NeRF [[]] employs pixel-wise depth estimation to generate pseudo point clouds but suffers from scale ambiguity and limited accuracy. Consequently, even with point cloud [E], [S], image-based pose-free NeRFs [T6.. 19, K remain challenging apply in large-scale scenes and are unable to perform point cloud NVS. Conversely, instead of utilizing point clouds as additional discrete supervision, recent advancements [66] have demonstrated that leveraging point clouds alone enables the reconstruction of continuous Neural LiDAR Fields. The LiDAR-based framework [|5, H6. [2] facilitates highly accurate and continuous geometric recon- structions. Among these efforts, GeoNLF [ extends LDDRR-Ficl to pose-free reconstruction task. Nonetheless, the inherent lack of texture information in point clouds, along with the sparse sampling and indistinct foreground-bacckgroum boundaries in the range map, continues to constrain the performance.
Regarding the challenges encountered in the aforementioned single-modality approach, Continuous Neural LiDAR Fields can provide pixel-wise depth supervision for images and directly propagate gradients to pose estimation, providing continuous geometric constraints. In turn, images offer rich textures and clear boundaries, which enhance the registration accuracy of sparse point clouds. Con- sequently, reconstructing both point clouds and images as continuous neural fields allows them to effectively complement each other in pose-free scenarios. Nevertheless, prior research [[Z]] has faced challenges due to the significant domain gap and uncoordinated convergence problems [27, 22, [2] between these modalities. Therefore, Alignmif [37]] employs independent hash- -riid for each modal- ity. However, in pose-free, ill-conditioned optimization, jointly optimizing the two distinct hash grids and poses is infeasible and yields suboptimal performance compared to the eoggltyyy model. The large discrepancy between two feature spaces leads to inconsistent gradients when prop- agated to poses, causing
[7]]] to fail to converge.
Consequently, in pursuit of effectively integrating the two modalities for unified pose-free recon- struction, we introduce MUPa framework that facilitates the simultaneous reconstruction of both point clouds and images via a unified neural field. Specifically, to mitigate local minima issues in single-modality approaches, we propagate the gradient of multi-modal rendering loss poses with varying emphasis at different optimization stages. Additionally, the MultiModal Geometric opti- mizer (MMG) guides pose optimization by leveraging geometric relations between multiview point clouds and incorporating point-to tont-to-ine to-image error as regularization term. To alleviate modality con- flicts [اР] and address the uncoordinated convergence problem, we introduce a multimodal-specifn coarse-to- to-se-tto-fine training approach [166], facilitating the utilization of singular hash grid for compact reconstruction. Moreover, to enhance color-depth consistency, we introduce a consistency constraint by projecting image pixels onto adjacent frames using depth derived from NeRF, Therefore, MUP

---

is capable of achieving geometry- aware, modality-consistenn and pose-free reconstruction in large-
scale scenarios.
We evaluate our method across diverse scenarios using the KITTI-360 [[.5] and NuScenes [ au- tonomous driving datasets. Comprehensive experiments demonstrate that MUP significantly outper- forms prior state-of-the-arr techniques and single-modality approaches by large margin in both registration and NVS.
In summary, our primary contributions can be delineated as follows: (1) pose-free framework that combines the advantages of two modalities for pose estimation and mul- timodal NVS in large-scale scenes, efficiently leveraging a compact neural representation without the need for accurate poses. (2) We introduce multimodal-speciine
We propose MUP, a unified
training approach, integrated with the MMG module and consistency constraint, to facilitate modality-consistery pose-free, and geometry-aware reconstruction. (3) We demonstrate the effectiveness of our method quantitatively and qualitatively through extensive experiments conducted on multiple datasets and scenes.
2 Related Work
NeRF for Single-Modality NVS. NeRF [20] and related works have achieved substantial progress in novel view synthesis. Diverse neural representations [21. MI, HJ KL, [[], techniques [22, 23, I.. [52]. and generalization methods [+]] for NeRF have been introduced to enhance its performance. Some methods incorporate depth priors [ BTC 5]]] point clouds as auxiliary data to ensure multi-view geometric consistency. However, relying solely on sparse depth supervision from point clouds fails to fully exploit their potential in expressing geometry. Consequently, researchers have extended the NeRFs to generate novel views from LiDAR [6G 12 5.S. 53% 2ZI, treating point clouds as range images. Nevertheless, sparse point clouds notably deficient in dense texture information. Ac- cordingly, we aim to leverage the complementary characteristics of both modalities, advancing a unified multimodal NeRF framework.
Multimodal Joint Learning in NeRF. NeRF framework facilitates the integration of wide range of attributes into the volumetric rendering pipeline, including color [21]], depth, intensity, and seman- tic labels [5 Recent advancements [G. EL I4. MI, ]0] exploit point clouds to provide depth priors but fail to offer pixel-wise supervision. Neural sensor simulator Unisim
[39] performs multimodal NVS via implicit fusion. However, all these methods rely on accurate poses. Very recently, Align- mif [GV] has proposed using distinct hash grids for separate modality reconstruction followed by fusion. However, its intricate structure with two hash grids fails to converge in pose-free optimiza- tion and is highly computationally expensive. Our approach employs
compact representation and introduces a novel strategy to achieve pose-free, multimodal reconstruction.
NeRF with Pose Optimization. Since iNeRF [50]] and subsequent works [177 X] demonstrated that NeRF can optimize the poses of new viewpoint images based on trained radiance fields, series of approaches have aimed to reduce NeRF's reliance highly accurate poses. NeRFmm
[+ and SCNeRF [ extend the method to intrinsic parameter estimation. BARF (IG. II employs a coarse- to-fine reconstruction scheme that gradually learns positional encodings, demonstrating notable effi- cacy. Additionally, several studies have expanded BARF to tackle more challenging scenarios, such sparse input [E8], dynamic scenes [55] [8]. and generalizable NeRF [.]. The coarse-to-fin tone train- ing method has been particularly inspiring for our work. However, these pioneering efforts primarily target indoor object-level scenes. Increasingly, pose-free methods have enhanced robustness by incorporating priors. In particular, [EL 68] use monocular depth or correspondence priors for con- straints. Recently, [ proposed LiDAR-only pose-free framework. Nonetheless, the sparse nature of point clouds, coupled with the absence of texture information, ray-drop characteristics, and inherent noise, imposes limitations on registration accuracy. Moreover, all of the aforemen- tioned methods are designed for a single modality. In the context of autonomous driving, they fail to fully exploit both the geometric information from point clouds and the texture information from images. As result, unified multimodal, pose-free framework remains absent.
3 Preliminaries
Pose-free NeRF for Images and Point Clouds. NeRF represents 3D scene implicitly by encoding the density along with additional data features such as color and intensity of the scene using an

---

World
MMG- ONGG-Optimizer
Coordinate
System
Frame
Fain Photomebric Irage
Frames,
P1
Pases
Frame,
Frame
2
POSE
Lanph
Distunce
paintiimags
Paso
Ground Truth
Nih
Gradient
Reconstruction Loss
Intendlyy
Paint Clead
Imaage
Depth
Prediction
Kay-drvo
Consistency constraints
MILF
Unified Neural LiDerrCamera Fields
Color
Dessit
Overview of our proposed MUP. MUP derives pose gradients through both im- plicit global optimization from the Unified Neural LiDAR-Camera Fields and our explicit MMG- optimizer, both of which effectively leverage complementary multimodal information. Besides, we integrate Unified Neural HIDAR-Camerr Fields with a multimodal -pecifii coarse-to- training strategy, along with consistency constraint to achieve geometry-awan free reconstruction.
Figure 2:
and pose-
implicit neural function (a, d), where a is the 3D coordinates and d is the view direction. NeRF pipeline compatible with both point clouds and images. For point clouds, converts LiDAR point clouds into a range image, then casts a ray with direction d determined by the azimuth angle 0 and elevation angle á under the polar coordinate system: = (cos 0 cos Φ, sin sin Φ cos a))" When performing NVS, NeRF employs volume rendering techniques to accumulate densities and the pixel depth value D along sampled rays. Using the same approach, NeRF predicts the color C for images or the intensity S and the ray-drop R for point clouds.
Traditional NeRF relies heavily accurate sensor poses and reconstruction accuracy can be sig- nificantly compromised with imprecise poses. Pose-free NeRF introduced solve this issue by treating sensor poses P = {P. = 0, 1...N - 1} optimizable parameters. Hence, the simultane- update via gradient descent of P and θ can be achieved by minimizing Ei | I, between the rendered and ground truth image or range image I:
, p arg min C(I, P Zt. 0.P
(I)
Problem Formulation. In large-scale autonomous driving scenarios, given time-synchronized se- quences of cameras and LiDAR data, denoted as I = {T&|s 0, 1,...,N 1} and Q = (Q.ls- 0, 0, N - 1}. the objective of MUP is to reconstruct the scene continuous implicit represen- tation based on a unified neural field. MUP is capable of performing NVS for both modalities, while also simultaneously recovering the vehicle poses P {P. 0, 1.... N 1}, which enables the global alignment of both images and point clouds. The relative poses of all sensors with respect to the vehicle are assumed to be known.
Pose Representation. Following [3], pose is modeled as a rotation R E SO(3) and translation e R3, This formulation allows for independent updates of the translation at the origin and the rotation around the origin. Rotation updates are computed in the Lie algebra of the special orthogonal group in three dimensions, Φ E so(3), while translations are updated in 3 Specifically, the updates are expressed as Φ' Φ + AΦ and = =t + At. Here, & satisfies R = En il ( where o² denotes the skew symmetriiii matrix representation of Φb
Challenges in Multimodal NeRF, Different modalities exhibit varying representations and con- verge at different rates [27] 92 [4] Specifically, point clouds often converge faster than images due to their sparsity and direct geometric supervision. This disparity causes uncoordinated convergence, with the model shifting focus to images once the point cloud error is sufficiently small.

---

Detach guddents of
Figure 3: Modality fusion in Bash-grids and geo-MLP. We truncate the gradients of each modal- ity separately in hash grids and geo-MLP. The results show that feature fusion across modalities primarily oceurs the hash grids rather than the geo-MLP.
4 Methodology
As shown in Fig. M. our framework is divided into two main modules: Unified neural fields that implicitly refines neural network and poses, and the Multimodal Geometric Optimizer (MMG) that explicitly optimizes poses. Both modules leverage geometric and texture information for registration and are exccuted alternately. In the following sections, we first present our LIDAR-Camee Fields in Section I. integrating unified neural fields with a -uleemodal-ppeciffic coarse-to-lnne training strategy for reconstruction and global implicit pose optimization. Then, we introduce our MMG module in Section 2 which provides explicit geometric guidance to avoid local optima. Finally, we present the proposed consistency constraint and the overall optimization pipeline in Section
4.1 Unified Neural LIDAR-Camert Fields
To address the issue of unbalanced convergence speeds mentioned in Section B. we design a Unified Neural LiDAR-Camer representation. Firstly, we introduce the Unified Neural Fields and identify that modality fusion occurs within the hash grids. Based on this observation, we propose a mul- timodal training method for optimizing the hash grid, which also stabilizes pose optimization and mitigates modality conflicts. Finally, a comprehensive analysis and discussion are provided.
Unified Neural Fields. Initially, we adopt i-NGP [2I]] as the base framework, leveraging multi- resolution hash grids to encapsulate the features, while a geometry-MLP (geo-MLP) is utilized to derive the density. In MUP, both the hash grids and the geo-MI.P are shared across the modalities. For the image modality, we use a lightweight MLP to refine the geo-ML.P output, helping reduce modality conflicts. To explore how modality features are fused, we independently truncate the gra- dients of reconstruction loss LCamera and LLIDAR to hash grids and geo-MLP. For hash grids, results indicate that truncating one modality prevents the Multimodal NeRF from learning the correspond- ing features. As shown in the upper images of Fig. E. when the image gradient is truncated, novel views lose texture and resemble colored point cloud projection, whereas truncating the point cloud gradient results in inaccurate geometry, resembling amage-based pseudo point clouds. The same experiment on geo-MLP reveals slight performance degradation, suggesting feature fusion primarily occurs in the hash grids. Thus, effectively controlling hash grid learning across modalities is crucial.
Multinodal-specficc Coarse-to-fine Training. To this end, we draw inspiration from the coarse-to- fine (C2F) strategy, which is widely used in pose-free NeRFs [87, I6. [.J. We extend this approach to a multimodal pose-free NeRF by adopting modality-speciffc C2F strategies, which helps to bal- ance the influence of each modality on the hash grid. Specifically, we progressively activate shared hash grids from low to high resolution, employing distinct activation speeds and initiation points for each modality, as described in Eq. (2))
(2) where Y'1 denotes the encoding of the L-th layer hash-grid, wt. is the coarse-to-fine mask, and can be any monotonic increasing function with a domain of [L - 1, L] and range of [0, 1), such as (z) = cl L + 1, 1) or (a) (1 cos (clip(a - L + 1,0, 1)m))/2 in most pose- free methods []G. 47 maj. a [0, L) is a controllable parameter proportional to the optimization progress. Notably, a varies across modalities. For images, all low-resolution hash grids are ini- tially activated, with higher-resolution grids progressively activated. For the point cloud, a similar coarse-to-fi approach is used but with slower activation starting from low-resolution grids. In our experiment, the OLiDAR is adjusted between 6-16, while the OCammea is adjusted between 12-16.
LiDAR ? L
wL (OLIDAR ) 1 ,7
LIDAR (Camera
wz (ACCmmra))) 1TL
Camera
5

---

Dense Point
Cloud
Reproject
i-nGs-Consisteny
Phocatto
Deychgrod
Other vicepaiint
I-NGP
aicegasind
Figure 4: Consistency constraint. We project rendered images onto other frames by depth obtained from NeRF to compute the photometric error. It's particularly effective for textureless regions.
Implicit Pose Optimization. In the Unified NeRF training, gradients also propagated to pose from reconstruction loss. However, in early optimization, geometric inaccuracies hinder texture- based pose refinement. In later stages, sparse point clouds limit registration accuracy, while images offer denser cues for alignment. Consequently, we adopt point cloud-based loss at the early stages and later employ image-based photometric loss to refine the poses. This is implemented by adjusting the learning rates of pose parameters across different modalities, depicted Eq. (E))
Pa+1 P. (1 w)IrGLAARR
1rGCamera
where G is the gradient of the corresponding modality, P,, denotes the pose the n-th iteration, is control variable increasing progressively during the training process, 1r denotes the learning rate.
Discussion. The C2F strategy is widely used in pose-free NeRFs [87) I6, ITO]. During ill-conditioned
optimization, minor perturbations pose can lead sigmificant deviations NeRF, potentially driv- ing it towards a local minimum. The C2F strategy alleviates this issue by blocking partial gradient propagation, thereby mitigating the impact of such perturbations. The Jacobian of 7 thus becomes:
@0 1.00xxo00 a)) (a) (0, x) (0, wL (a) (8,
(4)
00
00
Ox
8x
where 9 denotes the parameters of the hash grids, and the point x is associated with the pose. When (a) = O, the contribution to the gradient from the L-th (and higher) resolution component is nullified. As shown in Eq. ())) the optimization of both hash-grid and pose follows coarse-to-fine strategy. In the early stages of optimization, only the gradients from the coarse resolution of the hash grids contribute to pose optimization, while the finer resolutions further refine the pose.
Furthermore, our method utilizes a single shared hash grid and unified hash features across modali- ties. We adjusts the for each modality, ensuring consistent convergence speeds and balanced loss across modalities. Moreover, the varying a values guide high-resolution hash grids to capture fine image textures, while the LiDAR field refines geometry and primarily supervises oww-frqqueccy ge- ometric structures, mitigating cross-modal conflicts. In summary, our method ensures synchronous convergence and stable optimization, while also mitigating modality conflicts, by employing dis- tinct, compact, and efficient hash grid structure.
4.2 Multimodal Geometric Optimizer
Our MMG module leverages Image-enhanced Chamfer Distance combined with point-to-image reg- ularization. Unlike implicit optimization through NeRF, MMG explicitly optimizes poses by lever- aging both geometry and textures.
Explicit Pose Optimization. The closest-poing correspondences between two partially overlapping point clouds establish the most direct geometric relationship. which can be leveraged effectively for registration like ICP [D]. Chamfer Distance (CD) well-established loss derived from point correspondences and can be computed as Eq. (E);
CD
P.Q) CD(P.Q)
Σ D.EP
w; min Tpp: q.EQ
TQq
Σ QEeQ
w; min || TQq: TpP:
P.EP
(5)
where g.p in point cloud Q P are homogeneous coordinates. Tp, Tq represent the transforma- tion matrix to the world coordinate system. Additionally, we define uy to represent the weight of

---

Table 1: Quantitative comparison of NVS in pose-free setting. We conduct experiments under the pose-free setup. The estimated trajectory is aligned with the ground truth using Sim(3) for image- based methods. P.F: Pose-Free, RR: Reconstruction after Registration, I: Image-synthesizabll,
PZ: Image and Point cloud-synthesizable
Type
LIDAR Metries
Image Metrics
Posc Metrics
Methods
CDL
F-goore
MAETJ
PSNRT
SSIMT LPIPS).
RPE, (em)))
(Peg ATb
Experiments KITTI-360 []S
Colored-ICP
0.787
0.149
20.92
25.383
0.899
1.624
Nope NeRF
15
PF1I
19.82
0.337
0.592
83.223
14.412
0.653
BA-Aliammf
[0.
PFIPI
0.722
0.116
19.12
0,641
0.439
36.179
0.498
0.410
MUP(Ours)
PFIPI
0.079
0.942
0.096
23.46
0.759
0.287
1.471
0.025
0.187
Experiments NuSoenes
Colored-ICP M LX
RR. PI
0.930
0.599
0.047
19.21
0.438
0.644
14.380
0.599
1.170
Nope.NeRI NeRF
P..II
18.01
0.341
0.670
129.899
12.399
0.718
BA-Aligmmif
BA [0]
PF/PI
1.695
0.603
0.044
18.73
0.621
0.619
182.391
0.377
4.266
MUP(Ours)
PF/PI
0810
0.656
0.042
20.83
0.699
0.585
4.058
0.101
0.176
Nooe- TUNGS
CPKPP
BA Alginil
Figure 5: Qualitative comparison NVS. We compared MUP with pose-free and registration first methods. Nope-NeRF and Colored-ICP-assitee fail due to the large-scale scene. BA-Alignmif struggles to converge. All baselines fail entirely on certain sequences.
each correspondence. Our MMG module directly computes the inter-frame CD and propagates the gradient to the poses, guiding the optimization of the poses during the ill-conditioned optimization. However, CD overlooks the partial overlap of point clouds, merely minimizing CD does not neces- sarily improve pose accuracy. Leveraging the multimodal input, we exploit images alleviate the impact of non-overlapping regions. Specifically, we project point clouds onto time-synchronized images derive color information. During the calculation of CD, we re-weight the relevant corre- spondences by incorporating photometric information.
W% {
-C(F() S -C(F()
(6)
m,
where the C() denotes the color obtained by sampling from the image. F(x:) denotes the projection function that maps 3D point x onto a 2D image. Additionally, following [47]] we further construct
multi-frame graph (S, E), where vertex S represents frame of point cloud along with the time-
synchronized image, each edge & represents the CD. Finally, the Image-enhanced CD is computed as LICD E(ines CD.))))
Point-to- Point-to-Imagg Regularization. As shown in Fig. 0 (top right), we project point clouds onto im- ages from adjacent frames to establish pixel correspondences. Based on these correspondences, we introduce a ceint-to-image error using photometric loss, which serves regularization term.
CeinntImage (ICAF(R) - C, P.)
(7)
where P denotes the point cloud. Finally, the overall optimization loss for MMG is defined as:
LMMO LIRCD + CroinzZmagee
(8)
7

---

4.3 Overall Optimization Pipeline
Consistency Constraint. Due to the differing Fields of View (FoV) between LiDAR and cameras, plain LiDAR-Fields provide incomplete and limited depth supervision. Therefore, we further in- troduce a geometrie consistency constraint, which leverages reprojection error to constrain regions outside the LiDAR's FoV. Specifically, we extract dense point clouds from rendered depth maps and project them onto adjacent frame images to compute photometric errors. It is effective for large textureless regions, enforcing geometric-color consistency, and is calculated as Eq. (X):
Luoms
- - (15)
p))))
where F=, p) denotes the back- pock-projection, mapping a pixel p 3D point using depth 2.
Optimization. To optimize MUP, the total reconstruction loss formulated as a weighted sum of intensity loss Ls. ray-drop loss LR. LiDAR range image loss Lp. photometric loss Ligh from the image and consistency loss Leons:
(D(r) Σ
((r) D(+) (r) Σ [(((()) I(x))1
(10)
FER
FER
(s(r) Σ |S(r) -(()) (()))(2.(((T)
Σ [a(r)) H(F)llz
(11)
rER
FER
L = AoLD + Ao + AyLe + NnLa + deLeass
(12)
5 Experiment
5.1 Experimental Setup
Datasets and Experimental Settings. We conducted experiments on two public autonomous driving datasets: NuScenes [[]] and KITTI-360 [LS]] dataset, each with five representative time- synchronized LiDAR point cloud and image sequences. For the NuScenes dataset, it includes six cameras and a LiDAR sensor, with keyframes that are typically used, which are time-synchronized based on timestamps. Following [+ we selected 33 consecutive frames from keyframes as a single scene. KITTI-360 has three cameras and LiDAR, where each frame's 's point cloud and image are time-aligned. Following [33 55, [.7] we use the standard KITTI-360 dataset, all images and point clouds are sine-synchronized. For both datasets, only the front-facing single camera was utilized. Following [47, [G]), we perturbed poses of car with additive noise corresponding to a standard de- viation of 20 deg in rotation and 3m in translation. The relative poses of all sensors with respect to the vehicle are assumed to be provided.
Metrics. We evaluate our method for pose estimation and NVS. For pose estimation, we follow [+] employing standard odometry metrics: Absolute Trajectory Error (ATE) and Relative Pose Error (RPE), with rotational (RPE,) and translational (RPE,) components. Following [56 [55] for point cloud NVS, we adopt CD assess 3D geometric errors and the F-score with 5 em threshold. We also compute mean absolute error (MAE) for intensity in projected range images. Besides, we follow the approach in [3), 37]), employing PSNR, LPIPS [54]] and SSIM [[]] for image NVS.
Implementation Details. All experiments were conducted on a single NVIDIA GeForce RTX 3090 GPU. 768 points were uniformly sampled along each ray for two modalities. MUP optimization was implemented in PyTorch [2h]
the Adam optimizer [ The learning rates were set as follows: 1 10-2 decaying to 1 10-1 for NeRF; x 10-3 decaying to 10-3 á for translation; and 5 x 10-3 3 decaying to 5 x 10 10-5 for rotation. The weighting coefficients for each loss term are
defined as: Aa X 1000. X, = 10, An 2.5, X, = 150. Besides, all sequences are trained for 60K iterations in the pose-free setting and 30K iterations when ground truth poses are available. Additionally, after every mi epoch of Unified NeRF training, we proceed with m2 epochs of pure geometric optimization, where the ratio m2/m/ decreases from 10 to 1. Both values are tuned so that the maaree-to-fine strategy applied during the training progress between 0 and 0.3. In the training process, w increases from () to 1.
8

---

Methods/Pose- free)
LIDAR CD}
F-score7
PSNRR RPPE
Image
Pose Metres RPE, ATEL
Methods
LIDAR Metrics
Image Metries
F-score MAE
PSNR
SSIM
LPIPS
w/o MMG
10.592
0.731
19.27
26.201
0.433
0.805
w/o P22R
0.083
0.936
23.35
1,533
0.041
0.205
Experiments KITTI (CS), ENOP. iNNP
puint
w/o Imuge
0.089
0.937
1.668
0.062
0.224
w/o MSC2F
MUP(Ouss)
0.113
0.079
0.932
0.942
22.06
23.46
1.411
1.542
0.058
0.025
0.256
0.187
ENGP
23.12
0.791
0.223
L-NeRE
0.083
0.942
0.097
Methods/CT posc)
CD1 Foscaret MAE, PSNR SSIM" LPIPSL
LIDAR Metrics
Image Metrics
MUP(OOus) 00080
FNGP
0.945
0.089
21.29
23.23
0.794
0.812
0.211
0.220
wio MSC2T
0.931
0.102
23 33,66
0.227 0.812
Experiments on NuScenes
L-NGP. I-NGP w/ point cloud.
MUP(OUu))
00,000
24.29
ENGP F4T
20.78
0.667
0.530
Table 2: Ablation studies on the MMG mod-
L-NeRF
0.815
0.673
0.041
ule and image modality under the pose-free setting(top). MMG module plays pivotal role
EGGP
20,92
0.682
0.564
MUP(Ounn)
0.798
0.678
0.038
21.53
0.545
in pose optimization. Ablations of MSC2F Table 3: Quantitative comparison NVS
and consistency constraint under GT--pose with GT--poses. We conducted experiments un- (otggobooottttttee..i Point21mage Regular- der GT to demonstrate the effectiveness ization. MSC2F: Training Strategy.
of our method in modal fusion.
Singus
Figure 6: Ablation in pose-free setting. The first row illustrates registration results w/ and w/o the MMG, while the second row compares depth maps w/o and w/o the image modality.
Figure 7: Qualitative NVS results with GT- poses. MUP outperforms single-modal meth- ods i-NGP w/ and w/o point clouds and LiDAR- NeRF Our method achieves significantly better depth estimation and NVS quality.
5.2 Comparison and Ablation in Pose-free Setting
Baselines. In pose-free settings, all baselines utilize multimodal inputs. Methods are divided into two distinct categories: one where registration is performed prior to reconstruction, and another where pose-free free reconstruction simultaneously estimates the poses. Following prior pose-free NVS studies [IL 19 47 BL II p reconstruction is typically performed on short sequences without real-time constraints. Consequently, incremental SLAM systems are not directly comparable, and directly frame-to-frame registration is adopted instead [+7]]
For the first category, the registration method Colored-ICP [2E]] integrates both point clouds and images. We utilize it for registration and subsequently apply Aligmmif [37]] for multimodal recon- struction. Notably, For the second category, N Nope-NeRF takes images input and leverages DPT [56] to construct pseudo point clouds. Moreover, we reformulate the Alignmif method a pose-free framework, allowing gradient propagation to the poses for optimization. For all pose-free pose-free methods, we adopt the strategy from | 27] to obtain the poses of test views for rendering.
Comparison in NVS and Pose Estimation. The quantitative and qualitative results are presented in Table D1 and Fig. 1S Our method outperforms all approaches in both modalities. Previous pose- free methods NoRF [[3] primarily targeted small-scale scenarios, and only can perform images NVS. Consequently, when applied to autonomous driving environments, Nope-NeRF underperforms due to the lack of scale and the imprecision in depth estimation. Alignmif [[5Z] cannot be effectively used in ill-conditioned optimization. Its complex structure with multiple independent hash grids prevents efficient registration and reconstruction. As for the registration-fing approach, Colored- ICP [22]] exhibits limited accuracy in large-scale outdoor scenes. Our method achieves the highest pose estimation accuracy.
Ablation Study in frse-free setting. All ablation studies are conducted on KITTI-360 [LS]. We firstly exclude the images and perform reconstruction using only the LiDAR point clouds. Table E underscores the critical role of multimodal fusion in enhaneing accuracy. Besides, as shown in Fig. R the incorporation of images mitigates the limited LiDAR field of view, thereby enabling the

---

acquisition of depth maps (or point clouds) with broader FoV. It also introduces complementary information beyond LiDARs perspective, enhancing registration accuracy. We also conduct ablation studies on MMG module. The results indicate that relying solely on NeRF's implicit pose optimiza- tion fails to achieve accurate pose estimates and leads to convergence local optima. Besides, we conduct ablation studies on image enhancement in MMG and the modality-specife C2F training strategy (MSC2F). All modules demonstrate effectiveness in pose-free experiments.
5.3 Ablation Study with Ground Truth Poses
In the pose-free setting, the precision of the estimated poses and the efficacy of our MMG module are pivotal to performance. Therefore, we exclude the MMG and conduct ablation experiments with GT pose to further demonstrate the advantages of our Unified NeRF with the multimodal-spectnne to-fine training strategy (MSC2F) and the consistency constraint multimodal fusion. Additionally, to further demonstrate the effectiveness of our multimodal approach, We conduct comparative ex- periments with the single-modality LiDAR-NeRT [[36] and i-NGP [2I]], where i-NGP tested both with and without utilizing discrete LiDAR point clouds depth supervision.
coarse-
Ablation of MSC2F and Consistency Loss. Table 2 presents the ablation results on MSC2F and consistency constraint under the GT-Pose setting to verify the effectiveness of our method. By using reprojection operation link geometry and color, our method effectively ensures geometry-color coherence, resulting in improved reconstruction quality in both the image and point cloud NVS.
Comparision with Single-modality Methods. The quantitative and qualitative results presented in Table & and Fig. M Our MSC2F fusion approach, along with the color-depth consistency con- straint, effectively integrates features from both modalities. Thus, compared to Ainle-modality methods and i-NGP [21] that with and without point clouds for depth supervision, we achieve high- quality NVS and the best results across both modalities.
6 Limitation
MUP demonstrates strong performance in pose-free multimodal NVS and pose estimation under challenging large-scatl scale scenes. However, is primarily designed for sensor data within a sequence and relies on temporal correlations between frames. Additionally, it is not designed to handle dy- namic scenes, which is non-negligible limitation in autonomous driving scenarios.
Conclusion
We revisit the limitations of single-modaliy, pose-free methods in large-scale scenes. Subsequently, we propose a novel framework for Multimodal Unified Pose-free LIDAR-Camera NVS. Benefiting from the unified neural representation with MSC2F training strategy, the color-depth consistency constraint, the MMG module, and most importantly, the integration of different modalities and our pose optimization approach, achieve geometry-aware, modality-consiseen pose-free reconstruc- tion.
8 Acknowledgement
This work was supported by the National Key Research and Development Program of China (No. 2024YFE021 1000), in part by the National Natural Science Foundation of China (No. 62372329, 62402341), in part by the Shanghai Scientific Innovation Foundation (No. 23DZ1203400), in part by the China Postdoctoral Science Foundation (No. BX20250383, GZB20250385. 2025M771530 2025M77153 1539, GZC20241225. 2025M771513), in part by Tongii-Qomolo Autonomous Driving Commercial Vehicle Joint Lab Project, and in part by Xiaomi Young Talents Program.
10

---

References
[1] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin- Bauall,, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5855-5864, 2021.
[2] Paul J Besl and Neil D McKay. Method for registration of 3-d shapes. In Sensor fusion IV: control paradigms and data structures, volume 1611, pages 586-606. Spie, 1992.
[3] Wenjing Bian, Zirui Wang, Kejie Li, Jia-Wang Bian, and Victor Adrian Prisacariu. Nope- nerf: Optimising neural radiance field with no pose prior. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4160-4169 2023.
[4] Holger Caesar, Varun Bankiti, Alex Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621-11631, 2020.
[5] Eric R Chan, Connor Z. Lin, Matthew Chan, Koki Nagano, Boxiao Pan, Shalini Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, et Efficient geometry- aware 3d generative adversarial networks. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16123-16133, 2022.
[6] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European Conference Computer Vision, pages 333-350. Springer, 2022.
[7] Yu Chen and Gim Hee Lee. Dbarf: Deep bundle-adjusting generalizable neural radiance fields. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 24-34, 2023.
[8] Junyuan Deng, Qi Wu, Xicyuanli Chen, Songpengcheng Xia, Zhen Sun, Guoqing Liu, Wenx- ian Yu, and Ling Pei. Nerf-loam: Neural implicit representation for large-scale incremental lidar odometry and mapping. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 8218-8227, 2023.
[9] Kangle Deng, Andrew Liu, Jun- Yan Zhu, and Deva Ramanan. Depth-supervised nerf: Fewer views and faster training for free. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12882-12891, 2022.
[10] Hwan Heo, Taekyung Kim, Jiyoung Lee, Jaewon Lee, Soohyun Kim, Hyunwoo J Kim, and Jin- Hwa Kim. Robust camera pose refinement for multi-resolution hash encoding. In International Conference on Machine Learning, pages 13000-13016. PMLR, 2023.
[11] Wenbo Hu, Yuling Wang, Lin Ma, Bangbang Yang, Lin Gao, Xiao Liu, and Yuewen Ma. Tri- miprf: Tri-mip representation for efficient anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19774-19783, 2023.
[12] Shengyu Huang, Zan Gojeic, Zian Wang. Francis Williams, Yoni Kasten, Sanja Fidler, Kon- rad Schindler, and Or Litany. Neural lidar fields for novel view synthesis. arXiv preprint arXiv:2305.01643 2023.
[13] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 141.,694,,,690, 2014.
[14] Yixing Lao, Xiaogang Xu, Xihui Liu, Hengshuang Zhao, et al. Corresnerf: Image correspon- dence priors for neural radiance fields. Advances in Neural Information Processing Systems, 06:40504440522 2023.
[15] Yiyi Liao, Jun Xie, and Andreas Geiger. Kitti-360: A novel dataset and benchmarks for ur- ban scene understanding and 3d. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):3292-3311 2022.

---

[16] Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey. Barf: Bundle-adjuss neural radiance fields. In Proceedings the IEEE/CVE International Conference on Computer Vision, pages 5741-5751, 2021.
[17] Yunzhi Lin, Thomas Müller, Jonathan Tremblay, Bowen Wen, Stephen Tyree, Alex Evans, Patricio A Vela, and Stan Birchfield. Parallel inversion of neural radiance fields for robust pose estimation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9377-9384. IEEE, 2023.
[18] Yu-Lun Liu, Chen Gao, Andreas Meuleman, Hung- Yu Tseng, Ayush Saraf. Changil Kim, Yung- Yu Chuang, Johannes Kopf, and Jia-Bin Huang. Robust dynamic radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
13-23, 2023.
[19] Andreas Meuleman, Yu-Lun Liu, Chen Gao, Jia-Bin Huang, Changil Kim, Min H Kim, and Johannes Kopf. Progressively optimized local radiance fields for robust view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16539-16545 2023.
[20] Ben Mildenhall, Pratul Srinivasan, Matthew Taneik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Commu- nications of the ACM, 65(1):99-100) 106, 2021.
[21] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding, ACM transactions graphics (TOG), 41)) 1-15, 2022.
[22] Michael Niemeyer, Jonathan T Barron, Ben Mildenhall, Mehdi SM Sajjadi, Andreas Geiger, and Noha Radwan. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 5480-5490, 2022.
[23] Michael Oechsle, Songyou Peng, and Andreas Geiger. Unisurf: Unifying neural implicit sur- faces and radiance fields for multi-view reconstruction. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 5589-5599, 2021.
[24] Jaesik Park, Qian- Yi Zhou, and Vladlen Koltun. Colored point cloud registration revisited. In Proceedings of the IEEE international conference on computer vision, pages 143-152, 2017.
[25] Keunhong Park, Philipp Henzler, Ben Mildenhall, Jonathan Barron, and Ricardo Martin- Brualla. Camp: Camera preconditioning for neural radiance fields. ACM Transactions on Graphics (TOG), 42(6))1-11, 2023.
[26] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[27] Xiaokang Peng, Yake Wei, Andong Deng, Dong Wang, and Di Hu. Balanced multimodal learning via on-the-fly gradient modulation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8238 0044-8247440 2022.
[28] François Pomerleau, Francis Colas, Roland Siegwart, and Stéphane Magnenat. Comparing icp variants real-world data sets: Open-source library and experimental protocol. Autonomous robots, 34:133-148, 2013.
[29] Srikumar Ramalingam and Yuichi Taguchi. A theory of minimal 3d point to 3d plane registra- tion and its generalization. International journal of computer vision, 102:73-90, 2013.
[30] Barbara Roessle, Jonathan Barron, Ben Mildenhall, Pratul Srinivasan, and Matthias Nießner. Dense depth priors for neural radiance fields from sparse input views. In Pro- ceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 12892-12901, 2022.
12

---

[31] Szymon Rusinkiewiez and Mare Levoy. Efficient variants of the icp algorithm. In Proceedings third international conference on 3-D digital imaging and modeling, pages 145-152. IEEE, 2001.
[32] Johannes L Schonberger and Jan-Michael Frahm. Structure-fros frottmmr-roommootiion revisited. In Proceed- ings of the IEFE conference on computer vision and pattern recognition, pages 4104-4113, 2016.
[33] Liang Song, Guangming Wang, Jiuming Liu, Zhenyang Fu, Yanzi Miao, et al. Sc-nerf: Self- correcting neural radiance field with sparse views. arXiv preprint arXiv:2309.05008, 2023.
[34] Ya Sun, Sijie Mai, and Haifeng Hu. Learning to balance the learning rates between various modalities via adaptive tracking factor. IEFE Signal Processing Letters, 10111150-6444 2021.
[35] Zhen Tan, Zongtan Zhou, Yangbing Ge. Zi Wang. Xieyuanli Chen, and Dewen Hu. Td-nerf: Novel truncated depth prior for joint camera pose and neural radiance field optimization. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 372-379, IEEE, 2024.
[36] Tang Tao, Longfei Gao, Guangrun Wang, Peng Chen, Dayang Hao, Xiaodan I Liang, Mathieu Salzmann, and Kaicheng Yu. Lidar-nerf: Novel lidar view synthesis via neural radiance fields. arXiv preprint arXiv:2304. 10406, 2023.
[37] Tang Tao, Guangrun Wang, Yixing Lao, Peng Chen, Jie Liu, Liang Lin, Kaicheng Yu, and Xiaodan Liang. Alignmif: Geometry-aligned multimodal implicit field for lidar-camera joint the synthesis. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recog-
nition, pages 21230-21240, 2024.
[38] Prune Truong, Marie-Julie Rakotosaona, Fabian Manhardt, and Federico Tombari. Sparf: Neu- ral radiance fields from sparse and noisy poses. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4190-4200, 2023.
[39] Haithem Turki, Jason Y Zhang, Francesco Ferroni, and Deva Ramanan. Suds: Scalable urban dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12375-12385, 2023.
[40] Chen Wang, Jiadai Sun, Lina Liu, Chenming Wu, Zhelun Shen, Dayan Wu, Yuchao Dai, and Liangjun Zhang. Digging into depth priors for outdoor neural radiance fields. In Proceedings of the 31st ACM International Conference on Multimedia, pages 1221-1230, 2023.
[41] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprini arXiv:2 10689, 2021.
[42] Weiyao Wang, Du Tran, and Matt Feiszli. What makes training multi-modal classification networks hard? In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pages 12695-1270S 2020.
[43] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4): 600-612, 2004.
[44] Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. Nerf-: Neural radiance fields without known camera parameters. arXiv preprint 1:Xi::2002.07064 2021.
[45] Zirui Wu, Tianyu Liu, Liyi Luo, Zhide Zhong, Jianteng Chen, Hongmin Xiao, Chao Hou, Haozhe Lou, Yuantao Chen, Runyi Yang, et al. Mars: An instance-aware, modular and realistic simulator for autonomous driving. CAAI International Conference on Artificial Intelligence, pages 3-15. Springer, 2023.
[46] Ziyang Xie, Junge Zhang, Wenye Li, Feihu Zhang, and Li Zhang. S-nerf: Neural radiance fields for street views. arXiv preprint
2023.
13

---

[47] Weiyi Xue, Zehan Zheng, Fan Lu, Haiyun Wei, Guang Chen, and Changjun Jiang. Geonlf: Geometry guided pose-free free neural lidar fields. arXiv preprint arXiv:2407.05597, 2024.
[48] Jiawei Yang, Boris Ivanovic, Or Litany, Xinshuo Weng, Seung Wook Kim, Boyi Li, Tong Che, Danfei Xu, Sanja Fidler, Marco Pavone, et al. Emernerf: Emergent spatial-temporal scene decomposition via self superviseii... arXiv preprint arXiv:2311.00077, 20233.
[49] Ze Yang, Yun Chen, Jingkang Wang, Sivabalan Manivasagam, Wei-Chiu Ma, Anqi Joyce Yang. and Raquel Urtasun. Unisim: A neural closed-loop sensor simulator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1389-1399, 2023,
[50] Lin Yen-Chen, Pete Florence, Jonathan T Barron, Alberto Rodriguez, Phillip Isola, and Tsung- Yi Lin. inerf: Inverting neural radiance fields for pose estimation. In 2021 IEEE/RSJ Interna- tional Conference on Intelligent Robots and Systems (IROS), pages 1323-1330. IEEE, 2021.
[51] Zehao Yu, Songyou Peng. Michael Niemeyer, Torsten Sattler, and Andreas Geiger. Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. Advances in neural information processing systems, 005-20188220033 2022.
[52] Jian Zhang, Yuanqing Zhang, Huan Fu, Xiaowei Zhou, Bowen Cai, Jinchi Huang, Rongfei Jia, Binqiang Zhao, and Xing Tang. Ray priors through reprojection: Improving neural radiance fields for novel view extrapolation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 18376-18386 18386, 2022.
[53] Junge Zhang, Feihu Zhang, Shaochen Kuang, and Li Zhang. Nerf-lidar: Generating realistic lidar point clouds with neural radiance fields. arXiv preprint arXiv:2304.14811 2023.
[54] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[55] Zehan Zheng, Fan Lu, Weiyi Xue, Guang Chen, and Changjun Jiang. Lidar4d: Dynamic neural fields for novel space-time view lidar synthesis. arXiv preprint wxXiv2244.022444 2024.
[56] Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe. Unsupervised learning of depth and ego-motion from video. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1851-1858, 2017.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
This paper does discuss the limitations of the work performed by the authors.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairnsss.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: This paper fully discloses all the information needed to reproduce the main experimental results of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
This paper provide open access to the data and code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... CC/
ss ooloooooiiisso for more details.
While we encourage the release of code and data, we understand that this might not be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (nttps:
7/nips.. wtnppscc/ppu...... wwtppppccpp.p.....ccccccoooooo.p iie
des CCodeSooouueeeee bm ss Moooolliig icy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: This paper specify all the training and test details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: This paper report appropriate information about the statistical significance of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
This paper provide sufficient information on the computer resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't t make it into the paper).
9. Code Of Ethies
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httt://s/.ssuuu....c.///////aaaa
Question:
Answer: [Yes]
Justification: This research conducted the paper conform, evrery respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: While essential for autonomous driving, reconstruction and closed-loop simu- lation raise privacy and employment concerns, highlighting the need balance technolog- ical progress with societal responsibility.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards
allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We mentioned creators or original owners of assets and properly respected.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, papersw's thcode.cco.... com/ Gataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
19

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Question:
Answer: [Yes]
Justification:
This paper introduces new assets well documented.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects. and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
20

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The paper does not involve research related to LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer our LLM policy http::////eerrrsss.ccc/ooeeeeee (mno) for what should or should not be described.
2025/
21


Reviews:
[{"review_id": "uzeFN0rSiW", "reviewer": "Reviewer_UVxv", "summary": "This paper presents a multi-modal, unified framework for pose-free novel view synthesis tailored to autonomous driving scenarios. Unlike previous approaches that rely on a single modality, either RGB images or LiDAR point clouds. By integrating supervision from both RGB and LiDAR,  the proposed method improves pose optimization, depth estimation accuracy, and the overall quality of synthesized novel views.", "questions": "See weaknesses", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nThe paper introduces a Multimodal Geometric Optimizer that jointly leverages geometry and RGB information to refine camera poses explicitly. The proposed rigid alignment constraints significantly improve pose estimation accuracy, as evidenced by the results in Table 2.\n\nThe method demonstrates superior performance in geometry estimation, novel view synthesis, and pose prediction on two challenging datasets: KITTI-360 and NuScenes.\n\nDetailed ablation studies analyzed the efficacy of each design, like unified modality training, Multimodal-specific Coarse-to-fine Training, and Multimodal Geometric Optimizer.\n\n\n\n\nWeaknesses:\n\nThe technical contributions appear to be incremental rather than fundamentally novel from an academic perspective. (1) i-NGP also unified LiDAR-Camera observations, but it used separate hash grids for two different modalities. This work used a shared hash grid for both modalities. This modification seems to be a minor architectural variation and builds directly upon established techniques. \n(2) The multimodal-specific coarse-to-fine training is more like an effective training trick rather than a fundamentally new algorithmic contribution to modulate the contribution of geometric and photometric losses across scales.\n\nLack of Ground Truth in Figures: Figures 3, 4, 5, 6, 7 do not include ground truth (GT) renderings for reference. This omission makes it difficult to assess the visual accuracy or quality of the proposed method. Including GT in at least some visualizations would significantly improve interoperability.\n\nAmbiguity in Figure 6: The purpose and interpretation of the last row in Figure 6 are unclear.\n\nMissing Video Results: The paper only provides static novel view comparisons with a few selected viewpoints. Without video results of RGB and depth renderings, it's difficult to evaluate the method’s temporal coherence or robustness across continuous novel views. \n\nIncluding video examples (e.g., in supplementary materials) would strengthen the experimental section.\n\nLengthy Related Work in Introduction: The introduction section spends a significant amount of space reviewing related work, which could be streamlined. A more concise overview would help improve focus and readability.", "was_revised": true, "final_justification": "Thanks for the rebuttal! The clarification of technical contribution can be included into the introduction part. I understand that due to the rebuttal policy of this year, authors cannot upload any visuals. But it would improve the paper if the authors can add GT images in the figure comparisons in the final revisiom. Also, it will help readers better understand Fig. 6 if you have sperate visuals of LIDAR and RGB inputs, network outputs, and GT with clear captions.\n\nI went through the questions from other reviewers. The authors also provided additional comparisons and results to address their concerns.", "rebuttal": "Thank you very much for your careful reading and summary. Below, we provide our responses to your questions, and we sincerely look forward to engaging in in-depth discussions with you.\n\n**1. Technical Contributions:** We would like to offer some additional clarifications. Our core motivation is to leverage multimodal information to achieve mutual complementarity, both in reconstruction and registration.\n\n(1) We observe that using separate hash grids for each modality introduces geometric inconsistencies across modalities, as the density values at the same spatial location can diverge. Especially for the camera field, continuous geometric supervision from the LiDAR field becomes difficult due to modality-specific density representations.\nTo mitigate this, we adopt a shared density field to enforce cross-modal geometric consistency. In this setting, the camera field can obtain better geometric information from the LiDAR field (the density curve along the ray is sharper). Conversely, the LiDAR field benefits from the camera field by exhibiting increased continuity (enhanced interpolation capability). \n\n(2) In addition, maintaining independent hash grids increases both the optimization complexity and the risk of geometric misalignment in pose-free scenarios. By contrast, a unified hash grid imposes shared structure across modalities and proves to be more effective. From a technical standpoint, the instability inherent to ill-conditioned optimization limits the complexity of the methods that can be effectively applied. Thus, it is crucial to reduce the optimization burden as much as possible. \n\nConsequently, using a shared hash grid naturally offers an efficient solution to the aforementioned two challenges. However, direct fusion of modalities may introduce conflicts. For instance, we observe that if the LiDAR branch converges too rapidly, the optimization tends to overemphasize fine-grained texture details in the RGB modality, potentially compromising the geometric representation. To mitigate this, we propose the MSC2F strategy — a simple yet effective approach that progressively activates modalities in a coarse-to-fine manner, thereby balancing their contributions during training.\n\n**2.Ground Truth in Figures and Vedio example:** Thank you for pointing out the incompleteness of our presentation. As we aimed to compare different methods within limited space, some visual results were not sufficiently shown.\nIn fact, our reconstruction—especially in point clouds—shows minimal perceptual differences from the GT, with little blur or artifacts in the rendered images. The main discrepancy lies in textureless regions where the GT appears sharper.\n(**Due to rebuttal constraints this year (e.g., no links or images allowed), we will provide more detailed visualizations on the project webpage  that will be released in the future, and include GT images in the revised version to enhance clarity and persuasiveness.**)\n\n**3.Ambiguity in Figure 6:** We apologize for the lack of clarity in our explanation. As shown in the leftmost image of the last row in Fig. 6, the LiDAR field of view (FoV) is significantly smaller than that of the image, resulting in noticeable non-overlapping regions. Combined with the second row of Fig. 6, our intention was to illustrate that: thanks to the continuity of neural fields, **accurate and smooth depth can still be recovered in image-supervised regions beyond the LiDAR FoV.** This effectively extrapolates the LiDAR FoV and demonstrates a potential benefit of the unified framework. The rightmost image in the last row is intended to show the full input view from the current camera perspective. We will clarify the purpose of these images in the revised version.\n\n**4.Lengthy Related Work in Introduction:** Thank you very much for your suggestion. In our initial submission, we aimed to thoroughly present the motivation behind our work, which unfortunately came at the expense of readability and conciseness. In the revised version, we will refine the focus of our exposition and relocate some of the detailed discussions to the Related Work section to improve clarity and overall presentation."}, {"review_id": "GyXdKAjbOZ", "reviewer": "Reviewer_2Vdy", "summary": "The authors propose MUP (Multimodal Unified Pose-free) framework, based on NeRFs for multimodal (LiDAR and RGB images) 3d reconstruction. Key contributions include a multimodal-specific coarse-to-fine (MSC2F) training scheme that balances the very different convergence speeds of images and point clouds. They also propose a multimodal geometric optimizer that uses Chamfer distance with point-to-image photometric regularisation to give reliable gradients for the pose. They demonstrate results on static scene from KITTI-360 and NuScenes which outperform baselines reported in the paper.", "questions": "1. How does MUP behave on sequences with moving vehicles or pedestrians? Could MMG reject dynamic points automatically?\n2. Please answer queries in weaknesses section as well.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. Single hash grid for both modalities helps with better feature fusion and is a more elegant design choice.\n2. Use of Image Weighted Chamfer distance is neat, and provides better grounding for training the already ill-posed setting of being pose-free.\n3. I appreciate the ablations mentioned in the tables, they lay out the contributions well for each component.\n\n\n\nWeaknesses\n1. In the experiments I would like to see comparisons with some more recent baselines such as GeoNLF[1] (LiDAR only), and few other approaches like NeurAD[2], SplatAD [3]which are multimodal but use poses. In table 2 and 3 there are comparisons where authors use GT poses and therefore a comparison with methods that do multi-modal 3D reconstruction albeit with poses is warranted and will strengthen the contribution of the paper.\n\n2. I’m confused as to why only the front camera was used for experiments? I’d like to see consistency of results when all cameras are used. \n\n3. I would like to know more about the limitations of MSC2F such as mis-balances, or  where MMG fails due to low image-LiDAR overlap.\n\n4. Typically, the sensor calibration(LiDAR-RGB) on the AV has some errors. A sanity check would be to see if rendered depth and rgb images are perfectly aligned using your multi-modal training scheme. \n\n[1] Xue, Weiyi, et al. \"GeoNLF: Geometry guided Pose-Free Neural LiDAR Fields.\" Advances in Neural Information Processing Systems 37 (2024): 73672-73692.\n\n[2] Tonderski, Adam, et al. \"Neurad: Neural rendering for autonomous driving.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[3] Hess, Georg, et al. \"Splatad: Real-time lidar and camera rendering with 3d gaussian splatting for autonomous driving.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.", "was_revised": true, "final_justification": "I'm happy with the detailed responses to my questions, therefore raising the score to a 4", "rebuttal": "Thank you very much for your insightful suggestion, which helps improve the completeness of our work. The issues you raised have been well addressed, and we provide detailed responses below.\n\n**1. More comparisons:** \nBased on your suggestion, we provide comparisons with GeoNLF and NeuRAD on the KITTI-360 dataset, utilizing all stereo camera inputs. We evaluate on five static scenes. Note that NeuRAD's CD computation contains an error (see GitHub issue) and omits normalization by chunk size, we report results based on our corrected implementation. NeuRAD models the two modalities separately without explicit cross-modal information fusion. As a result, under ground-truth poses, our method performs slightly better in static scenes, while showing a significant advantage in the pose-free setting. Besides, compared to GeoNLF, our method benefits from the incorporation of multimodal inputs and achieves improved registration accuracy in the pose-free setting.\n\n|Method|CD↓|PSNR↑|SSIM↑|LPIPS↓|RPEₜ(cm)↓|RPEᵣ(°)↓|ATE(m)↓|\n|-|-|-|-|-|-|-|-|\n|NeuRAD|0.099|24.56|0.820|0.205|-|-|-|\n|Ours(GT-pose)(all cameras)|0.073|24.82|0.833|0.210|-|-|-|\n|GeoNLF|0.088| -| -| -|2.001|0.063|0.225 |  \n|Ours(pose-free)(all cameras)|0.076|24.02|0.771|0.269|1.489|0.023|0.181|\n\n**2. Results when all cameras are used:** Your feedback is greatly appreciated and has been carefully considered. Indeed, using multiple cameras is more aligned with real-world autonomous driving settings. In the KITTI-360, it contains two forward-facing stereo cameras (the fisheye is typically not used for reconstruction tasks). Moreover, the two forward-facing cameras in KITTI-360 have a high degree of overlap and observe very similar views. Therefore, we only adopted a single pinhole camera in our original experiments. To maintain consistency, we used a similar setting for NuScenes. \n\nMotivated by your suggestion, we additionally conduct experiments by incorporating all available pinhole cameras in both KITTI-360 and NuScenes. We assume that the relative extrinsics between the cameras and LiDAR are known, which is a practical assumption in autonomous driving scenarios. In both datasets, sensors are time-synchronized, and all settings—including optimization steps, hyperparameters, and test sets—remain unchanged. The only difference is a larger set of RGB images with known poses used for supervision. We retain the alternating training strategy and report the results as follows:\n\n|Method|Dataset|CD↓|PSNR↑|SSIM↑|LPIPS↓|RPEₜ(cm)↓|RPEᵣ(°)↓|ATE(m)↓|\n|-|-|-|-|-|-|-|-|-|\n|Single|NuScenes|0.810|20.83|0.699|0.585|4.058|0.101|0.176|\n|All| |0.806|22.31|0.721|0.486|3.001|0.099|0.173|\n|Single|KITTI-360|0.079|23.46|0.759|0.287|1.471|0.025|0.187|\n|All| |0.076|24.02|0.771|0.269|1.489|0.023|0.181|\n\nAs expected, incorporating additional supervised views improves reconstruction quality without introducing significant computational overhead. In particular, on the NuScenes dataset, leveraging six camera viewpoints leads to a notable performance gain. This improvement can be attributed to two key factors: (1) denser RGB supervision enhances photometric consistency; and (2) LiDAR points are projected onto a greater number of views, enabling our MMG module to more effectively identify and filter out occluded or non-overlapping regions.\n\n**3. Potential Limitations of MSC2F and MMG under Low Image-LiDAR Overlap:** \n\n(1) Effect of Non-overlapping Fields of View (FoV) on MMG.\nOur setup utilizes a 360-degree surround-view LiDAR, and thus the primary source of non-overlap arises from the mismatch in vertical FoV between the LiDAR and the RGB cameras (LiDAR typically has a much narrower vertical FoV). Despite this limitation, the LiDAR still provides rich geometric cues that can be leveraged for accurate pose optimization. As demonstrated in Tab.2 (w/o image), although the absence of image supervision leads to a significant performance drop, pose optimization still converges to a reasonable range, thereby establishing a lower bound for MMG’s performance. \n\n(2) Optimization Balance in MSC2F.\nMSC2F employs a hard-coded scheduling strategy in which each modality is gradually and fully activated during training. **This ensures that both image and LiDAR features are eventually optimized, regardless of their degree of FoV overlap.** Consequently, even under low-overlap scenarios between RGB and LiDAR, each modality is independently capable of contributing to the reconstruction. **The primary source of imbalance lies in the frequency content: LiDAR point clouds are inherently sparser than images**, whereas RGB inputs provide richer high-frequency details. We find that if the LiDAR branch converges too quickly, the optimization process may shift its focus toward fine-grained texture details in RGB, potentially undermining the representation of geometry. While this modality imbalance does not lead to failure in LiDAR-based reconstruction (as shown in Tab.2,w/o MSC2F), it can degrade geometric fidelity. To address this, we propose the MSC2F training strategy, which ensures stable convergence across all sequences.\n\nAnother implication of low-overlap conditions lies in the extrapolation of LiDAR-based geometry. As discussed in L306-308, thanks to the continuity of neural fields, our model can extrapolate depth estimates beyond the LiDAR’s limited FoV by leveraging supervision from overlapping images (see Fig. 6). Naturally, the extrapolation becomes less reliable if the LiDAR FoV is further reduced. Nonetheless, in the KITTI dataset, where the LiDAR has only approximately half the horizontal field of view compared to the camera, our method is still able to produce stable and accurate extrapolated depth.\n\n**3. Sensor Calibration concerns:**  We appreciate your insightful suggestion. Since the data in the dataset have been carefully processed and well-calibrated, such errors theoretically have minimal impact on the results, as shown in Fig.7(last row), where the depth maps and images are well-aligned. Furthermore, by combining the rendered depth with corresponding color information and back-projecting them into 3D space, we obtain dense colored point clouds that appear visually consistent and plausible.(**Due to rebuttal constraints disallowing images and external links, we will make these results available on our future project page.**)\n\nWe fully understand your concern, which is indeed critical in real-world applications. To address this, while retaining the original relative poses as priors, we introduce a small, learnable adjustment by enabling gradient propagation to the relative poses, analogous to pose optimization. Given that the initial relative poses are reasonably accurate, this refinement is performed at a later training stage or with an extremely small learning rate to maintain stability throughout the optimization process.\n\nWe conducted a simple experiment on a sequence in KITTI-360: the relative poses are initialized using the provided values, perturbed with an average angular error of 2 degrees, and optimized with a learning rate 20× smaller than that used for the extrinsic parameters. **The results are close to those obtained using the provided values, with a comparable Chamfer Distance, a PSNR drop of only 0.06, and a reduction in the relative rotation error from 2 degrees to 0.12 degree.**\n\nNonetheless, in practical deployments where relative poses are not guaranteed to be accurate, our framework retains an interface for such refinements, providing flexibility to adapt to different levels of pose uncertainty.\n\n**4. Dynamic scenes:** This is a valuable question. Since our method does not explicitly model dynamic objects, the NVS results exhibit noticeable blurring in dynamic regions. We have acknowledged this limitation in Appendix (L525–529).\nCurrently, most approaches handle dynamic scenes by separately reconstructing static and dynamic components—typically by masking out dynamic objects during static scene reconstruction and modeling dynamic objects independently. We consider this primarily an engineering aspect that requires introducing appropriate priors to mask dynamic elements, a task that has been extensively studied and is relatively straightforward to implement. Addressing this issue will be a key direction for future work to further improve our method.\n\nRegarding the second question, our response is: **Yes, MMG is effective at filtering out dynamic parts**, and even in the presence of dynamic objects, our pose optimization still achieves high accuracy. For instance, in the sequence 2350-2400 of KITTI-360 (which includes moving vehicles), our method successfully performs registration. This can be attributed to two main reasons:\n\n(1)In MMG, the CD is computed by finding the nearest neighbor $b \\in B$ for each $a \\in A$ after global alignment. We further project these points onto time-synchronized images and use the color differences to filter out non-overlapping regions (e.g., areas occluded in one frame but visible in the other). **This mechanism is also effective for dynamic objects because erroneous correspondences caused by motion are filtered out by MMG.** Concretely, if $a$ is a point on a static car (e.g., on a side mirror), its nearest neighbor $b$ in point cloud B after alignment will correspond to the same location on the car. Conversely, if $a$ belongs to a dynamic car, once aligned in the global coordinate system, its nearest neighbor in B is likely to be off the car (e.g., on a building), causing significant color differences that lead to filtering.\n\n(2)In autonomous driving scenarios, dynamic objects generally constitute a small portion of the overall point cloud, with the scene dominated by static geometry. Similar to how ICP can still register frames effectively in dynamic scenes given good initialization and sufficient overlap, our MMG optimization objective minimizes the CD over all edges in the graph, which is primarily governed by the static parts of the scene."}, {"review_id": "STDSymA0lK", "reviewer": "Reviewer_NasR", "summary": "This paper presents a unified framework of pose-free multimodal LiDAR-camera novel view synthesis. The view features and LiDAR features are unified in the hash grid of instant NGP. A multimodal geometric optimizer and a multimodal-specific coarse-to-fine training approach are utilized to guide the optimization of poses and neural fields.", "questions": "- MVS methods, e.g., colmap or other network-based methods, should also be compared with. \n\n- How does Nope-NeRF utilize multimodal inputs in your baseline setting?\n\n- Some important methods should be cited and well discussed. \n[1] MVSNeRF: Fast generalizable radiance field reconstruction from multi-view stereo.\n[2] DBARF: Deep bundle-adjusting generalizable neural radiance fields.\n[3] CRAYM: Neural field optimization via camera ray matching.\n[4] Self-calibrating neural radiance fields.", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- This paper presents a novel continuous and pixel-wise depth supervision.\n- The proposed method outperforms other methods on both reconstruction and view synthetic results. \n- The proposed method can better recover the poses of input views.\n\nWeaknesses:\n- The limitations of the proposed method are not well discussed. \n- The comparisons are not quite fair.", "was_revised": true, "final_justification": "All of my concerns are addressed, so I'm raising the score to 5 accept.", "rebuttal": "We sincerely thank you for your careful review. Your suggestions have been very helpful in improving our work and have provided us with valuable insights. In fact, the concerns you raised were considered at the early stages of this work. Below, we address each of them in detail.\n\n**1. Limitations of MUP:**  We briefly discussed some limitations of our method in Appendix(L525-L529), and here we provide a more detailed explanation.\n\nIn practical applications, the primary limitation of our approach lies in the reconstruction of dynamic objects in autonomous driving scenarios. Our current method is not specifically designed for 4D (i.e., dynamic) reconstruction. A commonly adopted strategy is to mask out dynamic objects for static scene reconstruction, and then perform separate reconstruction of the dynamic components. We consider this process more of an engineering issue, as it typically relies on prior knowledge or external cues to identify and mask dynamic objects—something that is feasible with existing techniques. In future work, we plan to extend our method to more explicitly address dynamic scene reconstruction.\n\nIn addition, there is a minor limitation related to the global optimization in MMG. The efficiency of this step relies on the sequential nature of the data: each frame’s point cloud (along with its corresponding image) is connected to a few temporally adjacent frames. If this sequential structure is absent, constructing the graph would require connecting each frame to all others, which would increase computational complexity and potentially degrade pose optimization performance. Fortunately, datasets in autonomous driving scenarios are typically captured as continuous sequences, allowing our method to operate efficiently in such settings.\n\n**2. Fairness of Comparisons:** As few existing baselines address multi-modal pose-free scene reconstruction, most of the baselines we compare against follow a \"register-then-reconstruct\" paradigm. We fully understand your concern regarding the fairness of such comparisons. To the best of our knowledge, point cloud registration generally achieves higher accuracy than image-based registration. Therefore, we choose to adopt Colored-ICP, which incorporates both image and point cloud information to improve alignment accuracy. To further address your concern, we provide additional experiments in our response to the next question, including both COLMAP-assisted and network-assisted registration strategies.\n\n**3. Comparison with COLMAP-assisted and Network-assisted Methods:** We fully agree with your points. In the early stages of this work, we carefully considered these comparisons. The reason we did not adopt COLMAP for the main evaluation is that Colored-ICP typically achieves higher registration accuracy than COLMAP. This is primarily because ICP-based method leverages geometric information for registration, while COLMAP is mainly based on sparse feature matching from images. Given that our method also utilizes geometric information (i.e., LiDAR point clouds), we believe that comparing with Colored-ICP is a more fair choice.\n\n| Method                    | CD↓   | F-score↑    | PSNR↑ | SSIM↑ | LPIPS↓ | RPE_t(cm) | RPE_r(degree) | ATE(m)   |\n|---------------------------|-------|-------|--------|--------|--------|--------|--------|--------|\n| Colored-ICP-assisted      | 0.492 | 0.787 | 20.92  | 0.698  | 0.459  | 25.383 | 0.899  | 1.624 |\n| COLMAP-assisted           | 0.532 | 0.781 | 20.96  | 0.675  | 0.488  | 32.383 | 1.962  | 1.016 |\n| GeoTransformer-assisted   | 0.314 | 0.811 | 22.21  | 0.701  | 0.333  | 6.024  | 0.393  | 0.254 |\n| **Ours**                  | **0.079** | **0.942** | **23.46** | **0.759** | **0.287** | **1.471** | **0.025** | **0.187** |\n\nTo further address your concerns, we have now included COLMAP-assisted registration results in the above table. As shown, Colored ICP consistently achieves better registration accuracy than COLMAP under our settings. And regarding learning-based registration methods, we note that point cloud-based approaches generally offer higher accuracy compared to image-only methods. Among them, GeoTransformer[1] is a representative and state-of-the-art method for point cloud registration. We therefore conduct additional experiments using GeoTransformer with official pretrained weights (trained on the KITTI dataset). The results are summarized in the table above. Although GeoTransformer achieves good registration results compared to other baselines, our method attains superior pose estimation accuracy, with translation error reduced to one-third and rotation error reduced to one-sixteenth of GeoTransformer’s, thereby yielding improved reconstruction outcomes.\n\n**4. Multimodal Inputs in Nope-NeRF:** We follow the official setup of Nope-NeRF, which, under the original RGB-only input setting, employs DPT to estimate a depth map for each image. This introduces geometric priors that enable pixel-wise depth constraints. However, due to the inherent scale ambiguity in monocular depth estimation, the depth maps require joint optimization of their scale and offset. For more details, please refer to Appendix L571-575.\n\nTherefore, although Nope-NeRF is nominally an RGB-only method, we consider it to effectively leverage multi-modal inputs by incorporating estimated depth as a geometric prior. To the best of our knowledge, existing pose-free reconstruction methods primarily focus on RGB-only inputs. Among them, Nope-NeRF and the recently proposed COLMAP-free Gaussian Splatting (CF-GS)[2] are the most relevant methods that utilize multi-modal information for pose-free rescontruction, such as RGB combined with depth. We supplement here a comparison with CF-GS to better contextualize our method within this line of work. The experimental results on KITTI-360 can be referred to in our response to Reviewer RGqZ.\n\n**5. Related Work to Be Discussed:** We sincerely appreciate your valuable suggestions. The works you mentioned are highly relevant, especially those concerning generalizable NeRFs and sparse-view reconstruction, which we have been closely following. We will include citations and further discussion of these works in the revised version.\n\n(1) Generalizable NeRFs and Sparse-View Reconstruction: MVSNeRF[3] and PixelNeRF[4] are representative methods in the domain of generalizable NeRFs. They incorporate strong priors to enhance performance under sparse-view settings. Specifically, MVSNeRF constructs a cost volume by warping 2D image features onto a plane sweep and extracts voxel features via 3D CNNs, the features of query points are then obtained from the corresponding voxel features. DBARF[5] is more closely related to our approach. It extends generalizable NeRFs to pose-free settings by integrating a pose and depth estimator into the IBRNet backbone. While these methods primarily focus on generalizable NeRFs in object-centric or indoor scenes, and indeed exhibit faster convergence and improved robustness to sparse views—their application in autonomous driving scenarios remains challenging. **Specifically, the complexity of driving environments calls for stronger priors (e.g., in feature extraction and pose estimation), whereas methods that generalize well to object-level or indoor reconstructions often fall short.**\n\n(2) Pose-free NeRFs:\nCRAYM[6] focuses on pose-free reconstruction, approaching the task through camera ray matching and offering a novel perspective. It also shares conceptual similarities with ray-based pose estimation methods such as Ray-Diffusion[7]. However, it is primarily applied to object-level reconstruction. Among these methods, SCNeRF[8] is most closely related to ours. It adopts an optimization-based approach, leveraging NeRF to learn scene geometry while incorporating photometric consistency constraints in conjunction with traditional geometric priors from classical self-calibration techniques. **While SCNeRF represents a notable step forward in pose-free NeRF methods, Nope-NeRF leverages geometric priors to attain superior performance, especially in outdoor environments, as evidenced by the results presented in its experiments. **Therefore, we adopt Nope-NeRF as our baseline.**\n\nAll these classical works have inspired our approach and are closely related to our method. Once again, we are grateful for the insightful suggestions and will incorporate the corresponding citations and discussions in the updated version.\n\n[1] Qin, Zheng, et al. \"Geotransformer: Fast and robust point cloud registration with geometric transformer.\" IEEE Transactions on Pattern Analysis and Machine Intelligence 45.8 (2023): 9806-9821.\n\n[2] Fu, Yang, et al. \"Colmap-free 3d gaussian splatting.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[3] Chen, Anpei, et al. \"Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo.\" Proceedings of the IEEE/CVF international conference on computer vision. 2021.\n\n[4] Yu, Alex, et al. \"pixelnerf: Neural radiance fields from one or few images.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.\n\n[5] Chen, Yu, and Gim Hee Lee. \"Dbarf: Deep bundle-adjusting generalizable neural radiance fields.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[6] Lin, Liqiang, et al. \"Craym: Neural field optimization via camera ray matching.\" Advances in Neural Information Processing Systems 37 (2024): 9950-9973.\n\n[7] Zhang, Jason Y., et al. \"Cameras as rays: Pose estimation via ray diffusion.\" arXiv preprint arXiv:2402.14817 (2024).\n\n[8] Jeong, Yoonwoo, et al. \"Self-calibrating neural radiance fields.\" Proceedings of the IEEE/CVF international conference on computer vision. 2021."}, {"review_id": "HFzXrZT41W", "reviewer": "Reviewer_RGqZ", "summary": "This paper proposes a novel unified filed for LiDar-image rendering using NeRF techniques. The authors first analyze key limitations of the methods on pose-free NeRFs and LiDar-NeRFs, then address them by introducing several adaptive contributions constructing a unified LiDar-camera field and an adaptive and effective optimization pipeline. Experiments show the significant improvement on reconstruction metrics (expecially LiDar metrics).", "questions": "See Weaknesses.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. Clear motivation and good technical contribution: The paper proposes a unified neural field for both LiDar and image rendering, as well as an effective and adaptive optimization skill. Current solutions on LiDar-NeRF design do need improvements like that to address key limitations.\n2. Good expression: In method part, the authors do necessary explainations and exhibitions on why the module is designed like this -- it's explainable, clear and experimentally effective.\n3. Strong evaluation results: the proposed method gains impressive results against related baselines, especially on LiDar metrics becaused of the unified field design.\n\nWeaknesses:\n\n1. Format advice: The words in Fig. 1 are too small to be clear enough. The format for Table 2 and 3 seems squeezed.\n2. Discussion and possible comparison with Gaussian Splatting counterparts: I'm not here to say that the technical contribution on NeRF filed is not as meaningful as previous. However, I believe the authors can put 1-2 baselines based on Gaussian representation to make the experimental results more persuasive. After all, the technical stream based on GS shows faster rendering and optimization time. I would recommend NopeNeRF like baselines like COLMAP-free GS, and Dust3R-inspired(-utilized) baselines like InstantGS for the setting of pose-free reconstruction. Moreover, GS-based methods considering LiDar inputs also should be referred to.", "was_revised": true, "final_justification": "The authors' rebuttal addresses my concerns very well. I'm glad to hold my original rating to borderline accept this paper. Presentation issues should be further considered upon the publication.", "rebuttal": "We sincerely appreciate your careful review. Your insightful and objective suggestions have been thoroughly considered, and the manuscript will be revised accordingly, including further experiments to enhance the persuasiveness of our method.\n\n**1. Formatting Issues:**  We aimed to present more comprehensive content in the main text, which unfortunately resulted in some formatting congestion. In future revisions, we will strive to improve the clarity of our figures and overall presentation. \n\n**2. Comparison with Gaussian Splatting:** We fully agree with your perspective, and accordingly, we selected the KITTI-360 dataset to evaluate the performance of COLMAP-free GS (CF-GS[1]) and DUSt3R[2]/VGGT[3]-assisted 3DGS reconstruction. For the latter, we select InstantGS[4] as the baseline method.\n|Method|CD↓|F-score↑|PSNR↑|SSIM↑|LPIPS↓|RPE_t(cm)|RPE_r|ATE(m)|\n|-|-|-|-|-|-|-|-|-|\n|CF-GS[1]|-|-|20.13|0.36|0.511|53.223|9.412|0.353|\n|InstantGS[4]|-|-|21.01|0.66|0.477|36.223|3.329|0.312|\n|Ours|0.079|0.942|23.46|0.76|0.287|1.471|0.025|0.187|\n\n2.1 Comparison with CF-GS[1]: Similar to Nope-NeRF, the CF-GS also utilizes DPT-predicted depth as a prior to initialize the local 3D Gaussians. As a result, it suffers from the same limitations. Specifically, DPT’s depth estimation in outdoor scenes is inaccurate, and the predicted depth maps suffer from inherent scale ambiguity. Although CF-GS performs well in indoor and object-level scenes, it introduces many erroneous floating points in autonomous driving scenarios. Moreover, some frames in long driving sequences exhibit misaligned camera poses, the pose estimation quality is clearly inferior to approaches aided by LiDAR point clouds, which can achieve centimeter-level precision. Consequently, while CF-GS performs slightly better than Nope-NeRF, it still falls short of the performance achieved by our method.\n\n2.2 Regarding the DUSt3R-inspired baselines, we sincerely appreciate your constructive suggestions. The results of InstantGS are shown in the table above. Due to the lack of robustness in end-to-end pose estimation for outdoor scenes, outlier frames occasionally occur. Thus, suboptimal initialization leads to degraded reconstruction quality in certain sequences. **Due to rebuttal constraints disallowing images and external links, we will make these results available on our future project page.** \n\nRecently, VGGT[3] has attracted considerable attention and provides a convenient API that enables direct pose estimation and point cloud generation from images for training 3DGS. To further evaluate the capability of end-to-end pose estimation in autonomous driving scenarios, we attempted to use VGGT outputs as initialization and subsequently performed 3DGS-based reconstruction. We observed that, although VGGT yields better performance than DUSt3R, it still struggles with long sequences in outdoor scenarios. Moreover, VGGT currently does not support using shared camera intrinsics as prior information, resulting in inconsistent intrinsic estimates across frames. These two factors together lead to notable radial distortions, similar to those observed in DPT and other monocular depth estimation methods. Consequently, the final reconstructed scenes contain noticeable artifacts and a large number of floaters.\n\n**3. Completeness of Related Work:** To improve the completeness of our related work, we will also include recent GS-based novel view synthesis methods of LiDAR point cloud in the updated version of the paper, such as GS-Lidar[5], LiDAR-GS[6] and others. Once again, we sincerely thank you for your valuable suggestions, which will help make our paper more thorough and comprehensive.\n\n[1] Fu, Yang, et al. \"Colmap-free 3d gaussian splatting.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[2] Wang, Shuzhe, et al. \"Dust3r: Geometric 3d vision made easy.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[3] Wang, Jianyuan, et al. \"Vggt: Visual geometry grounded transformer.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n[4] Fan, Zhiwen, et al. \"InstantSplat: Sparse-view Gaussian Splatting in Seconds.\" arXiv preprint arXiv:2403.20309 (2024).\n\n[5] Jiang, Junzhe, et al. \"Gs-lidar: Generating realistic lidar point clouds with panoramic gaussian splatting.\" arXiv preprint arXiv:2501.13971 (2025).\n\n[6] Chen, Qifeng, et al. \"Lidar-gs: Real-time lidar re-simulation using gaussian splatting.\" arXiv preprint arXiv:2410.05111 (2024)."}]

Author rebuttal:

Meta-review: The paper received unanimously positive reviews, consisting of two Borderline Accepts and two Accepts. It introduces a novel pose-free framework that effectively combines two modalities, leveraging a compact neural representation without the need for accurate initial poses. While reviewers initially raised concerns regarding the method's incremental nature, unaddressed limitations, and insufficient comparisons, the rebuttal and subsequent discussion successfully resolved the main technical issues. As a result, Reviewers 2Vdy and NasR raised their scores to a Borderline Accept and an Accept, respectively, while the other two reviewers maintained their initial positive ratings. The AC judges that a consensus for acceptance has been reached and strongly recommends that the authors incorporate all reviewer feedback into the final camera-ready version.
Decision: Accept (poster)