SimULi: Real-Time LiDAR and Camera Simulation with Unscented Transforms
Authors: Haithem Turki
Qi Wu
Xin Kang
Janick Martinez Esturo
Shengyu Huang
Ruilong Li
Zan Gojcic
Riccardo de Lutio
Venue: ICLR 2026 Poster (2026)
Keywords: neural rendering
3d gaussians
3d vision
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We design a factorized 3d gaussian representation that improves multi-sensor reconstruction and strategies to accelerate LiDAR rendering by 10x.

Abstract: Rigorous testing of autonomous robots, such as self-driving vehicles, is essential to ensure their safety in real-world deployments. This requires building high-fidelity simulators to test scenarios beyond those that can be safely or exhaustively collected in the real-world. Existing neural rendering methods based on NeRF and 3DGS hold promise but suffer from low rendering speeds or can only render pinhole camera models, hindering their suitability to applications that commonly require high-distortion lenses and LiDAR data. Multi-sensor simulation poses additional challenges as existing methods handle cross-sensor inconsistencies by favoring the quality of one modality at the expense of others. To overcome these limitations, we propose SimULi, the first method capable of rendering arbitrary camera models and LiDAR data in real-time. Our method extends 3DGUT, which natively supports complex camera models, with LiDAR support, via an automated tiling strategy for arbitrary spinning LiDAR models and ray-based culling. To address cross-sensor inconsistencies, we design a factorized 3D Gaussian representation and anchoring strategy that reduces mean camera and depth error by up to 40% compared to existing methods. SimULi renders 10-20$\times$ faster than ray tracing approaches and 1.5-14$\times$ faster than prior rasterization-based work (and handles a wider range of camera models). When evaluated on two widely benchmarked autonomous driving datasets, SimULi matches or exceeds the fidelity of existing state-of-the-art methods across numerous camera and LiDAR metrics.

Published as a conference paper at ICLR 2026
SIMULI: REAL- LIDAR AND CAMERA SIMULA- TION WITH UNSCENTED TRANSFORMS
Haithem Turki'
Qi Wu"
Xin Kang
Janick Martinez Esturo
Shengyu Huang
Ruilong Li
Zan Gojeie
Riccardo de Lutio
NVIDIA
(hturki, qiwu, roelutio
30.2 PSNR
137.7 MP/s
11.1
Figure SimULi. We design a factorized 3D Gaussian representation that encodes camera and LiDAR information into separate sets of 3D Gaussians joined via nearest-neighbor anchoring loss (left). To efficiently render LiDAR scans (middle), we extend 3DGUT (Wu al., 2025b) with an automated tiling strategy and ray-based culling. When compared to existing methods, we render 1.5- 20x faster and match or exceed LiDAR and camera quality (right) across a wide range of metrics.
ABSTRACT
Rigorous testing of autonomous robots, such as self-driving vehicles, essen- tial to ensure their safety in real-world deployments. This requires building high- fidelity simulators to test scenarios beyond those that can be safely or exhaustively collected in the real-world. Existing neural rendering methods based NeRF and 3DGS hold promise but suffer from low rendering speeds or can only render pinhole camera models, hindering their suitability to applications that commonly require high-distortion lenses and LiDAR data. Multi-sensor simulation poses additional challenges existing methods handle cross-sensor inconsistencies by favoring the quality of one modality at the expense of others. To overcome these limitations, propose SimULi, the first method capable of rendering arbitrary camera models and LiDAR data in real-time. Our method extends 3DGUT, which natively supports complex camera models, with LiDAR support, via an automated tiling strategy for arbitrary spinning LiDAR models and ray-based culling. To address cross-sensor inconsistencies, we design factorized 3D Gaussian repre- sentation and anchoring strategy that reduces mean camera and depth error by up 40% compared existing methods. SimULi renders 10-20x faster than ray tracing approaches and 1.5-10x faster than prior rasterization- bbasdd work (and handles wider range camera models). When evaluated on two widely bench- marked autonomous driving datasets, SimULi matches exceeds the fidelity of existing state-of of-the---te-rr methods across numerous camera and LiDAR metrics.
Equual contribution

---

Published as a conference paper at ICLR 2026
INTRODUCTION
With the rise of end-to-end policy models, accurate sensor simulation has become a critical com- ponent in the development and evaluation of autonomous vehicle (AV) systems. Data-driven meth- ods based on Neural Radiance Fields (NeRFs) (Mildenhall et al., (2000) offer a scalable solution for reconstructing high-quality, diverse simulation environments directly from real-world sensor data (Turki et al.) 2023; Yang et al. (023b). As they are optimized to match real-world observa- tions, they also exhibit a smaller domain gap compared traditional artist- grrist-generated simulators.
Multi-Sensor Simulation. In addition to cameras, most AVs are equipped with active sensors such as LiDAR to obtain 3D measurements of their surroundings. Early approaches focus on camera simulation (Rematas et al., 2022). using other sensors to constrain geometry for improved novel
view synthesis, or solely on LiDAR (Tao et al.) 2023 Huang et al., 2023). Recent efforts YYang
et al. 2023b Tonderski et al., 2024 Hess et al. 2025) jointly model both sensors, challenging
task as cross-sensor calibration inconsistencies due are impossible to eliminate in the real world. In
practice, they are forced prioritize camera quality at the cost of LiDAR accuracy (or vice-versa).
Sensor Rendering. NeRF-based methods (Yan et al., 2023b Tonderski et al. 2024 suffer from low rendering speeds, limiting their practicality for large-scale use. Replacing NeRFs with a 3D Gaussian Splatting (3DGS) formulation (Kerbl et al., (023) mitigates these challenges by enabling real-time rendering through a rasterization-bant visual fidelity, especially for larger scenes (Chen et al. 2025 Yan et al. 2024 Zhou et al. 2024).
pipeline, while maintaining, or even improving,
However, this comes the cost of limitations inherent to the rasterization paradigm. In particular, rasterization is constrained to idealized pinhole camera models and does not naturally support non- linear projection functions (e.g.. fisheye lenses) or time-dependent effects such as rolling shutter, both of which are common in data captured by AV sensor suites (Xiao et al. 202 Sun et al. 2020)
Caesar et al. 2020 Wilson et al.) 202I Liao et al. 202Ib. Rectifying training images and ignoring rolling shutter effects degrades reconstruction quality, while the inability to render non-pinhen connpinnole cam- era views introduces a significant domain gap for downstream models, which are typically trained on raw sensor data. LiDAR sensors are often neglected, their sparse, irregular sampling pattern and strong time-dependent effects make them difficult to model within the 3DGS framework.
SimULi. In this work, we propose a high-fidelity and efficient reconstruction pipeline that enables joint camera and LiDAR simulation for AV scenarios. Importantly we aim to: (i) natively support arbitrary camera and LiDAR models, including their time-dependent effects; (ii) accurately model sensor characteristics (such as color, LiDAR intensity, and ray drop effects), without prioritizing the accuracy of one sensor at the expense of others as in existing work; and (iil) achieve real-time inference while maintaining high visual fidelity.
We build upon the 3D Gaussian Unscented Transform (3DGUT) (Wu et all 2025b which inher- ently supports non-linear camera models and time-dependent effects such rolling shutter. In this work, we extend it further to support rotating LiDAR sensors with irregular sampling patterns. Specifically, (i) derive LiDAR measurement model that accounts for its dime-dependent be- havior, (ii) introduce a for handling the highly irregular structure of LiDAR measurements, and (iii) take advantage of the relative sparsity of LiDAR rays via a culling strategy that further accelerates rendering.
baasea algorithm to compute an optimal tiling scheme
To accurately model multi-sensor data, we encode the information relevant each modality into distinct sets of 3D Gaussians coupled via an efficient nearest-neighbt anchoring method. We de- rive several benefits. First, we find that this handles cross-sensor inconsistencies far more robustly than prior methods that encode all sensor data into a single NeRF (Yang et al. 2023b; Tonderski et al. 2024) or set of Gaussian particles (Hess et al. 2025). p, allowing us to match or exceed the accuracy of camera-only or LiDAR-only simulation. Second, this improves rendering efficiency, as the information needed to render each sensor is contained within a subset of Gaussians (accelerating LiDAR rendering by 2-3x). Third, by specializing the characteristics of each Gaussian set, we ben- efit from inductive biases inherent to each sensor. Since LiDAR returns typically intersect surfaces, encourage sparsity and binary opacity in LiDAR Gaussians, further improving rendering speed. Camera Gaussians remain unconstrained and benefit from the smooth optimization properties and high rendering quality of volumetric approaches (Wang et al. 2023] Turki al. 2024
2

---

Published as a conference paper at ICLR 2026
Contributions. We make the following contributions: (1) we extend 3DGUT with LiDAR support and introduce an automated tiling scheme from which we derive optimal tiling parameters for any spinning LiDAR model without relying handcrafted heuristics (Hess et al. 2025), (2) we design a factorized 3D Gaussian representation and anchoring strategy that mitigates the camera vs LiDAR accuracy tradeoff needed in prior methods, (3) we present results that surpass existing LiDAR and
camera-based SOTA, without relying on neural networks for refinement as in prior NeRF (Tonderski
2024 Yang et al.) 2023b, Zheng al. 2024) and 3DGS-based work (Zhou al. 2025 Hess et al. 2025p. Our simplified approach renders 1.5-10x faster than recent rasterization based methods (Hess et al. 2025). 10-20x faster than ray tracing approaches (Zhou al., 2025). and is, to our knowledge, the first to render in real-time and support arbitrary camera and LiDAR models.
al.
RELATED WORK
Recent neural reconstruction methods (Mildenhall et al. 2020 Kerbl et al.) 2023) have greatly influ- enced AV simulation by enabling the reconstruction large, high-fidelity environments from raw sensor data. We list a non-exhaustive selection of these methods along axes most relevant to ours.
Simulating Camera Observations. Early neural reconstruction methods in the AV domain fo- cused on modeling camera observations (Ost et al. 2021: Xie et al., 2023 Guo et al. 2023 Yang
et al. 2023a), often leveraging LiDAR data constrain geometry and enhance novel view synthesis. To adapt static NeRF representations to the AV setting, these approaches introduced domain-specing extensions such as dynamic scene graph parameterizations (Ost et al. 202T and specialized sky rep- resentations (Guo al. 2023 Yang et al., 2023a). With the advent of 3DGS (Kerbl et al. 2023), which enables faster rendering and better scalability, several of these ideas have been reimplemented within particle-based representations (Chen et al. 2025 Yan et al. 2024 Chen et al. 2023) While 3DGS-based methods yield faster frame rates, they also inherit all the limitations of rasterization.
Simulating LiDAR Observations. In parallel to camera-focused approaches, several works have adapted neural reconstruction methods for LiDAR novel view synthesis (Huang et al.] 2023 Tao
et al. 2023; Zhang et al. 2024h. NFL (Huang et al. 2023) introduced physically grounded way of modeling LiDAR characteristics within NeRFs. DyNFL (Wu et al. 2024 and LiDAR4D (Zheng al. 2024 extended NFL to dynamic scenes using the same scene graph parameterization cam-
era methods and improve rendering speed via acceleration structures (Müller et al. (022).
Adapting 3DGS for LiDAR rendering is more challenging, its rasterization assumes ideal pinhole camera models and its tiling strategy does not naturally accommodate the non-uniform sampling patterns of LiDAR sensors. Recently, 3DGRT (Moenne-Loccoz al., 2024 replaced rasterization with a ray tracing formulation, enabling support for more complex sensor models. LiDAR-RT (Zhou al. 2025) applies this framework specifically LiDAR simulation, while GS-LiDAR (Tang al., 2025 combines similar method with tile-based panoramic rendering. Both approaches render faster than NeRF-based approaches but remain several times slower than rasterization methods.
Joint LiDAR-Camera Simulation. Recent works jointly model LiDAR and camera novel view synthesis via a unified neural representation. UniSim (Yang et al. 2023b and NeuRAD (Tonder- ski et al. 2024) build upon NeRF and adapt the rendering formulation for different sensors using the same underlying model. AlignMif (Tag et al. 2024) encodes sensors into separate feature grids aligned via shared initialization and feature fusion. They remain impractical due to the slow rendering speed of NeRF and (other than AlignMiF) perform worse than camera-only or LiDAR reconstruction due to cross-sensor inconsistencies.
Closest to our work, SplatAD (Hess et al. 2025) extends 3DGS with a LiDAR rendering formu- lation that supports non-uniform sampling patterns (as does ours) and support for rolling shutter effects. While addresses several important challenges, it differs from our method in three key as- pects. First, 3DGS-basee based method, it remains limited to perfect pinhole camera models, whereas SimULi builds upon 3DGUT (Wu et al. 2025b), which enables native support for high-distortion camera models, such fisheye lenses. Second, it relies on manual heuristics to determine the tiling strategy for each LiDAR sensor. In contrast, we demonstrate how an automated strategy can adap- tively compute optimal tiling patterns across wide range of non-uniform LiDAR sensors. Third, it
3

---

Published as a conference paper at ICLR 2026
Representation
Rendering
Optimization
Sencoe-ppecffc
Tillng
Iepuis
Canations
g.
g.s.q.a 1.3
Somne Graph
Ven-Fquidistant Tiling Roy Culling
Anchoring Loss
Canu Reekeng
KNNN audgenents
LIBAR
Figure 2: Method Overview. We model the scene a dynamic graph (Ost et 2021 and pa- rameterize the background and each actor with camera and LiDAR 3D Gaussians (left). We render camera views similar 3DGUT (Wu al. 2025b) and derive an automated tiling strategy and ray- based culling to efficiently render LiDAR (middle). We sample image and LiDAR scan at each training step to optimize our representation (right). To improve camera novel view synthesis with LiDAR -upeevupe geometry, we anchor camera Gaussians near surfaces via nearest-neighbor loss.
encodes all sensor information into the same Gaussian set, and suffers from cross-sensor inconsis- tencies as prior NeRF-based works (Tonderski et al. 2024 Yang et al.) (023b), which we mitigate by encoding each sensor into its own particle set and using an anchoring loss. We illustrate the difference approaches via extensive benchmarking in Sec. 4
3
METHOD
Our goal learn controllable scene representation that simulates camera and LiDAR renderings from novel viewpoints in real-time (Fig. 2). We describe our representation in (Sec. 3.1 our general
rendering approach Sec. 3.2 LiDAR specifically
Sec.. 3.3 and optimization in Sec.
3.1 REPRESENTATION
Particle Representation. We encode camera and LiDAR attributes into separate unordered sets G. and G. of semi-transparent 3D Gaussian particles (Kerbl et al. 2023). Particles in both sets are parameterized by their 3D position μ4 E R opacity coefficient Σ, 33x3 Σ; decomposed into a rotation matrix R SO(3) and scaling matrix S R333 such that Σ = RSS R", to ensure that it remains positive semi-definite. We associate 3rd 3rd-order spherical harmonics coefficients SH e R848 camera Gaussians to encode view-dependent color, and SH E 18 to LiDAR Gaussians for view-dependent intensity and ray drop information.
R. and covariance matrix
Scene Representation. We adopt a graph decomposition similar to prior work (Ost et al. 2021 enable controllability in dynamic scenes. We assign each particle G. and G. dynamic object or the static background. Each object is associated with a 3D bounding box and a sequence of SE(3) poses adjusted with learnable offsets. The means and covariances the Gaussians assigned to objects are expressed in the objects' local coordinate system. At render time, they are transformed world coordinates by applying the SF(3) transformation corresponding the timestamp
3.2 RENDERING
We build our differentiable renderer upon 3DGUT (Wu al. 20256), as it supports complex cameras and time-dependent effects that common in autonomous driving data.
4

---

Published as a conference paper at ICLR 2026
256
10-
(Deanton (*)
0-
Elevatoo:
20
252
-10
80
250
-10
10
10
20
30
100
2000
Arimut (*)
Rebrive Elevation Angle(")
Relative Amtmuth
(a) Beam pattern
(b) Elevation angle to tile map
(c) Beam count per tile
Figure 3: LiDAR Tiling. As the measurement pattern of commonly used LiDAR sensors (Xiao al. p021) is irregular (left), rendering with equally spaced tiles highly inefficient. We compute the normalized CDF of elevation angles using a predefined histogram bin count and set elevation tiling boundaries at angles where the CDF values cross integer boundaries (middle). We then com- pute an azimuth tile count such that the beam count per tile differs at most by samples (right).
Volume Rendering. Given a camera ray r(r) = o--rd with origin o E IR3 and direction d E IR3, we volumetrically render foreground color op F and opacity E via numerical integration over the interacting Gaussian particles in Ge:
cr(o, = Σ (d)a,,, w(o,d) = ΣasTe aTT a; + iEG.
T.
aj.
JEGe
j=1
where Pj is the particle response described the next paragraph. We
the fore- ground color cy with background color сb obtained from learned texture environment map (Chen
et al. 2023 2025). We obtain the final color prediction c via an affine transformation from learned
bilateral grid A Wang al. 2024) that handles lighting variations across frames and cameras:
c(o, d) = (o d)cr(o, d) + (1 w(o, d))c,
(2)
We similarly render LiDAR features E IR3 from particles in G; as <(o, = EiEg, (d)
We decode beam intensity E IR from the first channel of ( and derive a ray drop probability 3 E R by applying the softmax function on the remaining two channels (Bais Strop) of Ç (Zou al. 2025
Particle Contributions and Response. As in 3DGUT (Wu et al. 2025b). we determine which particles contribute to each ray by approximating each Gaussian via the Unscented Transform (UT). We project 7 sigma points per Gaussian to estimate a 2D conic E' before applying tiling and culling as in 3DGS (Kerbl et al., 2023p. As sigma points are independently projected, we trivially handle deeendeenddnnt effects such as rolling-shutte by incorporating camera motion into the projection function FFg.. lTp. We measure the particle response function of each Gaussian 3D via the distance Tmaz
argmax, Po + rd) that maximizes p; along the ray r(r) "Moenne-Locoe et al. 2024)
3.3 LIDAR RENDERING
Projection. For a given LiDAR Gaussian in G₁, we project each of its 7 sigma points onto a 2D azimuth/elevation grid. We convert the 3D Euclidean coordinates of each sigma point from the world frame to the sensor frame and derive the corresponding spherical coordinates:
s arclan2(y,
arcsin((z/)) r= / +
where denotes the azimuth, w the elevation and r the range. We derive a 2D conic from the projected coordinates to use for tiling and culling, and handle time-dependent effects (such as the LiDAR-equivalent rolling shutter) in the same manner as with camera rendering.
Tiling. Effectively distributing work across tiles is crucial to achieving high rendering speeds. However, unlike cameras, the resolution of LiDAR measurements is irregular and differs greatly across dimensions Fig. 3a). Although azimuths cover the full 360 radius, the elevation angle
5

---

Published as a conference paper at ICLR 2026
Depth Loss (k=0.001) 1PSNR: 29.94, JCD: 0.46
Depth Loss (A=0.1) PPSNR: 27.53, [CD: 0.23
Anchoring Loss (Ours) 1PSNR: 30.09, JCD: 0.22
Figure 4: Depth Supervision. Prior work encodes camera and LiDAR into the same representation constrained with LiDAR-supervised depth loss. As cross-sensor data is not fully consistent, this forces the representation prioritize camera instead of LiDAR quality (left) the inverse (middle), as shown by PSNR and chamfer distance. We factorize each modality into its own particle set joined via nearest-neighbor anchoring loss, improving the quality of each (right).
usually much smaller (around 20%). Using equally sized tiles as with cameras is thus highly inefficient. We derive automated strategy that takes elevation angles input and computes an equalized elevation tiling and azimuth tile count such that each elevation tile contains roughly the same number of measurements and that the count per tile is under user-defined constraint M.
We compute the normalized cumulative distribution function C' using a pre-defined histogram bin count = 400 (Fig. 3b). We then set elevation tiling boundaries at elevation angles where the corre- sponding cumulative distribution function values cross integer boundaries. We finally compute the azimuth tile count and the maximum point count per tile constraint M. We use the azimuth tile count to uniformly tile points along the azimuth axis. This procedure happens once per sensor definition and we reuse the same tiling across rasterization calls. We provide pseudo-code in Procedure
Ray-based Culling. Compared to cameras, LiDAR scans consist of fewer rays that cover wider field of view. This results in a sparser pattern where naive tile-based rendering suffers from evaluat- ing numerous Gaussians that that do not contribute to any rays. To address this, we use two separate tile resolutions for rendering and culling T, and Te, adopting a denser resolution for the latter to cull as many particles as possible (rg 1600, r 8). After projecting a particle to the LiDAR angle space, we compute its extent in Te, check if it is intersected by any rays, and only include the particle for rendering (using its coarse extent T,) if so. We filter in constant time (4 memory reads +3 3 arithmetic operations) via 2D range query where we first construct 2D ray mask indicating whether rays intersect each culling tile and then build a summed-ares table using 2D prefix sum. We provide details in Sec. of the appendix.
Beam Divergence. LiDAR beams have a larger footprint than that of camera rays and diverge significantly as they travel away from the sensor. This causes "bloating" artifacts that are noticeable around reflective surfaces such as traffic signs where only a small portion of the beam hitting the surface is sufficient to obtain return. These spurious returns (that true zero-thickness ray would "miss") negatively impact the learned scene geometry when unaccounted for. To address this, we use 3D smoothing filter similar to AAA-Gaussians (Steiner et al. 2025) but with an important modification to avoid degrading depth. We provide details in Sec. Clof the appendix.
3.4 OPTIMIZATION
We jointly optimize the camera particles G., LiDAR particles Gr, bilateral grids A, and the environ- ment map by sampling a random input image and LiDAR scan at each training step. We minimize reconstruction loss, anchoring loss that encourages camera Gaussians in Ge to lie near the ADDAR-supervised scene geometry distilled into G,. and lower-level regularization terms such that + Lreg. with Aanchor 0.01.
the final loss C == Lrecon
Reconstruction Losses. We minimize Creeon via LI photometric loss Lphoto. SSIM loss LSSIM, LI distance loss Laists L1 intensity loss Lim, and binary cross-entropy ray drop loss Lya:
Lrecon (NphotooGphoto + ASSIM LSSIM) + (Ddist + Ninthing AraLrrd)
(4)
camera losses
LIDAR losses

---

Published as a conference paper at ICLR 2026
NeoRAD (5 FPS)
SIGUT (60 FPS)
SplatAD (30 FPS)
Ours (64 FPS
Ground Truth
10017
Scone 10099
Scene 10275
Figure 5: Static NVS. Projecting LiDAR as a sparse depth map causes inaccuracies that degrade 3DGUT's rendering of the pole (above), which we avoid by rendering LiDAR directly. Our bilateral grids also prevent floaters (middle), allowing us to render slightly faster. SplatAD's 'ss CNN does not properly recover the sign lettering (above) or rear car seat (below), while our simpler pipeline does.
Table I: Waymo Interp. SimULi renders the fastest, outperforms all baselines by 2dB PSNR, and gives better depth reconstruction than LiDAR-only LiDAR-RT Zhou et al. 2025
1.PIPS]
MeanRell.2/
Dropp
CD&
MPNNT
MRG1
Nerfacto
Tancik et
21.79
0.516
0.734
3DGRE
3024
27.49
0,860
0.264
0.008
0.008
0.195
3:23
1.39
3DGUT
Warerie A7230
27.23
0.856
0.271
1,252
1.200
2.698
135.47
OmniRe
26.68
0.833
0.256
0,173
0.167
0.560
24:15
StreetiS
curetss 5054
26.59
0,832
0.256
0,145
0.150
0,566
24.85
PVC Chen
27.19
0,838
0.303
11.97
2.148
3.766
6.65
1ing
1024
0,856
0.235
LiDAR-RT
et
UniSim ei X000
23.17
0.756
0.369
0.056
0,041
0.077
0.823
0,456
8:32
0.98
AlignMiF e
3044
24.22
0.777
0.484
010156
0.064
0.304
0.14%
0.20
0.20
NaiRAD
1024
27.49
0.810
0.237
0.005
0.049
0.061
0.907
0.165
13:21
1.62
SylatAD
50/6
27.82
0.839
0.246
0.008
0,013
0.061
0.916
0.175
49.98
2.40
SimULi
30.15
0.881
0.211
0,003
0.007
0.064
0.944
0.136
156.90
11.33
with Aphoto = 0.8, ÄSSIM = 0.2, Adist = 0.01, Aint = 0.1, and Ard = 0.05. Camera losses only require rendering and backpropagating through particles in G. (and LiDAR losses through Gi).
Anchoring Loss. Existing methods encode camera and LiDAR data into a single NeRF YYang ett al., 2023b; Tonderski et al. 2024) or set of 3D Gaussian particles (Hess et al. 2025). As cross- sensor data contains inconsistencies that are impossible to eliminate, this forces the representation to prioritize the reconstruction quality of one modality over the other based on loss weights Fig.
By encoding each sensor into its own particle set against which we apply separate reconstruction losses, avoiding this tradeoff is trivial. We still wish to improve camera novel view synthesis via connt-conttra geometry, and thus minimize the distance of camera Gaussians from the pre- dicted surfaces distilled into Gj. We define a nearest-neighhor anchoring loss on Gaussian means:
Lanchor 1
Σ JEG
N.NNNN NN G) argmin --'le
(5)
n
Since running full neirest-n on all Gaussians in Ge and G2 is computationally expensive, we assign each Gaussian in Ge to its K = 50 nearest neighbors in Gj. We minimize the K-distance loss each training iteration, and update the camera-to-LLiDA. assignments every 1000 iterations. Compared to the direct LiDARR-upervise model a greater degree of freedom addressing cross-sensor inconsistencies while still improving camera rendering via
depth loss used in prior work, this strategy allows the
scene geometry. We ablate its effectiveness in Sec. 4.3
Regularization. As LiDAR is typically reflected by surfaces, we apply an entropy loss to G, that encourages binary opacity and sparsification. To generate plausible renderings at novel timestamps,
7

---

Published as a conference paper at ICLR 2026
UniNim (4.5 FPS)
NesRAD 4.5 FPS)
SplatAD (2 FPS
Ours (69 FPS
Ground Trath
Waymo 10072
BOURBON
GOURBEN
Waymo 101000
Home
Punduset
Pandaset-
Figure 6: Dynamic Scenes. FPS numbers are averaged across Waymo Dynamic and PandaSet. Approaches that CNNs for upsampling (Yang al. 2023h Tonderski al. 2024 view dependence (Hess et al. 2025) struggle with blurriness and incorrect lettering (first two rows). Our rolling shutter strategy handles rapid movement that SplatAD's does not (third row). We capture background details and reflections (bottom) better than other methods.
Table 2: Waymo Dynamic. As with static reconstruction Table 1b. we render the fastest and report best or next-best results across every camera and LiDAR metric.
Method
PSNR
SSIM"
LPIPS.
Med1.2
MeanReil.2
Int. RMSE
KayDropAcct
CD
MPN7
MRSF
OmniRo Chen
3025
SteetGS
KU5S2
Defonilnztsy Tall ct
2024
28.82
0.898
0.173
1.898
0.697
1.299
12.90
LJDAR-KT Thou al
1025
0.003
0.046
0.066
0.946
0.176
0.93
UniSim Vane of 2024
24.70
0.799
0.347
0.023
0,055
0.072
0.823
0.671
8:77
1.06
NeuRAD
2011
29.61
0.853
0.148
0.008
0.084
0.875
0.199
9.39
1.19
SplatAD
0.008
0.223
0.922
we enforce smoothness across our affine color transformations .A and background, and regularize Gaussian scale and opacity as in MCMC (Kheradmand al. 2024). We provide details in Sec. D
EXPERIMENTS
We validate the effectiveness of method commonly used AV datasets across camera- only, LiDAR-only, and joint camera-LiDAR baselines. To measure against the widest possible set of baselines, we first measure novel view synthesis on static scenes used evaluate prior work (Huang et al. 2023) Secc. 4.T). As dynamic simulation is our main focus, we evaluate both reconstruction and novel view synthesis Sec. 4.2
We validate the efficacy of our components Sec. 4.3)
4.1 STATIC ENVIRONMENTS
Dataset. We perform experiments all four scenes of the Waymo Interp. benchmark (Huang ct al. 2023) and follow the suggested protocol of holding out every 5th frame for validation. We use the front three cameras and render images and LiDAR scans at full resolution.
Baselines. We compare SimULi to Nerfacto Tanciik et al. 2023) as a -fate-of-the-art NeRF ap- proach and adapt 3DGUT (Wu et al. 2025b take LiDAR depth supervision in the form
8

---

Published as a conference paper at ICLR 2026
NeuRAD (1.08 MR/s)
SplatAD (1.24 MR/)
Ours (11.10 MR/3)
Ground Truth
Seone
Figure 7: PandaSet. SimULi renders the fastest by 100x Compared other joint camera-LiDAR methods, ours provides the sharpest LiDAR renderings, especially near vehicles.
NeuRAD (1.19 MR/S)
Lidar-KT (9.93 MB33)
SpInAD (2.94 MIAA)
Ours (18.56 M13/)
Ground Truth
1883
1006
Figure 8: Waymo Dynamic. As in Fig. SimULi renders the fastest (as measured on an A40 GPU) and compares favorably to camera- LiDAR and LiDAR-only Zhou et al. 2025) baselines.
Table 3: PandaSet Reconstruction. We outperform all prior work by a wide margin, improving
upon the second-best method (SplatAD) by > IdB PSNR while rendering camera views 60% faster.
Method
PSNKT
SSIM
LPIPS]
McdL.2.
Memn
1nt. RMSE,
RayDropt
CD
MPist
MRNs
OmniRR
Ches
2025
25,82
0.779
0,322
0.131
0.083
0.970
56.03
StreetS Tellgteli sura
0.750
0.162
PVG (hhree or N115
0.723
0.464
30,859
6.763
Defoemnagggy Yang st al.
2014
25.00
0.770
0.349
0.359
0.159
0.289
16.52
UniSim Yann ef al. 2123
23.53
0.693
0.334
01047
0.079
0.087
0.917
1.326
10.05
1.07
NeuRAD
2024
26.50
0.767
0.241
0.006
0.044
0.055
0,973
0
9.71
1.10
SplaiAD
1.22
SimULi
29.76
0.881
0.195
0
0.006
0.034
0.997
0.206
137.74
11,10
of sparse depth maps. We also compare 3DGRT Moenne-Loccwe et al. 2024
AV-centric
methods in srivestudio (Chen et al.) 2025 Yan et al. 2024; Chen et al. 2023 Yang et al, 2024 and LiDAR-RT (Zhou al. 2025) as a LiDaR-only particle ray tracing method. We further measure SplatAD (Hess et al., 2025), NeuRAD (Tonderski et al. 2024 and neu couurrddstudii o's UniSim (Yang al.] 2023b) implementation as joint camera-L.iba L.DAR baselines, along with the of- ficial AlignMiF (Tang et al., 2024) implementation as a NeRF-based method that also addresses camera-LiDAR misalignment in static scenes.
Metrics. We evaluate image quality through PSNR, SSIM (Wagg et al. 2004). and the AlexNet variant of LPIPS (Zhang al. 2018). We list the median absolute depth error, mean relative depth accuracy, and chamfer distance of LiDAR predictions in meters, and intensity and ray drop accuracy for methods that support it. As 3DGRT and LiDAR-RT need ray tracing cores, we measure camera and LiDAR rendering speed (denoted in millions of pixels/rays per second) on an NVIDIA A40.
Results. We present results Table and qualitative examples in Fig. Visually, SimULi outper- forms all methods by >dd PSNR, and is best or nearly- across all other metrics. It outperforms LiDAR-RT (Zhou et al.] 2025). which solely targets LiDAR reconstruction, on all metries except ray drop accuracy (for which LiDAR-RT uses U-Net refinement network) and renders > 10x faster. Although our camera rendering builds upon 3DGUT (Wu et al. 2025b), the improvements in our pipeline (anchoring, appearance variation, environment map) improve rendering quality by 3dB PSNR and produce fewer "floaters", leading to slightly faster rendering. Compared to Align- MiF (Tag et al. 2024). which addresses camera- -umera-LiDAR misalignment via feature fusion that incurs a large inference- diference-time cost, our factorization actually improves rendering speed (as we only rasterize the subset of Gaussians relevant the sensor being rendered).

---

Published as a conference paper at ICLR 2026
Table 4: PandaSet NVS. Similar to Table
SimULi renders the fastest, provides the best PSNR, and outperforms next-best method SplatAD across every LiDAR metric.
Method
PSNKT
SSIM
LPIPS]
Medl.2.
MemuKell.2
Inr. RMSE,
KayLhopt
CD
MPPNT
MR3et
OmniRe Chea et ei 2025
25.13
0.757
0.351
0.425
0.113
1.126
53,19
StreetiS
0.756
0.318
64.67
PVG
Kho
0.709
0.462
30/666
Defoe
Yang st 2024
0.720
0.346
0.100
UniSim
Yann al. 2023
23.50
0.693
0.328
0.085
0.120
0.087
0.919
1.488
11.25
0.87
NeuRAU7 Lonnerii
2024
25.97
0.759
0.243
0.009
0.075
0.062
0:963
0.360
11.37
1.08
SplatAD
26.73
0.815
0.193
0.011
0.035
0.859
0.966
0.346
88.79
1.24
SimULi
4.2 DYNAMIC ENVIRONMENTS
We measure reconstruction on the same PandaSet scenes as SplatAD (Hess al. 2025 and novel view synthesis both PandaSet and the Waymo Dynamic benchmark (Wu et al. 2024). We compare to the methods in Sec. 4.1 that handle dynamics.
Datasets and Baselines.
Metrics. We evaluate Waymo Dynamic with the same metrics used in Sec. 4.1 On PandaSet, we measure timings on 80GB NVIDIA A100 GPUs to ensure that our comparisons are as close as possible to the setting described SplatAD (Hess et 2025). the baseline closest to our work.
We list reconstruction results in Table 3 and novel view synthesis in Tables and We provide camera visuals in Fig. and LiDAR in Figs. and In all cases, SimULi provides the best visual and depth quality. Compared to SplatAD (Hess et al. 2025), the method closest to ours, improve PSNR by 0.4-1.7 dB without relying on CNNs for view dependence, render cameras 1.5-3x faster, and accelerate LiDAR by 10x.
Results.
4.3 DIAGNOSTICS
Representation. We compare our factorized model to the alternative encoding all sensors into single particle set and vary the strength of LiDAR depth loss Xa (Fig. 4). We also ablate the bilateral grids and environment map used to improve camera rendering. We summarize results in Table 5 Not only does anchoring improve NVS compared to camera-only reconstruction (Aa = 0), but it outperforms the unified strategy across all metrics for all values of Ad, and renders LiDAR 2x faster. Bilateral grids and environment maps improve camera quality to different degrees.
LiDAR Rendering. We measure rendering speed on single scene while varying the maximum number of points per tile M and the number of elevation tiles Ng. The choice . 32, N₂- 16 gives the best LiDAR rendering speed (note that does not affect quality). This highlights the benefits of our automated strategy we can easily find optimal settings for any sensor via straightforward grid search, instead of manually specifying tile boundaries as in prior work (Hess et al. 2025).
5 CONCLUSION
We propose SimULi, the first method that renders arbitrary camera models and LiDAR sensors in real-time. Our automated LiDAR tiling and ray-based culling accelerate rendering > 10x relative to prior work. Our factorized representation greatly improves multi-sensor reconstruction quality and of interest to many downstream applications.
Table 5: Ablations. NVS metrics averaged across PandaSet.
Table 6: LiDAR Tiling (MR/s).
Method
Facterized Gonveians
Bibseral
Grd
Emy Map
PSNR1
CDL
MPNs|
MRN1
N₂ \
256
128
64
32
Direct Az
26.61
6.248
134.75
5,55
Direct A2 00001
26.70
0.718
125.22
5.01
7.22
14.24
Direct A2 0001
26.34
0.475
132.45
4,97
32
5.48
7.89
13.87
13.51
wio Bilatend Cold
Direct A. n.
v
x
<
<
25.99
25.78
0.303
0.337
131.04
150,63
12.07
4.73
16
6.49
12.02
13.64
15.75
wio Em. Map
x
26.81
01.33
125,30
11.53
9.24
15.12
Fall Method
27.12
0.331
136.33
11.02
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
Our approach facilitates the creation and rendering of high-fidelity neural representations. Conse- quently, the inherent risks mirror those found in other neural rendering research, mainly concerning privacy and security due to the potential capture of sensitive information, whether intentional or not. These concerns are particularly relevant in autonomous vehicle settings, which method targets, as training data may contain private details like human faces and vehicle license numbers. A poten- tial solution would be incorporate semantic distillation into the model's representation, as seen in prior work (Kobayashi et al., 2022 Tschernezki et al. 2022 Turki et al. [023]] KeIr et al. 2023 to filter out sensitive data during rendering. However, this information would still persist within the model itself. A more effective mitigation strategy would be to preprocess the input data used to train the model before it is ever ingested (Wang et al. 2017
REPRODUCIBILITY STATEMENT
We describe how to build our pipeline the paper and appendix, provide pseudo-code for our LiDAR tiling and culling strategies, list our training hyperparameters, and detail our experimental protocol. All data used for evaluation publicly available.
ACKNOWLEDGMENTS
We thank Nicolas Moenne-Loccon for assistance with renderer and the SplatAD authors for discussing their method and evaluation protocol.
REFERENCES
Sai Praveen Bangaru, Lifan Wu, Tzu-Mao Li, Jacob Munkberg, Gilbert Bernstein, Jonathan Ragan- Kelley, Frédo Durand, Aaron Lefohn, and Yong He. Slang.d: Fast, modular and differentiable shader programming. ACM Trans. Graph., 42(6), December 2023. ISSN 0730-0301. doi: 10.
1145/3618353. https:/ /owi///oii/////oo www.o.ggg..... /10. 1451 361 8353
Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR, 2020.
Yurui Chen, Chun Gu, Junzhe Jiang, Xiatian Zhu, and Li Zhang. Periodic vibration gaussian: Dynamic urban scene reconstruction and real-time rendering. arXiv preprint 1.X1522311.18561, 2023.
Ziyu Chen, Jiawei Yang, Jiahui Huang, Riccardo de Lutio, Janick Martinez Esturo, Boris Ivanovic, Or Litany, Zan Gojeic, Sanja Fidler, Marco Pavone, Li Song, and Yue Wang. Omnire: Omni urban scene reconstruction. In ICLR, 2025.
Tobias Fischer, Samuel Rota Buld, Yung-Hsu Yang, Nikhil Varma Keetha, Lorenzo Porzi, Norman Müller, Katja Schwarz, Jonathon Luiten, Marc Pollefeys, and Peter Kontschieder. FlowR: Flow- ing from sparse dense 3d reconstructions. arXiv preprint arXiv:2504.0 0165004..167,,... 2025.
Jianfei Guo, Nianchen Deng, Xinyang Li, Yeqi Bai, Botian Shi, Chiyu Wang. Chenjing Ding, Dongliang Wang, and Yikang Li. Streetsurf: Extending multi-view implicit surface reconstruction to street views. arXiv preprint 2rXiv:2306.04988 2023.
Georg Hess, Carl Lindström, Maryam Fatemi, Christoffer Petersson, and Lennart Svensson. Splatad: Real-time lidar and camera rendering with 3d gaussian splatting for autonomous driving. In CVPR, 2025.
Shengyu Huang, Zan Gojeic, Zian Wang. Francis Williams, Yoni Kasten, Sanja Fidler, Konrad Schindler, and Or Litany. Neural lidar fields for novel view synthesis. In ICCV, 2023.
Junzhe Jiang, Chun Gu. Yurui Chen, and Li Zhang. Gs-lidar: Generating realistic lidar point clouds with panoramic gaussian splatting. In ICLR, 2025.

---

Published as a conference paper at ICLR 2026
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), July 2023.
URL https://repo- sam http://ww.......uuua///////aaaa/
swwattinsito
Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. In ICCV, 2023.
Shakiba Kheradmand, Daniel Rebain, Gopal Sharma, Weiwei Sun, Yang-Che Tseng, Hossam Isack, Abhishek Kar, Andrea Tagliasacchi, and Kwang Moo Yi. 3d gaussian splatting markov chain monte carlo. Advances in Neural Information Processing Systems, 11:80965-8098 2024.
Sosuke Kobayashi, Eicchi Matsumoto, and Vincent Sitzmann. Decomposing nerf for editing via feature field distillation. In Advances in Neural Information Processing Systems, volume 35, 2022.
Yiyi Liao, Jun Xie, and Andreas Geiger. KITTI-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d. arXiv preprint UrXiv:2109.13410 2021.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020.
Nicolas Moenne-Loccoz, Ashkan Mirzaei, Perel, Riccardo de Lutio, Janick Martinez Esturo, Gavriel State, Sanja Fidler, Nicholas Sharp, and Zan Gojeic. 3d gaussian ray tracing: Fast tracing of particle scenes. ACM Transactions on Graphics and SIGGRAPH Asia, 2024.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics prim- itives with a multiresolution hash encoding. ACM Trans. Graph. 41(4)::02:11102:555 July 2022. doi: 10.11457332822333330177
URL https:/ htta:://doll. wtt..o//.....11133555353331
3530127
Julian Ost, Fahim Mannan, Nils Thuerey, Julian Knodt, and Felix Heide. Neural scene graphs for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2856-2865. June 2021.
Paszke. Pytorch: An imperative style, gigh-performance deep learning library. arXiv preprint arXiv:1912.01703, 2019.
Konstantinos Rematas, Andrew Liu, Pratul P. Srinivasan, Jonathan Barron, Andrea Tagliasacchi, Tom Funkhouser, and Vittorio Ferrari. Urban radiance fields. 2022.
Michael Steiner, Thomas Köhler, Lukas Radl, Felix Windisch, Dieter Schmalstieg, and Markus Steinberger. Aaa-gaussians: Anti-aliased and artifact-free 3d gaussian rendering, 2025. URL
2504. 12811
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov. Scalability in perception for autonomous driving: Waymo open dataset. In CVPR, June 2020.
Matthew Tancik, Ethan Weber, Evonne Ng. Ruilong Li, Brent Yi, Justin Kerr, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, David McAllister, and Angjoo Kanazawa. Nerfstudio: modular framework for neural radiance field development. In ACM SIGGRAPH 2023 Conference Proceedings, SIGGRAPH '23, 2023.
Tao Tang, Guangrun Wang, Yixing Lao, Peng Chen, Jie Liu, Liang Lin, Kaicheng Yu, and Xiaodan Liang. Alignmif: Geometry Geometry-aligned multimodal implicit field for lidar-camera joint synthesis. 2024.
Tang Tao, Longfei Gao. Guangrun Wang, Yixing Lao, Peng Chen, Zhao hengshuang, Dayang Hao, Xiaodan Liang, Mathieu Salzmann, and Kaicheng Yu. Lidar-nerf: Novel lidar view synthesis via neural radiance fields. arXiv preprint arXiv:2304. 100600060000 2023.
12

---

Published as a conference paper at ICLR 2026
Adam Tonderski, Carl Lindström, Georg Hess, William Ljungbergh, Lennart Svensson, and Christoffer Petersson. Neurad: Neural rendering for autonomous driving. In CVPR, pp. 14895- 14904, 2024.
Vadim Tschernezki, Iro Laina, Diane Larlus, and Andrea Vedaldi. Neural Feature Fusion Fields: 3D distillation of self-super 2D image representation. In Proceedings of the International Conference on 3D Vision (3DV), 2022.
Haithem Turki, Jason Y Zhang, Francesco Ferroni, and Deva Ramanan. Suds: Scalable urban dynamic scenes. In Computer Vision and Pattern Recognition (CVPR), 2023.
Haithem Turki, Vasu Agrawal, Samuel Rota Bulò, Lorenzo Porzi, Peter Kontschieder, Deva Ra- manan, Michael Zollhöfer, and Christian Richardt. Hybridnerf: Efficient neural rendering via adaptive volumetric surfaces. In Computer Vision and Pattern Recognition (CVPR). 2024.
Junjue Wang, Brandon Amos, Anupam Das, Padmanabhan Pillai, Norman Sadeh, and Mahadev Satyanarayanan. A scalable and privacy-aware IoT service for live video analytics. In Multimedia Systems Conference, pp. 38-49,2250 2017.
Yuehao Wang, Chaoyi Wang, Bingchen Gong, and Tianfan Xue. Bilateral guided radiance field processing. ACM Transactions on Graphics (TOG), 43(4):1-13, 2024.
Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions Image Processing, 13(4)x600-612, 2004. doi: 10.1109/TP.2003.880
Zian Wang, Tianchang Shen, Merlin Simier-David Nicholas Sharp, Jun Gao, Alexander Keller, Sanja Fidler, Thomas Müller, and Zan Gojeie. Adaptive shells for efficient neural radiance field rendering. ACM Trans. Graph., 42(6), 2023. doi: 011145/3618990 URL https: //doi.
org/10. 1145//6139/3
Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khandel- wal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, Deva Ramanan, Peter Carr, and James Hays. Argoverse 2: Next generation datasets for self-driving perception and forecasting. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS Datasets and Benchmarks 2021), 2021.
Hanfeng Wu, Xingxing Zuo, Stefan Leutenegger, Or Litany, Konrad Schindler, and Shengyu Huang. Dynamic lidar re-simulation using compositional neural fields. In CVPR, 2024.
Jay Zhangjie Wu, Yuxuan Zhang, Haithem Turki, Xuanchi Ren, Jun Gao, Mike Zheng Shou, Sanja Fidler, Zan Gojeic, and Huan Ling. Difix3d+: Improving 3d reconstructions with single-step diffusion models. 2025a.
Qi Wu, Janick Martinez Esturo, Ashkan Mirzaci, Nicolas Moenne- and Zan Gojeic. 3dgut: Enabling distorted cameras and secondary rays in gaussian splatting. CVPR, 2025b.
Pengchuan Xiao, Zhenlei Shao, Steven Hao, Zishuo Zhang, Xiaolin Chai, Judy Jiao, Zesong Li, Jian Wu, Kai Sun, Kun Jiang. Yunlong Wang, and Diange Yang. Pandaset: Advanced sensor suite dataset for autonomous driving. In 2021 IEEE International Intelligent Transportation Systems IEEE Press, 2021. doi: 10. 1091000 20029988.021.9995 995500999.9 www.1100//T/49993 2021 9565009
Conference (ITSC), pp. URL https::/ddoo
Ziyang Xie, Junge Zhang, Wenye Li, Feihu Zhang, and Li Zhang. S-nerf: Neural radiance fields for street views. In International Conference on Learning Representations (ICLR), 2023.
Yunzhi Yan, Haotong Lin, Chenxu Zhou, Weijie Wang, Haiyang Sun, Kun Zhan, Xianpeng Lang, Xiaowei Zhou, and Sida Peng. Street gaussians: Modeling dynamic urban scenes with gaussian splatting. In ECCV, 2024.
Jiawei Yang. Boris Ivanovic, Or Litany, Xinshuo Weng, Seung Wook Kim, Boyi Li, Tong Che, Danfei Xu, Sanja Fidler, Marco Pavone, and Yue Wang. Emerner:: Emergent spatial- ppatirlltemprra scene decomposition via self-supervision. arXiv preprint arXiv:231 1.02077, 2023a.
13

---

Published as a conference paper at ICLR 2026
Ze Yang, Yun Chen, Jingkang Wang, Sivabalan Manivasagam, Wei-Chiu Ma, Anqi Joyce Yang, and Raquel Urtasun. Unisim: A neural closed-loop sensor simulator. In CVPR, 2023b.
Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. In CVPR, 2024.
Zehao Yu, Anpei Chen, Binbin Huang, Torsten Sattler, and Andreas Geiger. Mip-splatting: Alias- free 3d gaussian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19447-19456, June 2024.
Junge Zhang, Feihu Zhang, Shaochen Kuang, and Li Zhang. Nerf-lidar: Generating realistic lidar point clouds with neural radiance fields. In AAAI, 2024.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiivnees of deep features as perceptual metric. In CVPR, 2018.
Zehan Zheng, Fan Lu, Weiyi Xue, Guang Chen, and Changjun Jiang. Lidar4d: Dynamic neural fields for novel space-time view lidar synthesis. CVPR, 2024.
Chenxu Zhou, Lvchang Fu, Sida Peng, Yunzhi Yan, Zhanhua Zhang, Yong Chen, Jiazhi Xia, and Xiaowei Zhou. LiDAR-RT: Gaussian-basee ray tracing for dynamic lidar re-simulation. In CVPR, 2025.
Xiaoyu Zhou, Zhiwei Lin, Xiaojun Shan, Yongtao Wang, Deqing Sun, and Ming- Ming-Hsuan Yang. Driv- inggaussian: Composite gaussian splatting for surrounding dynamic autonomous driving scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21634-21643, 2024.
14


Reviews:
[{"review_id": "dsBhAlOUVr", "reviewer": "Reviewer_6a2S", "summary": "This paper introduces a 3DGS-based simulator for real-time camera and LiDAR rendering in autonomous driving. The main contributions include a non-equidistant tiling strategy that efficiently handles arbitrary spinning LiDAR sensors, and a factorized 3D Gaussian representation that mitigates cross-sensor inconsistencies between camera and LiDAR modalities, thereby improving rendering realism. The experiments are comprehensive, with extensive comparisons on the Waymo and PandaSet datasets, demonstrating state-of-the-art performance and realism.", "questions": "* The paper claims improved cross-sensor consistency, but how does the method perform when modeling more accurate physical effects such as rolling shutter, motion blur, or calibration errors? This could also be evaluated under controlled conditions, for example using CARLA.\n* Can the proposed LiDAR tiling strategy generalize to non-spinning LiDAR sensors, such as solid-state LiDARs, where the sampling pattern is different?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper is well written and clearly motivated.\n* Experiments are thorough and cover multiple datasets and baselines. The proposed method achieves strong quantitative and qualitative performance, demonstrating competitive or superior rendering realism and efficiency.", "weaknesses": "* The novelty is limited. The method feels like a natural extension of 3DGUT, and the way LiDAR rendering is supported is conceptually similar to SplatAD.\n* The improvement in handling the camera–LiDAR accuracy tradeoff mainly comes from the decoupled representation, but the deeper issue—imperfect sensor modeling (e.g., motion blur, rolling shutter, or calibration)—is not really addressed. Prior work such as NeuRAD has shown that explicitly modeling these effects can significantly improve reconstruction quality.\n* This paper also omits related efforts such as AlignMiF, which also tackle multimodal alignment in autonomous driving simulation. A discussion or comparison with AlignMiF would make the contribution clearer and better positioned.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N73ENvFJyJ", "reviewer": "Reviewer_u66L", "summary": "this paper introduces a simulator method for autonomous driving that can render both complex camera models and LiDAR data in real time. It builds on 3DGUT and improves cross-sensor consistency, making it more accurate and faster than existing methods. The paper conducted experiments on two public datasets and show state-of-the-art performances.", "questions": "1. From what I understand, the authors equalize the elevation tiling using a 1D CDF of elevation angles, and then reuse the same azimuth tiling across the whole scan. Wouldn’t that implicitly assume that the LiDAR point distribution is separable between elevation and azimuth, and also static over time? In practice, the point density may vary a lot with azimuth (depending on the scene or motion), so a fixed azimuth tiling might lead to load imbalance — some tiles being overloaded and others almost empty. Did you observe this issue in your experiments, or did you use any mechanism to adapt the azimuth tiling dynamically?\n2. Since each camera Gaussian is softly constrained to stay close to its nearest LiDAR neighbor, how sensitive is the method to that K-nearest-neighbor choice (K = 50)? Also, do you ever observe issues where the anchoring loss pulls camera Gaussians toward noisy or missing LiDAR points, especially around thin structures or reflective surfaces?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well written and easy to follow.\n2. The proposed method consistently outperforms baselines according to both visualizations and tables. \n3. The proposed method achieves significantly faster rendering speed than baselines.\n4. The  factorized representation is interesting and innovative, which improves both the camera and lidar rendering accuracy.", "weaknesses": "would the factorized representation largely increase the total number of Gaussians in the scene? Does this lead to significantly higher memory usage compared to unified representations?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FuPzwWL3k9", "reviewer": "Reviewer_DQih", "summary": "SimULi describes a framework for real-time simulation of LiDAR and camera for autonomous driving scenes. The focus of this work is to improve the rendering speed and support arbitrary LiDAR and camera configurations. To do so, this work builds on 3DGUT to support arbitrary spinning LiDAR configuration. Furthermore, a factorized 3D Gaussian representation and anchoring strategy was proposed to address discrepancies between simulated LiDAR and camera data. The proposed method was benchmarked on two datasets to showcase the fidelity and efficiency in relation to existing works.", "questions": "1.\tHow does factorizing the Gaussian set impact training speed? How does the training time of the proposed work compare against existing works?\n2.\tFor the anchoring loss, the choice of 50 nearest neighbors and updating assignments every 1000 iterations are hyperparameters. How does varying the number of nearest neighbors and update frequency impact convergence?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper is well-written and easy to follow. The limitations of prior works and their relation to the proposed work are clearly highlighted.\n2.\tThe proposed method demonstrates strong rendering fidelity and significant boosts in speed compared to state-of-the-art methods.", "weaknesses": "1.\tThe proposed method has been evaluated on a relatively limited set of datasets. Common benchmarks used in prior works, such as nuScenes and Argoverse 2, would help demonstrate the robustness of the method across datasets and sensor setups.\n2.\tSplatAD encodes all sensor information into the same Gaussian set. SimULi proposes to encode each sensor into its own particle set to address the inconsistencies between LiDAR and camera. The impact of this on the training time and potentially memory requirements is not discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cLT1vso6Dw", "reviewer": "Reviewer_HQsC", "summary": "SimULi extends 3DGUT by incorporating LiDAR support, enabling simultaneous modeling of multimodal sensors in autonomous driving scenarios. The system achieves performance on both camera and LiDAR modalities that matches or surpasses the current state of the art.", "questions": "- How is LiDAR integrated into 3DGUT? Are the seven points projected and weighted following the LiDAR’s ray model? If so, could the authors provide a visualization similar to Fig. 7 in 2DGS to illustrate the validity of this approximation? As nonlinearity of LiDAR projection is typically stronger than that of camera projection.\n\n - Is the interaction between camera and LiDAR Gaussian primitives only achieved through the anchor loss? Would such a weak coupling lead to redundant Gaussian points?\n\n - Could the proposed approach be combined with recent generative-aided reconstruction methods (e.g., StreetCrafter, DriveX)?\n\n - Which component contributes most to the significant improvement in rendering speed?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Achieves joint modeling of multimodal sensors (camera and LiDAR) in autonomous driving scenes, with each modality outperforming existing single-modality methods.\n\n - Incorporates modeling of fisheye cameras and rolling-shutter effects, demonstrating a highly comprehensive design.\n\n - Significantly improves rendering speed.", "weaknesses": "- The interaction between camera and LiDAR Gaussian primitives relies solely on the anchor loss, which may lead to redundant Gaussian points.\n\n - The system demonstrates strong engineering merit, but the novelty is relatively limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)