WorldSplat: Gaussian-Centric Feed-Forward 4D Scene Generation for Autonomous Driving
Authors: Ziyue Zhu
Zhanqian Wu
Zhenxin Zhu
Lijun Zhou
Haiyang Sun
BING WANG
Kun Ma
Guang Chen
Hangjun Ye
Jin Xie
jian Yang
Venue: ICLR 2026 Poster (2026)
Keywords: Autonomous Driving
Video Generation
Primary area: applications to robotics, autonomy, planning

Abstract: Recent advances in driving-scene generation and reconstruction have demonstrated significant potential for enhancing autonomous driving systems by producing scalable and controllable training data. Existing generation methods primarily focus on synthesizing diverse and high-fidelity driving videos; however, due to limited 3D consistency and sparse viewpoint coverage, they struggle to support convenient and high-quality novel-view synthesis (NVS). Conversely, recent 3D/4D reconstruction approaches have significantly improved NVS for real-world driving scenes, yet inherently lack generative capabilities. To overcome this dilemma between scene generation and reconstruction, we propose \textbf{WorldSplat}, a novel feed-forward framework for 4D driving-scene generation. Our approach effectively generates consistent multi-track videos through two key steps: ((i)) We introduce a 4D-aware latent diffusion model integrating multi-modal information to produce pixel-aligned 4D Gaussians in a feed-forward manner. ((ii)) Subsequently, we refine the novel view videos rendered from these Gaussians using a enhanced video diffusion model. Extensive experiments conducted on benchmark datasets demonstrate that \textbf{WorldSplat} effectively generates high-fidelity, temporally and spatially consistent multi-track novel view driving videos.

Published as a conference paper at ICLR 2026
WORLDSPLAT: GAUSSIAN-CENTRA FEED-FORWARD 4D SCENE GENERATION FOR AUTONOMOUS DRIVING
Ziyue Zhu 12, Zhanqian Wu", Zhenxin Zhu? Lijun Zhou², Haiyang Sun?, ,ing Wang", Kun Ma², Guang Chen', Hangjun Ye?, Jin Xie Jian Yang
PPA Lab, VCIP, College of Computer Science, Nankai University Xiaomi EV SScoool of Intelligence Science and Technology, Nanjing University, Suzhou
zhuzi zhuziyue@mail www.nanniiie.mmm cn h csjxieenjj.eduu.cm
/ csjyangenankai. edu..
Project Page: http://m-rreee
ABSTRACT
Recent advances in driving-scene generation and reconstruction have demonstrated significant potential for enhancing autonomous driving systems by producing scalable and controllable training data. Existing generation methods primarily focus on synthesizing diverse and high-fidelity driving videos; however, due to limited 3D consistency and sparse viewpoint coverage, they struggle to support convenient and high-quality novel-view synthesis (NVS). Conversely, recent 3D/4D reconstruction approaches have significantly improved NVS for real-world driving scenes, yet inherently lack generative capabilities. To overcome this dilemma between scene generation and reconstruction, we propose WorldSplat, novel feed-forward framework for 4D driving-scene generation. Our approach effectively generates consistent multi-trace track videos through two key steps: (i) We introduce a 4D-aware latent diffusion model integrating multi- information to produce pixel-aligned 4D Gaussians refine the novel view videos rendered from these Gaussians using enhanced video diffusion model. Extensive experiments conducted on benchmark datasets demonstrate that WorldSplat effectively generates high-fidelity, temporally and spatially consistent multi-track novel view driving videos.
cerw-forward manner. (ii) Subsequently, we
INTRODUCTION
Synthesizing realistic driving-scene videos with controllable viewpoints is key challenge in au- tonomous driving and computer vision, crucial scalable training and closed-loop evaluation.
Recent generative models (Mao et al., 2024; Gao et al.) 2023 Wen et al.) 2024) have advanced
high-fidelity, user-defined video generation, reducing reliance on costly real data. Mcanwhile, urban
scene reconstruction methods (Chen et al.) 2024c) Yan et al. 2024) have improved 3D representa- tions (Mildenhall et al. 2021 Kerbl et al. 2023) and consistent novel-view synthesis.
Despite advancements, generation and reconstruction approaches face dilemma between unseen- environment creation and novel view synthesis. Existing video generators (Mao et al.) 2024 Gao
et al. 2023 Wen et al. 2024 Li et al. 2024at Gao et al. 2024b) work in the 2D image domain and often lack 3D consistency and novel-view controllability: they may look plausible from one angle but fail to stay coherent when generating from new viewpoints. Meanwhile, scene reconstruction methods (Yang et al. 2023 Yan et al.] 2024) Chen et al. 2024c) achieve accurate 3D consistency and photorealistic novel views from recorded driving logs. yet they lack generative flexibility, being unable to imagine scenes beyond the captured data. Although video generation followed by reconstruction (Gao et al. 2024a Lu al. 2024) Mao et al., (024) feasible, the quality of the resulting novel views remains constrained by both processes. Thus, bridging generative imagination with faithful 4D reconstruction remains an open challenge.
Corresponding authors: Jian Yang and Jin Xic.

---

Published as a conference paper at ICLR 2026
Novel View Video Consisiency (X)
Novel View Video Consistency (V)
Vidao Diflusion
Conesians
Diffusion
Previons Video World Model
Fewd-foward 4D Gaussians
Ours
Figure 1: Comparison of different driving world models. Previous driving world models (Jiang et 2024 Gao et al. 2023) focus on video generation, while our method directly creates controllable 4D Gaussians in a ford-forward manner, enabling the production of novel-view videos (e.g. shifting ego trajectory =Nm) with spatiotemporal consistency.
To address these challenges, we introduce WorldSplat, a feed-forward framework that combines gener- ative diffusion with explicit 3D reconstruction for 4D driving-scens synthesis. Our framework creates a dynamic 4D Gaussian representation and renders the novel views along any user-defined camera trajectory without per-scene optimization. By embedding 3D awareness into the diffusion model and using an explicit Gaussian-centric world representation, our method ensures spatial and temporal consistency across novel trajectory views. As shown in Fig. prior driving world models (Gao et 2023; Mao et al., 2024) Jiang et al. 2024) produce realistic videos but often lose coherence when synthesizing novel- view sequences due to their stochastic nature. contrast, WorldSplat directly outputs a 4D Gaussian field in a single forward pass, enabling stable, consistent novel-view rendering. For flexible generation control, the framework supports rich conditioning incuui-nn road sketchs, textual descriptions, dynamic object placements, and ego makii it highly controllable simulator for diverse driving scenarios.
Specifically, WorldSplat operates in three stages. First, without relying on real images (Ren et al., 2025) or LiDAR (Wang et al.) 2024d), but only on flexible user-defined control conditions, a 4D- aware latent diffusion model generates multi-view, temporally coherent latents inherently containing RGB, metric depth, and semantic channels, offering perspective-aware 3D information both visual appearance and scene geometry. Next, latent Gaussian decoder converts these latents into an explicit 4D Gaussian scene representation. In particular, it predicts a set of pixel-aligned 3D Gaussians, which separated into static background and dynamic objects and then aggregated into unified 4D representation, explicitly modeling a dynamic driving world. For clarity, we visualize this representation in Appendix Fig. 7 and argue that it provides more suitable basis than point maps (Wang et al., 2025a Guo et al. 2025) for consistent novel-view video generation. We then apply fast Gaussian splatting to project the Gaussians into novel camera views, enabling novel-track video synthesis with true geometric consistency. Finally, enhanced diffusion refinement model applied to the rendered frames to further improve realism by correcting artifacts and enhancing fine details (e.g., texture and lighting), yielding high-fidelity novel-view videos. To summarize, our main contributions include:
Framework of 4D scene generation. We introduce WorldSplat, feed-forward 4D generative framework that unifies driving-scene video generation with explicit dynamic scene reconstruction.
Feed forward Gaussians decoder. We propose a dynamic aware Gaussian decoder that directly infers precise pixel-aligned Gaussians from multimodal latents and aggregates them into 4D Gaussian representation with static-dynamic decomposition.
Comprehensive evaluation. Extensive experiments show that our framework generates spatially and temporally consistent free-viewpoint videos, achieves state-of- oftte----the-rn art performance in driving video generation, and provides significant benefits for downstream driving tasks.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Driving World Models. Recent world models for autonomous driving (Gao et al.
2023 2024c
Li et al. 2024a Swerdlow et al. 2024 Hu et al.) 2023 Jiang et al. 2024 Li et al.
2024b Wang
et | (024f) have advanced the simulation of realistic street scenes, aiming
generate diverse synthetic data for robust driving systems. Most approaches rely on vision as the primary modality and
focus on generating high-fidelity driving videos. For instance, GATA-1 (Hu et al. 2023) synthesizes realistic driving scenarios, while DriveDreamer (Wang et al.] 2024e) learns policies from real- world data. Vista (Gao et al. (224c) scales to large driving datasets, and MagicDrive (Gao et al. 2023
ensures cross-camera consistency via attention mechanisms. Subsequent works (Swerdlow ct al.
2024 Huang al.
2024a Chen et al. 2024a Ma et al. 2024 Wen al., 2024 Guo et al. 2024
Zeng et al. 2025 further improve controllability video length, and visual quality.
Beyond video generation, recent studies explore 3D and 4D scene modeling (Li et 2024a
et al. 2024 Mao et al. [0244] Zheng et al. 2024 Wang et al. 2024b Ren et al. 2024 2025| Wang al. 2024a). MagicDrive3D Gao al. 2024a) supports multi conti-con 3D scene generation. while InfiniCube (Lu et al.) 2024) produces unbounded dynamic 3D) scenes. DreamDrive (Mao et al. 2024)
further extends to generalizable 4D generation. However, these approaches often rely on video- first pipelines, leading to reconstruction artifacts and sparse-view inconsistencies, underscoring the need for direct generation of coherent 3D/4D representations.
Urban Scene Reconstruction. Urban scene reconstruction and novel-view synthesis are commonly tackled with neural 3D representations (Mildenhall et al., 2021 Barron et al. 2021 Kerbl et al. 2023), but driving scenes remain difficult due to sparse views and dynamic objects. Gaussian-
methods (Yan et al. 2024 (Zhou et al. 2024a b Chen et al. 2024c) use bounding boxes to reconstruct
static and dynamic parts, while self-supervised methods (ang et al. 2023 Huang et al. 2024b| Chen iet al. 2023 Peng et al. (2024) decompose them automatically. Feed-forward approaches (Wei et al.
2024 Yang et al. [022aa] Wang et al. 20256 further speed up reconstruction by avoiding per-scene optimization. However, these works focus on reconstruction rather than generation. Our method bridges this gap, enabling feed-forward 4D scene generation with high- nigh-quality novel views.
3 METHOD
3.1 OVERVIEW
As illustrated in Fig. 2 our framework comprises three key modules: 4D-aware latent diffusion model (Sec. 3.2) for multi-mode latent generation, a latent Gaussian decoder (Sec. 3.3 for feed- forward 4D Gaussian prediction with real-time trajectory rendering, and enhanced diffusion model (Sec. 3.4 for video quality refinement. The three modules are trained independently. and their architectures and training procedures detailed in Secs. 3.2 3.4 Finally, Sec. 3.5 describes how these modules are integrated to generate high-fidelity, spatiotemporally consistent videos.
3.2 4D-AWARE DIFFUSION MODEL.
Given a noise latent and fine-grained conditions (i.e., bounding boxes, road sketch, captions and ego trajectory), the 4D-Aware Diffusion Model performs denoising to generate multi-modal latents encompassing RGB, depth, and dynamic object information, which are subsequently used for 4D Gaussian prediction. In the following, we elaborate on the latents, control conditions, model architecture, and training strategy.
Multi-modal latent integration. Given K- K-view driving video clip with T frames I = [ we first extract a multi-view image latent Limg = E(I) using a pretrained VAE encoder (hpcai tech 2024). Next, we generate metric depth maps 'D with foundation depth estimator (Hu et al. (024). normalize them to | 1, 1], replicate across three channels, and encode them into depth latent Leepth- Prior works (Wei et al. 2024 Go et al. 2024 Yang et al.) (024b) have shown that incorporating depth latents improves both 3D reconstruction and generation quality. To further separate static and dynamic objects for 4D scene reconstruction,
obtain semantic mask latent Leeg from binary segmentation masks of dynamic class objects using SegFormer (Xie et al., 202I). Finally, we concatenate the three latents channel-wise to form the decoder input L concate Limg: Leepth, Loog)-
3

---

Published as a conference paper at ICLR 2026
Surrounding- View Vidoos
Noise
Conditions
Enkanced Novel Videos
Dynamic
Layout
Capcion
Box
Trajectory
Depth Map
Reprojection
RCiB
E
4D-Aware Diffusion Model
Enhanced Ditlusion Model
Train
Blocks
Conditiins
Dynamic Go
Spatal
Tempporal
Up Block
Attenioon
Attention
Static Eis
Denoised Latent
Lateni Gaussians
Decoder
Pinel- Alignel 3D Caussians
Aggregatedd 4D Ganesians
Novel Track Rendering
Figure 2: The overview of our framework. (1) Employing a 4D-aware diffusion model to generate multi-modal latent containing RGB, depth, and dynamic information. (2) Predicting pixel-aligned 3D Gaussians from the denoised latent using our feed-forward latent decoder. (3) Aggregating the 3D Gaussians with synamic-statio decomposition to form 4D Gaussians and rendering novel-view videos. (4) Improving the spatial resolution and temporal consistency of the rendered videos with an enhanced diffusion model. "E" and "D" denote the VAE encoder and decoder, respectively. The arrow and the ones denote the train-only and inference.
Multi-Condittons Control. The diffusion transformer conditions structured cues, including BEV layout S, instance boxes B, ego trajectory T, and text descriptions D, denoted collectively C [S, T. D). Following MagicDrive (Gao al. 2023 2025), derive the layout, box, and trajectory inputs. For fine-grained caption control, we introduce DataCrufter, which segments a K-view video into clips, scores them with VLM evaluator (Wang al. (202cc), generates per-view captions, and fuses them via a consistency module. The resulting structured captions capture both scene context (weather, time, layout) and object details (category, box, description), ensuring temporal coherence and spatial consistency across views.
The architectures of our 4D-Aware Diffusion models are ControlNet- based (Chen 2023) transformers built upon OpenSora v1.2 (hpcai tech, (024)). Specifically, we extend OpenSora with dual-branch Diffusion Transformer: a main DiT stream for spatiotemporal video latents L and multi-block ControlNet branch for the conditions C. To ensure multi-view coherence, we replace standard self-attention with cross-view attention.
Architecture.
Each ControlNet block integrates road sketch latents E(S) from pretrained VAE (bpcai tech 2024 and text embeddings from a T5 encoder (Raffel et al., 2020). The 3D boxes, ego trajectory, and text features are further fused through cross-attention into unified scene-level signal, enabling fine-grained guidance and consistent video synthesis across time and viewpoints.
Training. In training stages, replace the standard IDDPM scheduler with an Rectified Flow model to improve stability and reduce the number of inference steps. Let x ~ Pdata be a real sample of the clean latent L and (O, be noise sample. We introduce continuous mixing parameter s E [0, 1] and define the interpolated state
2ss (1 (1-s)E
5) + sc
We then train neural field gv (z, C), conditioned C, to recover the target vector by minimizing
[(+b) Ez.e, (x (a |
(2)
At test time, we discretize sy = k/N for k = =N=N..... N....,
..... 1 and step backward via
1 gu (z(ss), C).
z (sk-1) x(se)
(3)
At inference, rather than using VAE decoder to reconstruct video frames from denoised latents, employ our Latent 4D Gaussian Decoder (Sec. 3.3) to directly predict the 4D Gaussians for novel view videos rendering.
4

---

Published as a conference paper at ICLR 2026
3.3 LATENT 4D GAUSSIANS DECODER
Our Gaussians Decoder predicts pixel-aligned 3D Gaussians from the multi-modal latents L (Sec. 3.2) We then leverage the semantic information within the latents to distinguish dynamic from static objects and reconstruct the 4D scene from the 3D Gaussians.
Our transformer-basee decoder (Dosovitskiy et al. 2020 Yang al. 2024a Zhang et al. 2024) consists of multiple cross-view attention blocks and temporal attention layers across Irames, followed by hierarchy of up-sampling blocks predict per-pixel Gaussian parameters. As shown in Fig. 2, this design captures the apatiottmppora dynamics of 4D scenes and directly outputs pixel-aligned 3D Gaussians from the multi-m molti-modal latent input L. To further enhance 3D spatial cues, we incorporate the Plücker (Plucker 1865) ray map P. which encodes pixel-wise ray origins R. and directions R. derived from camera intrinsics and extrinsics.
Architecture.
Each 3D Gaussian (Kerbl et al. 2023) is parameterized as g = (H,, 8, Ot, c). where l E R., r E R., 8 E _3, a É IR, and c E 1 denote center, quaternion rotation, scale, opacity, and color, respectively. The final decoder layer predicts per-pixel offsets 8, rotation r, scale opacity color c, depth d. and logits m for static- dynamic classification. The Gaussian center is then computed μ1 R. + d R. + 6, where 5 E R is the learned offset. This process yields sequence of Gaussian sets G and mask M indicating dynamic class objects over time, which can be compactly written as:
D& (Ling: Leepth: Leeg P) + ((G,, M,) E RR*HHW>(1411))
Compared to prior feed-forward scene reconstruction models (Wei et al. 2024 Yang et al. 2024a Charatan et al. [024 Chen et al., 2024b), our decoder supports over 48 simultaneous input views, enabling more comprehensive reconstruction of complex scenes.
4D Gaussians Aggregation. By merging the 3D Gaussian estimates from each frame, we form an scene model that remains temporally aligned and coherent. We adopt straightforward 4D reconstruction scheme: Given the known ego trajectory T, all 3D Gaussians are transformed into a unified coordinate system with ego-coordinate transformations. At each time step, we fuse the static Gaussians gathered from every frame with the dynamic Gaussians extracted from the current frame.
T GDD {(G M.) ((G.O(L-M))
(5)
By integrating data from multiple time steps, our decoder captures the scene's complete geometry, appearance, and motion, enabling rendering from both new spatial viewpoints and different moments.
Supervision and Loss functions. Note that our Gaussian Decoder predicts both pixel-aligned 3D Gaussians and semantic masks to distinguish dynamic from static regions. The predicted semantic masks are supervised by those generated from SegFormer (Xie et al., (021) using a binary cross- entropy loss. After assembling the 4D Gaussians with predicted masks across all observed timesteps, we project them onto set of target rendering timesteps. During training, we randomly select a base timestep / and sample T' target timesteps {4, {4) and extract the corresponding clean Latent as input. For each t,, we render RGB images R and depth images and supervise them with the corresponding ground-truth truth signals: RGB inputs I and metric depth maps (Hu et al. (024)). RGB reconstruction guided by a combination of photometric L1 loss and perceptual LPIPS loss (Zhang et al. 2018), while depth predictions are supervised with an I, loss in metric space. The overall training objective is defined as a weighted sum of these losses:
C = Lrocon + Aj Lipips + X2 Ldopth A Leeg
At inference, the 4D Gaussians generated by our pretrained Gaussian Decoder are used to render novel-view videos R' following customized ego trajectories T'.
3.4 ENHANCED DIFFUSION MODEL
The Enhanced Diffusion Model refines the RGB videos rendered from the 4D Gaussians, with the generation process conditioned on both the original inputs C and the rendered videos. This refinement
5

---

Published as a conference paper at ICLR 2026
enriches spatial details and enforces temporal coherence, yielding the final high-fidelity novel-view sequences.
Unseen Region
Enhanced
Figure 3: Effectiveness of the enhanced diffusion model. During novel-view video synthesis, rendering quality may degrade due to unobserved regions or high ago-vehicle speed, resulting in missing content and artifacts. Our enhanced diffusion model can inpaint unobserved areas and sharpen fast-motion frames.
Reconstruction via Restoration. As introduced in Sec. 3.3 our Latent Gaussians Decoder recon- structs 4D scenes feed-forward manner. However, inherent limitations of Gaussian splatting result in low-quality renderings for unobserved regions. Additionally, without per-scene optimization, novel-view reconstructions can become blurred under strong ego motion. To address these issues, we design an enhanced diffusion model to improve the quality.
As illustrated in Fig. for better understanding, the upper-left novel-view rendering omits sky regions due to occlusions, and the lower-left rendering appears blurred due strong ego motion. In contrast, our model substantially enhances the fidelity and clarity of the final renderings.
Architecture and Training. Its overall architecture and training strategy remain consistent with the 4la-Aarre Diffusion Model Sec. 3.2 The objective to refine the rendering results R. in the latent space, with the ground truth being {I)) Thus, the regression target is the image latent E()), consistent with common latent diffusion models (Jiang et al. 20244 Gao et al.) 20025. The training pipeline mirrors that of the 4D-Aware Diffusion Model, differing only in the control conditions [R, S, B, T, D} and the regression target.
Due the limitations of Gaussian splatting, novel-view renderings inference often appear inferior to the source views used for training. ReconDreamer (Ni et al., (2024) reduces this gap by training with degraded renderings, but relying solely on degraded inputs weakens alignment between conditions and outputs. We instead adopt mixed-conditioning strategy, combining degraded and high-qualin views to improve both controllability and generation fidelity.
3.5 FRAMEWORK INFERENCE PIPELINE
During inference, the 4D-Aware Diffusion Model takes noise latents with control conditions C and outputs the denoised latent La. The Gs decoder then predicts 4D Gaussians from Lg, which are rendered into novel-view videos R' based on a customized ego trajectory T'. Sketches and boxes reprojected S' and B3, forming control conditions C {R', S' B', T' D). Taking noise latent and conditons C' as input, the Enhanced Diffusion Model refines R', producing high-quality novel-view videos.
Customized Trajectory Selection. Building on the ego-pose perturbation strategy of FreeVS (Wang al. 2024d), we generate of novel tracks by laterally shifting the vehicle's path. Specifically, given the original ego-trajectory [ we apply officts Ay E {+1m) m, +2 m,m, 44m) m) along the vehicle's 's y-axis to produce six perturbed trajectories (T: + (0, Ay. O) For each perturbed path, aggregated 4D Gaussians render high-quality novel-view videos.
Furthermore, when reconstructing a scene from real driving videos, the 4D-Aware Diffusion Model bypassed, and the Gs Decoder directly takes the clean latent input.

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUPS
Dataset and Metrics. We conduct experiments on the nuScenes benchmark (Caesar et al. 2020), which contains 1,000 urban driving scenes annotated 2 Hz. We upsample the annotations (e.g.. bounding boxes and road sketches) to 12 Hz following (Wang et al.) (023). The model is trained on 700 scenes and validated 150. We evaluate generation quality using Fréchet Video Distance (FVD) (Unterthiner et al. 2019) and Fréchet Inception Distance (FID). For downstream evaluation, we measure the domain gap on perception tasks and assess how generated data improves perception model training.
VAE-1.2 Implementation Details, We adopt the pretrained OpenSSrraVVA--1.2 (hpcai tech 2024) as the backbone, fine-tuning only the cross-view attention blocks (Gao al. 2023) in the diffusion trans- former. More architectural and training details are provided in Sec.
4.2 ORIGINAL VIEW VIDEO GENERATION
Quantitative Comparison. In Tab. we report the quantitative results of our video synthesis approach under three different conditioning schemes: (i) no first-frame guidance, (ii) first-frame guidance, and (iii) noisy latent initialization. Across all scenarios, method consistently delivers the best scores on both the FVD and FID metrics.
Without first-frame guidance, our model achieves 74.13 FVD multi and 8.78 FID surpassing DriveDreamer-2 (Zhao et al., 2024), MagicDrive-V2 (Gao et al. 2025). and Panacea (Wen et al. 2024). Incorporating the first frame further boosts performance 16.57 FVD and 4.14 FID, par with or better than DriveDre while maintaining temporal smoothness and structural detail. Under the noisy-latent protocol (6,019 clips), method reaches 60.87 FVD and 6.51 FID, establishing a new state of the art over UniScene (Li et al. 2024a).
Table 1: Video generation comparison on the nuScenes CCaesar et al. 2020 validation set, with green and blue highlighting the best and second-best values, respectively.
Method
Gen. Mode
Backhone
Video
Novel View
Sample Num
FVD
FiD
DriveDreumer-2
w/o first cond w/o first cond
SD-based CogVideoX
X
105.10
25. 25.00
MagicDrive-V.
94.84
20.91
MagieDrive3D
w/o first cond
SD-v1.5
164.72
20.67
Panacea
first cond
SD-based
139.00
16.96
DIVE
w/o first cond
OpenSora-v1.2
X
5369
94.60
Ours
w/o first cond
OpenSora-vl.2
/
5369
74.13
8.78
CoGien
first cond first cond
OpenSora-v1.2
/
x
5369
68.43
10.15
DriveDreamer-2
SD-based
/
x
55.70
11.20
first cond
OpenSoru-v1.2
/
$369
16.57
Vista*
noisy latent
SVD
6019
112.65
13.97
UniScene
nuisy lalent DiT-bused
6019
70.52
6.12
Ours
noisy latent
OpenSora-v1.2
6019
60.84
6.51
Qualitative Comparison. In Fig. 4 we compare our generated videos with two leading methods: MagicDrive (Gao al. 2023) and Panacea (Wen al.) 2024). We also show the real samples and the control inputs. As the results demonstrate, our approach produces more accurate shapes and positions for dynamic objects, and achieves much better consistency across multiple views. Overall, our method captures rich details with high realism while maintaining strong agreement between different viewpoints.
4.3 NOVEL VIEW SYNTHESIS
Following FreeVS (Wang et al. 2024d), we evaluate our method on novel trajectories using the FID and FVD metrics. Specifically, translate the camera by offsets of +1 +2 m, and #4 m, then compute FID and FVD between the generated RGB frames along each shifted trajectory and the original ground-truth frames.
7

---

Published as a conference paper at ICLR 2026
Real
Conds
MagicD
Panacea
Ours
Figure 4: Comparison with MagicDrive (Gao et al. 2023 and Panacea (Wnn et al. 2024). The top row shows real frames, the second row the corresponding sketches and bounding-box controls. Red boxes highlight areas where our method achieves the most notable improvements.
Quantitative Comparison. In Tab. we compare WorldSplat with six baselines on nuScenes under viewpoint shifts of +1, 22 and +4 meters. WorldSplat consistently achieves the best FID/FVD across all shifts-for for example, at 1 m outperforms DiST-4D and OmniRe, and even at =44 m remains clearly ahead of all baselines. These results demonstrate the robustness and fidelity of our 4D Gaussian representation for novel-view synthesis under varying viewpoint shifts.
Table Quantitative results of novel-view synthesis, reporting FID and FVD under viewpoint shifts of +1, +2, and #4 meters. Baseline metrics are taken from DiST-4D (Guo et al. 2025
Shift 4:2m
Shift 44m
Method
FID
FVD 4
FID
FVD J
FID
FVD |
PVG (Chen et ul 2023
48.15
246.74
60.44
356.23
84,50
501.16
EmerNeKIF Yang a.. 2023
37.57
171.47
52.03
294.55
76.11
497.85
StrectGausstan
32.12
153.45
43.24
256.91
67.44
429.98
OmniRe (Chen èt 2024
31.48
152.01
43.31
254.52
67.36
428.20
FreeVS" Wang el 20244
$1.26
431.99
62.04
497.37
77.14
556.14
DiST-4D Guo et al
2025
10.12
45.14
12.97
68.80
17.57
105.29
Ours
8.25
11.26
47.41
13.38
64.07
Qualitative Comparison. Furthermore, Fig. offers a quantitative comparison with the of the-art urban reconstruction model (Chen 2024c). demonstrating that method delivers superior spatial consistency. Our renderings are sharper and more detailed: OmniRe often loses fine elements such lane markings and railings, whereas our approach preserves these features accurately. Moreover, our background reconstruction shows significant improvements over OmniRe.
4.4 ABLATION STUDY
In Tab. report FID and FVD for novel-view synthesis with +22 ego shift across six variants to systematically validate each component's 's contribution.
We systematically validate each component's contribution: (1) 3D Gaussians Representation (Version A + B): Introducing 3D Gaussians as explicit scene representation significantly improves performance with FVD dropping from 260.07 to 75.26 (- 184.81) and FID from 41.40 to 16.31
8

---

Published as a conference paper at ICLR 2026
Real
OmniRe
Ours
Figure 5: Qualitative comparison of novel view synthesis against the state-of teef-the-rr art urban reconstruction method (Chen et al., 2024c). We translate the ego-vehicle by #2 m to generate the novel viewpoints. Red boxes indicate where method achieves the greatest improvements.
Table Ablation study of novel-view generation: "C-Reprojection" reprojects boxes and sketches; "3D Gs" Gaussians from single-frame reconstructions: "4D Gs" Gaussians from multi- frame reconstructions: "Mixed Aug" mixes renderings of varying quality during training; "Enhanced" employs the enhanced diffusion model improve video quality.
Version
C-Reprojection
3D Gs
4D Gs
Mixed Aug
Enhanced
FVD + 2m +
FID + 2m
A
/ 260.07
/
41.40
75.26
16.31
/ 50.73
11.60
107.58
26.73
/ $1.64
12.07
/ 47.41
11.26
(-25.09), demonstrating that 3D Gaussians enable spatiotemporally consistent videos with effective
novel-view synthesis capability. (2) 4D Gaussian Aggregation (Version B - C): Aggregating single-frame 3D Gaussians into unified 4D representation further improves performance with FVD from 75.26 to 50.73 ( (-24.53) and FID from 16.31 to 11.60 (-4.71). Single-frame Gaussians are often too sparse, producing holes and aliasing novel views, while 4D aggregation enhances spatiotemporal consistency by densifying the representation across time in dynamic scenes. (3) Mixed Augmentation (Version
F): This contributes FVD improvement from 50.73 to 47.41 (-3.32) and FID from 1 11.60 to 11 11.26 (- 0.334). As discussed in Line 292 of our paper, novel-view renderings inference often appear inferior to source views: by degrading training source view quality through mixed augmentation, we improve the enhanced diffusion model's 's robustness and overall generation quality. (4) Enhanced Diffusion (Version - F): The Enhanced Diffusion module provides substantial improvements with FVD dropping dramatically from 107.58 to 47.41 (-60.17) and FID from 26.73 to 11.26 15.47). As detailed in Section 3.4 and illustrated in Figure 3, this module addresses inherent limitations of Gaussian Moaatting-lo-guueee renderings in unobserved regions

---

Published as a conference paper at ICLR 2026
and blurred reconstructions under strong ego motion without per-scene optimization effectively inpainting missing content and sharpening fast-motion frames. (5) Condition Reprojection (Version - F): Removing reprojection conditions leads to 8.9% increase in FVD (47.41 + 51.64) and 7.2% increase in FID (1 11.26 > 12.07). While the rendering views from our 4D Gaussians already provide strong conditions, the condition reprojection (Section 3.5) further enhances performance by enforcing explicit 3D geometric constraints across views and frames.
4.5 DOWNSTREAM EVALUTAION
Beyond visual fidelity, we assess the domain gap on downstream tasks-3D) 3D detection and BEV map segmentation-folllee
MagicDrive (Gao et al. 2023) (Tab. 4a). With a pretrained BEVFormer (L.
cct al. 2024c). our generated inputs achieve 38.49% % mIoU and 29.32% mAP, outperforming DiVE by
2.53% and 4.79%.
Further, following the experimental setup of Panacea (Wen et al. 2024), we generate a new training dataset based on nuScenes and integrate the generated data with real data train the StreamPETR (Wang et al.] 2023) model. (Tab, 4b) reports the 3D object detection results, showing that our approach provides larger improvements over the bascline compared to Panacea.
Table 4: The applications of our method on the downstream tasks.
(a) Domain gap validation of generated data on driving perception with pretrained Former.
(b) Performance gains achieved by incorporating gen- erated data into the training of the StreamPETR.
mioU?
mAP1
Method
mAPt
NDS1
MagicDrive (Gao et al 2023
18.
11.86
Real
34.5
46.9
DaiicDriveTDD (Gai
18.27
Punacea Wen et
2024
22.5
36.1
MagicDrive-V2 y2 Use al. 2023
20.40
18.17
DIVE Jiang
35.96
24.55
Real Panacen Wen cl
2024
37.1 (+2.6)
49.2 (+2.3)
Ours Real
38.49
29. 34
Ours
29.2
41.7
44.09
34.51
Real Ours
38.5 (+4.0)
50.1 (+3.2)
FAILURE CASE STUDY
Our framework can fail when the customized ego trajectory enters completely unobserved regions (e.g., shifting a trajectory inside a building whose interior is never visible from original viewpoints). Without geometric priors from Gaussians' rendering, the enhanced diffusion model produces artifacts or blank regions. This limitation is inherent to view synthesis from limited observations and affeets all reconstruction-Be priors for heavily occluded regions.
methods; future work could incorporate geometric inpainting or learned
CONCLUSION
In this work, we presented WorldSplat, novel feed-forward framework that unifies the strengths of generative and reconstructive approaches for 4D driving-scene synthesis. By integrating a 4D- aware latent diffusion model with enhanced diffusion network, method produces explicit 4D Gaussians and refines them into high-iidelity, temporally and spatially consistent multi-track driving videos. Extensive experiments standard benchmarks confirm that WorldSplat outperforms prior generation and reconstruction techniques in both realism and novel-view quality.
Ackmowledgement This work supported by the National Key R&D Program of China No. 2024YFC3015801, National Science Fund of China under Grant Nos. 62361 166670, U24A20330, and 62276144.
REFERENCES
Hassan Abu Alhaija, Jose Alvarez, Maciej Bala, Tiffany Cai, Tianshi Cao, Liz Cha, Joshua Chen, Mike Chen, Francesco Ferroni. Sanja Fidler, et al. with adaptive multimodal control. arXiv preprint arXiv:2503. /4492, 2025.
Conditional world generation
10

---

Published as a conference paper at ICLR 2026
Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Portin-Brualla, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti- anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF international conference computer viston, pp. 5855-5864, 2021.
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 1121--1631,, 2020.
David Charatan, Sizhe Lester Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 19457-19467, 2024.
al. Chen. Controlnet: Adding conditional control diffusion models, 2023. ArXiv preprint, available at
Rui Chen, Zehuan Wu, Yichen Liu, Yuxin Guo, Jingcheng Ni, Haifeng Xia, and Siyu Xia. Unimlvg: Unified framework for multi-view long video generation with comprehensive control capabilities for autonomous driving. arXiv preprint urXiv:24/2.04882. 2024a.
Yuedong Chen, Haofei Xu, Chuanxia Zheng, Bohan Zhuang, Mare Pollefeys, Andreas Geiger, Tat-Jen Cham, and Jianfei Cai. Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images. In European Conference on Computer Vision, pp. 370-386. Springer, 2024b.
Yurui Chen, Chun Gu, Junzhe Jiang, Xiatian Zhu, and Li Zhang. Periodic vibration gaussian: Dynamic urban scene reconstruction and real-time rendering. arXiv preprint 1rXiv:2311.18561 2023.
Ziyu Chen, Jiawei Yang, Jiahui Huang. Riccardo de Lutio, Janick Martinez Esturo, Boris Ivanovic, Or Litany, Zan Gojeic, Sanja Fidler, Marco Pavone, al. Omnire: Omni urban scene reconstruction. arXiv preprint arXiv:2408. 16760, 2024c.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 11999, 2020.
Ruiyuan Gao, Kai Chen, Enze Xie, Lanqing Hong. Zhenguo Li, Dit- Yan Yeung, and Qiang Xu. Mag- icdrive: Street view generation with diverse 3d geometry control. arXiv preprint arXhv:2310002611, 2023.
Ruiyuan Gao, Kai Chen, Zhihao Li, Lanqing Hong, Zhenguo Li, and Qiang Xu. Magicdrive3d: Con- trollable 3d generation for any-view rendering in street scenes. arXiv preprint 2rXiv:2405.14475. 2024a.
Ruiyuan Gao, Kai Chen, Bo Xiao, Lanqing Hong, Zhenguo Li, and Qiang Xu. Magicdrivedit: High- resolution long video generation for autonomous driving with adaptive control. arXiv preprint arXiv:2411.13807 2024b.
Ruiyuan Gao, Kai Chen, Bo Xiao, Lanqing Hong, Zhenguo Li, and Qiang Xu. MagicDrive-V22 High- resolution long video generation for autonomous driving with adaptive control. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2025.
Shenyuan Gao, Jiazhi Yang, Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li. Vista: A generalizable driving world model with high fidelity and versatile controllability. arXiv preprint arXiv:2405. /7398, 2024c.
Hyojun Go, Byeongjun Park, Jiho Jang, Jin- Young Kim, Soonwoo Kwon, and Changick Kim. Splatflow: Multi-view rectified flow model for 3d gaussian splatting synthesis. arXiv preprint arXiv:2411. 2024.
11

---

Published as a conference paper at ICLR 2026
Jiazhe Guo, Yikang Ding, Xiwu Chen, Shuo Chen, Bohan Li, Yingshuang Zou, Xiaoyang Lyu, Feiyang Tan, Xiaojuan Qi, Zhiheng Li, et al. Dist-4d: Disentangled spatiotemporal diffusion with metric depth for 4d driving scene generation. arXiv preprint arXiv:2303.15208 2025.
Xi Guo. Chenjing Ding, Haoxuan Dou, Xin Zhang. Weixuan Tang, and Wei Wu. Infinitddrive: Breaking time limits in driving world models. arXiv preprint arXiv:2412.01522. 2024.
hpcai tech.
Opensora-vae-vl..
https://huugghnnffccccc///pp ttcch
OpenSora - VAE v1.2 2024.
Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. Gaia-l: A generative world model for autonomous driving, arXiv preprint arXiv:2309. 17080, 2023.
Mu Hu, Wei Yin, Chi Zhang, Zhipeng Cai, Xiaoxiao Long, Hao Chen, Kaixuan Wang, Gang Yu, Chunhua Shen, and Shaojie Shen. Metric3d v2: A versatile monocular geometric foundation model for zero-shot metric depth and surface normal estimation. IEEE Transactions Puttern Analysis and Machine Intelligence, 2024.
Binyuan Huang, Yuqing Wen, Yucheng Zhao, Yaosi Hu, Yingfei Liu, Fan Jia, Weixin Mao, Tiancai Wang, Chi Zhang, Chang Wen Chen, et al. Subjectdrive: Scaling generative data in autonomous driving via subject control. ar preprint 2tth:2403.114388 2024a.
Nan Huang, Xiaobao Wei, Wenzhao Zheng, Pengju An, Ming Lu, Wei Zhan, Masayoshi Tomizuka,
Kurt Keutzer, and Shanghang Zhang. S3gaussian: Self-supervised street gaussians for autonomous driving. arXiv preprint arXiv:2405.20323, 2024b.
Junpeng Jiang, Gangyi Hong, Lijun Zhou, Enhui Ma, Hengtong Hu, Xia Zhou, Jie Xiang, Fan Liu, Kaicheng Yu, Haiyang Sun, al. Dive: Dit-based video generation with enhanced control. arXiv preprint 4rXiv:2-09.015955 2024.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4):139-1, 2023.
Bohan Li, Jiazhe Guo, Hongsi Liu, Yingshuang Zou, Yikang Ding. Xiwu Chen, Hu Zhu, Feiyang Tan, Chi Zhang, Tiancai Wang, et al. Uniscene: Unified occupancy-centric driving scene generation. arXiv preprint arXiv:2412.054433 2024a.
Xiaofan Li, Yifu Zhang, and Xiaoqing Ye. Drivingdiifusion: Layout-guided multi-view driving scenarios video generation with latent diffusion model. In European Conference on Computer Vision, pp. 469-485. Springer, 2024b.
Zhiqi Li, Wenhai Wang. Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Qiao Yu, and Jifeng Dai. Bevformer: learning seyd'--eye-view representation from lidar-camera via spatiotemporal transformers. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024c.
Yifan Lu, Xuanchi Ren, Jiawei Yang, Tianchang Shen, Zhangjie Wu, Jun Gao, Yue Wang, Siheng Chen, Mike Chen, Sanja Fidler, et al. Infinicube: Unbounded and controllable dynamic 3d driving scene generation with world-guided video models. arXiv preprint
2024.
Enhui Ma, Lijun Zhou, Tao Tang, Zhan Zhang, Dong Han, Junpeng Jiang, Kun Zhan, Peng Jia, Xianpeng Lang, Haiyang Sun, al. Unleashing generalization of end-to-end autonomous driving with controllable long video generation. arXiv preprint arXiv:2406.01 2024.
Jiageng Mao, Boyi Li, Boris Ivanovic, Yuxiao Chen, Yan Wang, Yurong You, Chaowei Xiao, Danfei Xu, Marco Pavone, and Yue Wang. Dreamdrive: Generative 4d scene modeling from street view images. arXiv preprint arXiv:2501. 00601, 2024.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 1066 2021.
12

---

Published as a conference paper at ICLR 2026
Chaojun Ni, Guosheng Zhao, Xiaofeng Wang. Zheng Zhu, Wenkang Qin, Guan Huang, Chen Liu, Yuyin Chen, Yida Wang, Xueyang Zhang, et al. Recondreamer: Crafting world models for driving scene reconstruction via online restoration. arXiv preprint arXiv:2411.19544 2024.
Chensheng Peng, Chengwei Zhang, Yixiao Wang, Chenfeng Xu, Yichen Xie, Wenzhao Zheng, Kurt Keutzer, Masayoshi Tomizuka, and Wei Zhan. Desire-gs: 4d street gaussians for static- dynamic decomposition and surface reconstruction for urban driving scenes. arXiv preprint arXiv:2411.1192/1 2024.
Julius Plucker. Xvii. on a new geometry of space. Philosophical Transactions of the Royal Society of London, (155):725-791, 1865.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140): 2020.
Xuanchi Ren, Yifan Lu, Hanxue Liang, Zhangjie Wu, Huan Ling, Mike Chen, Sanja Fidler, Francis Williams, and Jiahui Huang. Scube: Instant large-scale scene reconstruction using voxsplats. arXiv preprint ar arXiv:2410.2000 20020000 2024.
Xuanchi Ren, Tianchang Shen, Jiahui Huang. Huan Ling, Yifan Lu, Merlin Simier-David, Thomas Müller, Alexander Keller, Sanja Fidler, and Jun Gao. Gen3c: 3d-informed world-consistent video generation with precise camera control. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.
Alexander Swerdlow, Runsheng Xu, and Bolei Zhou. Street-view image generation from a bird's s-ree view layout. IEEE Robotics and Automation Letters, 2024.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. Fvd: A new metric for video generation. 2019.
Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 1294-53001 2025a.
Lening Wang, Wenzhao Zheng, Dalong Du, Yunpeng Zhang, Yilong Ren, Han Jiang, Zhiyong Cui, Haiyang Yu, Jie Zhou, Jiwen Lu, and Shanghang Zhang. Stag- Towards realistic 4d driving simulation with video generation model. arXiv preprint
2024a.
Lening Wang, Wenzhao Zheng, Yilong Ren, Han Jiang, Zhiyong Cui, Haiyang Yu, and Jiwen Lu. Occsora: 4d occupancy generation models world simulators for autonomous driving. arXiv preprint arXiv:2405. 20337, 2024b.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing vision- language model's 's perception of the world at any resolution. arXiv preprint 2409v:2409.122 12191, 2024c.
Qianqian Wang, Yifei Zhang, Aleksander Holynski, Alexei A Efros, and Angjoo Kanazawa. Continu- ous 3d perception model with persistent state. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 10510-10522, 2025b.
Qitai Wang, Lue Fan, Yuqi Wang, Yuntao Chen, and Zhaoxiang Zhang. Freevs: Generative view synthesis on free driving trajectory. arXiv preprint arXiv:2410.1800 18079, 2024d.
Xiaofeng Wang, Zheng Zhu, Yunpeng Zhang, Guan Huang, Yun Ye. Wenbo Xu, Ziwei Chen, and Xingang Wang. Are we ready for vision-centric driving streaming perception? the asap benchmark. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9600-9610, 2023.
Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Jiagang Zhu, and Jiwen Lu. Drivedreamer: Towards real-worldddrive world models for autonomous driving. In European Conference on Computer Vision, pp. 55-72. Springer, 2024e.
13

---

Published as a conference paper at ICLR 2026
Yuqi Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14749-14759, 2024f.
Dongxu Wei, Zhiqi Li, and Peidong Liu. Omni-scene: Omni-gaussian representation for ego-centric sparse-view scene reconstruction. arXiv preprint arXiv:2412.06273, 2024.
Yuqing Wen, Yucheng Zhao, Yingfei Liu, Fan Jia, Yanhui Wang, Chong Luo, Chi Zhang. Tiancai Wang, Xiaoyan Sun, and Xiangyu Zhang. Panacea: Panoramic and controllable video generation for autonomous driving. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattem Recognition, pp. 6902 6912, 2024.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in neural information processing systems, 34:12077-12090, 2021.
Yunzhi Yan, Haotong Lin, Chenxu Zhou, Weijie Wang, Haiyang Sun, Kun Zhan, Xianpeng Lang, Xiaowei Zhou, and Sida Peng. Street gaussians: Modeling dynamic urban scenes with gaussian splatting. In European Conference on Computer Vision, pp. 156-173, Springer. 2024.
Jiawei Yang, Boris Ivanovic, Or Litany, Xinshuo Weng, Seung Wook Kim, Boyi Li, Tong Che, Danfei Xu, Sanja Fidler, Marco Pavone, et al. Emerner:: Emergent spatial-temporal scene decomposition via self-supervision. arXiv preprint wrXiv:231.002077 2023.
Jiawei Yang. Jiahui Huang, Yuxiao Chen, Yan Wang, Boyi Li, Yurong You, Apoorva Sharma, Maximilian Igl. Peter Karkus, Danfei Xu, et al. Storm: Spatio-temporal reconstruction model for large-scale outdoor scenes. arXiv preprint arXiv:2501.0060 2024a.
Yuanbo Yang, Jiahao Shao, Xinyang Li, Yujun Shen, Andreas Geiger, and Yiyi Liao. Prometheus: 3d-aware latent diffusion models for feed-forward text-to-3d scene generation. arXiv preprint ar arXiv:2412.21117, 2024b.
Kai Zeng. Zhanqian Wu, Kaixin Xiong, Xiaobao Wei, Xiangyu Guo, Zhenxin Zhu, Kalok Ho, Lijun Zhou, Bohan Zeng, Ming Lu, et al. Rethinking driving world model as synthetic data generator for perception tasks. arXiv preprint arXiv:2510. 2025.
Kai Zhang, Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, and Zexiang Xu. Gs-Irm: Large reconstruction model for 3d gaussian splatting. In European Conference on Computer Vision, pp. 1-19. Springer, 2024.
Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, p 586-595, 2018.
Guosheng Zhao, Xiaofeng Wang, Zheng Zhu, Xinze Chen, Guan Huang, Xiaoyi Bao, and Xingang Wang. Drivedreamer-2: An-ennanced world models for diverse driving video generation. arXiv preprint arXivv2403.06885 2024.
Wenzhao Zheng, Weiliang Chen, Yuanhui Huang, Borui Zhang, Yueqi Duan, and Jiwen Lu. Occworld: Learning 3d occupancy world model for autonomous driving. In European conference on computer vision, pp. 55-72. Springer, 2024.
Hongyu Zhou, Jiahao Shao, Lu Xu, Dongfeng Bai, Weichao Qiu, Bingbing Liu, Yue Wang, Andreas Geiger, and Yiyi Liao. Hugs: Holistic urban 3d scene understanding via gaussian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp.
11345, 2024a.
Xiaoyu Zhou, Zhiwei Lin, Xiaojun Shan, Yongtao Wang, Deqing Sun, and Ming- Ming-Hsuan Yang. Drivinggaussian: Composite gaussian splatting for surrounding dynamic autonomous driving scenes. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 21634-226111 (664,21643, 2024b.
14

---

Published as a conference paper at ICLR 2026
Appendix
A IMPLEMENTATION DETAILS
A.1 ARCHITECTURES
In Fig. 6 we provide a detailed view of our enhanced diffusion model (See. 3.4). To enable fine- grained control over video synthesis. we condition multiple signals: rendered RGBs from 4D Gaussians (Sec. 3.3), road sketches, 3D bounding boxes, ego-vehicle trajectories, and textual scene descriptions. The overall transformer backbone of our enhanced diffusion model is identical that of our 4D-awere diffusion framework (Sec. [3.2]; we simply adjust the input and output channel dimensions to suit different latent representations.
Generated Video
Rendering
Road Sketch
3D Boxes
Traeectoy
Caption
VAE Encoder
STDiT
XN
STDIT Control
Control
Proj
Proj
VAE Decodor
XN
STDiT
STDIT
Base
Noisy Latent
FFN
Spatial Temporal Compress
3D Boxes
Temporal Attention
STDIT Block
MLP
Cross. View Attention
Trajectory
Cross Attention
Text Encoder
Scene Caption
Spatial Self-Artention
Figure 6: The architecture details of our diffusion transformer.
Double-Branch Diffusion Transformer. Following DiVE (Jian al. 2024), first employ frozen variational autoencoder (VAE) to encode the input multi-view video clip into compact latent tensor xxx where V the number of camera views, T the number of frames, and H, W the spatial dimensions of each latent feature map. A 3D patch embedding module then aggregates these features to capture spatiotemporal correlations.
parallel, we introduce a dedicated ControlNet (Chen) 2023) branch to inject rendering and sketch guidance: the VAE encodes both signals into latent patches, which are aligned with the main 3D patch embedder. We interleave specialized ControlNet blocks alongside each DiT transformer stage, merging sketch information into the main feature stream to achieve precise structural control.
Spatial-Tene Temporal Diffusion Transformer Block. To enforce coherence across views without increasing parameter count, we replace standard self-attention with a cross-view attention mechanism. Concretely, given input of shape B V x T H W x C, reshape it to B x T (VHW) C, treating the flattened V HW dimension as the attention sequence length. This simple reordering enables riss-view interactions while keeping model size unchanged.
We further fuse 3D bounding boxes, ego-trajectory data, and scene captions via a single cross-attention layer. We project the 2D image-plane embeddings of each 3D box with 3D convolution, encode the
15

---

Published as a conference paper at ICLR 2026
ego trajectory through small MLP, and tokenize the textual caption using T5 backbone (Raffel iet al. (220)) These sedalltt-specciff embeddings are concatenated and passed through a final MLP to produce unified conditioning vector for the cross-attention block.
A.2 TRAINING DETAILS
The training is organized into four sequential stages on 32x H20 GPUs:
Table 5: 4D-Aware Diffusion Training Stages. Steps are reported for 8-GPU training. For 32-GPU training, divide by (e.g., Stage I: 15K steps).
Stage
Description
Steps*
Resolution (6 views)
Time
Stage
Fine-tune from OpenSora v1.2 with layout/sketch control
60K
256x
~>2hh
Stage 2
Mixed-resolution training with varying frame lengths
40K
144p-360p
~28h
Stage 3
Switch to rectified flow low resolution
20K
144p-360)
~14h
Stage
High-resolution fine-tuning with rectified flow
60K
480p-full
8-3h
Total
4D-Aware Diffusion Training
180K
~157
Stage 1: Starting from the OpenSora v checkpoints, we fine-tune for 60K iterations (15K steps on 32 GPUs) 256 fixed-resolution images establish layout and sketch control. At this stage, the ControlNet-Transfommeer spatial attention, and layout module (with spatial self-attention in the base layers) optimized.
Stage 2: We continue for 40K iterations (10K steps on 32 GPUs) using mixed resolutions (144p, 240p. 360p) and varying frame lengths, aligning the model to the nuScenes data distribution, still employing spatial self-attention.
Stage DDPM replaced with rectified flow. We first train for 20K iterations (5K steps on 32 GPUs) at low resolutions (144p-360p).
Stage 4: We finetune the model by 60K iterations (15K steps on 32 GPUs) higher resolutions (480p to full scale) with rectified flow.
Additional Training Stages:
Gaussian Decoder: ~22 hours (100K iters on 32xH20, using frozen diffusion latents)
Enhanced Diffusion: ~559 hours (50K iters on H220 initialized from 4D-Aware Diffusion
checkpoint)
A.3 GEOMETRIC CONSISTENCY AND MULTI- COHERENCE EVALUATION
To address concerns about geometric consistency and multi-view coherence, we evaluate our method using novel-view synthesis metrics following OmniScene's 's (Wei et al.. 2024) protocol. We compare against
3D reconstruction methods on the nuScenes validation set, measuring PSNR, SSIM, and LPIPS to assess the geometric fidelity and multi-vien consistency of the generated scenes.
Table 6: Geometric consistency and multi-view coherence evaluation. We follow OmniScene's
protocol for novel-view synthesis evaluation.
Model
PSNR
SSIM
LPIPS +
PixelSplat (Charatan et al. 2024)
21.51
0.616
0.372
MVSplat (Chen et al. (024b)
21.61
0.658
0.295
OmniScene Wei et 2024
24.27
0.736
0.237
Ours
24.59
0.912
0.147
As shown in Tab. 66 our method achieves superior performance across all metrics, demonstrating strong geometric consistency and multi-view coherence. The significantly higher SSIM (0.912 vs. 0.736) and lower IPIPS (0. (0.147 vs. 0.237) compared to OmniScene indicate that 4D Gaussian
16

---

Published as a conference paper at ICLR 2026
representation effectively maintains structural coherence across different viewpoints. These results confirm that our approach not only generates high-quality videos (as measured by FID/FVD in the main paper) but also preserves geometric consistency and multi-view coherence essential for autonomous driving applications.
B INFERENCE SPEED COMPARISON
In Tab. 7 we compare the inference speed of our pipeline with MagicDrive-V2 Gaa et al. 2025) and Cosmos-transfent (Alhaija al.) (2225) single NVIDIA H20 GPU, producing 17-frame, 6-view video at x24x800 800 resolution. Although our model employs two diffusion modules, the use of rectified flow with only 8 sampling steps keeps the inference speed comparable others, which typically require over 30 steps.
Table 7: Efficiency comparison on novel scene generation. We report runtime breakdown and GPU memory usage for different methods.
Method
Resolution Diff-1 (s) Gs Dec. (s) Diff-2 (s)
VAB Dec. (s)
Total (min)
Device
GPU Mem (GB)
MagicDrriv--2 V2
424x800
215.35
15.83
3.85
H20
26
Cossos-transfer
424x800
126.37
4.14
2.18
H20
17
Ours
424x800
66.56
0.84
66.35
16.10
2.50
H20
22
C MORE VISUALIZATION RESULTS
To better illustrate our Gaussian representation, we provide visualizations in Figs. strate its high structural fidelity.
which demon-
Figure 7: Visualizations of our Gaussians representation.
Further, our method produces fully controllable videos without relying on any reference frames, while simultaneously supporting high-quality novel-view synthesis. Figs. 8 showcase qualitative results of trajeiitrae generation without RGB inputs, clearly demonstrating the effectiveness and robustness of our model.
We include series of generated novel-view videos in the supplementary material to further validate the quality of our results. Specifically, the supplementary videos correspond to two novel trajectories parallel to the original path, shifted by +2 m the left and right.
D THE USE OF LARGE LANGUAGE MODELS (LLMs)
large language model (ChatGPT) was only used after completing the draft to polish wording and improve readability. It did not contribute to research or content creation. The authors take full responsibility for the final text.
17

---

Published as a conference paper at ICLR 2026
Real
Conds
Ours
Left Im
Right Im
Left 2m
Right 2m
Figure 8: Multi-trajectory generation (from center to # +lm/ / +2m lateral shifts) without RGB inputs.
Real
Conds
Ours
Left Im
Right Im
Left 2m
Right 2m
Figure 9: Multi-trajectory generation (from center to #1m / +2m lateral shifts) without RGB inputs.
18

---

Published as a conference paper at ICLR 2026
Real
Conds
Ours
Left Im
Right Im
Left 2m
Right 2m
Figure 10: Multi-trajectory generation (from center to #lm/ +2m lateral shifts) without RGB inputs.
LEGAL
Real
Conds
Ours
Left Im
Right Im
Left 2m
Right 2m
Figure 11: Multi-trajectory generation (from center to #1m/ +2m lateral shifts) without RGB inputs.
19

---

Published as a conference paper at ICLR 2026
Real
Conds
Ours
Left 1m
Right Im
Left 2m
Right 2m
Figure 12: Multi-trajectory generation (from center to #1m/ / #2m lateral shifts) without RGB inputs.
Real
Conds
Ours
Left Im
Right Im
Left 2m
Right 2m
Figure 13: Multi-trajectory generation (from center to #1m/ +2m lateral shifts) without RGB inputs.
20


Reviews:
[{"review_id": "96aGORQXKX", "reviewer": "Reviewer_VA3W", "summary": "Current Driving scene generation methods lack 3D consistency for novel-view synthesis, while reconstruction methods cannot generate unseen scenarios. This paper introduces WorldSplat, a feed-forward framework that bridges this gap using explicit 4D Gaussian representations. The approach employs: (1) a 4D-aware latent diffusion model to generate multi-modal latents (RGB, depth, semantic) from control conditions, (2) a Gaussian decoder that produces pixel-aligned 4D Gaussians with static-dynamic decomposition for efficient novel-view rendering, and (3) an enhanced diffusion model for refinement. Experiments on nuScenes demonstrate impressive performance in video generation and novel-view synthesis, with improvements on downstream perception tasks.", "questions": "Additional questions:\n\nQ1. In reference to W1, can the authors provide an comparison between methods using modern video backbones like OpenSora for FID and FVD?\n\nQ2. What amount of generated data was used in the mixed \"Real + Ours\" result in Table 4b?\n\nQ3. How do you prevent redundant Gaussians from multiple views of the same static regions?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The submission contributes a novel feed-forward Gaussian decoder which effectively uses the latents from the 4D latent diffusion model to create accurate static/dynamic decomposed 4D Gaussians.\n- The method effectively addresses a key limitation in the field by bridging the gap between video generation methods (which may contain artifacts and inconsistencies) and reconstruction methods (which lack generative capability). The use of explicit 4D Gaussian representations as an intermediate representation is clever and well-justified.\n- The paper demonstrates significant improvements across multiple metrics, including FVD/FID scores for video generation (Table 1), novel-view synthesis quality (Table 2), and  downstream driving tasks (Table 4)\n- The paper is well structured and lays out the rationale for the architecture clearly. The combination of a generative pipeline to create 4D Gaussians paired with a final diffusion model for inpainting unseen regions and overall refinement is well motivated by related work.", "weaknesses": "W1. While the improvements in FID and FVD compared to baselines even at varying degrees of camera displacements is solid, the work focuses on this improvement without examining or ablating how much of this improvement can be attributed to the backbone model (OpenSora v1.2) versus their own models.The downstream task comparisons also show meaningful improvements in mIoU and mAP for BEVFormer. Of note is the fact that the closest model in terms of performance to WorldSplat is DiVE, which also employs an OpenSora backbone in contrast to the MagicDrive works. The paper is missing a comparison of FID and FVD against DiVE despite it being a comparable model in terms of the end video diffusion model used to generate RGB frames.\n\nW2. While the paper shows successful examples, it lacks in-depth discussion of failure cases and ablations. Table 3 demonstrates the improvements when using all of WorldSplat's individual components, but doesn't provide any explanation or intuition behind the results. For example, Version D shows that without enhanced diffusion, FVD degrades from 47.41 to 107.58 (>2x worse), suggesting the end diffusion model is doing a lot of the heavy lifting to uphold the FID and FVD scores rather than the generated Gaussians (despite the consistently better performance with displacement compared to other methods). There's also little discussion of failure cases at all.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Pn0Chr8613", "reviewer": "Reviewer_t1xN", "summary": "This paper proposes a Gaussian-centric, feed-forward 4D scene generation method for autonomous driving, called WorldSplat. The work explores how to combine generative models with accurate 4D reconstruction. WorldSplat introduces three key modules: the 4D-aware latent diffusion module, which transforms user-defined control conditions into temporally coherent latents; the latent Gaussian decoder, which converts these latents into explicit 4D Gaussian scene geometry; and the enhanced diffusion refinement model, which further improves the quality of rendered frames. Compared to other methods, WorldSplat achieves state-of-the-art performance.", "questions": "- In L226, it says \"our decoder supports over 48 simultaneous input views.\" Could you clarify how the model supports 48 views? Why can't prior feed-forward reconstruction models achieve this capacity?\n- Both the introduction and the ablation study mention the \"Mixed Aug\" technology, but I don't believe this technology is a key contribution of the current work. Could you clarify why it is mentioned so frequently?\n- In WorldSplat, I would like to know how the performance is affected without the reprojection conditions in the Enhanced Diffusion Model. It seems that the restricted views sampled from the Aggregated 4D Gaussians are somewhat aligned with the conditions. Could this lead to performance degradation in novel-view synthesis?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- WorldSplat elegantly combines the generative ability of novel scene generation with the custom scene reconstruction capability of Gaussian scene (GS) models.\n- WorldSplat achieves state-of-the-art performance under various conditions, including both custom and novel-view synthesis, outperforming other leading methods.", "weaknesses": "- The ablation study in this work is somewhat simplistic, and the authors tend to overstate their claims.\n- The novelty of the module design is limited, and the methods section delves into well-known concepts, such as the diffusion process, without offering significant new insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FEullTbcDz", "reviewer": "Reviewer_9j61", "summary": "In this paper, the authors propose a novel feed-forward framework for 4D driving-scene generation, which can integrate multi-modal information via a 4D-aware latent diffusion model to produce pixel-aligned 4D Gaussians in a feed-forward manner. What's more, the authors use an enhanced video diffusion model to refine the novel view videos rendered from these Gaussians. The experiments show that WorldSplat can generate high-fidelity novel view driving videos.", "questions": "1. I am confused about Fig. 2. It seems that only the 4D-Aware Diffusion Model needs to be trained, but in the paper, the Latent Gaussian Decoder also needs to be trained.\n2. It is hoped that the authors can provide more descriptions regarding the training time of the entire framework and all its training stages.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors focus on the limitations of existing methods—2D video generation lacks 3D consistency, while 3D/4D reconstruction lacks generative capabilities—and select 4D Gaussians as the core representation to improve 3D consistency, bridging the gap between generation and reconstruction.\n2. The framework can generate a dynamic 4D Gaussian representation via a latent Gaussian decoder and then render novel-view videos through these Gaussians. This design effectively enhances both spatial and temporal geometric consistency, addressing the 3D inconsistency issue of existing 2D video generation methods.\n3. Experiments on the nuScenes dataset show that the rendered novel-view videos have high fidelity(Tab.2) and the framework exhibits strong generative capabilities(Tab.1)", "weaknesses": "1. It seems that the training of the Gaussian Decoder relies on the seg loss, which requires additional time consumption due to the external use of SegFormer. Could the authors discuss the total training time of the framework, particularly the time overhead introduced by SegFormer?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "prJZuXXKkj", "reviewer": "Reviewer_HFQu", "summary": "This paper addresses novel-view synthesis in driving-scene generation and introduces WorldSplat, a feed-forward 4D generation framework bridging video synthesis and 3D/4D reconstruction. Leveraging a 4D-aware latent diffusion model to produce pixel-aligned Gaussians, followed by refined video diffusion, it generates high-fidelity, temporally and spatially consistent multi-track novel-view videos. Experiments on benchmark datasets show its effectiveness in producing realistic and controllable driving scenes.", "questions": "1. With the rapid development of generative models, stronger models naturally lead to better generation performance. Compared with previous methods, is the performance improvement mainly due to the proposed approach or the use of a more powerful diffusion model?\n2. Are Diff1, the GS decoder, and Diff2 trained jointly, or are they optimized in separate stages? It’s not entirely clear from the Training Details section.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well written, and the figures are clear and informative.\n2. The method demonstrates strong performance in generation metrics.\n3. The motivation is reasonable — leveraging Gaussian splatting to ensure geometric consistency, while incorporating diffusion models to enhance quality on novel regions.", "weaknesses": "1. The main issue with the paper lies in the inconsistency between the motivation and the experiments metrics. The method is only evaluated using FID and FVD, which primarily measure generation quality. These metrics fail to capture geometric consistency, as well as multi-view and temporal coherence, all of which are missing from the validation. When comparing with NVS methods, it is usually also report reconstruction metrics (psnr, ssim) on the original viewpoints to ensure consistency. FID and FVD alone only reflect perceptual generation quality, while many details may differ significantly — as can be clearly observed in the appendix figures, where the generated backgrounds differ noticeably from the original real images.\n2. I have some concerns about the proposed method. It seems that the success of novel view rendering may largely stem from the strong conditioning provided. If the projected boxes and trajectories are given, the overall object layout and scene distribution are already well constrained. Previous generative methods could also leverage this conditioning to perform novel view generation. The real challenge lies in maintaining background consistency, such as the mismatch of elements like utility poles observed in the appendix visualizations. Therefore, it remains unclear how much geometric consistency is truly contributed by the Gaussian-centric module, and how much is simply due to the strong conditioning.\n3. Since the method is based on 4D Gaussian Splatting, it would be important to include experiments demonstrating temporal consistency. Qualitative visualizations could also help illustrate this aspect. Currently, the appendix only provides static rendering results, which are insufficient to evaluate the model’s performance on temporal coherence.\n4. Table 4(a) lacks a comparison with the BEVFormer baseline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)