DynamicCity: Large-Scale 4D Occupancy Generation from Dynamic Scenes
Authors: Hengwei Bian
Lingdong Kong
Haozhe Xie
Liang Pan
Yu Qiao
Ziwei Liu
Venue: ICLR 2025 Spotlight (2025)
Keywords: LiDAR Generation
Dynamic Scenes
4D Generation
Primary area: applications to robotics, autonomy, planning
TL;DR: DynamicCity is a versatile 4D scene generation model that generate high-quality occupancy scenes from sensory driving data.

Abstract: Urban scene generation has been developing rapidly recently. However, existing methods primarily focus on generating static and single-frame scenes, overlooking the inherently dynamic nature of real-world driving environments. In this work, we introduce DynamicCity, a novel 4D occupancy generation framework capable of generating large-scale, high-quality dynamic 4D scenes with semantics. DynamicCity mainly consists of two key models. **1)** A VAE model for learning HexPlane as the compact 4D representation. Instead of using naive averaging operations, DynamicCity employs a novel **Projection Module** to effectively compress 4D features into six 2D feature maps for HexPlane construction, which significantly enhances HexPlane fitting quality (up to **12.56** mIoU gain). Furthermore, we utilize an **Expansion & Squeeze Strategy** to reconstruct 3D feature volumes in parallel, which improves both network training efficiency and reconstruction accuracy than naively querying each 3D point (up to **7.05** mIoU gain, **2.06x** training speedup, and **70.84\%** memory reduction). **2)** A DiT-based diffusion model for HexPlane generation. To make HexPlane feasible for DiT generation, a **Padded Rollout Operation** is proposed to reorganize all six feature planes of the HexPlane as a squared 2D feature map. In particular, various conditions could be introduced in the diffusion or sampling process, supporting **versatile 4D generation applications**, such as trajectory- and command-driven generation, inpainting, and layout-conditioned generation. Extensive experiments on the CarlaSC and Waymo datasets demonstrate that DynamicCity significantly outperforms existing state-of-the-art 4D occupancy generation methods across multiple metrics. The code and models have been released to facilitate future research.

Published as a conference paper at ICLR 2025
DYNAMICCITY: LARGE-SCALE 4D OCCU- PANCY GENERATION FROM DYNAMIC SCENES
Hengwei Bian 1,2+a Lingdong Kong" 13 Haozhe Xie
Liang Pan Lit Yu Qiao'
Ziwei Liu
Shanghai Al Laboratory Carnegie Mellon University
National University Singapore
LS-Lab, Nanyang Technological University
T=N
Command-Driven
Dynamic Object
Scene Generation
Generation
Forward
Turn Left
4
Tumn Lefe
Forward
Trajectory-Guided Generation
Dynamic Scene Inpainting
Layout- Condut-Conditioned Generation
Befors
After
Figure 1: Dynamic occupancy generation from DynamicCity. We introduce new generation model that generates diverse 4D scenes of large spatial scales (80 x 80 6.4 meter") and long sequential modeling (up to 128 frames), enabling diverse set of downstream applications. For more detailed examples, kindly refer to our Project Page: https: / /dyyamicccccccccccoa citty .erl.gittin github. io.
ABSTRACT
Urban scene generation has been developing rapidly recently. However, existing methods primarily focus on generating static and single-frame scenes, overlooking the inherently dynamic nature of real-world driving environments. In this work, we introduce DynamicCity, novel 4D occupancy generation framework capable of generating large-scale, high-quality dynamic 4D scenes with semantics. Dynam- icCity mainly consists two key models. 1) A VAE model for learning HexPlane the compact 4D representation. Instead of using naive averaging operations, DynamicCity employs a novel Projection Module to effectively compress 4D features into six 2D feature maps for Hex construction, which significantly enhances HexPlane fitting quality (up to 12.56 mloU gain). Furthermore, we utilize Expansion & Squeeze Strategy to reconstruct 3D feature volumes in parallel, which improves both network training efficiency and reconstruction accu- racy than naively querying each 3D point (up 7.05 mIoU gain, 2.06x training speedup, and 70.84% memory reduction). 2) A DiT- diffusion model for HexPlane generation. To make HexPlane feasible for DiT generation, Padded Rollout Operation is proposed to reorganize all six feature planes of the Hex- Plane as a squared 2D feature map. In particular, various conditions could be introduced in the diffusion sampling process, supporting versatile 4D genera-
"Work done during internship Shanghai AT Laboratory.
Corresponding author.
Project lead.

---

Published as a conference paper at ICLR 2025
tion applications, such as trajectory- and command-driven generation, inpainting, and layout- layout-conditioned conditioned generation. Extensive experiments on the CarlaSC and Waymo datasets demonstrate that DynamicCity significantly outperforms existing state-of-the-art 4D) occupancy generation methods across multiple metrics. The code and models have been released to facilitate future research.
INTRODUCTION
Urban scene generation has garnered growing attention recently, which could benefit various related applications, such as robotics and autonomous driving. Compared to its 3D object generation coun- terpart, generating urban scenes remains an under-explored field, with many new research challenges such the presence of numerous moving objects, large-scale scenes, and long temporal sequences (Huang et al., 2021). For example, in autonomous driving scenarios, a scene typically comprises multiple objects from various categories, such as vehicles, pedestrians, and vegetation, captured over a long sequence (e.g., 200 frames) spanning a large area (e.g., 80 x 80 x 6.4 meters"). Although in its early stage, 4D occupancy generation holds great potential to enhance the understanding of the 3D world, with wide-reaching and profound implications.
Due to the complexity of occupancy data, many efficient learning frameworks have been introduced for large-scale 3D scene generation. X (Ren et al., 2024b) utilizes hierarchical voxel diffusion model to generate outdoor 3D scenes based on VDB data structure. PDD (Liu et al., 2023a) introduces a pyramid discrete diffusion model to progressively generate high-quality 3D scenes. SemCity (Lee et al., 2024) resolves outdoor scene generation by leveraging triplane diffusion model. Despite achieving impressive occupancy generation, they primarily focus on generating static and single-frame 3D occupancy, and hence fail to effectively capture the dynamic nature of outdoor environments. Recently, few works (Zheng al., 2024b; Wang al., 2024) have explored 4D scene generation. However, generating sugh-tuality long-sequene 4D scenes is still a challenging and open problem (Nakashima & Kurazume, 2021; Nakashima et al., 2023).
In this work, we propose a novel 4D occupancy generation framework, DynamicCity, enabling generating large-scale, high-quality dynamic occupancy scenes, which mainly consists of two stages: 1) a VAE network for learning compact 4D representations, i.e. HexPlanes (Cao & Johnson, 2023; Fridovich-Ko Keil al., 2023); 2) HexPlane Generation model based on DiT (Peebles & Xie, 2023).
VAE for 4D Occupancy, Given a set of 4D occupancy scenes, DynamicCity first encodes the scene 3D feature volume sequence with 3D) backbone. Afterward, we propose novel Projection Module based on transformer operations to compress the feature volume sequence into six 2D feature maps. In particular, the proposed projection module significantly enhances HexPlane fitting performance, offering an improvement of up to 12.56% mIoU compared to conventional averaging operations. After constructing the HexPlane based on the projected six feature planes, we employ an Expansion & Squeeze Strategy (ESS) to decode the HexPlane into multiple 3D feature volumes in parallel. Compared to individually querying each point, ESS further improves Hex Plane fitting quality (with up to 7. 7.05% mIoU gain), significantly accelerates training speed (by up to 2.06x), and substantially reduces memory usage (by up to a relative 70.84% memory reduction).
DiT-based framework for generating HcxPlane, enabling 4D occupancy generation. Training a DiT with token sequences naively generated from HexPlane may not achieve optimal quality, could overlook spatial and temporal relationships among tokens. Therefore, we introduce the Padded Rollout Operation (PRO), which reorganizes the six feature planes into square feature map. providing an efficient way to model both spatial and temporal relationships within the token sequence. Leveraging the DiT framework, DynamicCity seamlessly incorporates various conditions guide the 4D generation process, enabling a wide range of applications including hexplane-condittioall generation, trajectory reajcctory-guided generation, command-drivenn scene generation, layout-conditionel generation, and dynamic scene inpainting.
DIT for HexPlane. Using the encoded HexPlane,
Our contributions can be summarized as follows:
We propose DynamicCity, a high-quality, large-scale 4D occupancy generation framework consisting of a tailored VAE for HexPlane fitting and a baseb network for HexPlane generation, which supports various downstream applications.
2

---

Published as a conference paper at ICLR 2025
In the VAE architecture, DynamicCity employs a novel Projection Module to benefit in encoding 4D scenes into compact HexPlanes, significantly improving HexPlane fitting quality. Following, an Expansion & Squeeze Strategy is introduced to decode the HexPlanes for reconstruction, which improves both fitting efficiency and accuracy.
Building on fitted HexPlanes, we design a Padded Rollout Operation to reorganize HexPlane features into masked 2D square feature map, enabling compatibility with DiT training.
Extensive experimental results demonstrate that DynamicCity achieves significantly better 4D reconstruction and generation performance than previous SoTA methods across all evaluation metrics, including generation quality, training speed, and memory usage.
2 RELATED WORK
3D Object Generation has been central focus in machine learning, with diffusion models playing a significant role in generating realistic 3D structures. Many techniques utilize 2D diffusion mechanisms to synthesize 3D outputs, covering tasks like text-to-3D object generation (Ma et al., 2024), image-to- 3D transformations (Wu et al., 2024a), and 3D editing (Rojas al., 2024). Meanwhile, recent methods bypass the reliance on 2D intermediaries by generating 3D outputs directly in three-dimensional space, utilizing explicit (Alliegro et al., 2023), implicit (Liu al., 2023b), triplane (Wu al., 2024b), and latent representations (Ren et al., 2024b). Although these methods demonstrate impressive 3D object generation, they primarily focus on small-scale, isolated objects rather than large-scale, scene-level generation (Hong et al., 2024; Xie et al., 2025b). This limitation underscores the need for methods capable of generating complete 3D scenes with complex spatial relationships.
Urban Scene Generation extends the scope to larger, more complex environments. Earlier works used VQ- VQ-VAE (Zyrianov et al., 2022) and GAN-based models (Caccia et al., 2019; Nakashima et al., 2023) to generate LiDAR scenes. However, recent advancements have shifted towards diffusion models (Xiong et al., 2023; Ran et al., 2024; Nakashima Kurazume, 2024; Zyrianov et al., 2022: Hu et al., 2024; Nunes et al., 2024), which better handle the complexities of expansive outdoor scenes. For example, (Lee et al., 2024) utilize voxel grids to represent large-scale scenes but often face challenges with empty spaces like skies and fields. While some recent works incorporate temporal dynamics to extend fangle-framce generation to sequences (Zheng et al., 2024b; Wang et al., 2024), they often lack the ability to fully capture the dynamic nature of 4D environments. Thus, these methods typically remain limited to short temporal horizons or struggle with realistic dynamic object modeling, highlighting the gap in generating high-fidelity 4D scenes.
4D Generation represents a leap forward, aiming to capture the temporal evolution of scenes. Prior works often leverage video diffusion models (Singer al., 2022; Blattmann al., 2023) generate dynamic sequences (Singer et al., 2023), with some extending to multi-view (Shi et al., 2023; Xie al., 2025a) and single-image settings (Rombach al., 2022) to enhance 3D consistency. In the context of video-conditional generation, approaches such as (Jiang et al., 2023; Ren et al., 2023; 2024a) incorporate image priors for guiding generation processes. While these methods capture certain dynamic aspects, they lack the ability to generate long-term, high-resolution 4D scenes with versatile applications. Our method, DynamicCity, fills this gap by introducing a novel 4D generation framework that efficiently captures large-scale dynamic environments, supports diverse generation tasks (e.g., trajectory-guiide (Bahmani et al., 2024), command-drivee generation). and offers substantial improvements in scene fidelity and temporal modeling.
3 PRELIMINARIES
HexPlane (Cao & Johnson, 2023; Fridovich-Keil et al., 2023) is an explicit and structured rep- resentation designed for efficient modeling of dynamic 3D scenes, leveraging feature planes to encode spacetime data. A dynamic 3D scene is represented as six 2D feature planes, each aligned with one of the major planes in the 4D spacetime grid. These planes are represented as H [Pay: Prz, Pys. yy2,7 Ptr, Pry, Piz], comprising Spatial TriPlane (Chan et al., 2022) with Pay. Puz. and Pyz. and a Spatial-Tima TriPlane with Ptz. Pty. and Ptz. To query the HexPlane at a point extracted from the corresponding coordinates on each of the six planes
p (t, x. y, 2), features and fused into a comprehensive representation. This fused feature vector is then passed through a lightweight network predict scene attributes for p.
3

---

Published as a conference paper at ICLR 2025
4D Scene
HexPlane
c(p)
Space
PE(p)
30 Backbone
Projection
Head
Space- Time
Hadamard
Product
Probabilities
(a) Learning HexPlane as Efficient 4D Stene Representation
Diffusion
Decoder
Decoder
DIT
Corditions
Scene Q:
Denoising
Scene Qo
(b) HexPlane Diffussion with DIT Scone Generation
Figure 2: Pipeline of dynamic scene generation. Our DynamicCity framework consists of two key procedures: (a) Encoding HexPlane with an VAE architecture (ef. Sec. 4.1), and (b) 4D Scene Generation with HexPlane DiT (ef. Sec, 4.2).
Diffusion Transformers (DiT) (Peebles & Xie, 2023) diffusion- bastdd generative models using transformers to gradually convert Gaussian noise into data samples through denoising steps. The forward diffusion adds Gaussian noise over time, with noised sample at step & given by x, va,Xo + /1 Oige, e ~~ (O I), where controls the noise schedule. The reverse diffusion, using neural network aims to denoise X₃ to recover expressed as: X Va. (x,
vI - aica (x,, t)). New samples are generated by repeating this reverse process.
4
OUR APPROACH
DynamicCity strives generate dynamic 3D occupancy with semantic information, which mainly consists of a VAE for 4D occupancy encoding using HexPlane (Cao & Johnson, 2023; Fridovich- Keil al., 2023) (Sec. 4.1), and DiT for HexPlane generation (Sec. 4.2). Given 4D scene, i.e., a dynamic 3D occupancy sequence Q E Txxxxx xx xxxxxxxxxx where T, X, Y, Z, and C denote the sequence length, height, width, depth, and channel size, respectively. the VAE first aims to encode efficient 4D representation, HexPlane H = [Pay. Prz, Py yzz Ptr, Ptrr,t,,, Pty, Piz], which is then decoded for reconstructing 4D scenes with semantics. After obtaining HexPlane embeddings, DynamieCity leverages DiT-based framework for 4D occupancy generation. Diverse conditions could be introduced into the generation process, facilitating a range of downstream applications (Sec. 4.3). The overview of the proposed DynamicCity pipeline is illustrated in Fig. 2.
4.1 VAE FOR 4D OCCUPANCY
Encoding HexPlane. As shown in Fig. 3, the VAE could encode 4D occupancy Q as HexPlane H. It first utilizes a shared 3D convolutional feature extractor fe(-) to extract and downsample features from each occupancy frame, resulting in a feature volume sequence Xaayz E XTXXXXX2xX
To encode and compress Xtzyz into compact 2D feature maps of H. we propose a novel Pro- jection Module with multiple projection networks ht-)) To project a high-dimensional feature input Xim E R xxxxxxx--xxl x" xD) xDx x."" xo as a lower-dimensional feature output X E a R De DD the projection network hg. (.) first reshapes Xú, - 3-dimensional feature X',ss, E & by grouping the dimensions into the two new dimensions, i.e., Se the dimension that will be kept, and S, the dimension that will reduced, where S, = D x D x ... x D and S. DJ D2 x Dm Afterward, hg utilizes transformer-hased operation project the
4

---

Published as a conference paper at ICLR 2025
Encoding HexPlane
Space
xyz
h2
X txya
XYZ
h2
f
XYZ Feature Volume
ft
fo
Space-Time
ht
4D Scone
Projection Module
Axy
HexPlane 3{
Reconstruction Loss
Feature Volume Sequence
X tzyz
Hadamard
Expansion
Product
Reconstructed
Decoding
Expand
Pay-
Pr2.
yr
Pix
Poy.
Prr
bayz
HexPlane
Figure 3: VAE for encoding dynamic scenes. We use HexPlane H as the representation. Se and Is convolution- bbasdd networks with downsampling and upsampling operations, respectively. h(-) denotes the projection network based on transformer modules.
reshaped feature X, to X E R& which is then reshaped to the expected lower-dimensional feature output Nan- Formally, the projection network is formulated as:
x(D) x xxx xxC = hs (i
x x xx) x ".""""
(1)
where their feature dimensions are added as the upscript for X and Xoott respectively.
To construct the spatial feature planes Pay. Paz, and Pyz. the Projection Module first generates the XYZ Feature Volume Xayz hee (Xayzz) Rather than directly access the heavy feature volume sequence Ntzyzs h-(-), h.).), and h-r()) are applied Xryz for reducing the spatial dimensions of Kays along the z-axis, y-axis, and x-axis, respectively. The temporal feature planes Pte. Pty. and Pts are directly obtained from Kezyz by simultaneously removing two spatial dimensions with hay()) h.z(,), and hryy respectively. Consequently, we could construct the HexPlane H based on the encoded feature planes, including Pzys Paz. Pyz: Ptx, Ptys and Pis.
Decoding HexPlane. Based on the HexPlane 7t = [Pay: Pys, Ptx. Pty, Pizl, we employ an Expansion & Squeeze Strategy (ESS), which could efficiently recover the feature volume sequence by decoding the feature planes in parallel for 4D occupancy reconstruction. ESS first duplicates and expands each feature plane P to match the shape of Kezyz, resulting the list six feature volume Pix yPtts sequences: (X XPPrs Keys A2yss tzys: tzyss Afterward, ESS squeezes the corresponding six expanded feature volumes with Hadamard Product:
X
II Hadamard
(N &lyy &f & M &
Subsequently, the convolutional network (-(+) is employed to upsample the volumes for generating dense semantic predictions Q':
Q' (C PE(POs
(3)
where Concat(-) and PE(.) denote the concatenation and sinusoidal positional encoding, respec- tively. Pos(-) returns the 4D position of each voxel within the 4D feature volume X'myyz
Optimization. The VAE is trained with combined loss LVAE. including cross-entropy loss, Lovász-softmax loss (Berman et al., 2018), and a Kullback-Leible (KL) divergence loss:
Lvx LCe( (Q. Q') afrow Q') + BL (H,

---

Published as a conference paper at ICLR 2025
Space
Space-Time
Numeric Conditions
Image Conditions
XY
Trajectory
Command
Timestamp
HexPlane
Layout
MLP
Embedding
MLP
Patchify
Patchify
(a) HexPlane
XY
TY
YZ
Conditioning
Cross-Attention Conditioning
XZ
TZ
Inpu oberoo
Lavertorm
Scals Shift
Coss Atsoniinn
bayor Nomm
Stale Shift
Poituise
(b) Padded Rollout
Figure 4: Padded Rollout
Figure 5: Condition Injection for DiT
where LCR is the cross-entropy loss between the input Q and prediction Q', La is the Lovász- softmax loss, and CKL represents the KL divergence between the latent representation H. and the prior Gaussian distribution (0, I). Note that the KL divergence computed for each feature plane of H. individually, and the term LKL refers to the combined divergence over all six planes.
4.2 DIFFUSION TRANSFORMER FOR HEXPLANE
After training the VAE, 4D semantic scenes can be embedded as HexPlane H (Pry, Paz, Pyz, Ptay Pty: Ptz]. Building upon H. we aim to leverage DiT (Peebles Xie, 2023) model D, to generate novel HcxPlane, which could be further decoded as novel 4D scenes (see Fig. 2()). However, training a DiT using token sequences naively generated from each feature plane of HexPlane could not guarantee high generation quality, mainly due to the absence of modeling spatial and temporal relations among the tokens.
Padded Rollout Operation. Given that the feature planes of HexPlane may share spatial or temporal dimensions, we employ the Padded Rollout Operation (PRO) to systematically arrange all six planes into unified square feature map, incorporating zero paddings in the uncovered corner areas. As shown in Fig. 4, the dimension of the 2D square feature map is (** Z dz L)), which minimizes the area for padding, where dx, dz, and dr represent the downsampling rates along the X, Z, and T axes, respectively. Subsequently, we follow DiT to first "patchify" the constructed 2D feature map, converting it into sequence of N = (( dxx + 2 + 2, }/)) tokens, where is the patch size, chosen so each token holds information from one feature plane. Following patchification, we apply the frequency- based positional embeddings all tokens similar DìT. Note that tokens corresponding to padding areas are excluded from the diffusion process. Consequently, the proposed PRO offers an efficient method for modeling spatial and temporal relationships within the token sequence.
dy
Conditional Generation. DiT enables conditional generation through the use of Classifier- Free Guidance (CFG) (Ho & Salimans, 2022). To incorporate conditions into the generation process, we designed two branches for condition insertion (see Fig. 5). For any condition we use the N-Zero technique from DIT, generating scale and shift parameters from and injecting them before and after the attention and feed-forward layers. To handle the complexity of magge-based conditions, we add a cross-attention block to better integrate the image condition into the DiT block.
4.3 DOWNSTREAM APPLICATIONS
Beyond unconditional 4D scene generation, we explore novel applications of DynamicCity through conditional generation and HexPlane manipulation.

---

Published as a conference paper at ICLR 2025
Table Comparisons of 4D Scene Reconstruction. We report the mloU scores of OccSora (Wang et al., 2024) and our DynamicCity framework on the CarlaSC, Occ3D- Dcc3D-Waymo, and Occ3D-nuSeenes datasets, respectively, under different resolutions and sequence lengths. Symbol denotes score reported in the OccSora paper. Other scores are reproduced using the official code.
OccSora
Ours (DynamicCity)
Dataset
#Classes
Resolution
#Frames
(Wang al., 2024)
10
128x 128x8
41.01%
79.61 (+38.6%)
CarlaSC
10
x28x 128x8
39.91%
76.18% (+36.3%)
(Wilson 2022)
10
128x 128x8
16
33.40%
74.22% (+40.8%%
10
128x 128x8
32
28.91% %
59.31 % (+30.4%)
Dcc3D-Waymo (Tian et al., 2023)
200x200
36,38%
68.18% (+31.8%
200x
16
13.70%
56.93% (+43.2%)
OeceDD-nueeenes (Tian et al., 2023)
200x
32
13.51
42.60 % (+29.1 1%)
17
200x
32
13.41% 16
40.79% (+27.3%)
17
200x 16
32
27.40%
40.79 % (+13,4%)
Table 2: Comparisons of 4D Scene Generation. We report the Inception Score (IS), Fréchet Inception Distance (FID), Kernel Inception Distance (KID), and the Precision (P) and Recall (R) rates of OccSora (Wang et al., 2024) and our DynamicCity framework on the CarlaSC and Oce Dcc3D-Waymo datasets, respectively, in both the 2D and 3D spaces.
Metric
Metric
Dataset
Method
#Frames
18 11
FID
KID
+ p
R
107
FID
KID
"
1
CarlaSC
OceSora
1,030
28.55
0.008
0.224
0.010
2.257
1559
52.72
0.380
0.151
(Wilson et oll,, 2022)
Ours
1,040
12.94
0.002
0.307
2.331
354.2
19.10
0.460
0.170
Occ3D Waymo
OceSora
1,005
42.53
0.049
0.654
0.004
3.129
3140
12.20
0.384
0.001
16
(Tian 2023)
Ours
1,010
36.73
0.001
0.705
0.015
3,206
1806
77.71
0.494
0.026
First, showcase versatile uses of image conditions in the conditional generation pipeline: 1) HexPlane: By autoregressively generating the HcxPlane, we extend scene duration beyond temporal constraints. 2) Layout: We control vehicle placement and dynamics in 4D scenes using conditions leamned from bird' s-ede view sketches.
To manage ego vehicle motion, introduce two numerical conditioning methods: 3) Command: Controls general ego vehicle motion via instructions. 4) Trajectory: Enables fine-grained control through specific trajectory inputs.
Inspired by SemCity (Lee et al., 2024), we also manipulate the HexPlane during sampling to: 5) Inpaint: Edit 4D scenes by masking HexPlane regions and guiding sampling with the masked areas. For more applications and implementation details, kindly refer to Sec. A.5 in the Appendix.
EXPERIMENTS
5.1 EXPERIMENTAL DETAILS
Datasets. We train the proposed model on the 30cc000-ayaaam Waymo, "Occ3D-nuScenes and "CarlaSC datasets. The former two from Oce3D (Tian et al., 2023) are derived from Waymo (Sun et al., 2020) and nuScenes (Caesar et al., 2020), where LiDAR point clouds have been completed and voxelized to form occupancy data. Each occupancy scene has resolution of 200 x 200 x 16, covering region centered on the ego vehicle, extending 40 meters in all directions and 6.4 meters vertically. The CarlaSC dataset (Wilson et al., 2022) is synthetic occupancy dataset, with scene resolution of 128 x 128 x 8, covering a region 25.6 meters around the ego vehicle, with a height of 3 meters.
Implementation Details. Our experiments are conducted using eight NVIDIA A A100-80G GPUs. The global batch size used for training the VAE is S, while the global batch size for training the DiT 128. Our latent HexPlane H. is compressed the latent channels C 16. The weight for the Lovász-softmax and KL terms are set to and 0.005, respectively. The learning rate for the VAE is 10-3 while the learning rate for the DiT is 10-1
half the size of the input Q in each dimension, with
7

---

Published as a conference paper at ICLR 2025
T=8
T-8
T=8
T-8
T-B
T-11
T 16
T-16
Build
Vehicle
Ped
Road
Sidewalk
Barrier
Ground
Veg
Other
Pole
Dynamic Scene Generation Results. We provide unconditional generation scenes from the 1st, Sth, and 16th frames on Occ3D- Dcc3D-Waymo (Left) and CarlaSC (Right), respectively, Kindly refer to the Appendix for complete sequential scenes and longer temporal modeling examples.
Figure 6:
Evaluation Metrics. The mean intersection over union (mloU) metric is used to evaluate the reconstruction results of VAF. For DIT, Inception Score, FID, KID, Precision, and Recall are calculated for evaluation. Specifically, we follow prior work (Lee et al., 2024; Wang et al., 2024) by rendering 3D scenes into 2D images and utilizing conventional 2D evaluation pipelines for assessment. Additionally, we train the 3D Encoder to directly extract features from the 3D data and calculate the metries. For more details, kindly refer See, A.2 the Appendix.
5.2 4D SCENE RECONSTRUCTION & GENERATION
Reconstruction. To evaluate the effectiveness the proposed VAE encoding the 4D occupancy sequence, compare it with OccSora (Wang et al., 2024) using the CarlaSC, Oce3D- Waymo, and Occ3D-nuScenes datasets. As shown in Tab. 1, DynamicCity outperforms OccSora on these datasets, achieving mloU improvements of 38.6%, 31.8%, and 43.2% respectively, when the input number of frames is 16. These results highlight the superior performance of the proposed VAE.
Occ3D-Waymo,
Generation. To demonstrate the effectiveness of DynamicCity in 4D scene generation, we compare the generation results with OccSora (Wang et al., 2024) on the Occ3D-Waymo and CarlaSC datasets. As shown in Tab. 2, the proposed method outperforms OccSora in terms perceptual metrics in both 2D and 3D spaces. These results show that our model excels in both generation quality and diversity, Fig. and Fig. show the 4D scene generation results, demonstrating that our model capable of generating large dynamic scenes in both real-world and synthetic datasets. Our model not only exhibits the ability to generate moving scenes with static semantics shifting as a whole, but also capable of generating dynamic elements such as vehicles and pedestrians.
Applications. Fig. presents the results of our downstream applications. tasks that involve inserting conditions into the DiT, such generation, and layout-conditional generation, our model demonstrates the ability to generate reason- able scenes and dynamic elements while following the prompt to certain extent. Additionally, the inpainting method proves that our HexPlane has explicit spatial meaning, enabling direct modifica- tions within the scene by editing the HexPlane during inference.
command-ccnnitionn generation, trajectory-conditional
8

---

Published as a conference paper at ICLR 2025
Command-Drivee Scene Generation
Dynamic Object Inpainting Before
Turn Left
Forward
Turn Right
Static
After
88
T=1
T=1
T=
T-1
T=1
T-8
T-8
T-8
T-8
T-8
T=16
T-16
T= 16
Layout-Condittoned Scene Generation
Trajectory-Guided Generation
T=1
T=1
T=11
T=1
T=1
T-8
T-8
T-8
T-B
T-8
T= T=16
T= 1=
T=16
T-16
T=16
Build
Vehicle
Ped
Road
Sidewalk
Barrier
Ground
Veg
Other
Figure 7: Dynamic Scene Generation Applications. We demonstrate the capability of our model on diverse set of downstream tasks. We show the 1st, 8th, and 16th frames for simplicity. Kindly refer to the Appendix for complete sequential scenes and longer temporal modeling examples.
5.3 ABLATION STUDIES
We conduct ablation studies demonstrate the effectiveness of the components of DynamicCity.
VAE. The effectiveness of the VAE is driven by two key innovations: Projection Module and Expan- sion& & Squeeze Strategy (ESS). As shown Tab. 3, the proposed Projection Module substantially improves HexPlane fitting performance, delivering up to 12.56% increase in mloU compared traditional averaging operations. Additionally, compared to querying each point individually, ESS enhances HexPlane fitting quality with up to 7.05% mloU improvement, significantly boosts training speed by up to 2.06x, and reduces memory usage by a substantial 70.84%.
HexPlane Dimensions. The dimensions of HexPlane have direct impact both training efficiency and reconstruction quality. Table 4 provides comparison of various downsample rates applied to the original HexPlane dimensions, which are 16 128 128 x 8 for CarlaSC and 16 x 200 x 200 16 for Oce3D- Occ3D-Waymo. As the downsampling rates increase, both the compression rate and training efficiency improve significantll, but the reconstruction quality, measured by mloU, decreases.

---

Published as a conference paper at ICLR 2025
Table Ablation Study on VAE Network Structures. We report the mloU scores, training time (second-per ceaandd-ee--ieee per-iteration), and training-time memory consumption (VRAM) of different Encoder and Decoder configurations on CarlaSC and Oce3D- -ayyoo respectively. Note that "ESS" denotes "Expansion & Squeeze". The best and second-best values are in bold and underlined.
CarlaSC Time (8),
Oce3D- Waymo
Encoder
Decoder
mloUt
VRAM (G))
mloU?
Time (s)
VRAM (G),
Average Pooling
Query ESS
60.97%
0.236
12.46
49.37%
1.563
69.66
Average Pooling
68.02%
0.143
4.27
55.72%
0.758
20.31
Projection
Query ESS
68.73%
0.292
13.59
61.93%
2.128
73.15
Projection
74.22% %
0.205
5.92
62.57 %
1.316
25.92
Table 4: Ablation Study on HexPlane Downsampling (D.S.) Rates. We report the compression ratios (C.R.), mloU scores, training speed (seconds per iteration), and training-tima memory consump- tion CarlaSC and Occ3D- -ayymo. The best and second-best values are in bold and underlined.
D.S. Rates
CarlaSC
Dec3D-Waym
dT dx
dy
dz
C.R.t
mloUt
Time (s)
VRAM (G)
C.R.t
mloUt
Time (s))
VRAM (G)
1
1
5.78%
84.67%
1.149
21.63
Out-of-Memory
>80
2
2
1
17.96%
76.05%
0.289
8.49
38.42%
63.30 %
1.852
32.82
2
2
23.14%
74.22%
0.205
5.92
48.25%
62.37%
0.935
24.9
2
4
4
2
71.86%
65.15%
0.199
4.00
153.69%
58.13%
0.877
22.30
Table 5: Ablation Study Organizing HexPlane as Image Tokens. We report the Inception Score (IS), Fréchet Inception Distance (FID), Kernel Inception Distance (KID), and the Precision (P) and Recall (R) rates on CarlaSC. The best values highlighted in bold.
Metric
Metric
Method
D
FID 4
KID
pinn
R
Rf
18
FID
KID
pip
R
Direct Unlold
2.496
205.0
0.248
0.000
0.000
2.269
9110
723.7
0.173
0.043
Vertical Concatenation
2.476
12.79
0.003
0.191
0.042
2.305
623.2
26.67
0.424
0.159
Padded Rollout
2.498
10.96
0.002
0.238
0.066
2.331
354.2
19.10
0.460
0.170
To achieve the optimal balance between training efficiency and reconstruction quality, we select a downsampling rate of di dx dy d2 = 2.
Padded Rollout Operation. We compare the Padded Rollout Operation with different strategics for obtaining image tokens: 1) Direct Unfold: directly unfolding the six planes into patches and concatenating them; 2) Vertical Concat: vertically concatenating the six planes without aligning dimensions during the rollout process. As shown in Tab. 5, Padded Rollout Operation (PRO) efficiently models spatial and temporal relationships in the token sequence, achieving optimal generation quality.
6 CONCLUSION
We present DynamicCity, a framework for high-quality 4D occupancy generation that captures the temporal dynamics of real-world environments. Our method introduces HexPlane, compact 4D representation generated using a VAE with a Projection Module, alongside an Expansion & Squeeze Strategy to enhance reconstruction efficiency and accuracy. Additionally, our Masked Rollout Operation reorganizes HexPlane features for DiT-based diffusion, enabling versatile 4D scene generation. Extensive experiments demonstrate that DynamicCity surpasses tfte-of--he-arr methods in both reconstruction and generation, offering significant improvements in quality, training speed, and memory efficiency. DynamicCity paves the way for future research in dynamic scene generation.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This study is supported by the National Key R&D Program of China No.20222D0160102 This study is also supported by the Ministry of Education, Singapore, under its MOE ACRF Tier 2 (MOE- T2EP20221-0012, MOE-T2EP20223-00022 NTU NAP, and under the RIE2020 Industry Alignment Fund - Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s). This study is also under the programme DesCartes and is supported by the National Research Foundation, Prime Minister's ' Office, Singapore, under its Campus for Research Excellence and Technological Enterprise (CREATE) programme.
We sincerely thank our reviewers and ACs for the time and effort they devoted during the review.
REFERENCES
Antonio Alliegro, Yawar Siddiqui, Tatiana Tommasi, and Matthias Nießner. PolyDiff: Generating 3D polygonal meshes with diffusion models. arXiv preprint arXiv:2312.11417, 2023.
Sherwin Bahmani, Xian Liu, Yifan Wang, Ivan Skorokhodov, Victor Rong, Ziwei Liu, Xihui Liu, Jeong Joon Park, Sergey Tulyakov, Gordon Wetzstein, Andrea Tagliasacchi, and David B. Lindell. TC4D: Tnajectory-conditionnl text-to-4D generation. arXiv preprint arXiv:2403. 17920, 2024. 3
Maxim Berman, Amal Rannen Triki, and Matthew B Blaschko. The lovász-softmax loss: A tractable measure in neural networks. In
surrogate for the optimization of the intersection-over-une IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
2018.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution vidco synthesis with latent difffusion models. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22563-22575, 2023.
Lucas Caccia, Herke Hoof, Aaron Courville, and Joelle Pineau. Deep generative modeling of LiDAR data. In JEEE/RSJ International Conference on Intelligent Robots and Systems, pp.
5034-5040, 2019.
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuScenes: multimodal dataset for autonomous driving. In JEEE/CVF Conference on Computer Vision and Pallern Recognition, pp.
11621-11631, 2020. 7, 16
Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. In JEEE/CVF
Conference on Computer Vision and Pattern Recognition, pp. 130-141, 2023. 2,
Eric R Chan, Connor Z Lin, Matthew Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay. Sameh Khamis. et al. Efficient geometry-awar 3D generative adversarial networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16123-16133, 2022. 3
Christopher Choy, Jun YunYoun Gwak, and Silvio Savarese. 4D spatio-temporal convnets: Minkowski convolutional neural networks. In IEEE/CVF Conference Computer Vision and Pattern Recog- nition, pp. 3075 3084, 2019. 17
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with io-awareness. In Advances in Neural Information Processing Systems, volume 35, pp. 16344-16359, 2022. 18
Sara Fridovich-Keil. Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angioo Kanazawa. K-planes: Explicit radiance fields space, time, and appearance. IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 12479-12488, 2023. 3.4
Jonathan Ho and Tim Salimans. Classifier free diffusion guidance. arXiv preprint arXiv:2207.1259.
2022. 6, 19
11

---

Published as a conference paper at ICLR 2025
Fangzhou Hong, Lingdong Kong, Hui Zhou, Xinge Zhu, Hongsheng Li, and Ziwei Liu. Unified 3D and 4D panoptic segmentation via dynamic shifting networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(5):3480-3499 2024.
Qianjiang Hu, Zhimin Zhang, and Wei Hu. RangeLDM: Fast realistic LiDAR point cloud generation. In European Conference Computer Vision, pp. 115-135, 2024. 3
Siyuan Huang, Yichen Xie, Song-Chun Zhu, and Yixin Zhu. Spatio-temporal self-supervised representation learning for 3D point clouds. In IEEE/CVF International Conference Computer Vision, pp. 6535-6545, 2021.
Yanqin Jiang, Li Zhang, Jin Gao, Weimin Hu, and Yao Yao. Consistent4D: Consistent 360 dynamic object generation from monocular video. arXiv preprint arXiv:231 (tt::23/1.0028440 1,02848,, 2023. 3
Jumin Lee, Sebin Lee, Changho Jo, Woobin Im, Juhyeong Seon, and Sung-Eui Yoon. SemCity: Semantic scene generation with triplane diffusion. /EEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 28337-28344 2024. 2, 7, 8, 19, 22, 23
Yuheng Liu, Xinke Li, Xueting Li, Lu Qi, Chongshou Li, and Ming-Hsuan Yang. Pyramid diffusion for fine 3D large scene generation. arXiv preprint arXiv:2311. 12085, 2023a. 2
Zhen Liu, Yao Feng, Michael J. Black, Derek Nowrouzezahraii Liam Paull, and Weiyang Liu. MeshDiffusion: Score-based generative 3D mesh modeling. In International Conference on Learning Representations, 2023b. 3
Zhiyuan Ma, Yuxiang Wei. Yabin Zhang, Xiangyu Zhu, Zhen Lei. and Lei Zhang, ScaleDreamer: Scalable text-to-3D synthesis with asynchronous score distillation. In European Conference on Computer Vision, pp. 1-19, 2024. 3
Kazuto Nakashima and Ryo Kurazume. Learning to drop points for LiDAR scan synthesis. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 222-229, 2021.
Kazuto Nakashima and Ryo Kurazume. LiDAR data symthesis with denoising diffusion probabilistic models. In IEEE International Conference on Robotics and Automation, pp. 14724-14731, 2024.
Kazuto Nakashima, Yumi Iwashita, and Ryo Kurazume. Generative range imaging for learning scene priors of 3D LiDAR data. In IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1256-1266, 2023. 2,3
Lucas Nunes, Rodrigo Marcuzzi, Benedikt Mersch, Jens Behley, and Cyrill Stachniss. Scaling diffusion models to real-world 3D LiDAR scene completion. /EEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14770-14780, 2024. 3
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32: 8026-8037, 2019. 18
William Peebles and Saining Xie. Scalable diffusion models with transformers. International Conference Computer Vision, pp. 4195-4205, 2023. 2, 4,
IEEE/CVF
Haoxi Ran, Vitor Guizilini, and Yue Wang. Towards realistic scene generation with LiDAR diffusion models. In JEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14738-14748,
2024.7 3
Jiawei Ren, Liang Pan, Jiaxiang Tang, Chi Zhang, Ang Cao, Gang Zeng, and Ziwei Liu. DreamGaus- 4ia Generative 4D gaussian splatting. arXiv preprint 21214:2312.17142, 2023.
Jiawei Ren, Kevin Xie, Ashkan Mirzaei, Hanxue Liang, Xiaohui Zeng, Karsten Kreis, Ziwei Liu, Antonio Torralba, Sanja Fidler, Seung Wook Kim, and Huan Ling. L4GM: Large gaussian reconstruction model. arXiv preprint arXiv:2406, 10324, 2024a. 3
12

---

Published as a conference paper at ICLR 2025
Xuanchi Ren, Jiahui Huang. Xiaohui Zeng, Ken Museth, Sanja Fidler, and Francis Williams. XCube: Large-scale 3D generative modeling using sparse voxel hierarchies. In JEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 4209 2209-4219, 2024b. 2,
Sara Rojas, Julien Philip, Kai Zhang, Sai Bi, Fujun Luan, Bernard Ghanem, and Kalyan Sunkavall. DATENeRF: Depth-aware text-based editing of nerfs. arXiv preprint arXiv:2404.04555 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 10684-10695, 2022.
Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. MVDream: Multi-view diffusion for 3D generation. arXiv preprint urXiv:2308. 16512, 2023.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1555 2015. 17
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-A-Video Text-to- video generation without text-video data. In International Conference on Learning Representations, 2022. 3
Uriel Singer, Shelly Sheynin, Adam Polyak, Oron Ashual, lurii Makarov, Filippos Kokkinos, Naman Goyal, Andrea Vedaldi, Devi Parikh, Justin Johnson, and Yaniv Taigman. Text-to-4D dynamic scene generation. arXiv preprint 2tti::230..128800 2023. 3
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Alcksei Timofeev, Scott Ettinger. Maxim Krivokon, Amy Gao. Aditya Joshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov. Scalability in perception for autonomous driving: Waymo open dataset. In IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 2446 2454, 2020. 7, 16
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2818 2826, 2015. 17
Haotian Tang, Zhijian Liu, Shengyu Zhao, Yujun Lin, Ji Lin, Hanrui Wang, and Song Han. Searching efficient 3D architectures with sparse point-voxel convolution. In European Conference on Computer Vision, pp. 685-702, 2020. 17
Xiaoyu Tian, Tao Jiang, Longfei Yun, Yucheng Mao, Huitong Yang. Yue Wang, Yilun Wang, and Hang Zhao. Occ3D: A large-scale 3D occupancy prediction benchmark for autonomous driving. In Advances in Neural Information Processing Systems, volume 36, pp. 64318-64330, 2023. 7, 16, 21, 22, 23, 24, 26, 27
Lening Wang, Wenzhao Zheng, Yilong Ren, Han Jiang, Zhiyong Cui, Haiyang Yu, and Jiwen Lu. OccSora: 4D occupancy generation models as world simulators for autonomous driving. arXiv preprint arXiv:2405.203:: 2024. 2, 7, 8, 21, 22, 23, 31
Joey Wilson, Jingyu Song, Yuewei Fu, Arthur Zhang, Andrew Capodieci, Paramsothy Jayakumar, Kira Barton, and Maani Ghaffari. MotionSC: Data set and network for real-time semantic mapping in dynamic environments. IEEE Robotics and Automation Letters, 73)::4439-844, 2022. 7, 16, 21, 22, 23, 24, 25, 28, 29, 30, 31, 32,
Kailu Wu, Fangfu Liu, Zhihan Cai, Runjie Yan, Hanyang Wang, Yating Hu, Yueqi Duan, and Kaisheng Ma. Unique3D: High-quality and efficient 3D mesh generation from single image. arXiv preprint arXiv: 2024a.
Shuang Wu, Youtian Lin, Feihu Zhang, Yifei Zeng, Jingxi Xu, Philip Tor, Xun Cao, and Yao Yao. Direct3D: Scalable image-to-3D generation via 3D latent diffusion transformer. arXiv preprint a405v:2405. 14832, 2024b.
13

---

Published as a conference paper at ICLR 2025
Haozhe Xie, Zhaoxi Chen, Fangzhou Hong, and Ziwei Liu. CitDDreamer4D: Compositional genera- tive model unbounded 4D cities. arXiv preprint arXiv:2501.08983, 2025a. 3
Haozhe Xie, Zhaoxi Chen, Fangzhou Hong, and Ziwei Liu. Generative gaussian splatting for unbounded 3D city generation. In IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2025b.
Yuwen Xiong, Wei-Chiu Ma, Jingkang Wang, and Raquel Urtasun. UltraLiDAR: Learning compact representations for LiDAR completion and generation. arXiv preprint 1.Xi::3311.0144. 2023.
Wenzhao Zheng, Weiliang Chen, Yuanhui Huang, Borui Zhang, Yueqi Duan, and Jiwen Lu. Oceworld: Learning 3D occupancy world model for autonomous driving. In European Conference Computer Vision, pp. 55-72, 2024a.
Zehan Zheng, Fan Lu, Weiyi Xue, Guang Chen, and Changjun Jiang. LiDAR4D: Dynamic neural fields for novel space-time view LiDAR synthesis. IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5145-5154, 2024b. 3
Vlas Zyrianov, Xiyue Zhu, and Shenlong Wang. Learning to generate realistic LiDAR point clouds. In European Conference on Computer Vision, pp. 17-35, 2022. 3
14

---

Published as a conference paper at ICLR 2025
APPENDIX
In this appendix, we supplement the following materials to support the findings and conclusions drawn in the main body of this paper.
A Additional Implementation Details
16
A.I Datasets
16
A.2 DIT Evaluation Metrics
17
A.3 Model Details
18
A.4 Classifier-Free Guidance
19
A.5
Downstream Applications
19
Additional Quantitative Results
21
B.1 Per-Class Generation Results
B.2
Occupancy Forecasting Results
21
B.3 User Study
22
B.4 Model Stats
22
B.5
Comparisons with SemCity
22
C
Additional Qualitative Results
24
C.1 Unconditional Dynamic Scene Generation
24
C.2 HexPlane-Guided Generation
26
C33 Layout-Guided Generation
27
C.4 Command- & Trajectory- Generation
28
C.5 Dynamic Inpainting ................................
30
C.6 Comparisons with OccSora
31
C.7 Dynamic Outpainting
................................
32
C.8 Single Frame Occupancy Conditional Generation
33
Potential Societal Impact & Limitations
34
D.1 Societal Impact
34
D.2 Broader Impact
D.3 Known Limitations
34
Public Resources Used
35
E.1 Public Datasets Used
35
E.2 Public Implementations Used
35
15

---

Published as a conference paper at ICLR 2025
A ADDITIONAL IMPLEMENTATION DETAILS
In this section, we provide additional implementation details to assist in reproducing this work. Specifically, we elaborate on the details of the datasets, DIT evaluation metrics, the specifics of our generation models, and discussions on the downstream applications.
A.1 DATASETS
Our experiments primarily utilize two datasets: Occ3D-Wayme (Tian et al., 2023) and CarlaSC (Wil- son et al., 2022). Additionally, we also evaluate our VAE on Occ3D-nuScenes (Tian et al., 2023).
The Occ3D Occ3D-Waymo dataset is derived from real-world Waymo Open Dataset (Sun al., 2020) data, where occupancy sequences are obtained through multi-frame fusion and voxelization processes. Similarly, Occ3D-nuSeenes is generated from the real-world nuScenes (Caesar et al., 2020) dataset using the same fusion and voxelization operations. On the other hand, the CarlaSC dataset is generated from simulated scenes and sensor data, yielding occupancy sequences.
Using these different datasets demonstrates the effectiveness of our method on both real-world and synthetic data. To ensure consistency in the experimental setup, we select 11 commonly used semantic categories and map the original categories from both datasets to these 11 categories. The detailed semantic label mappings are provided in Tab. 6.
Table 6: Summary of Semantic Label Mappings. We unify the semantic classes between Car- laSC (Wilson et al., 2022), Occ3D- Occ3D-Waymo (Tian et al., 2023), and mns3eee (Tian et al., 2023) datasets for semantic scene generation.
Class
CarlaSC
Occ3D-Waymo
Oce3D-nuSeenes
Building
Building
Building
Manmadc
Barrier
Barrier, Wall, Guardrail
Barrier
Other
Other, Sky, Bridge, Rail track, Static, Dynamic,
General Object
General Object
Water
Pedestrian
Pedestrian
Pedestrian
Pedestrian
Pole
Pole, Traffic sign, Traffic light
Sign, Traffic light, Pole, Construction Cone
Traffic cone
Road
Road, Roadlines
Road
Drivable surface
Ground
Ground, Terrain
Other flat, Terrain
Sidewalk
Sidewalk
Sidewalk
Sidewalk
Vegetation
Vegetation
Vegetation, Tree trunk
Vegetation
Vehicle
Vehicle
Vehicle
Bus, Car. Construction
vehicle, Trailer, Truck
Bicycle
Bicyclist, Bicycle, Motorcyele
Bicycle. Motorcycle
Oce3D-Waymo, This dataset contains 798 training scenes, with each scene lasting approxi- mately 20 seconds and sampled at a frequency of 10 Hz. This dataset includes 15 semantic categories. We use volumes with a resolution of 200 200 x 16 from this dataset.
CarlaSC. This dataset contains 6 training scenes, each duplicated into Light, Medium, and Heavy based on traffic density. Each scene lasts approximately 180 seconds and sampled at a frequency of 10 Hz. This dataset contains 22 semantic categories, and the scene resolution is 128 X 128 x 8.
Oce3D-nuScenes.. This dataset contains 600 scenes, with each scene lasting approximately 20 seconds and sampled at frequency of 2 Hz. Compared to Occ3D- --ey-Wymoo and CarlaSC, OccSD-nuSeenes has fewer total frames and more variation between scenes. This dataset includes 17 semantic categories, with resolution of 200 x 200 x 16.
16

---

Published as a conference paper at ICLR 2025
A.2 DIT EVALUATION METRICS
Inception Score (IS). This metric evaluates the quality and diversity of generated samples using a pre-trained Inception model follows:
IS = exp (EQup, [PL(((()))) p(u))))
(5)
where Pa represents the distribution of generated samples. p(v]Q) is the conditional label distribution given by the Inception model for generated sample Q p(y) = f p(y Q),, dQ the marginal distribution over all generated samples. (pu(p(y)Q) p(3)) is the Dallback-Leibler divergence, defined as follows:
p(vlQQ)
(KL( p())) Σ p(ys Q) log
(6)
p(y)
Fréchet Inception Distance (FID). This metric measures the distance between the feature distribu- tions of real and generated samples:
FID = 1|, 1 + Tr E, + -g -
(7)
where p4 and Dn are the mean and covariance matrix of features from real samples. My and Mg are the mean and covariance matrix of features from generated samples. Tr denotes the trace of matrix.
Kernel Inception Distance (KID). This metric uses the squared Maximum Mean Discrepancy (MMD) with polynomial kernel as follows:
KID = MMD2 (Q,)),
(8)
where ((²,) and ((2,)) represent the features of real and generated samples extracted from the Inception model.
MMD with polynomial kernel k(x,)) v) = (x) T y + c) is calculated as follows:
MMD) m(m k( IT) + n(n Σ k(n.
Σ k(ai, wy),
(9)
iデj where X (Q Qm and Y (y1...., yn sets of features from real and generated samples.
Precision. This metric measures the fraction of generated samples that lie within the real data distribution as follows:
Precision
1
Σ 1 - (1) n.) S x²)
(10)
N
2=1
where fy is a generated sample in the feature space. JA, and E,, are the mean and covariance of the real data distribution. I1()) is the indicator function. x" is a threshold based on the chi-squared distribution.
Recall. This metric measures the fraction of real samples that lie within the generated data distribution as follows:
1 M
Recall = M
Σ [(f,, - (191-Ma)77 T mg) S xx²).
(11)
j=1
where: f. real sample in the feature space. My and Eg are the mean and covariance of the generated data distribution. I((-) is the indicator function. x² threshold based on the chi-squared distribution.
2D Evaluations. We render 3D scenes as 2D images for 2D evaluations. To ensure fair comparisons, we use the same semantic colormap and camera settings across all experiments. Fig. 8 shows an example of rendered 2D semantic colormap. We use an Inception V3 (Szegedy et al., 2015) model to compute the Inception Score (IS), Fréchet Inception Distance (FID), and Kernel Inception Distance (KID) scores, while Precision and Recall are computed using a VGG-16 16 (Simonyan & Zisserman, 2015) model. We train both 2D backbones using semantic colormap data.
3D Evaluations. For 3D data, we train MinkowskiUNet (Choy et al., 2019) as an autoencoder. We adopt the latest implementation from SPVNAS (Tang et al., 2020), which supports optimized sparse convolution operations. The features were extracted by applying average pooling to the output of the final downsampling block.
17

---

Published as a conference paper at ICLR 2025
Figure 8: Example of 2D Evaluation Rendering.
A.3 MODEL DETAILS
General Training Details. We implement both the VAE and DiT models using PyTorch (Paszke et al., 2019). We utilize PyTorch's mixed precision and replace all attention mechanisms with FlashAttention (Dao et al., 2022) to accelerate training and reduce memory usage. AdamW used as the optimizer for all models.
We train the VAE with a learning rate of 10-3, running for 20 epochs on Bec3D-Waymo and 100 epochs CarlaSC. The DiT is trained with learning rate of 10-4, and the EMA rate for DiT is set to 0.9999.
VAE. Our encoder projects the 4D input Q into a HexPlane, where each dimension is compressed
version of the original 4D input. First, a 3D CNN is applied to each frame for feature extraction and downsampling. with dimensionality reduction applied only the spatial dimensions (X, Y. Z). Next, the Projection Module projects the 4D features into the HexPlane. Each small transformer within the Projection Module consists of layers, and the attention mechanism has two heads. Each head has a dimensionality of 16, with dropout rate of 0.1. Afterward, we further downsample the T dimension to half of its original size.
During decoding, we first use three small transpose CNNs to restore the T dimension, then use an ESS module to restore the 4D features. Finally, we apply 3D CNN to recover the spatial dimensions and generate point-wise predictions.
Diffusion. We set the patch size p to for our DiT models. The Waymo DiT model has hidden size of 768, 18 DiT blocks, and 12 attention heads. The CarlaSC DiT model has hidden size of 384, 16 DiT blocks, and attention heads.
Discussion on VAE Structure Improvements. Some prior work utilizes sparse 3D structures to enhance the efficiency of their backbones. For example, XCube (Ren et al., 2024b) employs a fully sparse 3D encoder, significantly improving model efficiency. Similarly, VAE could potentially improve the 3D convolutional feature extractor f-(-) by adopting sparse convolution. However, using sparse convolution offers only limited efficiency gains, convolution accounts for only small
18

---

Published as a conference paper at ICLR 2025
portion of our VAE. Moreover, like XCube, we cannot apply sparse convolution in our decoder. In the future, we plan to explore more efficient operations to further optimize our 3D backbone.
A.4 CLASSIFER--REE GUIDANCE
Classifier-Free Guidance (CFG) (Ho & Salimans, 2022) could improve the performance of conditional generative models without relying on an external classifier. Specifically, during training. the model simultancously learns both conditional generation p(z|c) and unconditional generation p(x), and guidance during sampling is provided by the following equation:
i = (1 w) #x(()) ())
(12)
where (c) the result conditioned on c, (0) is the unconditioned result, and is weight parameter controlling the strength of the conditional guidance. By adjusting appropriate balance between the accuracy and diversity of the generated scenes can achieved.
A.5 DOWNSTREAM APPLICATIONS
This section provides a comprehensive explanation of five tasks to demonstrate the capability of our 4D scene generation model across various scenarios.
HexPlane. Since our model is based on Latent Diffusion Models, it is inherently constrained to generate results that match the latent space dimensions, limiting the temporal length of unconditionally generated sequences. We argue that a robust 4D generation model should not be restricted to producing only short sequences. Instead of increasing latent space size, we leverage CFG to generate sequences in an auto-regressive manner. By conditioning each new 4D sequence on the previous one, we sequentially extend the temporal dimension. This iterative process significantly extends sequence length, enabling long-term generation, and allows conditioning on any real-world 4D scene to predict the next sequence using the DiT model. Theoretically, our HexPlane conditional generation can model sequence of arbitrary length, but less stable generation may occur when generating very long sequences.
We condition our DiT by using the HexPlane from T frames earlier. For any condition HexPlane, we apply patch embedding and positional encoding operations to obtain condition tokens. These tokens, combined with other conditions, are fed into the adaLN-Zero and Cross-Attention branches to influence the main branch.
Layout. To control object placement in the scene, we train model capable of generating vehicle dynamics based on a bird's s-yeye view sketch. We apply semantic filtering to the bird's Baddsseyee view of the input scene, marking regions with vehicles as 1 and regions without vehicles as O. Pooling this binary image provides layout information as a x H x W tensor from the bird's-eye s-eye perspective. The layout is padded match the size of the HexPlane, ensuring that the positional encoding of the bird's 'eyde-ee layout aligns with the XY plane. DiT learns the correspondence between the layout and vehicle semantics using the same conditional injection method applied to the HexPlane.
Command. While we have developed effective methods to control the HexPlane in both temporal and spatial dimensions, critical aspect of 4D autonomous driving scenarios the motion of the ego vehicle. To address this, we define four commands: STATIC, FORWARD, TURN LEFT, and TURN RIGHT, and annotate our training data by analyzing ego vehicle poses. During training, we follow the traditional DiT approach of injecting class labels, where the commands are embedded and fed into the model via adeLN-
Trajectory. For more fine-grained control of the ego vehicle's motion, we extend the command- based conditioning into a trajectory condition branch. For any 4D scene, the XY coordinates of the trajectory traj E 7'x2 are passed through an MLP and injected into the adaLN-Zero branch.
Inpaint. We demonstrate that our model can handle versatile applications by training a conditional DîT for the previous tasks. Extending our exploration of downstream applications, and inspired by (Lee et al., 2024), we leverage the 2D structure of our latent space and the explicit modeling of each dimension to highlight our model's 's ability to perform inpainting 4D scenes. During DiT sampling, we define a 2D mask m E IR XxY on the XY plane, which is extended across all dimensions to mask specific regions of the HexPlane.
19

---

Published as a conference paper at ICLR 2025
At each step of the diffusion process, apply noise the input 7h and update the HexPlane using the following formula:
H4. m Hz + m) @ 7
(13)
where @ denotes the clement-wise product. This process inpaints the masked regions while preserving the unmasked areas of the scene, enabling partial scene modification, such as turning an empty street into one with heavy traffic.
Outpaint. Outpainting extends the spatial dimensions of given occupancy sequence. We use the same procedure for outpainting as we do for inpainting. Specifically, we mask half of the scene, shift the latent representation, and apply the inpainting process. Consequently, we could obtain larger scene with consistent dynamies.
Single frame occupancy. We apply the same procedure for single-frame occupancy conditional generation as for HexPlane conditional generation. Specifically, we preprocess the data, encode the first frame of each training sequence HexPlane, and fine-tune our HexPlane generation model for single-framd conditional generation.
20

---

Published as a conference paper at ICLR 2025
B ADDITIONAL QUANTITATIVE RESULTS
In this section, present additional quantitative results to demonstrate the effectiveness of our VAE in accurately reconstructing 4D scenes.
B.1 PER-CLASS GENERATION RESULTS
We include the class-wise IoU scores OccSora (Wang al., 2024) and our proposed DynamicCity framework on CarlaSC (Wilson et al., 2022). As shown in Tab. 7, our results demonstrate higher IoU across all classes, indicating that our VAE reconstruction achieves minimal information loss. Additionally, our model does not exhibit significantly low IoU for any specific class, proving its ability to effectively handle class imbalance.
Table Comparisons of Per-Class IoU Scores. We compared the performance of OceSora (Wang et al., 2024), and our DynamicCity framework on CarlaSC (Wilson et al., 2022) across 10 semantic classes. The scene resolution 128 128 The sequence lengths are 4, 8, 16, and 32, respectively.
Pedestrian
Vegetation
Building
Barrier
Ground
Sidewalk
Vehicle
Other
Road
Pole
Method
mloU
Resolution:
128
Sequence Length:
OccSora
41.009
38.861
10.616
6.637
19.191
21.825
93.910
61.357
86.671
15.685
55,340
Ours
79.604
76.364
31,354
68.898
93.436
87.962
98.617
87.014
95.129
68.700
88.569
Imprev:
38.595
37.503
20.738
62.261
74.245
66.137
4.707
25.657
8.458
$3.015
33.229
Resolution:
128 128 x
Sequence
Length: 8
OceSora
39.910
33.001
3.260
5.659
19.224
19.357
93.038
57.335
85.551
30.899
51.776
Ours
76.181
70.874
50.025
52.433
87.958
85.866
97.513
83.074
93.944
58.626
81.498
Improv:
36.271
37.873
46.765
46.774
68.734
66.509
4.475
25.739
8.393
27.727
29.722
Resolution:
128 x 128 x 8
Sequence
Length: 16
OccSora
33.404
19,264
2.205
3.454
11.781
9.165
92.054
50.077
82.594
18.078
45.363
Ours
74.223
66.852
51.901
49.844
79.410
82.369
96.937
84.484
94.082
58.217
78.134
Improv:
40.819
47.588
49.696
46.390
67.629
73.204
4.883
34.407
11.488
40.139
32.771
Resolution:
128
Sequence
Length: 32
OceSora
28.911
16.565
1.413
0.944
6,200
4.150
91.466
43.399
78.614
11.007
35,353
Ours
59,308
52,036
25,521
29,382
56,811
57.876
94.792
78,390
89,955
46.080
62.234
Improv.
30.397
35.471
24.108
28.438
50.611
53.726
3.326
34.991
11.341
35.073
26.881
B.2 OCCUPANCY FORECASTING RESULTS
We train our HexPlane conditional generation pipeline on Oce3D-nuSeenes (Tian et al., 2023) as an occupancy forecasting model. We set T = 4 to ensure the model receives HexPlane with context length of seconds, aligning with OceWorld (Zheng al., 2024a), and generates the next seconds for evaluation. As shown in Tab. 8, our model outperforms OccWorld on most metrics.
Table 8: 4D Occupancy Forecasting Performance. We compare the performance of Occ- World (Zheng al., 2024a) and proposed DynamicCity framework Occ3D-nuScena (Tian et al., 2023).
mIoU
IoU
Method
T=0 0
T=1
T=
2
T=
T=1 1
T
OcoWord Ours
66.38
25.78
15.14
62.29
34.63
25.07
80.52
26.18
16.94
67.64
34.12
25.82
21

---

Published as a conference paper at ICLR 2025
B.3
USER STUDY
We conduct a user study comparing OceSora (Wang et al., 2024) with our proposed DynamicCity. The study includes 20 samples, with 10 from each method. Participants rate each sample four metrics: 1) overall quality, 2) time consistency, 3) background quality, and 4) foreground quality. Ratings range from to 5, with being the highest. We collect results from volunteers and get 840 valid scores in total, as shown in Tab. 9, Our method receives better user feedback across all metrics.
Table 9: User Study Results. We conduct user study comparing OccSora (Wang et al., 2024) and DynamicCity. The rating is of scale 1-5, the higher the better.
Method
Overall Quality
Time Consistency
Background Quality
Foreground Quality
OccSora 2.21
2.05
2.17
2.11
Ours
4.03
4.02
3.95
4.04
B.4 MODEL STATS
We compare the training speed, inference speed, training VRAM, and inference VRAM of Oce- Sora (Wang et al., 2024) and DynamicCity. The results presented in Tab. 10, Tab. 11, Tab. 12, and Tab. 13. While some of our models may be slightly slower and consume more memory compared OccSora, they achieve significantly better performance. We also compare the total model size of OceSora and our model in Tab. 14. Our model significantly smaller than OccSora while achieving superior performance.
Table 10: VAE Model Statisties CarlaSC Dataset (Wilson et al., 2022). We compare the training time, inference time, training VRAM, inference VRAM of OccSora (Wang et al., 2024) and our DynamicCity.
Method
Training Time (s).
Inference Time (s)
Training VRAM (G)
Inference VRAM (G))
OccSora
0.36
0.21
4.86
3.25
Ours
0.21
0.41
5.92
1.43
Table 11: VAE Model Statistics on Occ3D-Waymo Dataset (Tian et al., 2023). We compare the training time, inference time, training VRAM, inference VRAM of OccSora (Wang et al., 2024) and our DynamicCity.
Method
Training Time (8).
Inference Time (s).
Training VRAM (G).
Inference VRAM (G).
OccSora 0.63
0.21
10.05
3,93
Ours
0.94
0.54
24.90
4.62
B.5
COMPARISONS WITH SEMCITY
We compare the generation quality of SemCity (Lee al., 2024) and our DynamicCity in Tab. 15. Despite using a more compact latent representation and generating dynamic scenes, our model outperforms SemCity on most metrics.
22

---

Published as a conference paper at ICLR 2025
Table 12: DiT Model Statisties on CarlaSC Dataset (Wilson et al., 2022). We compare the training time, inference time, training VRAM, inference VRAM of OccSora (Wang et al., 2024) and our DynamicCity.
Method
Training Time (8),
Inference Time (s)
Training VRAM (G)
Inference VRAM (G)
OccSora
0.19
6.10
1.50
1.15
Ours
0.19
3.91
10.22
1.28
Table 13: DiT Model Statistics Dec3D-Wayne Dataset (Tian et al., 2023). We compare the training time, inference time, training VRAM, inference VRAM of OceSora (Wang et al., 2024) and DynamicCity.
Method
Training Time (8).
Inference Time (s).
Training VRAM (G).
Inferenee VRAM (G))
OceSora
0.35
6.09
15.16 16
1.15
Ours
0.45
4.41
22.33
1.29
Table 14: Model Size. We compare the total model size of OceSora (Wang et al., 2024) and our DynamicCity.
Dataset
Method
Model Size (M)
OccSora
169.1
CarlaSC
Ours
44.7
OccSora
174.2
Dec3D-Waymo
Ours
45.6
Table 15: Comparisons of 2D and 3D Evaluation Metrics. We report the Inception Score (IS), Fréchet Inception Distance (FID), Kernel Inception Distance (KID), and the Precision (P) and Recall (R) rates for SemCity (Lee et al., 2024) and our method in both the 2D and 3D spaces.
2e
Metric 3D
Method
IS
FID,
KIDJ
P
Rt
IST
FIDI
KIDJ
Pt
R1
SemCity
1.039
35.40
0.010
0.213
0.058
2.288
1113
53.948
0.253
0.787
Ours
1.040
12.94
0.002
0.307
0.018
2.331
427.5
27.869
0.460
0.170
23

---

Published as a conference paper at ICLR 2025
C ADDITIONAL QUALITATIVE RESULTS
In this section, we provide additional qualitative results on the Occ31>-Waymo (Tian et al., 2023) and CarlaSC (Wilson al., 2022) datasets to demonstrate the effectiveness of our approach.
C.I UNCONDITIONAL DYNAMIC SCENE GENERATION
First, we present full unconditional generation results in Fig. 9 and 10. These results demonstrate that our generated scenes are high quality, realistic, and contain significant detail, capturing both the overall scene dynamics and the movement of objects within the scenes.
T=1
T=2
T=3
T=4
T=8
T=7
T=6
T=5
T=9
T=10
T=11
T=12
T= T=16
T=15
T= 14
T=13
Build
Vehicle
Ped
Road
Sidewalk
Barrier
Ground
Veg
Other
Pole
We provide qualitative examples of total of 16 consectutive frames generated by DynamicCity on the Oce31- Occ3ID-ayymo (Tian et al., 2023) dataset. Best viewed in colors and zoomed-in in for additional details.
Figure Unconditional Dynamic Scene Generation Results.
24

---

Published as a conference paper at ICLR 2025
12 1
T=1
T=2
T=3
T=4
T=8
T=7
T=6
T=5
T=9
T=10
T=12
T= 16
T=15
T=14
T=13
Build
Vehicle
Ped
Road
Sidewalk
Barrier
Ground
Veg
Other
Pole
Unconditional Dynamic Scene Generation Results. We provide qualitative examples of a total of consectutive frames generated by DynamicCity on the CarlaSC (Wilson et al., 2022) dataset. Best viewed in colors and zoomed-in for additional details.
Figure 10:
25

---

Published as a conference paper at ICLR 2025
C.2 HEXPLANE DUIDDD GENERATION
We show results for our HexPlane conditional generation in Fig. 11. Although the sequences are generated in groups of 16 due to the settings of our VAE, we successfully generate long sequence by conditioning on the previous one. The result contains 64 frames, comprising four sequences, and depicts T-intersection with many cars parked along the roadside. This result demonstrates strong temporal consistency across sequences, proving that our framework can effectively predict the next sequence based on the current one.
4
5
B
16
15
13
12
11
10
18
19
20
21
22
23
24
32
31
30
29
28
27
26
33
34
35
36
37
38
39
40
48
47
46
45
44
43
42
41
50
52
53
54
55
56
64
63
62
61
60
59
58
57
Build
Vehicle
Ped
Road
Sidewalk
Barrier
Ground
Veg
Other
Pole
Figure 11: HexPlane-Guided Generation Results. We provide qualitative examples of a total of 64 consectutive frames generated by DynamicCity on the Occ3I-Waymo (Tian et al., 2023) dataset. Best viewed in colors and zoomed-in for additional details.
26

---

Published as a conference paper at ICLR 2025
C.3 LAYOUT-GUIDER GENERATION
The layout conditional generation result is presented in Fig. 12. First, we observe that the layout closely matches the semantic positions in the generated result. Additionally, the layout changes, the positions of the vehicles in the scene also change accordingly, demonstrating that our model effectively captures the condition and influences both the overall scene layout and vehicle placement.
T=2
T=3
T=6
T=5
T=4
B=T
G=L
T=10
T= 4
T= 5
Build
Vehicle
Ped
Road
Sidewalk
Barrier
Ground
Veg
Other
Figure 12:
Layout-Guided Generation Results. We provide qualitative examples of total of 16 consectutive frames generated by DynamicCity on the Foc3D-Wamma (Tian al., 2023) dataset. Best viewed in colors and zoomed-in for additional details.
27

---

Published as a conference paper at ICLR 2025
C.4 COMMAND- & TRAJECTORY- TRAJECTORYGUIDDE GUIDED GENERATION
We present command conditional generation in Fig. 13 and trajectory conditional generation in Fig. 14. These results show that when we input command, such as "right turn," sequence of XY-plane coordinates, our model can effectively control the motion of the ego vehicle and the relative motion of the entire scene based on these movement trends.
Turn Right
Turn Right
Turn Right
Turn Right
T=1
T=2
T=3
T=4
Turn Right
Turn Right
Turn Right
Turn Right
baspia
T=8
T=7
T=6
T=5
Turn Right
Türn Right
Turn Right
Turn Right
T=9
T=10
T=11
T=12
dap Turn Right
Turn Right
Cia Turn Right
Turn Right
T=15 15
T= 4
T= 1 3
Build
Vehicle
Ped
Road
Sidewalk
Barrier
Ground
Veg
Other
Pole
Figure 13: Hommand-Guidden Scene Generation Results. We provide qualitative examples of a total of 16 consectutive frames generated under the command RIGHT by DynamicCity on the CarlaSC (Wilson al., 2022) dataset. Best viewed in colors and zoomed-in for additional details.
28

---

Published as a conference paper at ICLR 2025
Seatiot
Yeans
Pea
T=1
T=2
T=3
T=4
T=8
CRA
T=7
T=6
T=5
T=9
T=10
T=11
T= 1 6
T=15
T= 14
T=
Build
Vehicle
Ped
Road
Sidewalk
Barrier
Ground
Veg
Other
Pole
Figure 14: Trajector-GGuided Scene Generation Results. We provide qualitative examples of total of 16 consectutive frames generated by DynamicCity on the CarlaSC (Wilson et al., 2022) dataset. Best viewed in colors and zoomed-in for additional details.
29

---

Published as a conference paper at ICLR 2025
C.5 DYNAMIC INPAINTING
We present the full inpainting results in Fig. 15, The results show that our model successfully regenerates the inpainted regions while ensuring that the areas outside the inpainted regions remain consistent with the original scene. Furthermore, the inpainted areas seamlessly blend into the original scene, exhibiting realistic placement and dynamics.
Before
After
Before
After
Before
After
T=1
T=2
T=3
Before
After
Before
After
Before
After
T=6
T=5
T=4
Before
After
Before
After
Before
After
T=7
T=B
T=9
Before
After
Before
After
Before
After
T=12
T=11
T=10
Before
After
Before
After
Before
After
13
T=14
T=15
Build
Vehicle
Ped
Road
Sidewalk
Barrier
Ground
Veg
Other
Pole
Figure 15:
Dynamic Inpainting Results. We provide qualitative examples total of 16 consectu- tive frames generated by DynamicCity on the CarlaSC (Wilson et al., 2022) dataset. Best viewed in colors and zoomed-in for additional details.
30

---

Published as a conference paper at ICLR 2025
C.6 COMPARISONS WITH OCCSORA
We compare our qualitative results with OceSora (Wang et al., 2024) in Fig. 16, using a similar scene. is evident that result presents realistic dynamic scene, with straight roads and complete objects and environments. In contrast, OccSora's result displays unreasonable semanties, such as pedestrian in the middle of the road, broken vehicles, and lack of dynamic elements. This comparison highlights the effectiveness of our method.
OceSora
Ours
OceSora
Ours
OceSora
Ours
T=1
T=2
T=3
OceSora
Durs
OcéSora
Ours
OctSort
Ours
T=6
T=5
T=4
OceSora
Ours
OceSora
Ours
OctSona
Ours
T=7
T=8
T=9
OceSora
Ours
OceSo
Ours
OctSon
Ours
T=12
T=11
T=1 T=10
OceSona
Ours
OcéSoral
Ours
OceSo
Ours
T= 113
T=14
T= 1-15 5
Build
Vehicle
Ped
Road
Sidewalk
Barrier
Ground
Veg
Other
Pole
Figure 16: Comparisons of Dynamic Scene Generation. We provide qualitative examples of a total of 16 consectutive frames generated by OceSora (Wang et al., 2024) and our proposed DynamicCity framework on the CarlaSC (Wilson et al., 2022) dataset. Best viewed in colors and zoomed-in for additional details.
31

---

Published as a conference paper at ICLR 2025
C.7 DYNAMIC OUTPAINTING
We present the full outpainting results in Fig. 17. The results demonstrate that our model can extend scene into larger dynamic scene.
Build
Vehicle
Road
Sidewalk
Barrier
Ground
Veg
Other
Pole
T=1
T=2
T=3
T=6
T=5
T=4
T=7
T=8
T=9
T=12
T=11
T=10
T=13
T=14
T=15
Figure 17: Dynamic Outpainting Results. We provide qualitative examples of a total of 16 consectutive frames generated by DynamicCity on the CarlaSC (Wilson al., 2022) dataset. Best viewed in colors and zoomed-in for additional details.
32

---

Published as a conference paper at ICLR 2025
C.8 SINGLE FRAME OCCUPANCY CONDITIONAL GENERATION
We present the results of generating frames based on single-frame occupancy condition in Fig. 18. The results demonstrate good temporal consistency with the condition frame, highlighting our model's ability to condition on easily accessible data.
Build
Vehicle
Ped
Road
Sidewalk
Barrier
Ground
Veg
Other
Pole
N
hu
Condition
T=2
T=3
T=4
N
T=B
T=7
T=6
T=5
T=9
T=10
T=11
T=12
5
T= 14
Figure 18: Single Frame Occupancy Conditional Generation Results. We provide qualitative examples of a total of 16 consectutive frames generated by DynamicCity on the CarlaSC (Wilson et al., 2022) dataset. Best viewed in colors and zoomed in for additional details.
33

---

Published as a conference paper at ICLR 2025
D POTENTIAL SOCIETAL IMPACT & LIMITATIONS
In this section, we elaborate on the potential positive and negative societal impact of this work, as well as the broader impact and some potential limitations.
D.1 SOCIETAL IMPACT
Our approach's ' ability to generate high-quality 4D occupancy holds the potential to significantly impact various domains, particularly autonomous driving, robotics, urban planning, and smart city development. By creating realistic, large-scale dynamic scenes, our model can aid in developing more robust and safe autonomous systems. These systems can be better trained and evaluated against diverse scenarios, including rare but critical edge cases like unexpected pedestrian movements or complex traffic patterns, which are difficult to capture in real-world datasets. This contribution can lead to safer autonomous vehicles, reducing traffic accidents, and improving traffic efficiency, ultimately benefiting society by enhancing transportation systems.
In addition to autonomous driving, DynamicCity can be valuable for developing virtual reality (VR) environments and augmented reality (AR) applications, enabling more realistic 3D simulations that could be used in various industries, including entertainment, training, and education. These advancements could help improve skill development in driving schools, emergency response training, and urban planning scenarios, fostering safer and more informed society.
Despite these positive outcomes, the technology could be misused. The ability to generate realistic dynamic scenes might be exploited to create misleading or fake data, potentially undermining trust in autonomous systems or spreading misinformation about the capabilities of such technologies. However, we do not foresee any direct harmful impact from the intended use of this work, and ethical guidelines and responsible practices can mitigate potential risks.
D.2 BROADER IMPACT
Our approach's 's contribution to 4D scene generation stands to advance the fields of autonomous driving, robotics, and even urban planning. By providing a scalable solution for generating diverse and dynamic scenes, it enables researchers and engineers to develop more sophisticated models capable of handling real-world complexity. This has the potential accelerate progress autonomous systems, making them safer, more reliable, and adaptable to a wide range of environments. For example, researchers can use DynamicCity to generate synthetic training data, supplementing real-world data, which is often expensive and time-consuming to collect, especially in dynamic and high-risk scenarios.
The broader impact also extends to lowering entry barriers for smaller research institutions and startups that may not have access to vast amounts of real-world occupancy data. By offering a means to generate realistic and dynamic scenes, DynamicCity democratizes access to high-quality data for training and validating machine learning models, thereby fostering innovation across the autonomous driving and robotics communities.
However, it is crucial to emphasize that synthetic data should be used responsibly. As our model generates highly realistic scenes, there is risk that reliance synthetic data could lead to models that fail to generalize effectively in real-world settings, especially if the generated scenes do not capture the full diversity or rare conditions found real environments. Hence, it's important to complement synthetic data with real-world data and ensure transparency when using synthetic data in model training and evaluation.
D.3 KNOWN LIMITATIONS
Despite the strengths of DynamicCity, several limitations should be acknowledged. First, our model's ability to generate extremely long sequences is still constrained by computational resources, leading to potential challenges in accurately modeling scenarios that span extensive periods. While we employ techniques to extend temporal modeling, there may be degradation in scene quality or consistency when attempting to generate sequences beyond a certain length, particularly in complex traffic scenarios.
34

---

Published as a conference paper at ICLR 2025
Second, the generalization capability of DynamicCity depends the diversity and representativeness of the training datasets. If the training data does not cover certain environmental conditions, object categories, dynamic behaviors, the generated scenes might lack these aspects, resulting in incom- plete or less realistic dynamic occupancy data. This could limit the model's effectiveness in handling unseen rare scenarios, which critical for validating the robustness of autonomous systems.
Third, while our model demonstrates strong performance in generating dynamic scenes, it may face challenges in highly congested or intricate traffic environments, where multiple objects interact closely with rapid, unpredictable movements. In such cases, DynamicCity might struggle to capture the fine-grained details and interactions accurately, leading to less realistic scene generation.
Lastly, the reliance on pre-defined semantic categories means that any variations new object types included in the training set might be inadequately represented the generated scenes. Addressing these limitations would require integrating more diverse training data, improving the model's s adaptability, and refining techniques for longer sequence generation.
PUBLIC RESOURCES USED
In this section, we acknowledge the public resources used, during the course of this work.
E.
PUBLIC DATASETS USED
nuScenes
CC BC-NC-SA 4.0
nuScenes-devkit
Apache License 2.0
Waymo Open Dataset
Waymo Dataset License
CarlaSC
MIT License
Occ3D
MIT License
E.2
PUBLIC IMPLEMENTATIONS USED
SemCity
Unknown
OccSora
Apache License 2.0
MinkowskiEngine
MIT License
TorchSparse"
MIT License
SPVNAS
MIT License
spcony
Apache License 2.0
hhtrppss/weammm.commmeee
35


Reviews:
[{"review_id": "vneoHWtRIp", "reviewer": "Reviewer_D4sh", "summary": "This paper proposes DynamicCity, a novel 4D LiDAR scene generation framework that supports large-scale dynamic reconstruction and generation. It introduces HexPlane as the compact 4D representation with effective decomposition to enhance the reconstruction quality. In order to improve the query efficiency, the authors further employ an expansion & squeeze strategy (ESS) to decode features in parallel. During the generation stage, this paper proposes a padded rollout operation to reorganize the six feature planes into a square feature map for better spatial and temporal awareness. Based on the VAE and DiT pipeline, DynamicCity achieves leading performance on both 4D reconstruction and generation, which also enables long sequential modeling and diverse conditional generation.", "questions": "According to the weaknesses above, there are some concerns to be addressed:\n\n1. It's noted that there are some recent works like XCube using advanced 3D sparse structure to improve the efficiency. Thus, it needs more explanation and comparison for the 3D backbone.  \n\n2. Does the model have the generalization ability and generative diversity? Overfitting to a few samples may reduce its significance. It would be better to provide more (conditional) generation results on the test set and multiple sampling results.\n\n3. It would be much more beneficial to incorporate more practical conditions such as images, text or single-frame point clouds, which is also helpful for the assessment of generalization ability.\n\n4. Given that the title is LARGE-SCALE LIDAR GENERATION, it may be plausible to include the generation or simulation of LiDAR point clouds (beyond occupancy) in the application. Or to avoid ambiguity.\n\n5. How to better demonstrate temporal consistency compared to baselines, which is difficult to reflect in current metrics?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Compared to existing methods that lack the ability of long-term dynamic generation, this paper utilizes Hexplane as the compact 4D representation and reorganizes into one feature map to achieve efficient reconstruction and generation. \n\n- The decoding manner in parallel proposed in Expansion & Squeeze Strategy (ESS) alleviates the problem of dense queries and further improves the generation efficiency.\n\n- Based on the VAE and DiT pipeline, the authors introduce diverse conditions for generation (e.g., command, trajectory and layout), demonstrating the potential of the model and its broad applications.\n\n- The overall paper is easy to follow with excellent illustration and clear statements of contributions, making it very comfortable to read.", "weaknesses": "- Despite the compact HexPlane and parallel decoder, the dense feature volume and projection module of autoencoder are still very heavy, which limits its efficiency and scalability. \n\n- The sample of the dataset is quite limited, which may lead to overfitting and memorization of the data by the generation model. This paper also lacks clarification of the division of training and test sets, as well as experiments and comparative results for their generalization ability and generative diversity. \n\n- Although the authors provide diverse control conditions for generation, there is a lack of some more simple and practical conditions such as images, text or single-frame point clouds that are easily accessible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jggFOyXfPR", "reviewer": "Reviewer_nfdL", "summary": "The paper proposes a diffusion-based 4D LiDAR scene generation method. This task\nis sometimes referred to as \"LiDAR world modeling\". The approach performs\ndiffusion in a HexPlane latent space, and the diffusion can be conditioned on\npast data in order to perform autoregressive synthesis, or on semantics in order\nto perform layout-guided generation. This makes the approach amenable to\nclosed-loop simulation.\n\nThe encoding stage follows a VAE framework, leveraging a LiDAR backbone followed\nby a novel projection module which produces spatio-temporal HexPlanes as the\noutput.\n\nThe decoding stage starts with HexPlanes, decodes them into spatio-temporal\nfeature volumes, and then finally into 4D semantic occupancy. The tokenization\nand diffusion-based generative modeling is performed in this HexPlane latent\nspace.\n\nAt generation time, the diffusion transformer (DiT) component generates new\nscenes by performing diffusion in the HexPlane space, optionally conditioning\nthe generation on a semantic map, or on past generation results to achieve\nautoregressive synthesis. These samples then get decoded into 4D semantic\noccupancy using the method described above.\n\nThe authors compare the approach to OccSora, another recent 4D generative\nmodeling technique, and show improved results on synthetic (CARLA-based) and\nreal (Waymo- and nuScenes-based) datasets.", "questions": "- [Q0] How do you calculate metrics like FID in 2D? What specific features do\n  you use for the computation? BEV? What is the network used in these metrics\n  originally trained on? I could not find this info in the references provided\n  at the end of Section 5.1. If the pre-trained InceptionV3 and VGG-16 networks\n  from A.2. are pre-trained on natural images, are they a good fit for comparing\n  (what I assume to be) BEV semantic images?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- [S0] Flexible modeling approach which seems to scale well to large scenes while\n  still allowing a wide range of rich conditioning methods.\n- [S1] The proposed approach outperforms OccSora, a very modern competitor, in a\n  wide range of metrics, including FID, precision, and recall.\n- [S2] Some interesting implementation tricks could potentially be applied to\n  other related tasks. For example, diffusion runs on a 2D setting with a clever\n  tiling of the six HexPlanes into a single plane (Fig 4 - the \"Padded Rollout\").\n- [S3] Overall, the paper is very well-written and provides thorough experiments,\n  architectural details, and discussions. The appendix is likewise\n  well-structured and I found it very easy to navigate.", "weaknesses": "- [W0] The pretrained networks used to calculate IS, FID, and KID should be\n  motivated more thoroughly, especially in the 2D case.\n  - For example, it is not clear why it is meaningful to use a CNN presumably\n    trained on ImageNet or COCO to reason about samples consisting of semantic\n    color maps. Unless this 2D CNN is trained to process semantic color maps as\n    inputs, passing semantic color maps to such a CNN would produce OOD feature\n    maps.\n- [W1] One conceptual limitation is the fact that the method does not explicitly\n  model uncertainty when forecasting a future scene conditioned on the present.\n  Is this something that can be modeled by sampling multiple futures from the\n  latent space?\n- [W2] The core applications of 4D world modeling are tasks like simulation and\n  motion forecasting. Presenting some results in this area could strengthen the\n  paper. For example, this could include demonstrating that the world model\n  performs well on a motion forecasting benchmark, or that it can be used to\n  supplement training data for an end-to-end autonomous driving model.\n- [W3] No source code seems to be promised.\n- [W4] In the current stage, the approach is LiDAR-only. This is a minor\n  limitation, though, and I am primarily mentioning it for completeness.\n- Minor suggestions:\n  - L245: \"first generate\" -> \"first generates\"\n  - L866: Missing parentheses around the PyTorch citation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i4sQr9zn8p", "reviewer": "Reviewer_a6Q2", "summary": "This paper presents DynamicCity, a novel 4D scene generation method for driving environments. DynamicCity utilizes a VAE model to compress 4D scenes into a HexPlane, integrating strategies such as a Projection Module and an Expansion & Squeeze Strategy to enhance performance and efficiency. A DiT-based diffusion model is then employed for generating the HexPlane. The method supports various 4D scene generation applications, including trajectory, command, and layout conditions. Basically this is a strong submission with impressive results, while I have several further concerns.", "questions": "Please refer to the Weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-structured and easy to follow.\n2. The generated results are impressive, with an accompanying demonstration video that effectively showcases the method’s capabilities.\n3. The proposed method has a clear motivation and presents a well-reasoned pipeline.", "weaknesses": "1. I’m a little confused by the title and the task definition. While the authors state that the proposed method is designed for “LiDAR” generation, the results seem more akin to “occupancy” generation. These concepts are distinct, despite both representing the scene’s geometry. For true LiDAR generation [1][2], the outputs should be LiDAR point clouds that reflect the sampling properties of LiDAR sensors (e.g., ray drop, ray-based sampling, etc).\n2. The authors note that the method can support long sequential modeling of up to 128 frames, but the factors limiting sequence length (e.g., GPU memory) are not discussed. Additionally, the inference running time is not mentioned.\n3. Can the proposed method support outpainting similar to SemCity? While inpainting results are shown to demonstrate the spatial significance of the HexPlane, outpainting results could further demonstrate the method’s ability to expand scenes. Furthermore, the inpainting examples in the paper are confined to small regions.\n4. Regarding the experimental results, I suggest to provide quantitative comparison with SemCity in static scenes. The caption of Table 2 points out the authors compare the method with SemCIty, but I do not see results in the table.\n\n[1] Ran, Haoxi, Vitor Guizilini, and Yue Wang. \"Towards Realistic Scene Generation with LiDAR Diffusion Models.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2024.\n\n[2] Zyrianov, Vlas, et al. \"LidarDM: Generative LiDAR Simulation in a Generated World.\" *arXiv preprint arXiv:2404.02903* (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "op24PbgzUT", "reviewer": "Reviewer_URnX", "summary": "This paper proposes DynamicCity for large-scale driving environments for 4D generation. The paper proposes a Projection Module to efficiently 4D features into six 2D feature maps and an Expansion & Squeeze Strategy to reconstruct 3D feature volumes in parallel. A Padded Rollout Operation is also proposed to reorganize all six feature planes of the HexPlane as a squared 2D feature map.", "questions": "Same as Weakness.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The writing is easy to follow.\n\n2. The ablation is comprehensive and validates the efficiency of Padded Rollout, Projection Module, and ESS in Table 3 and Table 5.\n\n3. The results are good compared to the baseline method over different datasets.\n\n4. The trajectory-guided generation, dynamic scene inpainting, and layout-conditioned generation show various downstream applications.", "weaknesses": "1. The Dynamic Object Inpainting Results seem to make the ground have holes(Figure 1). Could you please explain this phenomenon and discuss any potential limitations or artifacts of this inpainting approach?\n\n2. The results seem to reflect semantic occupancy, with training data voxelized into occupancy format using LiDAR point clouds. Could you clarify why \"LiDAR generation\" is used in the title, abstract, and introduction? It would be helpful to explain the distinction between \"LiDAR generation\" and \"semantic occupancy generation\" as used throughout the paper, as well as the relationship between LiDAR data and the voxelized occupancy representation. This clarification would help ensure that the terminology accurately represents the method and results.\n\n3. It’s recommended to add a comparison of model size across different methods, as the proposed approach appears to require substantial training resources.\n\n4. While a training time comparison is provided in the ablation study, it’s still suggested to include a comparison of training time and GPU requirements specifically against the baseline method, like Occ-sora. These comparisons could help readers better understand how much computational power the proposed method needs compared to other established methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)