CityDreamer4D: Compositional Generative Model of Unbounded 4D Cities
Authors: name: Haozhe Xie, name: Zhaoxi Chen, name: Fangzhou Hong, name: Ziwei Liu

3D scene generation has garnered growing attention in recent years and has
made significant progress. Generating 4D cities is more challenging than 3D
scenes due to the presence of structurally complex, visually diverse objects
like buildings and vehicles, and heightened human sensitivity to distortions in
urban environments. To tackle these issues, we propose CityDreamer4D, a
compositional generative model specifically tailored for generating unbounded
4D cities. Our main insights are 1) 4D city generation should separate dynamic
objects (e.g., vehicles) from static scenes (e.g., buildings and roads), and 2)
all objects in the 4D scene should be composed of different types of neural
fields for buildings, vehicles, and background stuff. Specifically, we propose
Traffic Scenario Generator and Unbounded Layout Generator to produce dynamic
traffic scenarios and static city layouts using a highly compact BEV
representation. Objects in 4D cities are generated by combining stuff-oriented
and instance-oriented neural fields for background stuff, buildings, and
vehicles. To suit the distinct characteristics of background stuff and
instances, the neural fields employ customized generative hash grids and
periodic positional embeddings as scene parameterizations. Furthermore, we
offer a comprehensive suite of datasets for city generation, including OSM,
GoogleEarth, and CityTopia. The OSM dataset provides a variety of real-world
city layouts, while the Google Earth and CityTopia datasets deliver
large-scale, high-quality city imagery complete with 3D instance annotations.
Leveraging its compositional design, CityDreamer4D supports a range of
downstream applications, such as instance editing, city stylization, and urban
simulation, while delivering state-of-the-art performance in generating
realistic 4D cities.