UrbanWorld: An Urban World Model for 3D City Generation

## Abstract

Abstract Cities, as the essential environment of human life, encompass diverse physical elements such as buildings, roads and vegetation, which continuously interact with dynamic entities like people and vehicles.
Crafting realistic, interactive 3D urban environments is essential for nurturing AGI systems and constructing AI agents capable of perceiving, decision-making, and acting like humans in real-world environments.
However, creating high-fidelity 3D urban environments usually entails extensive manual labor from designers, involving intricate detailing and representation of complex urban elements.
Therefore, accomplishing this automatically remains a longstanding challenge.
Toward this problem, we propose UrbanWorld, the first generative urban world model that can automatically create a customized, realistic and interactive 3D urban world with flexible control conditions.
Specifically, we design a progressive diffusion-based rendering method to produce 3D urban assets with high-quality textures.
Moreover, we propose a specialized urban multimodal large language model (Urban MLLM) trained on realistic street-view image-text corpus to supervise and guide the generation process.
UrbanWorld incorporates four key stages in the generation pipeline: flexible 3D layout generation from OSM data or urban layout with semantic and height maps, urban scene design with Urban MLLM, controllable urban asset rendering via progressive 3D diffusion, and MLLM-assisted scene refinement.
We conduct extensive quantitative analysis on five visual metrics, demonstrating that UrbanWorld achieves state-of-the-art generation realism.
Next, we provide qualitative results about the controllable generation capabilities of UrbanWorld using both textual and image-based prompts.
Lastly, we verify the interactive nature of these environments by showcasing the agent perception and navigation within the created environments.
We contribute UrbanWorld as an open-source tool available at https://github.com/Urban-World/UrbanWorld .

## 1 Introduction

Cities are the most complex human-centric environments, characterized by their intricate spatial structures, heterogeneous components such as buildings, infrastructure, and public spaces, and dynamic interactions between these components and human activities.
Creating near-realistic 3D urban world environments is a fundamental technique for broad research and real applications across various domains such as Artificial
General Intelligence [], AI agents [], embodied AI [], urban simulation [] and metaverse [].
Traditionally, achieving this involves expensive labor costs for human designers on detailed asset modeling, texture mapping, and scene composition.
With the advancement of generative AI, there have emerged some automatic approaches for 3D scene generation based on volumetric rendering[, ] and diffusion models[, ].
These approaches have revolutionized the paradigm of 3D scene generation, alleviating the high costs of manual design.
However, the crafted 3D scenes are limited to the video format and unable to provide embodied and interactive environments.
Regarding this issue, a recent series of methods known as world models have emerged, preliminarily focusing on autonomous driving scenes [, ].
These models are shown to possess the capability of understanding the scene dynamics and predicting future states, uplifting the interactivity of 3D scene generation.
However, there is still a large gap between the created urban environments and the real urban world in which humans live.
To sum up, there is still a long way from the actual “urban world models”, which we define as models able to create urban environments that are (1) realistic and interactive (2) customizable and controllable (3) capable of supporting embodied agent learning.
Table 1 provides a comprehensive review of existing 3D city generation methods from these key perspectives, highlighting that no existing approach can fully satisfy these requirements.

Urban world models are of great significance in developing embodied intelligence and Artificial General Intelligence (AGI). Firstly, it is promising to bridge the gap between virtual environments and the real world, enabling embodied agents to interact with and learn from richly detailed, realistic urban environments.
Secondly, by crafting synthetic 3D urban environments, researchers can gain complete control over data generation, with full access to all generative parameters.
Machine perceptual systems can thus be trained on tasks that are not well suited to conduct in the real world or require various environments.
Finally, a sophisticated urban world model can simulate a wide variety of environments, from bustling city centers to quiet residential neighborhoods, with realistic visual appearances of physical infrastructures such as buildings, roads, and natural spaces.
This is crucial to avoid overfitting and creating agents with high generalization in diverse and dynamic environments.
However, there are no specialized urban world models for automatically crafting interactive 3D urban environments, hindering the advancement of AI abilities toward general intelligence through embodied learning in complex open-world environments.

Toward this issue, we propose UrbanWorld, a generative urban world model that can automatically create realistic, controllable and embodied 3D urban environments from user instructions and urban layout data in various format such as OpenStreetMap(^†^†https://www.openstreetmap.org/) (OSM) and layouts with semantic and depth maps  [, ].
In detail, there are four key modules in the framework of UrbanWorld.
Firstly, UrbanWorld automatically generates untextured 3D layouts with the above-mentioned input data and conducts detailed asset processing via Blender(^†^†https://www.blender.org/).
Then, UrbanWorld adopts a fine-tuned urban-specific multimodal large language model (called Urban MLLM) to effectively plan and design urban scenes following user instructions, generating detailed textual descriptions of urban elements.
Next, UrbanWorld integrates a 3D asset renderer based on texture diffusion and refinement, flexibly controlled by textual and visual conditions.
Finally, to further optimize the visual appearance, it utilizes Urban MLLM to scrutinize the crafted 3D urban environment, generating detailed suggestions for refinement and activate an additional iteration of rendering.

Our framework is highly flexible to support generating two typical types of 3D urban environments.
On the one hand, UrbanWorld can generate a highly accurate replica of the real urban environment with real street-view imagery as the generation condition, which holds significant potential for studies of urban planning and geographic information systems.
On the other hand, it can also generate fully customized urban environments with textual descriptions as the generation condition.
This capability is valuable for simulating and exploring hypothetical urban scenarios, especially in areas such as virtual city design, gaming, and emergency response planning.
In the experimental part, we begin by conducting a comprehensive quantitative evaluation using five visual metrics, validating the state-of-the-art realism of UrbanWorld’s generated environments.
We then showcase diverse generation results from various textual and image prompts, highlighting the superior controllability of UrbanWorld.
Finally, we emphasize the interactive nature of the generated 3D urban environments by demonstrating agent perception and navigation within them.
We contribute UrbanWorld as an open platform to support the creation and manipulation of more advanced 3D urban environments, facilitating the advancement of broad AI communities.

The contributions of this work can be summarized as follows:

- •
We present UrbanWorld, the first urban world model for automatically creating realistic, customized and interactive embodied 3D urban environments with flexible controls.
- •
UrbanWorld demonstrates its superior generative ability to craft high-fidelity 3D urban environments, greatly enhancing the authenticity of interactions in the environment.
- •
We provide UrbanWorld as an open-source platform to develop 3D urban environments, which can support a wide range of research including embodied intelligence and AI agents, further laying a foundation for the advancement of AGI.

## 2 Related Works

**Table 1: Comparison between existing works for 3D city generation and UrbanWorld from four aspects: text-controllable, image-controllable, whether new assets can be created, and interactive.**
| Type | Method | Text-controllable | Image-controllable | Creating new assets | Interactive |
| --- | --- | --- | --- | --- | --- |
| Neural Rendering | SceneDreamer [] | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ |
| PersistentNature [] | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ |  |
| Infinicity [] | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ |  |
| CityDreamer [] | $\times$ | $\times$ | $\checkmark$ | $\times$ |  |
| Diffusion | CityGen [] | $\times$ | $\times$ | $\checkmark$ | $\times$ |
| 3D Modeling Software | MetaUrban [] | $\times$ | $\times$ | $\times$ | $\checkmark$ |
| SceneCraft [] | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ |  |
| CityCraft [] | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ |  |
| Comprehensive | UrbanWorld | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |

### 2.1 3D Urban Scene Generation

3D urban scene generation aims to create realistic 3D urban environments with sophisticated urban planning and visual element design, usually requiring high human efforts such as complex asset modeling, texture mapping, and
scene composition.
With the advancement of deep learning techniques, recently there are three lines of work trying to achieve this in an automated way, including neural rendering-based methods [, , ], diffusion-based methods [, , ] and 3D modeling software-based methods [, , ].
An overview of the method comparison is shown in Table 1.
Neural rendering-based methods implicitly represent the urban scene and perform the volumetric rendering for the neural fields. For example, CityDreamer [] first separates the scene into buildings and backgrounds then introduces different types of neural fields for asset rendering.
These methods can produce a high-quality visual appearance while potentially losing geometric fidelity.
Diffusion-based methods utilize diffusion models to generate city layouts or urban scenes.
CityGen [] provides an end-to-end pipeline to create diverse 3D city layouts with Stable Diffusion.
These methods are creative in generating scene images or videos, but hard to obtain embodied 3D environments, limiting the practical usages.
Recently, some professional software script-based methods have been proposed, trying to develop an automatic agentic workflow using LLMs to control the professional software for scene creation.
CityCraft [] adopts LLMs in designing and organizing 3D urban environments from off-the-shelf asset libraries.
Such approaches are effective but only create urban environments with existing 3D models, unable to flexibly create new assets when necessary.
By comparison, our model can freely create new 3D urban assets in a highly controllable way, allowing for crafting diverse urban environments.

### 2.2 3D World Simulator

A persistent objective in AI research has been to develop machine agents capable of engaging with various environments in 3D space like humans.
Toward this goal, researchers have been devoted to building various interactive world simulators in the format of videos [] or embodied environment [].
Existing world simulation environments and platforms are mostly for indoor scenes [, , , ]. Differently, Threedworld [] paid attention to creating outdoor environments by retrieving and compositing objects from an existing asset library.
When it comes to urban scenes which is the most important open environment, existing works mostly focus on the generative world model for autonomous driving capable of learning scene dynamics and understanding the geometry of the physical world [, , ].
However, these models can only generate new scenes in the format of videos, hard to provide an embodied and interactive urban environment for real use.
UGI [] conceptualized some relevant ideas toward urban world simulation, proposing to develop an embodied urban environment for agent development, but still lacks practical implementation.
More recently, MetaUrban [] developed a simulation platform for embodied agents in the urban environment, while the provided environment is limited to a fixed style without controllability.
To address these challenges, we propose UrbanWorld, which is expected to facilitate the construction of diverse embodied urban environments with controllable and refined visual appearance, supporting agent development or simulation in various urban scenes.

## 3 Methodology

Figure: Figure 1: Illustration of the whole framework of UrbanWorld, including four key components: (A) Flexible 3D urban layout generation; (B) Urban MLLM-empowered scene design; (C) Diffusion-based urban asset texture rendering; (D) MLLM-assisted scene refinement.
Refer to caption: x1.png

There are three main challenges to solve for building a real “urban world model”: efficient interactive environment construction, elaborate urban scene planning and high-quality texture rendering.
Towards these objectives, UrbanWorld follows a novel “map-design-render-refine” generation pipeline, accomplished by the collaboration of a specialized urban MLLM and an urban asset rendering module.
In detail, there are four key stages in UrbanWorld: (1) Flexible 3D urban layout generation, achieving automatic 2.5D-to-3D mapping based on various urban layout metadata such as globally open-accessible OSM data and urban layouts with semantic and height maps, which can address the first challenge.
(2) Urban MLLM-empowered scene design, which exploits the superior urban environment understanding ability of a fine-tuned urban MLLM to draft realistic urban scenes emulating human designers for addressing the second challenge.
(3) Controllable diffusion-based urban asset texture renderer, achieving flexible urban asset rendering based on 3D diffusion supporting both textual and visual prompts.
(4) MLLM-assisted urban scene refinement, exploiting the urban MLLM to conduct reflection to refine the generated urban environment, mimicking the iterative revision in the standard operation process of human designers.
The last two components contribute to high-fidelity textures of 3D assets, effectively tackling the third challenge.
The overview framework of UrbanWorld is illustrated in Figure 1.

### 3.1 Flexible 3D Urban Layout Generation

UrbanWorld supports any type of 2.5D layout data as input to efficiently build the untextured 3D urban environment, such as easily accessible OSM data and AI-generated urban layout data.
These 2.5D layout data record the metadata of an urban area including the geographic locations and attributes of diverse urban elements such as roads, buildings, vegetation, forests, and water.
All urban assets will be automatically separated as independent objects for subsequent element-wise rendering.
In this step, UrbanWorld also records the object center location for further reorganization of assets, making it match the real urban layout.

### 3.2 Urban MLLM-empowered Scene Design

Aiming to effectively craft customized urban environments, UrbanWorld integrates an advanced MLLM fine-tuned on extensive real-world urban imagery data, named Urban MLLM.
In detail, we collected approximately 300K urban street-view images from Google Maps and used GPT-4 to generate associated textual descriptions.
These descriptions were then manually reviewed and low-quality data were filtered out. Subsequently, we fine-tuned an advanced open-source MLLM, VILA-1.5 [] with the curated dataset to improve its understanding of urban environments.
In the workflow of UrbanWorld, Urban MLLM is utilized to act as a human-like designer, which automatically drafts high-quality and detailed urban scene descriptions, ensuring the crafted urban environment are visually coherent and instruction-following.
Specifically, taking a simple textual instruction (e.g., “a teaching area in the university”) from users as input, UrbanWorld calls Urban MLLM with carefully designed prompts and returns diverse detailed descriptions about visual appearance and materials for each asset.
The produced asset descriptions will be used to control the condition of the later texture rendering process.

### 3.3 Controllable Diffusion-based Urban Asset Texture Rendering

Figure: Figure 2: Illustration of the urban asset rendering method in UrbanWorld, mainly including two stages: depth-aware UV texture generation with flexible control under textual and visual prompts and UV position-aware texture refinement.
Refer to caption: x2.png

Rendering a large-scale urban scene is challenging due to the existence of complex elements and relations, whereby the scene-level rendering will inevitably result in texture mismatching and low-resolution issues.
Therefore, we adopt the element-wise rendering strategy to ensure the rendering quality.
Simultaneously, in order to speed up the rendering process, we merge some urban element types and finally define four main categories: buildings, roads and paths, forest and vegetation, and water.
We implement the rendering with a controllable diffusion-based method consisting of two stages: UV texture generation and texture refinement as shown in Figure 2.

After the 3D environment generation detailed in Section 3.1, we have obtained the untextured 3D mesh of an urban region $S$ and contained assets $\{S_{1},S_{2},...,S_{i}\}$.
For each type of asset, UrbanWorld takes the associated textual description $t_{i}$ from Urban MLLM or a reference image $r_{i}$ as prompts to control the generation.

We first set a series of camera views $v_{i}$ = $\{v_{i}^{k}\}_{k=1}^{N}$ to capture multi-view appearances of the asset $S_{i}$.
Next, we utilize the depth-aware ControlNet [] to control a 2D diffusion model $F$ to generate an image $I_{i}$ showing the visual appearance of $S_{i}$ on different views, controlled by the condition $c_{i}\in\{t_{i},r_{i},(t_{i},r_{i})\}$:

$$ $I_{i}=F(c_{i};d_{i};z),$ (1) $$

where $z$ is the latent embedding for the diffusion process, the depth map from different views $d_{i}$ is obtained from the rendering process $d_{i}=P(S_{i},v_{i})$.
Then we crop $I_{i}$ into patches where each represents a unique view $\{I_{i}^{k}\}_{k=1}^{N}$ of the rendered asset.
Then we conduct a reverse process $P^{-1}$ of rendering to back-project $I_{i}$ into the UV texture space:

$$ $U_{i}^{k}=P^{-1}(v_{i}^{k};I_{i}^{k};S_{i}),$ (2) $$

Subsequently, we merge the texture maps from different views into a single texture map $U_{i}$:

$$ $U_{i}=\sum_{k=1}^{n}M_{i}^{k}\odot U_{i}^{k},$ (3) $$

where $M_{i}^{k}$ denotes the corresponding mask in the UV space from the view $v_{i}$.

Up to now, we have obtained the preliminary texture map for the asset $S_{i}$, while in practice we found that there are still some untextured areas on the object due to the discrete sampling of camera views, especially for assets with many faces.
Inspired by the inpainting capability of diffusion models, we introduce an additional UV texture inpainting process to get more complete and natural texture.
However, such inpainting can not be directly achieved with general diffusion-based inpainting since the inpainting should be restricted to follow the position relation in the UV texture space.
Therefore, we introduce a position map-guided building UV inpainting process, inspired by the inpainting process of general 3D objects  [].

Figure: Figure 3: Illustration of the evolution of created urban environments, including the untextured urban scene, initial textured urban scene and refined urban scene.
Refer to caption: x3.png

To be specific, we add a UV position map encoder $E_{V}(\cdot)$ to encode the position map $V_{i}\in\mathbb{R}^{H\times W\times 3}$, indicating the adjacency relation of the UV texture fragments, where $E_{V}$ is set to have the same architecture of the image encoder in the diffusion model.
Then we curate a set of paired-up UV position maps and UV texture maps for urban assets with complex surfaces and train the position encoder following the pipeline of ControlNet [].
With the control of UV position maps, it’s expected to achieve accurate and natural inpainting for UV texture maps.
Denote $U_{i}^{*}$ as the inpainted UV texture map, the UV inpainting process is formulated as follows:

$$ $U_{i}^{*}=F(c_{i};U_{i};E_{V}(V_{i})).$ (4) $$

With the above texture generation and completion, UrbanWorld can produce coherent and high-fidelity textures for various urban elements.
For better visual aesthetic of the rendering, we further conduct image upscaling with ControlNet-tile to enhance the structure sharpness and realism of the texture map, contributing to more detailed and realistic appearances of urban assets.

### 3.4 MLLM-assisted Urban Scene Refinement

After urban asset rendering, UrbanWorld automatically reorganizes the assets guided by the location information extracted from the metadata of the urban layout, effectively recovering the original urban layout.
Inspired by the standard operation process of human designing, where experts will take an overview of the result and conduct iterative adjustments.
To mimic such an effort, UrbanWorld resorts to Urban MLLM again to scrutinize the crafted 3D urban environment, especially focusing on the texture details.
Specifically, we prompt Urban MLLM to identify the inconsistencies between the generated result and the prompts for generation.
Finally, Urban MLLM will provide sophisticated suggestions for further refinement, including elements to be modified and refined design prompts.
Then the rendering module described in Section 3.3 will be activated and the involved elements will be rendered under the refined text prompts and updated in the 3D environment.
With such a refinement process, UrbanWorld can further align the generated urban environment to user instructions.
Here we provide a visualization example in Figure 3, showing the evolution of the created urban environments happened in the running process of UrbanWorld.
It can be seen that UrbanWorld works in an iterative refinement manner to create high-fidelity urban environments, where the low-quality textures will be automatically identified and refined with the powerful Urban MLLM.

## 4 Experiments

Figure: Figure 4: Qualitative comparisons of generated 3D urban environments from Infinicity, CityGen, CityDreamer and UrbanWorld. By comparison, our method can craft more diverse and realistic 3D urban environments enabling dynamic interactions with humans (walking) and vehicles (driving).
Refer to caption: x4.png

In this section, we first introduce the experimental setup and implementation details (see Section 4.1), and then provide some quantitative evaluations of the created urban environments to demonstrate the superiority of UrbanWorld (see Section 4.2).
Finally, we present the generation results of UrbanWorld for qualitative estimation (see Section 4.3).

### 4.1 Implementation Details

UrbanWorld incorporates three key techniques: Blender as the professional 3D modeling software, diffusion-based rendering and Urban MLLM-empowered scene design and refinement.
Specifically, we use Blender-3.2.2 for Linux systems and the compatible Blosm addon to handle the OSM transformation.
In terms of diffusion-based rendering, we utilize Stable Diffusion-1.5 [] as the fundamental diffusion backbone, combined with ControlNet-Depth [] when generating multiple views of 3D assets.
We also introduce IP-Adapter [] to support taking reference images as the additional generation condition.
We use ControlNet-inpainting [] as the diffusion controller in the UV texture refinement and ContrlNet-tile [] in the realness enhancement stage.
For the hyper-parameter settings in the rendering part, we set the number of camera views $N=4$, which can basically satisfy the rendering needs of most urban assets.
The number of inference steps in all diffusion processes is set as 30 by default.
The UV maps of 3D assets are unwrapped in the “smart projection” mode operated in Blender.
All experiments are conducted on a single NVIDIA Tesla A100 GPU.

### 4.2 Quantitative Evaluations

**Table 2: Quantitative evaluation of existing works for 3D urban scene generation and UrbanWorld on depth error, homogeneity index and realistic score.**
| Method | FID (↓) | KID (↓) | DE (↓) | HI (↓) | PS (↑) |
| --- | --- | --- | --- | --- | --- |
| SGAM [] | 453.81 | 0.522 | 0.575 | 0.872 | 5.6 |
| PersistentNature [] | 441.65 | 0.319 | 0.326 | 0.742 | 4.8 |
| SceneDreamer [] | 389.90 | 0.284 | 0.152 | 0.817 | 6.2 |
| CityDreamer [] | 418.38 | 0.210 | 0.147 | 0.830 | 6.0 |
| UrbanWorld (text) | 377.65 | 0.187 | 0.089 | 0.683 | 6.5 |
| UrbanWorld (image) | 368.72 | 0.154 | 0.082 | 0.665 | 6.7 |

To better demonstrate the superior generation performance of UrbanWorld, in this section, we provide quantitative results on five metrics: Frechét Inception Distance (FID), Kernel Inception Distance (KID), Depth error (DE), Homogeneity index (HI) and Preference score (PS). (more details can be found in Appendix A.1).

We test two versions of our method including UrbanWorld (text) which takes textual prompts for generation and UrbanWorld (image) which takes reference images as generation conditions.
The compared methods for urban scene generation include SGAM [], PersistentNature [], SceneDreamer [] and Citydreamer [], representing the most advanced performance of automatic 3D urban environment generation.
We don’t provide results from other methods such as CityGen [], CityCraft [] and SceneCraft [] because the source codes are not open up to now.

From the results presented in Table 2, we can observe that UrbanWorld outperforms on each quantitative metric compared with baselines.
UrbanWorld with real street-view images as conditions demonstrates improved performance compared to the version relying solely on textual conditions.
In detail, compared with the most competitive baseline, UrbanWorld has 5.4% and 26.7% improvement on FID and KID, indicating better realism of generated results.
Besides, UrbanWorld achieves 44.2% improvement on depth error, demonstrating the geometry-preserving ability of UrbanWorld.
By comparison, rendering methods such as SceneDreamer and CityDreamer can produce visually appealing urban scenes, but commonly lose geometry consistency.
In terms of the homogeneity index, UrbanWorld has
10.4% improvement compared with baselines.
Consistent with the observation on the qualitative results in Section 4.3, the generated scenes from existing methods exhibit high homogeneity, limited to the style of training data.
By comparison, UrbanWorld can produce more diverse urban environments according to user instruction, effectively achieving customized creation.
This makes it possible to craft any type of 3D urban environment that adapts to the needs of different urban environments.
Lastly, the created urban environments from UrbanWorld obtain higher preference of GPT-4 outperforming all compared methods on the preference score.

**Table 3: Study of the effectiveness of three key designs in UrbanWorld with text prompts as generation conditions: Urban MLLM-empowered scene design, texture enhancement and MLLM-assisted scene refinement.**
| Method | FID (↓) | KID (↓) | DE (↓) | HI (↓) | PS (↑) |
| --- | --- | --- | --- | --- | --- |
| UrbanWorld (text) | 377.65 | 0.187 | 0.089 | 0.683 | 6.5 |
| w/o Urban MLLM design | 401.58 | 0.237 | 0.104 | 0.701 | 6.1 |
| w/o texture enhancement | 382.09 | 0.197 | 0.125 | 0.687 | 6.2 |
| w/o scene refinement | 393.73 | 0.202 | 0.096 | 0.690 | 6.3 |

**Table 4: Study of the effectiveness of three key designs in UrbanWorld with real street-view images as generation conditions: Urban MLLM-empowered scene design, texture enhancement and MLLM-assisted scene refinement.**
| Method | FID (↓) | KID (↓) | DE (↓) | HI (↓) | PS (↑) |
| --- | --- | --- | --- | --- | --- |
| UrbanWorld (image) | 368.72 | 0.154 | 0.082 | 0.665 | 6.7 |
| w/o Urban MLLM design | 373.21 | 0.158 | 0.104 | 0.701 | 6.5 |
| w/o texture enhancement | 384.10 | 0.172 | 0.125 | 0.687 | 6.5 |
| w/o scene refinement | 374.52 | 0.159 | 0.096 | 0.690 | 6.6 |

Effectiveness of three key designs.
To further validate the effectiveness of key designs in UrbanWorld, we conduct ablation studies and show the results on UrbanWorld (text) and UrbanWorld (image) in Table 3 and 4, respectively.
We explore the influence of three designs on the performance, including Urban MLLM-empowered urban scene design, texture enhancement and MLLM-assisted scene refinement.
The results indicate that all these techniques contribute to the final generation performance.
Specifically, the scene design from Urban MLLM contributes most to the quality of generated urban environments, benefiting from the rich urban environment knowledge of Urban MLLM.
The texture completion and enhancement operation have the most notable effect on the estimated depth error because better texture fidelity can help with geometric perception.
Besides, the final scene refinement process leads to a gain in all evaluation metrics, further promoting the generation quality.

Figure: Figure 5: Illustration of the controllable generation of diverse architecture styles when prompting with reference images (upper left) in UrbanWorld.
Refer to caption: x5.png

Figure: Figure 6: Illustration of the controllable generation of diverse urban functional scenes guided by different textual prompts in UrbanWorld.
Refer to caption: x6.png

### 4.3 Qualitative Results

We present generation results of UrbanWorld produced with textual prompts in Figure 4, including three representative urban functional spaces: residential areas, commercial blocks and parks.
For intuitive comparison, we also provide some generation samples from Infinicity [], CityGen [] and CityDreamer [].
The results of Infinicity and CityGen are taken from the original paper because the codes are not open-source.
It can be seen that scenes from Infinicity are short of clear textures and well-maintained building structures.
Scenes from CityGen feature homogeneous styles without clear characteristics of urban functions.
Similarly, the visual appearance of urban elements (especially buildings) in the environments from CityDreamer lacks diversity and is hard to distinguish.
Besides, there are also clear geometric distortions of the building boundaries in CityDreamer.
These issues will pose great challenges for the real interactions between subjects and urban environments. For example, embodied agents are hard to be trained to conduct urban navigation because the surrounding elements are too similar to recognize.
By comparison, the urban elements created by UrbanWorld possess high visual diversity, conveniently controlled by user instructions.
Moreover, we explore generating realistic urban assets by prompting UrbanWorld with realistic images as prompts.
Figure 5 presents generated urban buildings in various styles guided by realistic images.
We can observe that the generated asset possesses high fidelity to the real ones, which is rarely reached by existing approaches.
We further extend to generate large-scale urban environments, as illustrated in Figure 6, which presents four representative urban functional spaces, each with a visually consistent and contextually appropriate appearance.
These results highlight the superior controllability of UrbanWorld which supports flexible prompts as generation conditions.

### 4.4 Interactivity Demonstration

Figure: Figure 7: Illustration of the interactive nature of created environments from UrbanWorld, showcasing two representative types of interactions: agent perception and navigation.
Refer to caption: x7.png

To demonstrate the interactive characteristics of 3D urban environments produced by UrbanWorld, we provide a case study about how UrbanWorld provides information feedback and embodied activity spaces for the agents.
We focus on two representative interaction modes between agents and urban environments: perception and navigation, following existing works about urban agents [, ].
As shown in Figure 8, the created urban environment can provide multimodal observations for agents including RGB imagery, depth maps and semantic maps.
Such information can be utilized to enhance the perception ability of agents in various complex urban environments, benefiting in accomplishing various embodied tasks such as navigation and object manipulation.
We also evaluate the agent’s navigation capabilities within the generated urban environment, where the task is to navigate to specified target coordinates in 3D environments.
Here we utilize the Rapidly-exploring Random Tree (RRT) algorithm for path planning using the start coordinates of agents, target coordinates and obstacle coordinates in 3D space.
We record the visual observation trajectory of the agent during navigation as shown in Figure 8, demonstrating the effective interaction between agents and environments.

## 5 Conclusion

We propose UrbanWorld, the first generative urban world model to create realistic, customized and interactive 3D urban environments with flexible control conditions in a fully automatic manner.
Integrating the powerful urban environment understanding ability of an urban MLLM and the controllable generation ability of diffusion models, UrbanWorld can effectively craft high-fidelity and customized urban environments outperforming existing 3D city generation methods.
For practical usage, the created urban environments can provide high-fidelity data and interactive environments for developing embodied intelligence and AI agents.
We have contributed UrbanWorld as an
open-source tool to benefit broad research communities, which we believe can pave a new way to efficiently establish 3D urban world, accelerating the development of AGI.

## Appendix A Appendix

### A.1 Details of the metrics for quantitative evaluation

Frechét Inception Distance (FID) and Kernel Inception Distance (KID).
Both metrics quantify the similarity between the distribution of generated images and real images, where lower values indicate better image quality of generated results.
We compute FID and KID between frames sampled from the generated scene and an evaluation set comprising 1000 real street-view images randomly sampled from Google Maps.
These metrics can effectively measure the realism of generated results.

Depth error (DE).
Depth error is utilized to evaluate the 3D geometry accuracy, following the implementation of EG3D [] and CityDreamer [].
Specifically, we use the pre-trained depth estimation model [] to obtain the “ground truth” of depth maps via density accumulation.
DE is then calculated as the L2 distance between the normalized predicted depth maps and the “ground truth”.
The final result is averaged on the result from 100 captured frames of generated urban scenes.

Homogeneity index (HI).
Realistic cities are featured by complex elements with diverse visual appearances, indicating various functional uses of different urban areas.
In order to capture this key character, we propose to evaluate the homogeneity of generated scenes, mainly measuring the variance of different urban scenes.
To be specific, we first extract the visual feature of each generated scene image with ResNet [].
The homogeneity index is then calculated as the averaged cosine similarity of each pair of scenes in the feature space.
The smaller value of the homogeneity index means a higher diversity of generated urban environments.

Preference score (PS).
Another widely used approach to evaluate the generated result is utilizing powerful LLMs as the evaluator [, ], here we prompt GPT-4 to score the snapshots taken from the generated 3D environment, mainly considering the texture sophistication and geometric completeness (ranging from 1 to 10).
A higher score indicates stronger preference for the generated result.

### A.2 Designed Prompts for urban scene design using Urban MLLM

Figure: Figure 8: Designed prompt template for urban scene design using Urban MLLM.
Refer to caption: x8.png

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: See Weakness.
soundness: 2
strengths: 1. The task of 3D urban generation is important.
2. The method is reasonable and looks to have better quantitative results than previous models.
3. The writing is clear and easy to follow.
summary: This paper proposes a generative urban world model that can automatically create a customized, realistic, and interactive 3D urban world with flexible control conditions. The code of this work was released.
weaknesses: 1. Claim of World Model. This work belongs to the 3D urban generation. It is over-claimed to be a world model and barely related to AGI. Authors should precisely identify the task and topic. Then, focus on the specific topic and make it comprehensive rather than claim some large topics.

2. Technical contributions. The motivation of the generation pipeline is unclear. Why do you need a vision language model? What are the special designs in your work different from others, and why do you need them? What is the special challenges that lead you to design the method? So far, the pipeline looks like a combination of recent advanced techniques, i.e., diffusion model and vision language model.

3. Visual results. The visual results are insufficient. From only a few images, it can not be convinced that the visual quality is better than other models. Also, Figure 6 and 4 have some reduplicate results.

4. Evaluation of interactive environments. The evaluation of interactive environments is coarse. The navigation tasks are not really evaluated. From an image, nothing can be evident. What are the quantitative results, and what are the video results? How do you make the simulation of physics? What is the physics engine? What is the training speed? What is the model, RL or IL? What are the evaluation metrics?

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Please refer to weaknesses.
soundness: 3
strengths: 1. The paper is written fluently and is easy to understand.
2. The proposed method shows relatively better results in generating city scenes with assets that have new appearances.
3. The authors effectively showcase various capabilities of the pipeline.
summary: This paper introduces a method for 3D urban scene creation called UrbanWorld, which facilitates customized and interactive 3D urban world generation. UrbanWorld uses Blender to create untextured 3D layouts from 2D maps and incorporates Urban MLLM to generate textual descriptions for assets. A diffusion-based method is then applied to generate and refine the geometry of the 3D assets.
weaknesses: 1. While the authors state that the method achieves “customized, realistic, and interactive 3D urban world generation,” the results appear more simulation-style and fall short of true realism. The texture quality, as seen in Fig. 3 and 4, is not particularly impressive, and there are no significant improvements over CityDreamer.
2. The absence of video results is notable. For a 3D generation task, video demonstrations would better illustrate the quality and realism of the generated scenes.
3. Fig. 4 includes scenes with humans and vehicles, but the method of incorporating these assets is unclear. Details on how these elements are introduced and animated within the scene are missing.
4. Most visual results focus on limited, local areas. For a city-level generation, it would be beneficial to include bird’s-eye-view results covering larger spatial regions, similar to CityDreamer.
5. Including a user study comparison would provide a clearer assessment of the visual quality of the generated scenes.
6. Although the authors claim the ability to create new assets, this appears limited to the level of appearance, with geometry remaining unchanged from the asset library. Given the importance of geometry in 3D generation, this aspect should be addressed.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 2
questions: 1. In Figure 3, texture refinement only shows marginal improvement for buildings. Authors should provide more examples, including other objects and views.

2. In Figure 4, the authors shows existence of human and vehicles, how are these generated?  Are they also assets generated at some stage? Or done by manually post-processing? It is not mentioned anywhere in the paper, and this indicate the visual quality comparison with other methods is completely unfair.

3.Since the framework generate 3D scenes, I suggest the authors to submit videos or at least multi-views of the same scene to demonstrate quality and view consistency of the generated scenes.
soundness: 2
strengths: UrbanWorld introduces a pipeline that integrates generative diffusion models with an urban-specific MLLM to achieve realistic urban scene creation. This combination allows for controlled generation of 3D assets and adaptive urban design.
summary: The paper introduces UrbanWorld, a generative model designed for the automatic creation of realistic, customizable, and interactive 3D urban environments. UrbanWorld employs a progressive, four-stage generation pipeline: flexible 3D layout creation, Urban Multimodal Large Language Model (Urban MLLM)-based scene design, diffusion-based asset rendering, and MLLM-driven scene refinement.
weaknesses: 1. The authors claimed section A is flexible urban layout “generation”. However, this is not like generation methods where the distribution of urban layouts are learned from real-world data [1][2][3]. It seems like the authors are just using OSM’s GT data (AIGC-layout is not explained anywhere in the paper). No detail is given on how did the authors transform the OSM data or AIGC data into untextured 3D urban environment. Is there any generation models or other networks involved?  In short, if you are just using GT data and Blender add-on to import it, you can’t call the process “generation”.

2. In section 3.2 and the Appendix A.2, the authors shows a general urban generation prompt is converted into prompts for different categories of urban objects.  However, the same prompt is generated for all objects of the same class. Doesn’t that indicate they would have exact same style and appearance? For example, if there were 50 buildings in the scene, and they all share the same T2I prompt, they end up looking the same. Meanwhile, the authors introduced descriptions for all categories are generated by an MLLM, but did not explain where does the reference image comes from.

3. For a single asset, the authors generated textures from different views conditioned on the same text and reference image, then merged all textures. This approach cannot guarantee consistency between textures as no 3D condition has been used to guide the 2D diffusion model. Meanwhile, it cannot be called “3D diffusion renderer”, since the authors are only inferencing iteratively from pretrained 2D diffusion models. 

[1] Infinicity: https://arxiv.org/abs/2301.09637
[2] CityDreamer: https://arxiv.org/abs/2309.00610
[3] CityGen: https://arxiv.org/abs/2312.01508

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: Please refer to the weakness part.
soundness: 3
strengths: 1. The paper is well-written and easy to follow. 
2. The code has been released. 
3. The overall framework is technically sound. Based on a pre-defined set of common objects in the urban scenario, the framework bridges the gap between the 3D world and 2D views via pre-trained diffusion models. The pipeline is interesting. 
4. The framework achieves controllable and customizable scene generation, which can support tasks that require agent-environment interactions.
summary: The paper introduces UrbanWorld, a generative model designed to automatically create realistic and interactive 3D urban environments, addressing the challenges of manual labor in urban scene design. UrbanWorld employs a progressive diffusion-based rendering method and a specialized urban multimodal large language model (Urban MLLM) trained on street-view image-text data to guide the generation process. The model consists of four key stages: flexible 3D layout generation, urban scene design using Urban MLLM, controllable asset rendering, and scene refinement. Extensive evaluations demonstrate that UrbanWorld achieves state-of-the-art realism and interactivity, outperforming existing methods like Infinicity and CityGen in generating diverse urban environments suitable for embodied agents.
weaknesses: 1. Even though superior quantitative results are reported, the generated images are not realistic enough based on the demonstration in the paper. 
2. It would be better if the authors could provide more diverse qualitative results generated by the proposed method. The proposed system is claimed to be for 3D city generation. It would be good if a sequence of images/video captured with a moving camera is included to show the scene-level generation capability. 
3.  I am confused about the UV unwrapping and UV wrapping parts. How can you ensure the wrapping process can align the texture perfectly to the mesh model? For objects of different types and shapes, I believe this process can be hard to model by the diffusion model. The UV unwrapping is usually not unique. Is there any mechanism to enforce the equivariance to different unwrapping manners? 
4. I noticed that the Position-awareTexture Completion module is applied to refine the texture map. Can you provide some qualitative results (visualization) to compare the results before and after the refinement?
5. The section 4.4 is a little bit vague. How does your generated environment support navigation? How far is the longest distance your navigation can achieve? It could be better to show a bird-eye-view of your navigation environment.

Meta-review: **Summary**

The paper presents UrbanWorld, a generative world model for creating interactive 3D urban worlds. The generation consists of four stages: 1) generation of untextured 3D layouts based on input 2D layout, 2) using MLLM to generate textual descriptions detailing the appearance of the assets 3) texturing of urban assets using diffusion based on the generated description 4) scene refinement using a MLLM.

**Strengths**

Reviewers noted the following strengths of the work:
1. The task of 3D urban generation with controllable and customizable scene generation is important [YwTn,tHU8]
2. The method seems to be reasonable [YwTn,tHU8]
3. Code is provided [tHU8]
4. Comparisons show the proposed method to generate better results than previous models [YwTn,gfwE,eF1T]
5. Reviewers mostly found the paper to be well-written and easy to follow [tHU8,eF1T,YwTn]

**Weaknesses**

Reviewers were negative on the submission, and noted that some claims are unsubstantiated, with the main weaknesses being:

1. Inaccurate use of terms.  For instance, claims of the "World Model", "AGI" is not appropriate [YwTn].  The proposed frame is also not a generative model that learns the distribution, and the use of term "3D diffusion renderer" is inaccurate [eF1T]
   - Generated worlds does not seem realistic [gfwE, tHU8]
2. Weak evaluation
   - It's hard to determine from the few visual images (and no videos or birds-eye visuals) that the generated world is actually better than prior work [YwTn, gfwE, tHU8]
   - There was no evaluation of whether the generated environments can be interacted with [YwTn]
   - No human evaluation [gfwE]
3. Some aspects are unclear  
   - How are humans and vehicles incorporated? [gfwE,eF1T]
   - How are different appearances for different buildings / objects of the same class obtained? [eF1T]
   - Details of UV unwrapping and wrapping [tHU8]
   - Details of how navigation is supported [tHU8]
4. Design decisions and challenges are not clearly motivated or explained [YwTn]

**Recommendation**

As all reviewers were negative on the submission, and there was no author response, the AC recommends reject.

Decision: Reject