UrbanVerse: Scaling Urban Simulation by Watching City-Tour Videos
Authors: Mingxuan Liu
Honglin He
Elisa Ricci
Wayne Wu
Bolei Zhou
Venue: ICLR 2026 Poster (2026)
Keywords: Simulation
Real-to-Sim
Sim-to-Real
Digital Twin
Robot Navigation
Reinforcement Learning
Primary area: applications to robotics, autonomy, planning

Abstract: Urban embodied AI agents, ranging from delivery robots to quadrupeds, are increasingly populating our cities, navigating chaotic streets to provide last-mile connectivity. Training such agents requires diverse, high-fidelity urban environments to scale, yet existing human-crafted or procedurally generated simulation scenes either lack scalability or fail to capture real-world complexity. We introduce UrbanVerse, a data-driven real-to-sim system that converts crowd-sourced city-tour videos into physics-aware, interactive simulation scenes. UrbanVerse consists of: (i) UrbanVerse-100K, a repository of 100k+ annotated urban 3D assets with semantic and physical attributes, and (ii) UrbanVerse-Gen, an automatic pipeline that extracts scene layouts from video and instantiates metric-scale 3D simulations using retrieved assets. Running in IsaacSim, UrbanVerse offers 160 high-quality constructed scenes from 24 countries, along with a curated benchmark of 10 artist-designed test scenes. Experiments show that UrbanVerse scenes preserve real-world semantics and layouts, achieving human-evaluated realism comparable to manually crafted scenes. In urban navigation, policies trained in UrbanVerse exhibit scaling power laws and strong generalization, improving success by +6.3% in simulation and +30.1% in zero-shot sim-to-real transfer comparing to prior methods, accomplishing a 300 m real-world mission with only two interventions.

Published as a conference paper at ICLR 2026
URBANVERSE: SCALING URBAN SIMULATION BY WATCHING
VIDEOS
Mingxuan Liu 12,
Honglin He
Elisa Ricci Wayne Wu'
Bolei Zhou'
University of California, Los Angeles
University of Trento Fondazione Bruno Kessler
g aithub.io. io/
Videos
Urban Simulation Grounded in the Real World
Tarnger
Cape
Beijing
Los Angeles
latanbul
Tongler
Cope
TOWN
LOS Angoles
Istanbul
Zero-Shot Sim-to Real in Mapless Urban Navigation
Figure I: Urban Verse system converts world urban scenes from city-tour videos into physics-awand aware, inter- active simulation environments, enabling scalable robot learning in urban spaces with real-world generalization.
ABSTRACT
Urban embodied AI agents, ranging from delivery robots quadrupeds, are in- creasingly populating our cities, navigating chaotic streets to provide last-mile connectivity. Training such agents requires diverse, high-fidelity urban environ- ments to scale, yet existing human-crafted procedurally generated simulation scenes either lack scalability or fail to capture real-world complexity. We introduce Urban data-driven real-to-sim system that converts crowd-sourced city- tour videos into physics-aware, interactive simulation scenes. Urban Verse consists of: (i) Urban Verse- a repository of 100k+ annotated urban 3D assets with semantic and physical attributes, and (ii) UrbanVerse-CG Verse-Gen, an automatic pipeline that extracts scene layouts from video and instantiates metric-scale 3D simulations using retrieved assets. Rumning in IsaacSim, Urban Verse offers 160 high-quality constructed scenes from 24 countries, along with a curated benchmark of 10 artist-designed test scenes. Experiments show that Urban Verse scenes preserve real-world semantics and layouts, achieving human-evaluated realism comparable to manually crafted scenes. In urban navigation, policies trained in Urban Verse exhibit scaling power laws and strong generalization, improving success by +6.39 6. in simulation and +30.1% in zero-shot sim-to-real transfer comparing to prior methods, accomplishing a 300 m real-world mission with only two interventions.
"Equal contribution. Work conducted while Mingxuan Liu Visiting Rescarcher UCLA.

---

Published as a conference paper at ICLR 2026
1 INTRODUCTION
Today's is urban spaces have emerged as key arenas for the rise of micromobility systems (Oeschger et al., 2020). Small, lightweight Embodied AI (E-Al) agents (Zhang et al., 2024), such as wheeled delivery robots, quadrupeds, and humanoids, are increasingly navigating city streets to provide last-mile transportation and urban services. These mobile robots are reshaping the dynamics of city streets by improving logistical efficiency and reducing carbon emissions. Yet, the environments they operate in are often highly cluttered, with street infrastructure, sidewalks frequently blocked by parked cars, and narrow passageways, all of which pose significant challenges for E-Al agents to generalize across diverse real-world settings.
In efforts to improve generalizability, data scaling has been validated in both vision (Radford et al., 2021) and language (Kaplan et al., 2020), where large and diverse web corpora consistently lead to better generalization. In contrast, current urban navigation datasets remain limited in both scale and quality. Collecting high-quality data through human demonstrations in public spaces (e.g., sidewalk traversing) is unsafe, labor-intensee intensive, and often impractical, and thus cannot scale (Shah et al., 2023). Passive real-world data, such city-tour videos shared daily on social media like YouTube, captures diverse environments but lacks associated action labels. Moreover, both types of data are non-interactive and lack causal action-effect dynamics, which crucial for robust real-world decision- Thus, urban simulators (Wu et al., 2025a) have emerged as compelling alternative, offering interactive and virtually unlimited scenes for E-AI training. Yet, current simulators either rely on hand-crafted scenes (Dosovitskiy et al., 2017) or procedurally generated layouts defined by hard-coded rules (Wu et al., 2025a;b). The former lacks scalability, while the latter produces rigid, template-driven scenes that deviate from real-world distributions, such as randomly parked scooters or cars. However, simply increasing its quantity will lead to sustained generalization if the data does not faithfully reflect real-world distributions. This issue prompts the central question of this work: Can we build realistic, interactive urban scenes from real-world videos for scalable robot navigation?
To address this question, we propose VeranVerse, a data-driven real-to-sim system that bridges the gap between synthetic simulators and the complex, real-world streets. As shown in Fig. I (top), Urban reconstructs interactive urban scenes with real- world distributional fidelity from worldwide city-tour videos (walking or driving), enabling the training of "street-smart urban agents. At its core, Urban Verse reconstructs digital cousin (Dai al., 2024) scenes, by mapping a 2D scene to a 3D simulation-ready virtual world, with layouts, semantics, and physics aligned to real-world statistics. This combines the diversity of real data with the interactivity of simulation, enabling unlimited scene generation while preserving street-level distributions. Urban builds on two complementary pillars. The first is Urban Verse-100K (Fig. 2), a curated repository of 102,444 high-quality metric-scale urban object assets, 306 skyboxes, and 288 ground materials for roads and sidewalks. Each asset is organized within three-level urban ontology and annotated with 33 semantic, affordance, and physical attributes (e.g., mass, friction), forming the foundation for open-world, physics-ready scene construction. The second is UrbanVerse-Gen (Fig. 4), an automatic pipeline that accurately distills semantics, layouts, ground appearance, and sky from videos into urban scene graph representation, retrieves matched assets from Urban 100K, and instantiates multiple digital cousin simulation scenes in IsaacSim (NVIDIA, 2025). Using You' YouTube city-tour footage spanning 24 countries across six continents, Urban Verse produces a library of 160 simulation scenes with real-world street distributional fidelity ready for E-AI training. Together, we also introduce a 3D artist-designed set of 10 realistic urban scenes as test-only environments for closed-loop evaluation.
Urban Verse is evaluated through both scene generation quality assessment and its applicability to urban E-AI policy learning. Validated on 45 video sequences from KITTI-360 (Liao et al., 2022). our approach achieves high-fidelity scene reconstruction. recovering object semantics with 93.0% accuracy and localizing objects within 1.4 m error over 198.7 7 m scene horizon. Building on this real- world distributional fidelity, we train mapless urban navigation policies using a 160-scene simulation library generated by Urban Verse. We show that data scaling power-law emerges when training on our high-fidelity scenes, enabling simple policies to generalize to diverse, unseen urban spaces. Deployed across 16 real-world streets two embodiments (a wheeled robot and quadruped) in a zero-shot manner, policies trained on Urban Verse scenes consistently outperform state-of-the-an navigation foundation models, reaching up to 89.7% success in sim-to-red transfer. Finally, our policy completes a 337 m long- fong-horizon mission in public spaces with only two human interventions. All assets, scenes, and code of Urban Verse will be open-sourced to accelerate embodied AI research.
2

---

Published as a conference paper at ICLR 2026
Simulator
Creation
Realism
Scone Diversity
Physiis
VOhject
Ollject
8ky Mage
Ground Materiak
&ahhot Types
Training Paradigos
Embodiid
CARLA
Crafiaa
Hand
Realistk
106
935
30
Rf. n.
VIA
Navigation VQJA
MrtaUrban
Generation
Proeedral
Unrealistic
Templates
39
10,000
1
RL IL
Navigation
UshanSim
Procedural Generation
Unmeaietic
Templotes
39
15,000
RL. R.
Novigation
UrbunVerse
Anto
Data-driven
Realistic
+00
YLA
Mobile Manipulation
Navigation, VQA
Table I: Comparison of Urban with existing urban embodied-Al Al simulators.
RELATED WORK
Urban Navigation. Deep reinforcement learning (Mirowski et al., 2016) has demonstrated strong po- tential for goal-based mapless urban navigation by removing the dependency on pre-built maps (Chap- lot et al., 2020). Recent advances have introduced ussion-based navigation foundation models (Shah et al., 2023; Sridhar et al., 2024; Hirose et al., 2025), which leverage cross-sensor capabilities and large-scale offline vision data to improve generalization across robot platforms and camera setups. A major limitation of these approaches, however, is the lack of environmental interaction in the training data. As we demonstrate in Sec. 4.3, this results in poor obstacle avoidance (Liu et al., 2025). S2E (He et al., 2025) tackles this issue by training in interactive simulation scenes, rather than relying on passive data for extend both coverage and realism by training vision- based position- navigation policies in our real-world grounded Urban Verse scenes, leading to improved robustness and generalization.
and achieves real-world obstacle avoidance. In this work, we
Urban Embodied AI Simulators. Mainstream simulators focus either on indoor environ- ments (Kolve et al., 2017; Li et al., 2023a) or on driving domains centered on roads and high- ways (Kothari et al., 2021; Li et al., 2022), with only CARLA (Dosovitskiy et al., 2017) extending to city neighborhoods but relying on non-scalable, hand-crafted scenes (15 in total). The rise of micromobility (A (Abduljabbar et al., 2021) agents, such e-scooter delivery robots, has motivated the development of urban simulators like MetaUrban (Wu et al., 2025a) and UrbanSim (Wu et al., 2025b), which share our goal of modeling richer city spaces. However, these simulators face three key limitations: i) layoul realism: procedurally generated scenes deviate from real-world distributions; ii) asset diversity: limited object categories; iii) physies annotation: objects lack physical properties and remain static props. We provide a functional comparison between Urban Verse and existing urban simulators in Tab. 1. Building upon UrbanSim simulation platform, we address these gaps with our real-world grounded scene creation pipeline and semantically-rict physics-annotatte asset library.
Simulation Scene Creation. Automating scene creation for F-AI learning has traditionally relied on either enhancing procedural generation rules (Deitke et al., 2022; Li et al., 2023b; Yang et al., 2024; Wu et al., 2025a;b) using high-precision 3D scans to replicate real-world environments (Deitke et al., 2023a; Huang et al., 2025; Yu et al., 2025). More recently, 3D Gaussian Splatting (3DGS) methods like OmniRe (Chen et al., 2024) and Vid2Sim (Xie et al., 2025) reconstruct 3DGS digital twin environments from RGB videos for coarse simulation. However, current 3DGS based digital twins designed for one-to-on reconstruction and produce single fused radiance field without complete geometry, object instances, semantics, or physical attributes, making them unsuitable for editing, interaction, or physics-based simulation. Our work shares conceptual similarities with digital cousin approaches (Dai et al., 2024; Maddukuri et al., 2025; Melnik et al., 2025), which generate multiple virtual indoor scenes from a calibrated RGB image while preserving key semantics, geometry, and layout to improve manipulation policies. In contrast, Urban Verse constructs large-scale, street-level urban digital cousin scenes from uncalibrated RGB videos.
METHODOLOGY
To convert worldwide city-tour videos into physics-aware simulations, two indispensable elements are required: i) large-scale 3D asset database with physical annotations that match the magnitude of real-world semantic richness and appearance diversity, and ii) an automated open-vocabulary pipeline that extracts semantic and spatial layouts from any uncalibrated videos to generate simulation scenes. To this end, in Sec. 3.1, we first describe the data collection and semi-automatic annotation pipeline in Urban Verse for building the Urban Verse- database. Next, in Sec. 3.2, we present Urban Verse-Gen, our automated pipeline that extracts detailed scene representations from videos
3

---

Published as a conference paper at ICLR 2026
102,444 Densely Annotated Urban Objects at True Metric Scale
288 Ground Materials
306 Sky Maps
Articulated Objects
Navigable Buildings
Detailed Street Litter
Rich Curbside Infra
Diverse Barriers
Figure 2: Example instances from our large-scale urban asset database Urban V00K.-100K. Assets range from 0.03 crushed
200 skyscraper, all annotated metric scale; note the realistic relative scales between objects, with road and sidewalk materials and sky maps ensuring realistic ground appearance and illumination.
to ground simulated scene generation. Finally, in Sec. 3.3, we show how Urban Verse leverages crowd-sourced videos construct a large-scale scene library for policy learning and testing.
3.1 URBANVERSE-I00K ASSET DATABASE
Urban Verse-100K is large-scale, high-quality 3D asset database designed for urban simulation and beyond. To comprehensively model real-world scenes, require not only vast collection of on-ground 3D object assets but also diverse materials to render ground surfaces (e.g., cobblestone sidewalks snowy roads) and realistic lighting conditions across different times of day. Each of these significantly influences robot perception and interactivity. For this, as shown in Fig. 2, Urban Verse- 100K comprises three collections: (i) Object: 102,444 GLB objects spanning 659 categories, each annotated with 33 semantic, physical, and affordance attributes true metric scale; (ii) Ground: 288 photorealistic PBR materials (98 road, 190 sidewalk) for ground plane texturing; and (iii) Sky: 306 HDRI sky maps for realistic global illumination and immersive 360 backgrounds. We next outline the collection and annotation strategies for Urban 100Ke-100K, with additional details in App. C.
Object Collection. The recent development of large-scale 3D object repositories (Deitke et al., 2023b), such as Objaverse (Deitke al., 2023c), provides valuable resources for constructing sim- ulation scenes. However, due to their web- crawled nature, these repositories suffer from sev- cral critical issues: (i) most assets are unrelated to urban environments; (ii) many assets are rupted (e.g., missing textures, incomplete 3DGS reconstructions, or paper-thin geometry); (iii) as- sets often have non-melric scales, with examples such a cucumber being as large car; and (iv) assets lack semantic and physical attribute annota- tions. Fig. 25 and Fig. 26 in the appendix illustrate these issues. To this end, we curate high-quality urban subset from the 800K noisy 3D assets the Objaverse dataset (Deitke et al., 2023c) through asset viewer interface (Fig. 22)
tegh
ambie
Salety Clparance 155m
Charge Port Location Letrona quan
3 Re
2-3-300
friction
Aftundances:
Interotion
Boad inbed (eeahoo linns (Clasand mesl Piaee
Figure 3: Example annotated object attributes.
three-stage semi-aute autimaattoaaicc pipeline. First, we build Three.js- and employ human annotators to efficiently filter out corrupted or low-quality assets. After filtering, we retain 158K high-quality assets. Next, we build three-level urban ontology derived from the OpenStreetMap tag structure (Bennett, 2010) and expanded with categories from driving and scene understanding datasets (Zhou et al., 2017: Cordts et al., 2016; Caesar et al., 2020; Gupta et al., 2019; Kuznetsova et al., 2020), resulting in 659 leaf-level categories. Next, each asset is classified into a leaf category using CLIP (Radford et al., 2021) its thumbnail, followed by manual verification to remove non-urban items (e.g., game weapons, spaceships) and correct misclassifications, yielding our final curated set of objects. Lastly, leverage the world knowledge of GPT-4.1 (OpenAI, 2025) to
4

---

Published as a conference paper at ICLR 2026
Real world Scans Distilattion
Scane Mater ialization Diverffication Urbaan Graph
Simutation Scane Creation
Impult City-tour
Sisurd Froing and PBR Kanvial Tecuring
Sor 2013
100mm
MASTER
CO
Cijeet
Parter
Ground
UrbanVerse- 100k
Hascing
PeriMSS Maiching
Riviig
Aspestencs Selector
Deject Placemer and Prysicas Dratzed
Bound Dightl Cousin Assete
Ros HIRI iraans
Read Teatuene
Oejecn Horal
Bry Crep
Semenbee Wnseacee
Outgut Fullyintsrocop Simulation Soorics
Figure 4: Overview of the Urban Verse-Gen pipcline. Figure better seen magnification.
annotate each asset with semantic, affordance, and physical attributes (e.g., size, mass). We prompt it with the object thumbnail and four rotated snapshots. Fig. 3 shows an example of few annotated attributes. Using the annotated size and front-view, as displayed in Fig. 2 (left), we standardize all assets in our database to metric scale and a consistent orientation.
Ground and Sky Collections. To provide the simulated ground plane with realistic and diverse appearances, we collect 4K-quality PBR road and sidewalk materials from free-licensed reposi- tories (FreePBR, 2025; AmbientCG, 2025), each with thumbnail and spanning wide range of conditions, as shown in Fig. 2 (middle). For lighting, we gather artist-desinne HDRI sky maps (Fig. 2 (right)) for urban settings from PolyHaven (2025), each accompanied by rendered thumbnail and description. These maps enable image- based lighting, providing realistic global illumination with precise control over color, intensity, and direction, while also supplying immersive background.
3.2 URBANVERSE-GEN SCENE CONSTRUCTION PIPELINE
With our richly annotated asset database, we design Urban Ver,e-Gen, an automatic open- ppen-vocabulary pipeline capable of extracting real-world 3D scene layouts from uncalibrated RGB city-tour videos and generating fully interactive simulations. To structure a scene, we first introduce a unified 3D urban scene graph V (O,G, (O, that encodes: (i) O object nodes (e.g., cars, buildings) with category, location, orientation, and appearance; (ii) G ground nodes (road/sidewalk) with spatial extent and appearance; and (iäi S sky node capturing illumination and distant background. Distilled from real-world videos, this graph serves as compact blueprint for guiding simulation scene generation.
Overview. As shown in Fig. 4, Urban-erse-GGi Verse-Gen operates in three stages: (1) distillation, where object semantics and 3D layout, ground composition (road/sidewalk) and appearance, lighting, and distant background are extracted from the input video into unified scene graph representation; (2) materialization and diversíficatio, where multiple digital cousin assets from Urban Verse- 100K matched and bound to each graph instance; and (3) generation, where object, ground, and sky instances are assembled with their extracted spatial information into physically plausible scenes. We detail each stage below. Further technical specifics in App. E.
Real-World World Scene Distillation. To accurately parse semanties and estimate 3D layouts from uncalibrated open-world videos, we design a distillation module that integrates 2D open open-vocabulary foundation models with SIM through 3D lifting. Given an RGB city-tour video I = {11hh yT we sample every third frame and query GPT-4.1 (OpenAI, 2025) enumerate visible categories and form a candidate vocabulary. The video is lifted to metric 3D by estimating depth, intrinsics, and SE(() poses using MASt3R (Leroy et al., 2024). With these semantic and geometric estimates, we assemble an open-vocabulary object parser using YoloWorld (Cheng et al., 2024) and SAM 2 (Ravi et al., 2024)) to obtain fer-frame instance masks, which are lifted to metric 3D. Identical detections are fused across frames by semantic similarity and plont-cloud overlap, yielding N persistent object nodes O = {o, = (I,, c, b, M,) J Ji=1- where 1, is the category, c; the centroid, b; the oriented 3D box, 0, the yaw, and M, {ma,h the 2D object crops. In parallel, we segment
5

---

Published as a conference paper at ICLR 2026
Dopth
Eepth
Homss
Normel
Instafic
Figure 5: Diverse dynamic agents in Urban Verse scenes. Two example scenes populated with pedestrians, cars, wheelchair users, and scooter riders, shown across multiple observation modalities.
road and sidewalk using a panoptic Mask2Former ground parser (Cheng et al., 2022) trained on Cityscapes (Cordts al., 2016), lift and fuse results into metric point clouds p. and define ground nodes g = { (road, Prd. Mrd)), (sidewalk, Psw. the sky parser captures global illumination and distant background by cropping the upper half of each frame, stored in the sky node as S = {X1)==1
Sau:Maan MMsw) }, where .M preserves ground masks. Finally,
Scene Materialization and Diversification. With the distilled scene graph, our goal is to materialize each instance using digital-cousin assets from UrbanVerse- 100K that are semantically aligned, geometrically consistent, and visually faithful yet diverse, enabling varied appearances for stronger policy generalization. To meet these requirements, we retrieve Keousin assets for each object node o, E O through three corresponding steps: (i) semantic matching selects the best-matched asset category via CLIP similarity (Radford al., 2021) with its label 1; (ū) geometry filtering ranks candidates within that category by minimal Bounding Box Distortion (mBBD) between b; and candidate boxes, retaining the top 1,000; (iii) appearance selection re-ranks these candidates using DINOv2 similarity (Oquab et al., 2023) between M₁ and asset thumbnails, keeping the top-kcousin as final matches. For ground nodes, we retrieve kcousin PBR materials by comparing pixel-wise MSE between road/sidewalk crops MMra/Msw and rendered thumbnails. For the sky node, we select Keoasin HDRIs by matching HSV histograms sky crops }X,} to HDRI thumbnails, reproducing illumination and distant background.
Simulation Scene Creation. Finally, we instantiate the materialized graph into Keousin simulated scenes in UrbanSim (IsaacSim backend) by: (i) ground fitting and texturing, where road and sidewalk planes are fitted from Pra/Paw: sidewalks are elevated by 15 cm, and surfaces are textured with the selected PBR materials: (ii) lighting and surroundings, where a matched HDRI sky map is used as both a dome light source and spherical environment to provide realistic skies and distant context; (iii) object placement, where each object is positioned at its centroid C; aligned to the canonical front and heading 0., and adjusted to avoid penetrations. We then assign annotated physical parameters (e.g., mass, friction) and enable rigid-body dynamics, producing stable, interactive scenes. With metrically scaled assets from Urban Verse- 100K aligned to extracted layouts, the resulting scenes are true-to-scale, grounded in real-world layouts, and ready for E-AI agents to explore.
Interactive Dynamic Agent Population. Urban also supports dynamic agents such pedestri- ans, cars, wheelchair users, and scooter riders. Following UrbanSim (Wu et al., 2025b), we use a GPU-accelerated ORCA-based planner (Van Den Berg et al., 201 1) to populate scenes with agents that move realistically and interact with the robot. For each scene, we build a 2D occupancy map, sample start-goal pairs, and compute collision-free paths; agents then continuously adjust their veloc- ities during simulation for smooth, collision-awar motion. As shown in Fig. 5, this scene-agnostic mechanism enables diverse dynamic agents across all Urban Verse environments.
3.3 URBANVERSE SCENE LIBRARY
Urban Verse Scene Library Construction. Using Urban Verse, we build a training library of 160 simulation scenes grounded in real-world distribution. We have collected 32 city-tour videos from YouTube under Creative Commons License, spanning continents. 24 countries, and 27 cities. Each 3-min clip is distilled into a aayout-groundes scene using U Verse- and expanded into Keowsin 5 digital cousin variants, yielding 5 x 32 = 160 scenes. See construction details in App. D.I.
Urban Benchmark. Further, to enable closed-loop evaluation, we construct a benchmark that comprises: (i) AutoBench, 10 scenes automatically generated from hold-out city-tour videos using

---

Published as a conference paper at ICLR 2026
Scowax Navigaton
BeEllgg W Exgedellg
SigoR Collection
Figure 6: Urban embodied-Al tasks supported by the Urban Verse simulation platform.
Urban Verse- Gene and (ii) CruftBench, 10 artist-designed scenes reserved for test-only evaluation. As shown in Fig. 32 appendix, CraftBench spans diverse scenarios, layouts, cultural context, and safety-critical edge cases. To avoid bias, designers had no access to our assets and scenes.
Urban Verse Tasks. In this work, we focus on urban navigation as the primary case study in our experiments because it most clearly demonstrates the benefits of UrrsnVerse's s realistic layouts and asset distributions. However, all assets in Urban Verse include semantic labels, physical parameters, and affordance tags, enabling a wide range of urban embodied tasks. As the few examples illustrated in Fig. 6, Urban Verse naturally supports multi-agent interaction, mobile manipulation, and expert data collection for imitation learning, opening broader research opportunities.
EXPERIMENTS
We evaluate Urban Verse on three aspects: (i) scene construction capability, assessing fidelity and quality of scenes constructed from real-world videos (Sec. 4.1): (ii) scaling capability, examining whether training on Urban Verse scenes follows scaling laws that improve policy generalization (Sec. 4.2); and (iii) sim-to-real transfer capability, testing whether policies trained in Urban Verse enable robust and stable transfer to real-world environments (Sec. 4.3).
Policy learning for mapless urban navigation. In our study, following Wu et al. (2025b), we focus on RL to exploit scene interactivity, and study the task of position-goes urban navigation: the agent starts from a known ground-plane pose, receives a goal and waypoints sampled every 5 m from GPS projected to a local metric frame, and must learn a policy to reach the goal within distance tolerance while avoiding collisions and staying on traversable surfaces. Using the 160-scene Urban Verse library, we train vision-only navigation policies using PPO (Schulman al., 2017) to show the effectiveness of our weal-world grounded simulation scenes. During training, we load 16 different scenes at time and repeat each scene 4-6 times depending on the available GPU memory. The set of scenes is changed every 100 RL episodes to expose the policy to diverse environments. In both training and testing, the agent is provided with only RGB observation, its relative position to the goal, without access to the global map. See model architecture and training details in App. L.3.
4.1 SCENE CONSTRUCTION FIDELITY AND QUALITY
Scene fidelity evaluation. We first evaluate whether UrbanVerse can faithfully recover scene semantics and layouts from video. Using 45 KITTI-360 (Liao et al., 2022) sequences (average length 198.7 m) of residential and city streets, we generate digital cousin scenes with CrrbaVeerse Gen. Scene reconstruction fidelity measured by comparing the nearest digital-cousin scene, built with I matched assets, against ground-trute -tuth annotations: semantic fidelity by the proportion of correctly preserved categories; layout fidelity by per-object pose errors (L2 distance and orientation difference); geometric fidelity by bounding-box volume difference; and overall recovery by 3D detection mAP25 (Kumar et al., 2024). Appearance fidelity cannot be directly measured since real-world asset replicas are unavailable. Instead, we input walkthrough videos of ten simulated CraftBench scenes into Urban Verse-Gen and report the proportion of objects whose 3D models are exactly retrieved. For comparison, we evaluate two SfM models (VGGT (Wang et al., 2025) and our default MASt3R) and two open-vocabulary parsers (GroundedSAM2 (Ren et al., 2024) and our default AMM, combining YoloWorld with SAM2). Lastly, we present side-by-side comparisons of city-tour videos and their reconstructed scenes in Fig. 7.
Quantitatively, Tab. 2 shows that the MASt3R with our YoWorldSAM2 parser yields the best overall results, which adopt as Vrbsn'erse'ss default. With this setup, 93.1% % of object categories correctly preserved; reconstructed objects deviate by only 1.4 m in position. 19.8 orientation, and 0.8 m² in volume. Asset retrieval achieves 75. accuracy, indicating effective matching of
7

---

Published as a conference paper at ICLR 2026
SfM
Scene Parser
Cat. (%)}
Dist. (m) 4
Ori. ())
Scale (m"))
mAP25t
Ast. (%)
GroundedS AM2
88.2
2.4
21.5
1.5
7.5
67.5
VGGT
YoWoridSAM2 91.5
2.1
20.1
1.3
9.4
70.6
GroundedSAM2 93.1
2.1
19.9
1.1
24.3
68.5
MAS13
YoWoridSAM2
1.4
19.8
0.8
28.2
75.1
Table 2: Scene reconstruction fidelity evaluation of nearest digital-cousin generation. We report average results on KITTI-360, including: Category recovery (Cat.), the fraction of correctly categorized objects; Distance (Dist.), Orientation (Ori.), and Scale, the mean differences in centroid position, heading, and volume between recovered and ground-truth bounding boxes; and 3D detection mAP. Asset recovery (Ast.), the fraction of correctly retrieved objects evaluated from CraftBench scene videos, also reported.
Urba Verse Scone
UrbarVa ree Scane
Figure 7: Qualitative scene generation results of Urban Verse. Scenes generated from Cape Town (left) and Morocco (right) city-tour videos in our library. Highlighted details are shown in the circled areas. See Fig. 28 in the Appendix for additional qualitative results.
visually similar 3D assets from our database. Qualitatively, Fig. shows that Urban Verse produces physically plausible scenes that preserve fine object placement details from the original videos, such as cranes located in the distance motorcycles parked along the roadside. Given the long horizon length the evaluated street scenes, these results demonstrate that Urban Verse can faithfully capture real-world semantics and layout distributions from casual city- city-tour videos, enabling the generation of high-fidelity simulated scenes that reflect real-world street distribution.
Human Evaluation of Scene Quality. We evaluate whether Verse scenes align with human impressions of everyday streets through two user studies with 32 undergraduates. In the first comparative study, 100 Urban Verse scenes sampled from our library are paired with 100 UrbanSim 's procedurally generated (PG) scenes generated from the same Urban Verse- 100K assets. Participants view shuffled overview images and choose which scene is better (or equally good) in terms of object diversity, layout coherence, and overall realism. In the second study, participants rate the overall realism of 30 scenes, 10 from Urban Verse, 10 from PG, and 10 aarist-designe oracles from CraftBench, by watching 360 walkthrough videos and scoring them 1-5 scale. Further details provided in App. M.
(a) Object Diversity
[b) Layout Coherence
Equal
UtbanVenso 71 .92
UrbanWerse
(06)
(2,279)
UmbanSi 2%
79.0%
(2,577]
UrbanSim 13 77
(c) Overall Realism
(d) Scene Quality Rating
Equal
1.9%
UrbanWerce 5%
(60)
2.58
UrbanSim
Designad
[2.400]
20.04 (680)
Werbe
1.96
Urbansum
Figure 8: Human evaluation results.
As shown in Fig. 8 (a-c), participants consistently preferred Urban Verse over PG, with more than 70% favoring it across Object Diversity, Layout Coherence, and Overall Realism. In the second study, the human ratings of artist-designed scenes (4.08/5.00) in Fig. 8 (d) suggest that even hand-crafted scenes imperfect, underscoring the challenge of creating highly realistic outdoor environments. Given this difficulty, the 3.58/5.00 score of Urban Verse scenes is satisfactory for an automated approach.
4.2 SCALING URBANVERSE FOR POLICY GENERALIZATION
We next examine the scaling properties of using the Urban Verse 160-scene library for policy general- ization in mapless urban navigation on wheeled robot. We first study how the number of training layouts influences generalization, then analyze the effect of expanding each layout with more digital cousins. Policy performance is measured by success rate (SR), route completion (RC), and collision times (CT), with all evaluations conducted in unseen environments from AutoBench and CraftBench. We provide detailed experimental setups and metric definitions in App. I..
8

---

Published as a conference paper at ICLR 2026
PPO Uoterkeme
PD Uitrleeeeeeeeeeeoooo deceeeeeeeerrrr
10P
0.5
Parrson in -0403
Pearson -0-033
0.4
Eo0 500
0.0
ROUnaniin on Crafercch
PRO Ule Reale
18
(2) NJ/ubber Training Layouts
Number of Training Cousins
Power-law Treining
(d) Power-law Fit: Training Cousins
Figure 9: Generalization by scaling training layouts and digital cousins per scene.
Scaling with more unique layouts. We fix the number of cousins per layout and vary the number of unique layouts. From the 160-scene Urban Verse library, select 32 unique layouts (each from a distinct city-tour video), each expanded with five cousins. Training sets are constructed with N E(,1,, 16, 32} layouts, corresponding to 5, 40, 80, and 160 scenes. For comparison, policies on matched PG scenes in UrbanSim with identical layout counts and per-layout variants. Policies are then evaluated on the 10 CraftBench scenes with five runs per scene. As shown in Fig. (a), increasing the number of Urban Verse layouts consistently improves generalization, with success rates rising sharply as coverage expands. The flat slope at small N highlights how limited layout diversity constrains generalization. In contrast, policies trained PG layouts show very limited improvement, indicating that hard-coded templates lack the diversity needed to scale.
train
Scaling with more digital cousins. We fix the number of unique layouts and vary the number of digital cousins per layout. Using 32 layouts, we select m E 2,,3,3, 3, 4, 5) top-ranked cousins, yielding training sets of 32-160 scenes. Policies trained on these sets are evaluated on 10 AutoBench scenes and 10 CraftBench scenes. As shown in Fig. 9 (b), increasing the number of cousins consistently further boosts success rate, confirming that intra-layout diversity complements inter-layout diversity to reinforce generalization. Performance is overall higher on AutoBench, reflecting distributional familiarity with scenes generated by the same pipeline, while the lower scores on CraftBeenc highlight the difficulty of artist-desineed environments. Notably, the narrowing gap between the two testbeds indicates that greater per-layout diversity improves robustness to distribution shift.
Validating scaling power laws, We next validate whether performance gains follow a pawer-law scaling relationship (Lin et al., 2025b). Defining test error as E (number of layouts or cousins), power law holds if F = 3N-a, which becomes linear after log transform: log F = aa log N + log B. As shown in Fig. (c, d), linear fits in log-log space confirm clear power-law behavior for both layout and cousin scaling, with strong Pearson correlations.
SR and training scale as N
Benchmarking overall generalization. We compare our strongest policy (PPO-Urban Verse,, trained on all 160 Ur- ban Verse scenes, against foundation models MBRA (Hi- rose et al., 2025), City Walker (Liu et al., 2025), S2E (He al., 2025), and PPO policy trained 160 UrbanSim PG scenes, all evaluated on CraftBench. As an overfit- ting reference, we also train policies directly on each test scene. As shown in Tab. 3, PPP--rbaan Verse, despite its simple architecture, consistently outperforms all baselines, achieving a +6.3% SR gain over the second-best model MBRA. Policies trained directly on test scenes perform poorly on altered routes, revealing strong overfitting and underscoring the need for diverse training scenes to enable true generalization.
Method
SR f CT 1 RC f
MBRA
35.6
25.6
52.9
City Walker
29.2
38.2
48.6
S2E
33.1
27.7
55.7
PPO-UrbanSim
9.1
31.5
19.4
PPO-Urban Verse
41.9
35.5
62.4
Overfitting
26.5
32.2 40.6
Table Results on CraftBench.
4.3 ZERO-SHOT SIM-TO-REAL POLICY TRANSFER
Zero-shot transfer across urban spaces and embodiments. We evaluate our strongest policy, trained on all Urban Verse scenes, in 16 unseen real-world urban scenarios averaging 24.6 m per route (see Fig. 35 for examples). The same policy is deployed two embodiments: the Coco wheeled delivery robot (Coco Robotics, 2024) and the Unitree Go2 quadruped (see Fig. 36 for robot configuration). We benchmark against navigation foundation models NoMad (Sridhar et al., 2024), S2E, and a PPO policy trained on PG scenes. Each evaluation is repeated three times, with distance-to-gon (DTG) also reported. As shown in Tab. 4, PrO-Urban Verse significantly outperforms

---

Published as a conference paper at ICLR 2026
R Wheeled
SR
CT
RCT1
DTG 4
NoMad
33.3
66.7
57.4
CityWalke
25.0
75.0
42.7
12.0
S2E
47.9
54.2
59.6
8.2
34.6
11.8
Veoee
77.1
22.9
83.4
3.6
Start
95 Quadruped
SR 1
CT
4
RC1
DTG
Goal
NoMad
37.5
62.5
54.4
12.5
CilyWalker
31.3
66.8
42.6
13.4
bb
S2E
58.6
41.7
71.4
6.3
15.4
Veo-UrbanVerse
89.7
10.4
86.4
2,5
Start
Table Real-world results.
Figure 10: Visualization of real- world results.
Humen thupover.
Hum takeoven
Curcous pedestrians
Start the didwssk
Noisy GPS
Cross the street
constructio
D Shart
h
Robon Platform
TeleOpp Station
Goal
Weyoolms
Autonomy Humen-Controlidd (Takeovart, Coltey-Conroliedd (Autopliot)
Time
409
Figure 11: Long-horizon mapless urban navigation. The PPO-Urban Verse policy pilots the Coco robot across 337 of public urhan space, reaching the goal with only two human interventions and callisions.
all baselines, despite its simple architecture, it surpasses the second-best model (S2E) by +29.2% % SR on Coco and +31.1% % SR on Go2. Foundation models like City Walker trained on large-scale but non-interactive data, succeed mainly in obstacle-free cases and fail when obstacles appear. In contrast, PPO-Urban Verse consistently demonstrates robust obstacle avoidance (e.g.. bypassing bollards after turns while crossing streets). These results show that interactive capabilities learned via RL in UrbanVerse transfer effectively and reliably to real-world settings in a zero-shot manner.
Long-horizon mapless urban navigation. To stress-test policy stability, we deploy PPO-Urban Verse on the Coco robot for 337 m mission real urban spaces. The robot follows GPS waypoints 10m intervals. For safety in public streets we implement a human-Al shared autonomy TeleOp system that allows real-time human intervention when needed. As the route recording shown Fig. 11, successfully completes the task with only two interventions Completing such challenging task in public streets demonstrates the robustness policies trained on Urban scenes, stability we attribute in part to its long training routes (s:200 m) that encourage generalizability for long-horizon tasks. This highlight Urban Verse's potential for training versatile, practical urban agents.
CONCLUSION
We introduce Urban Verse, a data-driven real-to-sim system that brings our daily messy streets into interactive simulation environments. Leveraging the curated UrbanVerser 100K and the automated Urban Verse-Gen pipeline, Urban Verse can mass mass-produce simulated scenes that faithfully capture real- world distribution, enabling effective policy scaling and more generalizable urban Al embodiments.
Limitation. Urban Verse is currently tailored to street-level urban environments; extending it to parks, campuses, or indoor-outdoor transitions would require additional terrain modeling and access structures, which consider promising direction for future work. In addition, Urban Verse-Gen real-to-sing pipeline can be affected by rare challenging video conditions, such as low light, fast motion, heavy occlusion that may introduce depth and pose drift or imperfect object placement.
Urban
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This project was supported by NSF Grants CNS-2235012, IIS-2339769, CCF-2344955, TI-2346267, and ONR grant NM00142512166 Mingxuan Liu and Elisa Ricci were partially supported by the EU Horizon projects ELIAS (No. 101 101120237) and ELLIOT (No. 101214398). Honglin He was supported by the Amazon Trainium Fellowship. We gratefully acknowledge Coco Roboties for their generous equipment donation. We thank Yukai Ma for his support with the robotic infrastructure and Jack He for his assistance with the VR demo. We also acknowledge CINECA and the ISCRA initiative for providing high- -eeh--eerfrmaccee computing resources.
ETHICS STATEMENT
All city-tour videos used in this work are sourced from YouTube platforms that provide free Creative Commons licenses. Prior use, apply automated and manual filtering to remove any frames containing human faces, license plates, or other identifiable information, ensuring that no personally sensitive data is retained. Our focus is solely urban layouts, object distributions, and physical attributes. While our system enables scalable simulation for embodied AI, potential misuse (e.g.. surveillance applications) must be acknowledged. We therefore emphasize responsible use of our released assets, code, and data strictly for research purposes in urban simulation and embodied AI.
REPRODUCIBILITY STATEMENT
To ensure reproducibility, we will release the full Urban Verse-100K (annotated 3D assets, ground materials, and skyboxes), the Urban Verse-Gen implementation code, and the 160-scene VrbanVerse library constructed from city-tour videos. All experiments are documented with dataset splits, training details, and hyperparameters. Scripts for preprocessing, scene generation, and policy training will be included, along with instructions for reproducing results on KITTI-360 and real-worke deployment tests. By open-sourcing our resources, we aim support and accelerate embodied AI research in urban environments
REFERENCES
Rusul L Abduljabbar, Sohani Liyanage, and Hussein Dia. The role of micro-mobility in shaping sustainable cities: A systematic literature review. Transportation research part D: transport and enviromment, 2021. 3
AmbientCG. ambienteg. https: httt:s://amiientett /t/////mbiinngggce/ee/ cww. 2025. 5
Jonathan Bennett. OpenStreeiMap. Packt Publishing Ltd, 2010. 4, 25
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: multimodal dataset for autonomous driving. In CVPR, 2020. 4, 25
Holger Caesar, Juraj Kabzan, Kok Seang Tan, Whye Kit Fong, Eric Wolff, Alex Lang, Luke Fletcher, Oscar Beijbom, and Sammy Omari. nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles, (rXiv:216.118//1 2021. 27
Devendra Singh Chaplot, Dhiraj Gandhi, Saurabh Gupta, Abhinav Gupta, and Ruslan Salakhutdinov. Learning explore using active neural slam. arXi::2004.05/55 2020. 3
Ziyu Chen, Jiawei Yang, Jiahui Huang. Riccardo de Lutio, Janick Martinez Esturo, Boris Ivanovic, Or Litany, Zan Gojeic, Sanja Fidler, Marco Pavone, et al. Omnire: Omni urban scene reconstruction. arXiv:2408. 16760, 2024. 3
Bowen Cheng, Ishan Misra, Alexander G Schwing. Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In CVPR, 2022. 6
Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, and Ying Shan. Yolo-world: Real-time open-vocabulary object detection. In CVPR, 2024. 33
11

---

Published as a conference paper at ICLR 2026
Coco Roboties. Coco Robotics, 2024. URL htwph:tpp//..m
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016. 2,
Tianyuan Dai, Josiah Wong, Yunfan Jiang, Chen Wang, Cem Gokmen, Ruohan Zhang, Jiajun Wu, and Li Fei-Fei. Automated creation of digital cousins for robust policy learning. In CoRL, 2024. 2,
Matt Deitke, Winson Han, Alvaro Herrasti, Aniruddha Kembhavi, Eric Kolve, Roozbeh Mottaghi, Jordi Salvador, Dustin Schwenk, Eli VanderBilt, Matthew Wallingford, et al. Robothor: An open simulation-to-real embodied ai platform. In CVPR, 2020. 27
Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. Procthor: Large-scale embodied ai using procedural generation. In NeulPS, 2022. 3, 27
Matt Deitke, Rose Hendrix, Ali Farhadi, Kiana Ehsani, and Aniruddha Kembhavi. Phone2proc: Bringing robust robots into our chaotic world. In CVPR, 2023a.
Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, al. Objaverse-xl: A universe of 10m+ 3d objects. In NeurlPS, 2023b. 4, 26
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: universe of annotated 3d objects. In CVPR, 2023c. 4, 25, 26
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In CoRL, 2017. 2, 3, 27
FreePBR. Free pbr. https:: http:///eeeebbrree com/. 2025. 5
Chuang Gan, Jeremy Schwartz, Seth Alter, Damian Mrowca, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, et al. Threedworld: A platform for interactive multi-modal physical simulation. vrXiv:2007.04995 2020. 27
Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: dataset for large vocabulary instance
segmentation. In CVPR, 2019. 4, 25
Honglin He, Yukai Ma, Wayne Wu, and Bolei Zhou. From seeing to experiencing: Scaling navigation foundation models with reinforcement learning. wwXiv:2507.22028, 2025. 3,9
Noriaki Hirose, Lydia Ignatova, Kyle Stachowicz, Catherine Glossop, Sergey Levine, and Dhruy Shah. Learning to drive anywhere with model- reannotation. 255i::2505.0592,, 2025. 3,
Zhening Huang, Xiaoyang Wu, Fangcheng Zhong, Hengshuang Zhao, Matthias Nießner, and Joan Lasenby. Litereality: Graphics-ready 3d scene reconstruction from rgb-d scans. arXiv:2507.02861, 2025. 3
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv:2001.0836/ 2020. 2
Bernhard Kerbl, Georgios Kopanas, Thomas Leimküüler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 2023. 26
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, Kembhavi Aniruddha, Gupta Abhinav, and Farhadi Ali. Ai2-thor: An interactive 3d environment for visual ai. arXiv:1712.05474, 2017. 3, 27
Parth Kothari, Christian Perone, Luca Bergamini, Alexandre Alahi, and Peter Ondruska. Drivergym: Democratising reinforcement learning for autonomous driving. arXiv:211 1.6689, 2021. 3, 27
12

---

Published as a conference paper at ICLR 2026
Abhinav Kumar, Yuliang Guo, Xinyu Huang, Liu Ren, and Xiaoming Liu. Seabird: segmentation in bird's view with dice loss improves monocular 3d detection large objects. In CVPR, 2024. 7
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset v4: Unified image classification, object detection, and visual relationship detection scale. I/C,, 2020. 4, 25
Vincent Leroy, Yohann Cabon, and Jérôme Revaud. Grounding image matching in 3d with mastär. In ECCV, 2024. 5, 33
Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín- Martín, Chen Wang, Gabrael Levine, Michael Lingelbach, Jiankai Sun, et al. Behavior-1ke A benchmark for embodied ai with 1,000 everyday activities and realistic simulation. CoRL, 2023a. 3,27
Quanyi Li, Zhenghao Peng, Lan Feng, Qihang Zhang, Zhenghai Xue, and Bolei Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. TPAMI, 2022. 3
Quanyi Li, Zhenghao Mark Peng, Lan Feng, Zhizheng Liu, Chenda Duan, Wenjie Mo, and Bolei Zhou. Scenarionet: Open-source platform for large-scale traffic scenario simulation and modeling. In NeurlPS, 2023b.
Yiyi Liao, Jun Xie, and Andreas Geiger. Kitti-360; A novel dataset and benchmarks for urban scene understanding in 2d and 3d. TPAMI, 2022. 7,7
Chendi Lin, Heshan Liu, Qunshu Lin, Zachary Bright, Shitao Tang. Yihui He, Minghao Liu, Ling Zhu, and Cindy Le. Objaverse++: Curated 3d object dataset with quality annotations. arXiv:2504.07334 2025a. 27
Fanqi Lin, Yingdong Hu, Pingyue Sheng, Chuan Wen, Jiacheng You, and Yang Gao. Data scaling laws in imitation learning for robotic manipulation. In ICLR, 2025b. 9
Xinhao Liu, Jintong Li, Yicheng Jiang, Niramjan Sujay, Zhicheng Yang, Juexiao Zhang, John Abanes, Jing Zhang, and Chen Feng. Citywalker: Learning embodied urban navigation from web-scale videos. In CVPR, 2025. 3,
Abhiram Maddukuri, Zhenyu Jiang, Lawrence Yunliang Chen, Soroush Nasiriany, Yuqi Xie, Yu Fang, Wenqi Huang, Zu Wang, Zhenjia Xu, Nikita Chernyadev, et al. Sim-and-real co-training: A simple recipe for vision-based robotic manipulation. arXiv:2503.2455 243661 2025. 3
Mark Martinez, Chawin Sitawarin, Kevin Finch, Lennart Meincke, Alex Yablonski, and Alain Kornhauser. Beyond grand theft auto v for training, testing and enhancing deep learning self driving cars. arXiv:1712.01333 2017. 27
Andrew Melnik, Benjamin Alt, Giang Nguyen, Artur Wilkowski, Qirui Wu, Sinan Harms, Helge Rhodin, Manolis Savva, Michael Beetz, et al. Digital twin generation from visual data: A survey. arXiv:2504. 13159, 2025. 3
Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, al. Learning navigate in complex environments. arXiv:1611.03675 2016. 3
Mayank Mittal, Pascal Roth, James Tigue, Antoine Richard, Octi Zhang, Peter Du, Antonio Serrano- Munoz, Xinjie Yao, René Zurbrügg. Nikita Rudin, et al. Isaac lab: gpu-accelerated simulation framework for multi-moda. robot learning. arXiv:2511.04833 2025. 40, 41
NVIDIA. NVIDIA Isaac Sim, 2025. URL https::/ http:///r...iiiaaaacc/aaaaaaa//a wtt.a/a.c....aaaeaaaaaeaaaaaaaat
Giulia Oeschger, Páraic Carroll, and Brian Caulfield. Micromobility and public transport integration: The current state of knowledge. Transportation Research Part D: Transport and Environment, 2020. 2
13

---

Published as a conference paper at ICLR 2026
OpenAL. Introducing GPT-4.1 in the API, 2025. URL gpt-4-1/. 5,
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv:2304.071933 2023. 6, 33
Philip Polack, Florent Altché, Brigitte d' Andréa Novel, and Arnaud de La Fortelle. The kinematic bicycle model: A consistent model for planning feasible trajectories for autonomous vehicles? In IEEE intelligent vehicles symposium (IV), 2017. 40
PolyHaven. Poly haven hdris. https::t////m /ppl httpss://pollyyaveenn wwthaaaww...oohhheeeeeeeeeeeeeee
2025. 5
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021. 2, 4, 6, 25, 33
Nikhila Ravi, Valentin Gabeur, Yuan-Time Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv:2408. 044408..0744 14, 2024. 5, 33
Tianhe Ren, Shilong Liu, Ailing Zeng. Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, al. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv:2401.1 14159, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.
2017. 7, 40
Dhruv Shah, Ajay Sridhar, Nitish Dashora, Kyle Stachowicz, Kevin Black, Noriaki Hirose, and Sergey Levine. Vint: A foundation model for visual navigation. arXiv:2306. 1.Xiv:2306.14466,, 2023. 2,
Ajay Sridhar, Dhruv Shah, Catherine Glossop, and Sergey Levine. Nomad: Goal masked diffusion policies for navigation and exploration. In ICRA, 2024. 3, 9, 40, 41
Andrew Szot, Alexander Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Singh Chaplot, Oleksandr Maksymets, al. Habitat 2.0: Training home assistants to rearrange their habitat. In NeurIPS, 2021. 27
Jur Van Den Berg, Stephen J Guy, Ming Lin, and Dinesh Manocha. avoidance. In ISRR, 2011.
Reciprocal n-body collision
Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In CVPR, 2025. 7
Wayne Wu, Honglin He, Jack He, Yiran Wang, Chenda Duan, Zhizheng Liu, Quanyi Li, and Bolei Zhou. Metaurban: An embodied Al simulation platform for urban micromobility. In ICLR, 2025a. 2, 3, 27
Wayne Wu, Honglin He, Chaoyuan Zhang, Jack He, Seth Z Zhao, Ran Gong, Quanyi Li, and Bolei Zhou, Towards autonomous micromobility through scalable urban simulation. In CVPR, 2025b. 3. 6, 7, 27, 42
Ziyang Xie, Zhizheng Liu, Zhenghao Peng, Wayne Wu, and Bolei Zhou, Vid2sim: Realistic and interactive simulation from video for urban navigation. In CVPR, 2025. 3
Jie Xu, Viktor Makoviychuk, Yashraj Narang, Fabio Ramos, Wojeiech Matusik, Animesh Garg, and Miles Macklin. Accelerated policy learning with parallel differentiable simulation. arXiv:2204.07137, 2022. 40, 41
Yue Yang, Fan-You Sun, Luca Weihs, Eli VanderBilt. Alvaro Herrasti, Winson Han, Jiajun Wu, Nick Haber, Ranjay Krishna, Lingjie Liu, et al. Holodeck: Language guided generation of embodied ai environments. In CVPR, 2024.
14

---

Published as a conference paper at ICLR 2026
Huangyue Yu, Baoxiong Jia, Yixin Chen, Yandan Yang, Puhao Li, Rongpeng Su, Jiaxin Li, Qing Li, Wei Liang, Cong-Chun Zhu, et al. Metascenes: Towards automated replica creation for real-world 3d scans. In CVPR, 2025. 3
Xiang Zhang, Haojie Sun, Xiaoyang Pei, Linghui Guan, and Zihao Wang. Evolution of technology investment and development of robotaxi services. Transportation Research Part E: Logistics and Transportation Review, 2024.
Bolei Zhou, Hang Zhao, Xavier Puig. Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017. 4, 25
15

---

Published as a conference paper at ICLR 2026
APPENDIX
A Demonstration Video and Interactive Visualization
17
User-side Pipeline
17
Details of (rrae-erre-100k Asset Database
18
C.I Urban Verse- 100k Statistics
18
C.2 Details of Urban DerbanVerree100 Annotation
25
C,3 Discussion on Asset Scale and Quality Issues of Existing Databases
26
C.4 Validation of Physical Attribute Annotations
28
D Details of Urban Scenes
29
D.1 Details of Versn Scene Library Construction
29
D.2 Details of CraftBench Scene Creation
32
Implementation Details of Urban Verse-Gen Pipeline
33
F Failure Case Analysis
34
G Real-world Testing Scene Selection
35
Additional Experiments on Training Scene Horizon Length
36
I Additional Results on Customizable Real-to-Sim-to-Real Policy Learning and Transfer
36
J Additional Zero-shot Sim-to-real Transfer Results
37
System Computational Analysis
37
Experiment Setup Details
39
L.1
Definition of Urban Navigation Evaluation Metrics
39
L,2
Details of Training and Evaluation Setup
40
L,3 Details of Policy Learning
40
L,4 Robot Platform Configurations in Simulation
40
L,5
Robot Platform Configurations Real-world
M Human Evaluation Details
42
16

---

Published as a conference paper at ICLR 2026
This appendix provides additional demonstrations, visualizations, statistics, experiments, and im- plementation details that complement the main paper. App. summarizes the supplementary demonstration videos and interactive visualizations. App. B presents the complete user-side pipeline for generating and using Urban Verse simulation scenes. App. C. provides extended statistics and anno- tation details of the Urban Verse- database, together with discussion of scale and quality issues in existing datasets and a quantitative validation of our physical attribute annotations. App. D offers further construction details for the 160-scene Versn library and the artist-designed CraftBench benchmark, along with additional qualitative examples. App. E describes implementation details of the Urban Verse Verse-Gen pipeline. App. analyzes typical failure cases and reconstruction challenges. App. G visualizes the real-world testing scenes and outlines their selection strategies. App. H provides additional studies on the horizon length of training scenes. App. further reports results on policy learning and transfer in customized real-world scenes, while App. J provides expanded real-world results for each test scene. App. K presents computational and scalability analyses Urban Verse. App. L reports the evaluation metrics, training and evaluation setup, policy learning details, and robot configurations both simulation and the real world. Finally, App. describes the human evaluation protocols and interface.
LLM USAGE DECLARATION
In our system, GPT-4.1 was employed tool for category listing from input videos (with all identity information masked) and for annotating semantic, affordance, and physical attributes of virtual 3D assets. For writing, GPT-4o was used to check grammar mistakes.
A DEMONSTRATION VIDEO AND INTERACTIVE VISUALIZATION
We encourage readers to explore the videos and interactive demonstrations available on our project page here, which showcase detailed examples of the scene distillation process, generated simulation scenes, the proposed asset database, and real-world navigation policy performance.
B USER-SIDE PIPELINE
In this section, we outline the full user-side pipeline for using the Urban Verse simulation platform. As illustrated in Fig. 12, Urban Verse supports both automatic generation of new simulation scenes from raw video inputs and direct use of built-in scene repositories, enabling a broad range of embodied AI applications.
Generating Custom Simulation Scenes. Users can create their own simulation environments directly from raw inputs. Urban Verse-Gen accepts diverse sources such as YouTube city-tour videos. mobile- recorded walk-through clips, or folders of RGB frames. After providing the input, users simply the UrbanVerse-GG Verse-Gen API automatically convert the video into fully interactive, physics- ready simulation scene. The pipeline handles all steps internally- from video normalization to layout extraction and scene materialization-rreuuee generating multiple "digital cousins" from one video.
manual editing. Urban Verse-Gen supports
Using Built-in Scene Repositories. Urban also provides two ready-to-use simulation libraries. Urban UrbanVerse-11fr 160 contains 160 automatically generated real-to-sim scenes extracted from city-tour videos across the world. CraftBench provides 10 high-fideling artist-designed scenes for benchmark ing robustness, generalization, and navigation difficulty. Both repositories can be loaded directly without running Urban Verse-Gen.
Downstream Tasks Supported by VeranVerse With either mustom-generatee scenes or the built- in repositories, users can seamlessly train and evaluate embodied agents. Urban Verse supports reinforcement learning (e.g., PPO) for navigation and interaction, as well as imitation learning through expert demonstration collection via beyboard, joystick, gamepad, or VR teleoperation. The simulator also facilitates large-scale multimodal dataset collection (RGB, depth, normals, segmentation, LiDAR, and poses) for perception training. Finally, trained policies can be evaluated in closed-loop within Urban Verse scenes and deployed to real robots for zero-shot sim-to-real transfer.
17

---

Published as a conference paper at ICLR 2026
(A) Generate Your Simulation Scenes
(8) Buiit-in Verse Repositories
Input Preparation
KITTI 360
LuGcensa
Pre existing Datasets
YouTube Citytour Video
Record Phone
J
UrbanWerse-Gen Pipalino
UrbanVarra--Ce
Dratling 20 samantic layouts
Images, Camera Posse, Semantic Instance Paint Cloudi, Bioons
UrbanVerse Ro Sim
UbbunVerap-Ger: Retrieving aallull-oosinnsseet
Urbaniierm-100 100K
ChjercLacotione with Bound Assees, FOR Ground Monoriate, HDRI Sky Maps
UnbunVVeras Gere
Grestingsimmlltiinn soone bascdin
J
Digita Cousin
Load Built-in intt--n CraftBench Scenes
Simulation Scene's
10 Artist crafted Scenes
Leverage UrbanVerse Scenes for Downstream Applications
Interactive RL Training
Expert Data Collection fL Multimodal Data Collection
Closed-loop Evaluation
Sim2Real Transfer
Figure 12:
User-side pipeline of the CrbanVerse simulation platform.
Overall, Urban Verse provides a practical, flexible, and complete user-side pipeline that spans real- to-sim scene generation, large-scale simulation assets, embodied task execution, and real-world deployment.
C DETAILS OF URRBAVERSEE-10N ASSET DATABASE
C.I URBANVERSE-LOOE STATISTICS
To provide an overview of the rich semantic coverage of Urban Verse-100K, we visualize the category distribution of its curated 102,444 object assets under our introduced three-level urban ontology, along with few examples sampled from the database.
Fig. 13 shows the hierarchical breakdown of categories across all the three levels, spanning broad groups such as buildings, vehicles, street users, barriers, amenities, and urban objects, with finer- grained divisions into 659 leaf categories. Complementing this, Fig. 14 presents word cloud of all 659 leaf-level categories, where font size reflects category frequency.
Finally, we showcase few 3D urban object assets sampled from the object collection of Urban Verse- 100K in Fig. 15, Fig. 16, Fig. 17, and Fig. 18; several ground PBR materials sampled from the ground collection in Fig. 19 (road) and Fig. 20 (sidewalk); and few sky HDRI maps sampled from the sky collection in Fig. 21. These examples illustrate the quality and diversity of the assets that constitute the foundation of urban generation pipeline.
18

---

Published as a conference paper at ICLR 2026
E30000 ougo
Shetggun
bovil
abriig SeX
barrel bockat
haamne
personal Item
armotia
misc adjects
kichermaree
chass
solaa
desik
stuffed toy
biniem
doll
tood
chair
toys
table
decoramons
ufban object
cabinet or shaft
neture
vegetation
user
pUNN flwer
buikding
potted
vehicle
residential buikding
person
bind
commerciaa bullding
amenity
Gerceftt veleete
PURKI VERICA
peddosriin
privite voricce
Bubles building
faciities aweenity
combiA fectical veracte
Counsio i amenny
obstacle berriet
dirplane
TONCEINNE Pulding
coupe Car
won
Buind stone
store Blakk
sculpture
Figure 13: Hierarchical category distribution of Urban Verse-100K within our curated three-level urban.
bookcase fountain radio street
hondare store
nattele cover
plate
research
institute
record
player doverneent
berracal
tartie
lantern Storage
box
storage
iapple
propeter sitter
bazeball
batt
baseba
computeri monit
Jersey Barrier
do
flower
pickup
mobile
stool
hostel
sandugh
airplane
keyboard
playgr
receiver
light
swivel
tottats slayd
filing
coupe
bedestrian
I
arisot apa par tment
Face Truck
phone
curs
fatrbaack car
water well" paee
t
Buatar,
theth de
table
chair
all-
cabinet
cab
wsede
Cal Pontect
toolbox
OW skateboard
factory buildings
teleyssion
teerrain vehicle
police car
radiator
shrub
aideaas
station
encaie convert rible
out
de
Urar
aau
reels
building
axe
arii
armchair
penant trash bin
plano
pliers
aammy
shoes
S
sedan
scoonee
suv
gane
Jewelry
car
kioskuire
drue
van
traffic
san
pates
venting
Cread
cabinet
grave
yard
grave
bowl
police
cocorie
wrench
wwa e
cbetrack
cabinet
Incemnt
Caafe
bottl
S
officer
stone
cotorcycle
patzo
trafttic Light
Bes
block
restaurane
of-helicopter
e
the Welln
stone
Cabinets
to centrol
beer
Line player
Loy
S stuffed
red
tod Of
1aptop
warer rerouuuee
Sraftic
Chessboard
sconce
tree bonsa
Aine glass
cup
teapot
pagoda
plant
ladder
camapy
couch
sien
lotion
amistar
out
bucket
of
shelf
pot
corded phone
cas Faltte
contitionr
barn
book
stone e
s
d'd
guard
attractions
hydrant
f
affert thine tound
pitchfork
cuttion Buard
Fracer
r
snacks dos
backpack chotsser
pizza
bicycle
Lightwan hat
Sootes
electric fan first sid
candle
kit
donuts
pumpkin
handbage
train
inftr
intrrattoon seant
baskehhe
be
fence
bea
crustacean
screwesive
temple
coffee
sound systens
pienie tableG
ruffin
air ballon
strver
tearthy gard
vending
nachine christras,
desk
e
team drawers
Indectien
ship
other ee electro
solaa contaiser
sport ts car
pair of
the locher gy
rock
glasses
tent
infrastructto
wall
house
trafaforeg oper
sculptur
obelisk
kettle
berbacue
sk
humanoid
clock
gritl"
evar
sgarragg
aillow
.obot
mushr
storage
hanburger ger
cake
postiice
truck
iel
chaise
Lank
texterr oven
he Lemeet
r
commnty
ba
Bottonag
thar
ted
potted
Lamp
sterea
combat tactical
bollard d
hot
dig
pebdi
Vehicle
ehurth
Bddareed oa
Fire
office
extinguisher
building
sheelchat user
nights
telephone boota clock
perfune
tower
tripod
Figure 14; Word cloud of Urban Verse-100K category distribution over the 659 leaf-level categories.
19

---

Published as a conference paper at ICLR 2026
Jersey Barrier
Bollard
Traffic Barricade
Traffic Cones
Figure few examples Verse- objects Urhan Barriers.
20

---

Published as a conference paper at ICLR 2026
Dumpster
Garbage Bag
50
Street Cabinet
60
Traffic Sign
Figure 16: A few examples of Urban Verse-Vers-100K objects Urban Street Objects.
21

---

Published as a conference paper at ICLR 2026
Bulldozer
Police Car
Sports Car
Motorcycle
Figure 17: A few examples of Urban 10rse-100K objects Urban Vehicles.
22

---

Published as a conference paper at ICLR 2026
Apartment Building
Skyscraper
Historical Building
Utility Pole
Figure 18: A few examples of UrbanVerse-1000 100K objects Urban Structures.
23

---

Published as a conference paper at ICLR 2026
Figure A few examples Urban SeraenVerre-O01 ground PBR materials Road.
Figure 20: A few examples of Urban ffere-110KK ground PBR materials Sidewalk.
Figure 21: few examples of Urban V0rse-100K HDRI sky maps.
24

---

Published as a conference paper at ICLR 2026
GLB Asset Gallery
Annotation Task: Check and tick bad 3D assets based on the instruction
Figure Efficient Three.js-based annotation interface asset quality filtering.
C.2 DETAILS URBANVERSE- BOOK ANNOTATION
Our goal is curate high-quality 3D urban asset database with accurate and semantically rich annotations from the 800K assets Objaverse (Deitke et al., 2023c), thereby addressing the quality and scale issues discussed in Sec. C.3. Concretely, described in Sec. 3.1, design efficient semi-automatic annotation pipeline consisting of three steps: (1) Non-simulatable asset filtering: in-house human annotators manually remove low-quality assets that could corrupt simulation; (2) Urban ontology and categorization: we construct an ontology of common urban semantics and categorize assets accordingly; and (3) Attribute annotation: we employ GPT-4. (OpenAI, 2025) to automatically annotate semantic, affordance, and physical attributes of the curated assets. We now describe each step in detail.
(1) Non-simulatable asset filtering. We first eliminate assets that are likely fail in simulation by filtering out eight common corruption types. To support this process, we built lightweight Three.js- based GLB gallery interface that allows annotators to quickly inspect assets and perform binary quality tagging. Ten in-house annotators worked for three weeks, resulting in curated set of 158k simulatable 3D objects. shown Fig. 22, our interface enables rapid quality inspection and efficient removal of unusable assets. Prior to annotation, annotators underwent training to ensure consistent identification of low-quality assets.
(2) Urban ontology and categorization. from the OpenStreetMap (OSM) tag structure (Bennett, 2010). OSM is collaborative, open-source project that provides freely accessible maps of the world, created by volunteers using GPS devices, aerial imagery, street-level photos, and local knowledge. Its tag structure includes common urban amenities, objects, and landmarks. Building on this structure, we expand the leaf level with categories drawn from ADE20K (Zhou al., 2017), Cityscapes (Cordts et al., 2016), nuScenes (Caesar et al., 2020), LVIS (Gupta et al., 2019), and OpenImagesV? (Kuznetsova et al., 2020). After deduplication and refinement, the resulting ontology contains top-level, 59 mid-level, and 659 leaf categories. Assets are automatically classified into leaf classes using CLIP (Radford et al., 2021) applied to thumbnails, followed by human verification to prune non-urban objects (e.g., weapons, spaceships) and correct misclassifications This process yields 102,444 assets organized under our ontology.
We next construct a three-level urban ontology seeded
(3) Semantic, affordance, and physical attribute annotation. Finally, guided by the question "How would a robot interact with this object?", we annotate each asset with 36 attributes spanning semantic, affordance, and physical properties in metric units, enabling physically plausible interactions and richer semantics. For each asset, we provide G (OpenAl, 2025) with its thumbnail and four yaw snapshots at 0" /900 /180 /270" prompting it to produce attribute values. This step completed
25

---

Published as a conference paper at ICLR 2026
Object Attributes Annotation Prompt
sooumingt
stmee, sideweli, reighbohhodd gronge an park wir). failmwing
"description very
description catagaryy olject propar
Don't the
the
1-2
descript leep option and visual.
"description view <b>:
Hhi cotegrry 17 ooiject from pngrigghhoompee feam different from other viewe)).
"categary": i Detegory
"beight" upposimatte hright of this sateggry RL cbjjcct maters (m). Dapurt the height the otject's cnientation shown the images. Fora scanding human male this
could be "1,25;
"max, dimension" uporosimate musimum dimansion of this Catoggrry 18 sbjject metera (m). This is the longest dimenaion this L_ 11___ object, regerdlass of orientation. This shoud alwaya be g cater equal the height.
"materiats" a Python list of the metwrtale thet this cenegiry D | |njject arpwors to be mode of, racing imo socoumt the sisitie natwrlor and alen inneiie (raughly in order
of most coed mathrral to lesst used; includee "air the object inter on dosartt sears completely seclef).
"materiala compasitonn: as Pyttoo ist with the sopreent vatumm moture of the materill wis sbote (inslos the list sum to Tk.
"mass" : approcmaae musor of this [_ cataggrygy St pojeett i logaam (eg) considering typicall derre trs for the materials, For human being this could be '/2";
"recepticce": a boolean indicuting whather or not thi cutaprry [____ ooject is recoptacls tegg. bowl, cup, sans, a bes, a bag, enc). Ratum trae or false with no explenations,
"Tront/lew" iniegerr inder of the dew that reprosents the from of thix categgrryy I3 object. This in typically the siewo from which you would approsch the object to innenett with
"quality" a number, (-10, mndcarting the quality of thes I peingoryy 1S_.. objject vary lao quality (emmtturrih, conriusing missing bectune, a 3D scann with manyy halee, wti)) 1o very high quality (prctessiioall detuiled stc).
"movable": bossean indicating whether this categery_ 18 object movibls fisddrattt unvronment, Reture tras fulse explanations,
"mequreed approsdmate frice
miee category ccbject
your
onject' shaps.
aepacent matwrrel indirating agents g.
cleady immoattl lng, abiiding or strictuu)
pallgeryy [i_.. diject (ng,
Battur
soplamations
agents plosicall
Lateggoyy [R__ dbjact dl. docrwwyy,
Return
"affordancee": Python lazt of High-deool fumetional affordences this
in objject prosides "attatrer "opansblee "droeshie" "ceessativ" "toggleable"
"orivbble: "robeaable" "punbatior "prlabbee "utteble"), Retur the most nesuar whlnddenzes. Fartidde affurdencne auch "enterable" "wellottwr and "mmmetro: they annolated slaewihere Datum
lnt, without replamation addition bett.
a bostasn indicating whather Longgoryyy D. object can phyrically xupport other sojecte placad on (eg. tablex, platiorms, roofs). Beturn trum
futso th suparations,
"interactive parts a Pythor Ust of distine: functional parts or com ponepotontts of this Cotagory, B legject that can be interccoed with (e.g. handa"" "drawrr, "whoot, "daor,
"ouitann Only include parts vioble or molled from genmatry Baturn only the lie, oe or other wonds,
"trawrnability aa string labal descriting how an agent might tnvers thi catagryy II [1___otee Jnttect. Charss one of: 'paece thoough "push throuht, or "obxtaccle"
"troweraable by" a Pythor bet apjntt typer that can toaverse or pass thruuh this Leceagrry.. 15 object Ing, pensorf "whesled mubuot "dronw", 3. Only include spertt for wiech traversal la propicallyy fersitaa the diject is not traversable ayy agent, return un emply last (M.. [B.
"colors':a u Pytton ist of the vis bas colora of thi coteggry R. otject (og. ('winaa "ou. "blue", ". Focus on domtinant and distinct catora viaible from thu extarios
"colors composition a Python lie floats recrsenntnng the app cadmane volume compottion of the colors listed abown. Ersure the fot sume to 1.0 and comrepoonds to the
onder of "caiens"
"surface. hardnwss" a string desctibing the
haedhwood of the surfscw of this [ pategrey lojject Chaose anw "xatt: "swmivatt"" or "tart" "surface roughness" a fload the sange indissing the micro-bectre mughmsee the surfece this Lanngggoyyy 13 objjett D mesrs perfucily smooth (kaj. poladred
ghanel, masns sctummlly roaj (w g. caarse stone).
"surface finiah' str descriong a cutegory Bt_.. ooject's aurface tactilelviuull quality Chocse one of "rough" "matto"" "amcotti" "Meaayy "sleect or ipiopy:
"reflectivtty
range (0. 1) that controus
light reflected by
the] conegory (ojject
reflection,
refrection': floet reproseniing
optical
HOO Latespryy It_.. eljject. "ypiroi nange (er) srglee,
hys. Higher
"younga,
spprosimate forstesl),
catggery [3__ staject Magepascala
typea (euf..
"friction conffictent" positive flost reproseeting the estimuned frlciion confficlen, based this thin ontegory in Jotject's material and su turence finish polisted
01, pleatic m..3 wood nitterr n.S.
representing the expecied alaeriiity the pategory 1S sejject upen impastt H gher values indicare more beurne gl rubber bull 0.5), white lowse valuse indicatte maremnel no bounce teg, stoom--d.... Ii, "recommended dedddddddeeeee clearance's approsimate butter distance metara] that should suround thi [_ www.grrr.. _i__ stject when placed acarmu. This hwips averd collason interference with agents, "aseet composition type": i string dexpribing the structural naturs aD axset. Chonse one ot: "lingle" ((s stancation atomic ohject,, "group" (1 small collection relsted otjecte), or "scanw" hull compoxte scane with with leymuty.
"bouncinese" i ficse in the
Figure 23: Prompt for object attribute annotation.
at total API cost of $1,334. The exact annotation prompt is shown in Fig. 23, and the full set of annotated attributes is presented in Fig. 24.
C.3 DISCUSSION ON ASSET SCALE AND QUALITY ISSUES OF EXISTING DATABASES
Asset Quality Issues. High-quality 3D assets are essential for constructing realistic and physically accurate simulation environments. The recent proliferation of large-scale 3D repositories has made it possible to efficiently assemble datasets for diverse scene construction. Objaverse (Deitke et al., 2023c), for instance, provides over 800K 3D objects sourced from the Internet, and its extension Obj (Deitke et al., 2023b) scales this collection to 10.2M unique objects from sources such as GitHub. However, because these assets are primarily scraped from the web, their quality highly inconsistent and largely uncontrolled.
In the early stage of our project, we systematically examined Objaverse's 800K assets and identified nine recurring forms of corruption, which affect more than half of the collection. As illustrated in Fig. 25, these issues include:
1. Bad mesh: incompletely reconstructed assets (often from 3D Gaussian Splatting (Kerbl et al., 2023)), resulting in noisy, broken geometry;
26

---

Published as a conference paper at ICLR 2026
Thumbnail
0o
90"
180
270
Category Vehicle (L11) Private Vehicle (L2) Electricc Car (L3)
Front View
180*
Wordnet pickup.n.01
Mass 3000 kg
Friction Coefficient
0.9
Required Force
23.5 kN
LxWxH x.1x2.4x 1.9
Young's Modulus 210 GPa
Reflectivity
0.18
Index of Refraction
1.52
Safety Clearance
1.5
Bounciness 0.05
[0. 1]
Finish Metal
Surface Roughness
0.18 [0.
Surface Hardness Hard
Receptacle False
Support Surface False
Enterable
True
Ground Mounted
False
Movable
True
Affordances
Drivable
Openable
Closable
Toggleable
Pressable
Traversability
Obstacle
Interactive Parts
Door
Wheel|
Window
Trunk | Charging Port MMirrrr
Asset Quality 9 [0, 10]
Colors
Silver Gray (85%)
Black (10%)
Red (3%) Yellow (2%)
Asset Composition Single
Materials
Steel (70%) Glass (15%) Rubber (12%) Plastic (3%)
License Plate Design None
Description
metal, silver gray, angular electric pickup with large tinted windows and armored design
Manufacturer
Tesla
Model
Cybertruck
Emblem
None
Charge Port Location Left rear quarter panel
Figure 24; Example of full annotated attributes for each object asset in Urhan Verse-100F
2. No texture: pure meshes without surface textures, lacking visual realism;
3. Paper-like; thin, hand-authored background props with negligible mesh depth, unsuitable for physical simulation;
4. With base: assets embedded in oversized base meshes, producing inaccurate occupancy and collisions;
5. Terrain maps: large terrain-like assets that cannot be meaningfully used in urban embodied
AI simulation:
6. Inconsistent names: category names directly inherited from web tags, often written in multiple languages, idiosyncratic codes, or designer-speciff terms;
7. CAD-like: CAD models lacking textures and physical realism, unsuitable for direct use in interactive simulation;
8. Non-single objects: assets that are entire scenes or contain multiple unrelated objects rather than single entity;
9. Non-uniform scales: assets not in metric scale, which makes them unusable for physically grounded simulation.
The non-uniform scale issue is particularly problematic: as shown in Fig. 26, direct import of such assets lead to absurd scenarios (e.g., fire hydrant larger than building). Similar issues have also been noted in Objaverse++ (Lin et al., 2025a). To address these problems at their root, we curated a new repository, Urban Verse- 100K, by employing human annotators carefully filter low-quality assets. This manual quality control ensures that only simulation-rest -eady objects are included. Our curated database will be open-sourced facilitate reliable and reproducible embodied Al research.
Asset Scale Issues. Due to these quality issues, existing simulators such as Meta (Wu et al., 2025a), UrbanSim (Wu et al., 2025b), indoor simulators (Deitke et al., 2022; Gan et al., 2020; Deitke al., 2020; Szot al., 2021; Kolve al., 2017; Li al., 2023a), and driving simulators (Dosovitskiy et al., 2017; Martinez et al., 2017; Kothari et al., 2021; Cacsar et al., 2021) 2021)-typicall typically rely relatively small, manually curated asset repositories. Human annotators must painstakingly adjust object scales and orientations one by one, which is not scalable. As summarized in Tab. 5, current urban simulators rarely exceed 15K curated assets, limiting the diversity and richness achievable in simulation environments.
In contrast, our work introduces a hybrid annotation pipeline that combines efficient human filtering with large language model (baee) automatic annotation. Human annotators perform rapid
27

---

Published as a conference paper at ICLR 2026
Bad With
Paper-like
'домаыкaa Aren
Non-single Object
Mon-Uniform Object Scales
Figure 25: Typical quality issues in existing 3D asset databases.
Figure Non-metric scale problems existing asset databases.
binary tagging to eliminate poor-quality assets, while LLMs contribute common-sense knowledge to automatically annotate metric scales, canonical front views, semantic categories, and physical attributes. This hybrid process enables us to construct significantly larger, physically grounded urban asset database at scale, bridging the gap between raw Internet collections and high-quality simulation-readt repositories. In the next section, we detail our hybrid annotation pipeline.
C.4 VALIDATION PHYSICAL ATTRIBUTE ANNOTATIONS
Validating the physical attributes annotation in Urban Verse- 100K is essential, yet direct human annotation of true object dimensions or mass generally infeasible without expert knowledge. Prior simulators (e.g., MetaUrban, UrbanSim) typically rely anchor-based visual calibration, where objects are manually resized based on appearance and relative proportions. Following this practice, main paper presents large-scale qualitative validation by placing hundreds of assets side-by-side to demonstrate consistent, physically plausible relative scales in Fig.
To complement these qualitative checks, in this section, conduct quantitative evaluation 17 categories comprising 1,335 objects for which reliable specifications or commonly agreed real- world dimensions are publicly available (e.g., Tesla Cybertruck, vending machines, traffic cones, laptops).
28

---

Published as a conference paper at ICLR 2026
Seemario Simulator
Scene
Physies
Parallel Training
#
# of Categories
# of Assets
Asset Physics
of Sky boxes
# of Ground Materials
Creation
Engine
Scenes
AI2-THOR
Manual
Lnity
120
3,578
Indoor
POR
PG
Unity
+00
108
1,633
/
X
Habitat 3.0
Manual
Bullet
211
18,656
X
x
Holodeck
Manual
Unity
400
108
1,633
X
X
X
Behavior
Manual
PhysX
50
1,900
9,000
/
X
X
Unity
X
Driving
CARLA
Manual
Unreat5
935
X
MetaUrban
PG
Panda3D
400
39
10,000
X
5
Urban
Urban-Sim
PG
IsaaeSim
400
39
15,000
X
8
Vebsn
Real2sim
IsaacSim
+00
659
102,444
/
306
288
Table 5: A systematic comparison of urban Embodied AI simulators.
Category
# Objects
MAPE (%)
MAPE L ())
MАPE W (%)
MAPE M (%)
Average
5,88 3.85
6,14 3.86
7.00 4.11
19.58 + 12.11
Lamborghini Hurucan STO
12
0.49 = 0.30
0.66 + 0.40
0.51 0.30
2.02 1.30
McLaren 600LT Spider
0.59 + 0.40
0.78 + 0.50
0.62
0.40
2.47 1.50
Tesla Cybertruck
1.97 + 1.40
2.50 1.70
2.02 1.40
9.05 6.50
Land Rover Defender
2,99 2.00
3.72
1.40
12.56 7.50
Electric Scnoter
68
5.97 4.00
9.38 7.00
12.32 6.00
29.96 19.00
Bicycle
7.03 5.00
5.05 3.00
7.75 5.00
17.78 13.00
Vending Machine
7,94 5.00
5.00
7.18 7.18=500
19.60 + 10.00
Street Cabinet
43
20.00 +
23.00 + 13.00
22.00 12.00
14.86 + 10.00
Parking Meter
16
7.94
10.71 7.00
6.56 5.00
22.94 14.00
Fire Hydrant
160
1.47 1.47+1.00 1.00
1.53 1.00
1.53 1.00
7.14 4.00
Traffic Cone
12.57 +.00
20.40 + 13.00
20.40 + 13.00
51.07
25.00
Jersey Barrier
95
5.06 3.000
7.07 5.00
25.83 + 13.00
75.00 50.00
Egg
188
1.09 0.70
1.03 0.70
1.03 0.70
12.45 8.00
Cigarette
28
1.00 - 0.60
1.00 0.60
1,00 = 0.60
15.10 = 9.00
Laptop
171
20.46 + 13.00
5.06 + 3.00
4.84 3.00
25.02 + 17.00
Foothall
99
1.50 + 1.00
1,50 + 1.00
1.50 1.00
6.83 4.00
Basketball
45
1.92 + 1.00
1.92 + 1.00
1.92 1.00
8.97 6.00
Table 6: Evaluation of annotated physical attributes. We report the MAPE and standard deviation against ground-trus tound--ruthh attribute values 17 object categories for Height (H), Length (L), Width (W). and Mass (M).
For each category, we compute the Mean Absolute Percentage Error (MAPE),
MAPE 100% N
Annotation, GT,
GT,
over height, length, width, and mass.
As summarized in Tab. 6, geometric attributes are highly accurate typically within 1-8% MAPE and often 1-3% for rigid objects such as cars, hydrants, and balls. Mass values exhibit larger variation due to material uncertainty but remain within a reasonable error range (mean 19.58%). Overall, these results demonstrate that our automatic annotation pipeline yields reliable physical attributes scale, enabling high-fidelity. physics-aware simulation without manual labeling.
D DETAILS OF URBANVERSE SCENES
D.1 DETAILS OF URBANVERSE SCENE LIBRARY CONSTRUCTION
City-tour video collection. To construct scene library that captures diverse and realistic layouts reflective real-world urban settings, we collect 32 city-tour videos from YouTube released under Creative Commons licenses. These videos span continents, 24 countries, and 27 cities, providing geographically and culturally comprehensive coverage for our city-tour video inputs. We present a few examples of the collected city-tour videos in Fig. 27. The distribution is follows:
29

---

Published as a conference paper at ICLR 2026
Figure 27: A thumbnail montage showcasing a subset of the city-tour videas collected worldwide for grounding scene generation.
Continents:
Africa, Asia, Europe, Middle East, North America, Oceania, South America
Egypt, Kenya, Morocco, Nigeria, South Africa, China, India, Japan, Kazakhstan, Singapore, South Korea, Vietnam, France, Iceland, Italy, Netherlands, Spain, Sweden, Saudi Arabia, United Arab Emirates, Canada, Mexico, United States, Australia, New Zealand, Argentina, Brazil, Colombia
Countries:
Cities: Cairo, Nairobi, Tangier. Rabat, Lagos, Cape Town, Beijing, Shijiazhuang, New Delhi, Tokyo, Kyoto, Almaty, Singapore, Scoul, Ho Chi Minh City, Paris, Reykjavik, Naples, Amsterdam, Barcelona, Stockholm, Riyadh, Dubai, Toronto, Puerto Vallarta, Los Angeles, Sydney, Auckland, Buenos Aires, São Paulo, Rio de Janeiro, Bogotá
UeranVerse simulation scene library. By applying our Urban Verse-Gen pipeline to the collected city-tour videos, we generate library of 160 urban simulation scenes for embodied AI training. Here, we present a few examples from the generated scene library in Fig. 28.
Urban Verse scene diversity. The diversity of Urban Verse scenes emerges from the wide range of retrieved object assets, ground materials, and sky maps integrated by our automatic real-to- sim Urban Verse- Gen pipeline. From the collected city-tour videos, Urban Verse-Grn generates multiple digital cousin scenes that share the same underlying layout but differ in visual and physical composition, while preserving the key geometry and affordances. Fig. 29 showcases variations in lighting conditions, distant backgrounds, and ground appearances within the simulator. By retrieving multiple digital cousin assets from UrbanVerse- 100K for each object instance, Urban Verse-Gen can produce numerous diverse urban scenes from a single city-tour video, varying in illumination, ground materials, object geometry, and texture. We present the top-5 digital cousin scenes generated from city-tour videos filmed in Beijing, China, and Tangier, Morocco, in Fig. 30 and Fig. 31, respectively.
30

---

Published as a conference paper at ICLR 2026
Figure 28: A thumbnail montage showcasing a subset of the generated Urban Verse simulation scenes.
31

---

Published as a conference paper at ICLR 2026
Lighting and Distant Background Diversity
Ground Diversity
Figure 29: Examples Illustrating the diversity of lighting conditions, distant backgrounds, and ground materials in Urban Verse scenes.
G
Input RGB Video: Beijing, China
Genersted Digital Cousin Scene 01
Generated Digital Cousin Scene 02
Generated Digital Cousin Scene 03
Generated Digital Cousin Scene 04
Generated Digital Cousin Scene 05
Figure 30: Qualitative results of the top-5 digital cousin scenes generated by Urban Verse-Gerr from walking city-tour video of Beijing, China.
Input RGB Video: Tangier, Morocco
Generated Digital Cousin Scene 01
Generated Digital Cousin Scene 02
Cenersted Digital Cousin Scene 03
Cenersted Digital Cousin Scene 04
Gene rater Digital Cousin Scene 05
Figure 31: Qualitative results of the top-5 digital cousin seenes generated by Urban Verse-Gen from driving city-tour video of Tangier, Morocco.
D.2 DETAILS OF CRAFTBENCH SCENE CREATION
To complement automatically generated scenes, we commission professional 3D artists to design a suite of fideliiyyiity environments that serve the CraftBench benchmark for closed-loon evaluation. These scenes are carefully crafted to balance realism and diversity, capturing not only the everyday orderliness of urban streets but also the messy, safety-critical edge cases that real-world agents
32

---

Published as a conference paper at ICLR 2026
Residential Sidewalks Carbage Bags
Alley:
Residential Single
Scone Sccoter Sidewalk
Scene 05 Commercial Street: Dual Sidewelks with Bike Lane
Scene 06 CBD Inteersectoonn with Construction Sites
Scone 07 Commercial Street: Asymmetric Dual Sidewalks
Scone 08 CBD Cross Intersection with Diverse Obstacles
Scana 09 Tourist Street: Sidewalk Blocked by Cara
Scane 10 Urban Park: Dogs and Basketball Courts
Figure 32: Examples of all CraftBench test seenes.
frequently encounter. As shown in Fig. 32, the benchmark includes diverse urban layouts such as residential streets with garbage bags on sidewalks, narrow alleys lined with food courts, bar streets with fallen scooters obstructing walkways, commercial districts with bike lanes, and CBD intersections under active construction. The scenes also depict edge cases such as asymmetric sidewalks, sidewalks blocked by illegally parked cars, and public parks populated with both dogs and basketball courts. By combining realistic details with safety-critical anomalies, these artist-created scenes provide challenging yet authentic environments for evaluating embodied urban navigation.
E IMPLEMENTATION DETAILS OF CRBANVERSE-GEN PIPELINE
In this section, we provide implementation details of the proposed Urban Verse-Gen pipeline. For coucture-from-mont we adopt MAS13R (Leroy et al., 2024) with ViT-Large backbone to estimate camera intrinsics, metric depth, and camera poses. For 2D object semantic parsing, we use the state- of-the-art open-vocabulary detector YOLO- MOLOO-Wrldy2 XL (Cheng et al., 2024) to obtain on-ground object bounding boxes. Subsequently, SAM 2.1 Large (Ravi et al., 2024) refines these detections by generating pixel-level semantic instance masks conditioned on the YOLO-Worldy2
Worldy2 predictions. To match objects with database assets, employ CLIP ViT-L/14 (Radford et al., 2021) for textual semantic similarity, and DINOv2 ViT-B/32 (Oquab et al., 2023) to measure visual similarity between input object masks and the thumbnails of 3D assets in our database.
33

---

Published as a conference paper at ICLR 2026
lrss
cliner
Terrn
Figure 33: Typical challenging input video conditions.
Input City-tour Vidao
Generated Digital Cousin Scene
Figure 34: Example failure cases from Urban Verse-Gen. The sidewalk billboard that originally placed on the sidewalk in the input video shifted toward the road due inaccurate depth and pose drift
F FAILURE CASE ANALYSIS
Challenging Input Conditions. Urban Verse-Gen is generally robust across diverse city-tour videos, but certain input conditions can still degrade reconstruction quality. As the examples shown in Fig. 33, extremely low light, distant shot, heavy clutter, or complex terrain may affect depth estimation and geometric consistency. prevent clearly unrecoverable clips from entering the pipeline, we screen every 10th frame with 4.1 during sange-scale generation and automatically filter out problematic segments.
Depth and Pose Drift. Fast camera motion or unstable handheld recordings may introduce depth pose drift, occasionally causing misplaced objects (e.g., sidewalk billboard shifted toward the roadway). Multi-view aggregation substantially reduces such errors, though they remain the most common failure mode. As illustrated in Fig. 34, sidewalk billboard shifted toward the road due to inaccurate depth and pose drift. on the sideIncorporating spatial constraints between objects and their plausible placement areas a promising direction for further improving stability.
Imperfect Asset Retrieval. Visually complex rare objects may not always retrieve perfect appearance match from Urban V0ers--10KK. Our three-stage matching geometry filtering, and appearance selection-ensuree that the retrieved asset maintains correct category. affordance, and collision geometry. As result, appearance mismatches do not harm physical interaction and often serve as benign domain randomization.
matching,
Mis-segmentation Under Heavy Occlusion. Severe occlusion by pedestrians or vehicles can lead to incomplete masks from open- -pecbvocayylary detectors. However, because our layout extraction fuses multi-view evidence rather than relying on single-frame segmentation, reconstructed scene geometry typically remains stable.
Impact on Downstream Learning. Despite these failure modes, we observe minimal negative impact on downstream policy learning. Multi-view fusion and strict geometry filtering provide stable scene layouts when some frames noisy, and both simulator evaluations and real-world sim-to-real results confirm that policies trained in Urban Verse generalize reliably. This indicates that Urban Verse provides sufficient scene fidelity for large-scale robot training.
34

---

Published as a conference paper at ICLR 2026
Scene 01 Go straight
Scene 02 Turn left
Scene 03 Turn right
Scene 04 Go straight
Scene 05 Turn left
Scene 06 Tum right
Scene 07 Walkway w. obstacles
Scene 08 Sidewalk w. obstacles
Scene 09 Obstacle after turn
Scene 10 Obstacle after turn
Scene 11 Sidewalk blocked by cars
Scene 12 Plaza w. lampposts
Scene Narrow gate
Scene
Scene Street crossing
Scene 16
Narrow access ramp
Street crossing
Figure 35: Real-world testing scenarios. We evaluate zero-shot sim-to- real policy transfer across 16 diverse urban scenes and two embodiments (Coco wheeled delivery robot and Unitree Go2 quadruped). spanning challenges such as straight paths (Scenes 01, 04) and turns (Scenes 03, 05, 06) open ground or sidewalks, walkway and sidewalk obstacles (Scenes 07, 08), obstacles appearing after turns (Scenes 09, 10), sidewalk blockage by cars (Scene 11), structural elements including lampposts and narrow access points (Scenes 12-14), and street crossings (Scenes 15, 16). Key challenges for each scene highlighted in Red at the upper-right corner.
G REAL-WORLD TESTING SCENE SELECTION
To rigorously evaluate zero-shot sim-to-real transfer, we carefully select 16 real-world testing scenes that expose diverse challenges for tested robot embodiments: the Coco wheeled delivery robot and the Unitree Go2 quadruped, as shown in Fig. 35. These scenarios assess each robot's ability to follow trajectories on open ground and sidewalks, negotiate turns both with and without obstacles, and cope with safety-critions events such as obstacles appearing after sharp turns, sidewalks blocked by parked cars, and walkways cluttered with objects. Structural challenges such as narrow gates, access ramps, lampposts, and urban street crossings are also included to test mobility and accessibility. Concretely, the set spans straight paths (Scenes 01, 04), turns (Scenes 02, 03, 05, 06), walkways and sidewalks with obstacles (Scenes 07, 08), obstacles appearing after turns (Scenes 09, 10), sidewalk blockage by cars (Scene 11), lampposts and plazas (Scene 12), narrow gates and ramps (Scenes 13, 14), and street crossings (Scenes 15, 16). By covering both trajectory- fowlooiig and obstacle- -nviigattnnniiionnnnn tasks across two distinct embodiments, these scenes provide a comprehensive benchmark for evaluating policy robustness and generalization in realistic urban environments.
35

---

Published as a conference paper at ICLR 2026
H ADDITIONAL EXPERIMENTS ON TRAINING SCENE HORIZON LENGTH
In this section, further study how the spatial horizon of training scenes affects policy learning. Specifically, we compare Half-Length UrbanVerse scenes, where each scene is truncated to roughly half of its original spatial extent m), with Full-Length Urban Verse scenes that retain the complete layout (~2200 m). Both settings use the full set of 160 Urban Verse training scenes (320 truncated scenes in the half-length case). Policies are trained on wheeled robot and evaluated on the ten CraftBench test scenes.
As shown in Tab. 7, full-length scenes lead to clear improvements across all metrics, rais- ing Success Rate by +9.6, reducing Collision Time by -5.0, and increasing Route Comple- tion by +11.6. We attribute this to two fac- tors. First, longer scenes expose the agent to richer spatial structures, denser object config- urations, and longer-range dependencies that better match the complexity of real-world nav- igation and the artist-designed CraftBench lay- outs. Second, longer episodes provide PPO with more diverse transitions and more chal- lenging decision points per rollout, improving both collision avoidance and global route planning.
Training Scene Length
SR 1
CT +
RC f
Half Length (~ 100m)
32.3
40.5
50.8
Full Length (~~00mm
41.9
35.5
62.4
Table 7: Effect of training scene horizon length. Policies are trained on all 160 Urban Verse scenes; the half-length setting truncates each scene to ~00 m, while the full- length setting uses the complete ~200 m layouts. Evalua- tion is conducted on the ten CraftBench test scenes using wheeled robot.
These results highlight that increasing scene horizon- and thereby spatial complexity is effective way to improve generalization in Urban Verse-trained policies.
I ADDITIONAL RESULTS ON CUSTOMIZABLE POLICY LEARNING AND TRANSFER
In certain practical deployments, the robot's target operational environment is known beforehand. We therefore investigate whether sim-to-real policy transfer can be further improved through customized policy fine-tuning within digital cousin simulation scenes of the target environments, generated by our Urban
real-to-sim Aaplline-frrming a fully automated, end-to-ent real-to-sim-to- sim-to-real policy learning framework. To this end, we focus on the Scene 12 real-world test scenario (see the site image in Fig. 35) as the target operational environment. This scene is particularly challenging, as the agent must navigate through a dense arrangement of lampposts to reach the goal while avoiding randomly parked e-scooters, bollards, and shrubs along the sides of the path. In this environment, our original PPO-Urban Verse policy on the Coco wheeled robot failed entirely, achieving success rate of 0%, as reported in Tab. 9. To construct the corresponding digital cousin scenes for fine-tuning, recorded a short handheld RGB video of the target environment and used Urban Verse- Greee-Gnnnn to generate eight customized simulation scenes tailored to it. We then fine-tuned the PPO-Urban Verse model (pre-trained on 160 Urban Verse scenes) on these customized scenes (denoted as Urban Verse + 'arget) and compared it against two baselines: the original pretrained policy without fine-tuning (denoted as UrbanVerse) and policy trained solely on the target scenes (denoted as Target Only).
As shown in Tab. 8, only the policy pretrained on the 160-scene Urban Verse library and subse- quently fine-tuned on the 8 customized digital cousin scenes succeeded in completing the task, effectively navigating the challenging sequence of lampposts and e-scooters- ponptters-ddmoostratiiin rapid real-to-sim-to-redl transfer. Interestingly, al- though both the pretrained and target-only poli- cies failed to reach the goal, PPO- Verse still outperformed the latter in route completion (RC) and distance-to to-goal (DTG). This suggests that the broader and more diverse training distribution provided by Urban Verse acts as an implicit regularizer, preventing overfitting a small number of customized scenes and mitigating the sim-to-real gap.
Training Scenes
SR t
CT
RC t
DTG 1
Urban Target Only
0.0
100.0
72.9
5.3
0.0
100.0
23.6
20.0
Urban Verse + Target 80.0 20.0
96.5
0.3
Table 8: Results of real-to-simm-o-real customized policy learning and transfer. Performance compari- son across different training setups in the Scene 12 real-world test scenario. All results are evaluated on
the F wheeled robot.
36

---

Published as a conference paper at ICLR 2026
Scene
Scene
03
Scene
Route Length
22.1 m
25.8 m
22.9
23,1 m
Method
SR
CT 4 RC 1 DTG
SR
CT L RC 1 DTG +
SR
CT
RC
DTG 4
SR 1
CT
4. RC
DTG
NoMad
100.0
100.0
99.6
100.0
1.1
100.0
0.0
95.1
1.3
CityWalker
100.0
93.3
100.0
97.5
100.0
0.0
96.3
0.8
74.3
5.5
100.0
90.2
100.0
95.9
66.7
66.7
89.4
2.9
100.0
0.0
90.5
2.0
PPO- UrbanSinn
100.0
0.0
90.9
2.0
100.0
0.0
65.0
6.0
100.0
0.0
87.1
4.3
0.0
100.0
67.3
6.0
PPO Beo-Urram Verse
100.0
0.0
90.9
2.0
100.0
0.0
95.1
2.1
100.0
0.0
93.1
2.0
100.0
0.0
90.7
2.0
Scene
Scene 06
Scene 07
Scene
Ronte Length
13.9
13.7
24.8
26.2 m
Method
SR
CT
RC 1 DTG
SR
CT
RC 1 DTG 4
SR
CT
RC
DTG
SR 1 CT J. RC
DTG
NoMad
0.0 100.0
53.2
5.4
0.0
100.0
69.3
4.9
0.0 100.0
30.0
15.0
33.3
66.7
63.6
10.0
CallWaler
0.0 100.0
38.3
0.0
100.0
31.1
6.8
0.0
100.0
36.5
14.8
0.0
100.0
14.8
22.1
S2E
0.0 100.0
9.5
9.0
100.0
0.0
82.7
2.0
66.7
33.3
69.8
7.0
66.7
33,3
66.5
8.0
PPO-UrbanSim
0.0 100.0
32,5
6.8
0.0
100.0
17.3
7,9
0.0
100.0
23.5
16.3
0.0
100.0
12.5
22.8
PPO Urban
100.0 0.0
85.6
2.1
100.0
0.0
80.4
2.1
100.0
0.0
94.2
2.0
100.0
0.0
88.1
3.1
Scene 09
Scene 10
Scene 11
Scene 12
Length
m
m
Method
SR 1 CT
RC DTG
SR
CT
RC f DTG
SR
CT
RC
DTG
SR t
CT J RC
DTG
NoMad
0.0 100.0
6.9
28.4
0.0 100.0
56.3
7.9
0.0
100.0
52.5
10.1
100.0
0.0
94.3
2.0
City Walker
0.0 100.0
0.0
30.7
0.0 100.0
19.1
9.3
33.3
66.7
66.6
9.1
0.0 100.0
15.1
17.2
82B
0.0 100.0
12.8
26.9
0.0 100.0
62.2
5.5
100.0
0.0
90.4
2.0
0.0 100.0
25.7
12.8
PPO-UrbanSim
0.0 100.0
20.9
11.8
0.0
100.0
8,4
10.6
0.0
100.0
82.9
3.5
0.0
100.0
10.0
15.0
PPO-Urbant Verse
33.3
66.7
70.2
9.1
33.3
66.7
79.4
4.0
33.3
66.7
52.4
9.7
0.0
100.0
72.9
5.3
Scene 13
Scene 14
Scene 15
Scene 16
Route Length
28.0 m
21.8 m
25.9 m
31.7 m
Method
SR
CT
RC DTG
SR
CT
1 DTG
SR CT
DTG
SR CT
RC DTG
NoMad
0.0 100.0
9.2
18.1
0.0
100.0
10.0
17.9
0.0 100.0
68.1
7.7
0.0
100.0
17.4
24.9
CityWalker
0.0 100.0
10.0
20.5
0.0
100.0
6.9
17.9
0.0 100.0
74.4
4.4
0.0
100.0
8.3
24.9
S2B
0.0 100.0
11.2
18.2
33.3
66.7
35.1
12,4
0.0 100.0
72.6
5.4
33.3
66.7
49.0
13.5
PPO-UrbanSan
0.0 100.0
10.7
17.6
0.0
100.0
9.2
17.4
0.0
100.0
5.0
14.9
0.0
100.0
10.5
25.2
PPO-UrbanVerse
100.0 0.0
92.7
66.7
33.3
79.6
3.9
100.0
0.0
85.8
2.1
66.7
33.3
83.9
4.6
Table Expanded real-world results of Coco wheeled robot on each scene. Best performance is colored in
Blue
J ADDITIONAL ZERO-SHOT SIM-TO-REAL TRANSFER RESULTS
We provide per-scene real-world experimental results for both robot embodiments. Specifically, Tab. 9 reports results for the Coco wheeled delivery robot, while Tab. 10 presents results for the Unitree Go2 quadruped. All experiments are conducted three times for each method.
K SYSTEM COMPUTATIONAL ANALYSIS
We provide detailed analysis of the computational cost and scalability of Urban Verse, covering: (i) real-to-sim scene generation with Urban Verse-Gen, Gen, (ii) large-scale object annotation in Urban Verse- 100K, and (iii) comparison with the procedural workflow of UrbanSim. Results are summarized in TablesTab 11, Tab. 12, Tab. 13, and Tab. 14.
Urban Verse-Gen Scene Generation. As shown in Tab. 11, the computational cost of Urban Verse- Gen scales almost linearly with input video length. The dominant runtime component is MASt3R- based 3D reconstruction, while GPT-4.1 queries are kept lightweight by sampling every third frame for object categorization. Short clips (10-40s) s) require 4-14 multimodal LLM calls and 12-114 s of processing, and longer clips (80-180s) require 27-60 calls and 289-1135 s single NVIDIA H100 GPU. Using 4 H100 GPÜs, the system generates 160 fully interactive scenes from 180 s city-walk videos in 1. 1.26 hours (Tab. 12), demonstrating strong practical scalability.
37

---

Published as a conference paper at ICLR 2026
Scene
Scene
Ronte Length
22.1
25.8
22.9
23.1
Method
SR
CT
L. RC 1 DTG.)
SR
CT
RC 1
DTG L
SR
CT
4. RC
1 DTG
SR
CT
RC 1 DTG
NoMad
97.5
100.0
100.0
City CayWalker
100.0
100.0
100.0
90.0
S2E
100.0
100.0
100.0
67.1
PPO UrbunSim
100.0
0.0
92.3
2.0
100.0
0.0
45.6
11.4
100.0
0.0
68.3
0.0
100.0
1.5
20.1
PPO Urrun
100.0
0.0
88.7
2.1
100.0
93.9
2.0
100.0
95.2
2.1
100.0
0.0
90.3
2.0
Scene
Scene
Scene
Scene 08
Ronte Length
13.9
13.7
24.8
26.2
Method
SR
CT
RC 1 DTG +
SR
CT
RC
DTG
SR
CT
RC 1 DTG
SR
CT 4. RC 1 DTG
NoMad
100.0
90.7
100.0
0.0
100.0
22.4
18.0
0.0
100.0
44.2
13.1
CayWalker
100.0
0.0
93.0
2.4
0.0
100.0
30.8
0.0
100.0
9.1
20.7
0.0
100.0
2.7
22,8
S2B
0.0
100.0
78.5
3.3
100.0
0.0
79.0
2.1
66.7
33.3
94.6
2.1
100.0
0.0
87.0
3.1
PPO-UrbanSim
0.0
100.0
15.9
8.8
0.0
100.0
65.8
3.3
0.0
100.0
30.8
15.1
0.0
100.0
14.4
20.2
PPO Verun
100.0
0.0
88.0
2.0
100.0
0.0
78.0
2.1
100.0
0.0
92.3
2.1
100.0
0.0
91.4
2.0
Scene 09
Scene 10
Scene 11
Scene 12
Route Length
m
35.7
Method
SR
CT
RC f DTG
SR t
CT
RC DTG
SR
CT
RC T DTG
SR
CT
RC 1 DTG
NoMad
100.0
74.4
10.1
0.0 100.0
83.8
13.4
0.0
100.0
36.8
13.2
0.0
100.0
18.4
18.2
Cit Calker
0.0
100.0
37.1
19.1
0.0
100.0
5.2
20.3
0.0
100.0
23.1
14.9
0.0
100.0
25.8
16.6
$2B
33.3
66.7
57.4
13.6
100.0
0.0
96.2
2.1
66.7
33.3
63.3
2.7
0.0
100.0
53.0
10.5
PPO-UrhanSin
0.0
100.0
11.7
26.7
0.0
100.0
0.0
21.6
0.0
100,0
58.0
4.6
0.0
100.0
12.8
22.1
PPO Vebun
100.0
0.0
93.5
2.1
100.0
0.0 96.4
2.0
66.7
33.3
65.1
2.3
100.0
0.0
89.8
2.3
Scene 13
Scene 14
Scene 15
Scene 16
Route Length
28.0 m
21.8 m
25.9 m
31.7 m
Method
SR t
RC DTG
SR
RC DTG
SR
CT
RC DTG
SR CT
1 DTG
NoMad
0.0 100.0
1.5
20.8
0.0 100.0
33.6
13.9
0.0 100.0
0.0
26.4
0.0
100.0
12.2
32.4
City Walker
0.0 100.0
1.5
20.8
0.0 100.0
20.4
16.6
0.0 100.0
48.5
10.4
0.0
100.0
588
32,3
S2B
100.0
69.3
33.3
66.7
33.9
13.8
0.0
100.0
53.6
33.3
66.7
34.3
17.6
PPO-UrbunSim
100.0
23.4
15.7
0.0
100.0
10.0
21.2
0.0
100.0
8.0
14.5
0.0
100.0
7.4
32.7
PPO-UrbanNerse
100.0
94.9
66.7
33,3
85.0
33.3
66.7
81.4
66.7
33.3
58.8
6.0
Table 10: Expanded real-world results GO2 quadruped robot on each scene. Best performance is colored
Blue
Video Duration (sec)
Video Length (# frames)
L.M Calls
Scene Generation Wall Time (sec)
10
10
4
12.38
40
40
14
14.46
80
80
27
289.80 80
180
180
60
1135.20
Table 11: Veree-erse real-to-sim generation time with varying input video lengths
NVIDIA H100 GPU.
# City-tour Videos
# Cousin Scenes Layout
Unique Layouts
# Total Scenes
Wall Time
Setting
Single layout with
5
5
18.92 min
5 digital cousins
160 Scenes
32
5
32
160
10.08 hrs
(1 x NVIDIA H100)
160 Scenes
32
5
32
160
1.26 hrs
(4 x NVIDIA H100, default)
Table 12: Overall computational time of Urban Verse scene generation.
38

---

Published as a conference paper at ICLR 2026
# Objects
GPT-4.1 Call Counts
API Wall Clock Time
API Cost
0.0003 hrs
$0.018
2
0.0005 hrs
$0.029
102,444
102,444
65.5 hrs
$1,334
Average
o/object
2.3 sec / object
$0.013 1..01//ojjett
Table 13: Annotation cost and runtime statistics of the Urban annotation pipeline.
Aspect
UrbanSim
Urban Verse
How to Add
Manual annotation (metric resizing, mass setup)
Fully automated annotation pipeline
New 3D Asset
Time to Add
6600 sec
2.3 sec
a New 3D Asset
How to Add
Manual creation of scene templates
Automatic scene generation via Urban Verse-Gen
a New Scene
Time Create
118.9 min
240 min
a New Scene (~2200m)
(NVIDIA H100)
Rendering Efficiency
99 FPS
~9 FPS
(RGB, Single Env, L40S)
Table 14: Comparison of UrbanSim and Urban for asset creation, scene generation, and rendering efficiency.
UrbanVerss-1000 Verse-100K Asset Annotation. The Urban Verse- 100k annotation pipeline also scales ef- ficiently. As summarized in Tab. 13, each 3D asset is annotated with exactly one GPT-4.1 call, averaging 2.3 and $0.013 per object. Annotating the full dataset of 102,444 assets requires 65.5 hours of API wall time and S1,334 in total cost, providing an economical path to large-scale semantic and physical labeling without manual intervention.
Comparison with UrbanSim. UrbanSim relies on procedural templates that are hand-designed by developers and require manual asset annotation. While procedural sampling itself incurs little computational cost, this workflow limits realism and diversity. As shown in Tab. 14, Urban Verse automates both scene creation and asset annotation, reducing scene generation time from 240 min (UrbanSim) to 18.9 min, and reducing per-object annotation time from ~~600 to 2.3 s. Both systems achieve similar rendering throughput in Isaac Sim (94 FPS), but Urban Verse provides orders-of- magnitude higher scalability and produces scenes grounded in real-world distributions.
L EXPERIMENT SETUP DETAILS
L. DEFINITION OF URBAN NAVIGATION EVALUATION METRICS
In this section, we formally define the evaluation metrics used in our urban navigation experiments.
Success Rate (SR; %). The percentage of episodes in which the robot successfully reaches the goal without collision, averaged across scenes and runs. Higher values indicate better navigation performance.
Collision Time (CT; %). The fraction of total episode time during which the robot is in contact with any obstacle. Lower values indicate safer navigation.
Route Completion (RC; %). The percentage the planned evaluation route completed before termination (goal reached, fatal collision, off-road deviation, or timcout). Higher values reflect more reliable progress.
Distance to Goal (DTG; m). The final Euclidean distance from the robot to the goal at episode termination. Lower values indicate precise goal reaching.
39

---

Published as a conference paper at ICLR 2026
L.2 DETAILS OF TRAINING AND EVALUATION SETUP
For training and evaluation, agents are initialized within the annotated traversable regions in Ur- ban Verve scenes. The goal point is randomly sampled distance between 10 and 30 m from the starting pose. Each episode runs for a fixed horizon (60s@ during which the policy must navigate toward the goal while avoiding collisions and staying within the traversable area. Rollouts are collected with specified horizon length (i.e., 32 steps) for training updates. For both training and evaluation, an episode terminated either when the robot collides with any object in the scene or when the maximum time horizon is reached.
L.3 DETAILS OF POLICY LEARNING
Model architecture. For RL training, we adopt an actor-critic architecture with continuous action space, trained using PPO (Schulman et al., 2017). Each observation is the relative position of the goal point and an RGB frame of size 135 x 240 with three channels. The convolutional encoder consists of three layers with depths [16, 32, 64), each followed by ReLU activation. The goal point is encoded using a MLP. No special regularization is applied. The encoder output is passed through a MLP with three hidden layers of size 128 and ELU activations. Actor and critic share the same backbone. The output distribution uses Gaussian parameters with unconstrained and σ. initialized respectively with the default initializer and constant (0.0).
Reward fuunctions The reward is designed to encourage goal-reaching while penalizing unsafe behaviors. Specifically:
R= RA + Rc
Rp + Ry
(1)
Arrived reward RA: A large positive reward (+2000) is given when the agent successfully reaches the goal.
Collision penalty Ro: A penalty (-200) is applied if the agent collides with any obstacle.
Position tracking Rp: A shaping reward based on the error between the commanded and actual position, with two scales: coarse (std = 5.0, weight 10) R.. and fine (std = 1.0, weight=500 Rp,--
Velocity reward Rv: A reward (weight = 10) encourages matching the target velocity command, defined as the cosine similarity between the current velocity and target velocity between the robot position and target position.
This combination balances sparse terminal signals (arrival, collision) with dense shaping terms (tracking error, velocity), stabilizing training and guiding exploration.
Optimization and training hyperparameters. We provide the detailed hyperparameters in Tab. 15.
L.4 ROBOT PLATFORM CONFIGURATIONS IN SIMULATION
Our simulated platforms are implemented in NVIDIA IsaaeSim (Xu et al., 2022; Mittal al., 2025), GPU-accelerated environment that provides chysics-accurall interactions and photorealistic rendering. Each robot model instantiated from its official URDF specification and equipped with RGB sensing for visual input.
Locomotion is governed by a layered control architecture composed of low-level joint controller and high-level policy optimized through reinforcement learning. Policies are trained with Proximal Policy Optimization (PPO) (Schulman et al., 2017) under curriculum learning and terrain random- ization. The reward design promotes balance, velocity tracking, and energy-efficient movement, while penalizing collisions and falls. To enhance sim-to-real transfer, domain randomization is applied across textures, friction parameters, and mass properties. This unified framework is applied consistently across all robot embodiments, supporting scalable and embodiment- training.
Wheeled robot. The wheeled platform is modeled kinematic formulation (Polack et al., 2017). Linear and angular velocities (v, w) are generated from waypoints using an ideal PD controller (Sridhar al., 2024). These commands propagated in
differential-die robot controlled via
40

---

Published as a conference paper at ICLR 2026
Parameter
Value
Learning rate
x 10
(adaptive schedule)
Discount factor y
0,99
GAE parameter
0.95
PPO clipping
0.2
KL threshold
0.01
Entropy coefficient
0.002
Critic loss coefficient
1.0
Gradient norm clipping
1.0
Horizon length
32
Minibatch size
512
Mini-epochs
Bounds loss coefficient
0.01
Training epochs
1500
Device
Single GPU (L40S), mixed precision
Table 15: Optimization and training hyperparameters.
Power bank
WI-FI Router
dhaa Fachhrttayyyccoom
Comaes
02 Grite
Front Fishage Camara (model
OPS
Actuated
Whoels
(a) Delivery
[b) Quadruped
Figure 36: Robot platform configurations used in real-world experiments: (a) Coco wheeled delivery robot and (b) Unitree Go2 quadruped.
IsaacSim's 's rigid-body physics engine, where wheel-ground frictional contact governs the realized motion.
Unitree Go2 quadruped. The quadruped embodiment (Unitree Go2) is designed for agile locomotion in unstructured environments. Control actions are expressed as (Uz, wy. again produced from waypoints through an ideal PD controller (Sridhar et al., 2024). The locomotion module itself is compact MIP trained on IsaaeSim's standard quadruped training setups (Xu et al., 2022; Mittal et al., 2025), enabling stable gait generation across diverse terrains.
L.5 ROBOT PLATFORM CONFIGURATIONS IN REAL-WORLD
For real-world experiments, we deploy two distinct robot platforms: the Coco wheeled delivery robot and the Unitree Go2 quadruped, as shown Fig. 35. These platforms represent complementary embodiments for urban navigation- compact, wheeled sidewalk delivery robot and legged quadruped capable of traversing uneven terrain.
Coco wheeled robot. As shown in Fig. 36 (a), the real Coco robot is equipped with four actuated wheels, differential-drine odometry, GPS unit, and multiple fisheye cameras (front, rear, left, and right) that provide panoramic perception. Its sensing and communication stack includes 4G router for remote teleoperation connectivity and delivery box as payload. The front fisheye camera serves
41

---

Published as a conference paper at ICLR 2026
Section 1/10
Q1: Below are three front-view images captured from different simuleted urben
Please click to play waich
Please compare the three scanes (A, based the following criteria, and choose one. difference unclear or they are uqually good, to choose "Equal":
Evaluation Criteria
Layout Caherence: Which displays realittic logically layout of 30 asozts, based on or nalplex of real-world urbon planning and apotial organization? Consider object alignment spacing, and naturally the enxtronmmnt flowss
Object Diversity: Which ocene includes a more varied reprecentative set brb eljects, contribuiing to the nicnnees, completeness, and authenticty arvfrommment ayourt Cohanence
Overall Urban Scene Realiam Which scene best resemblee a believbbe real-
world urban acenaria overal? Considor the combined effects asost
appropr fataness, layour oot coharence, object verrsiyy and visual reafsm.
02: Now would you rate the quality of thes dignal simalated uban scene in terma of Overal Urban Soane Realion Asset Contentual Appropiiooness Layour Raeltor, and Object Diversity?
Frpull
Object Diversty
Average
-5 Exoellert
Layout
Overal Rea lom
Dealle:
(a) User Preference Study
(b) Scene Quality Rating Study
Figure 37: User study interface example.
the primary input to our policy models. The robot is controlled via the same kinematic model used in simulation, where linear and angular velocities (v, w) are computed from waypoints using an ideal PD controller. Odometry is used for real-time position estimation and to continuously update the target position during navigation.
Unitree Go2 quadruped. As shown in Fig. 36 (b), the Unitree Go2 offers a contrasting embodiment with articulated legs and onboard compute support. It is equipped with a fisheye perception camera, Wi-Fi router for connectivity, and an extended power system consisting of a power bank, voltage boost module, and router charger to support sustained experiments. Low-level locomotion is handled by the controller provided by Unitree. Instead of executing joint-level commands from a trained policy, we interface with the Go2 through its built-in velocity control API, sending high-level commands (Ues Uy, w) that leverage its native gait generation and stability modules. LiDAR-bar based odometry is used for real-time position estimation and to continuously update the target position during navigation.
Together, these two platforms enable to examine sim-to-real transfer across distinct locomo- tion modalities and hardware configurations, thereby providing a broader evaluation of embodied navigation in diverse urban settings.
M HUMAN EVALUATION DETAILS
To better understand how human judgments align with the realism of generated scenes, we designed two complementary user studies using a custom-buil online interface, as illustrated in Fig. 37.
Comparative User Preference Study. For the first study, participants were presented with pairs of static overview images randomly sampled from dataset. Each pair consisted of one scene generated by Urban and one procedurally generated (PG) scene using UrbanSim (Wu et al., 2025b) built from the same Urban Verse- 100K assets. As shown in Fig. 37(a), participants compared
42

---

Published as a conference paper at ICLR 2026
the two scenes side-be bide-by-side and were asked to select which scene performed better (or "Equal") across three criteria: (1) Object Diversity - the richness and representativeness included urban objects; (2) Layout Coherence the realism and logical arrangement of objects based real-world urban design principles; and (3) Overall Realism - the degree to which the scene resembled a plausible real-world street environment. Responses were collected through simple three-choice interface (A better, B better, Equal) for each criterion.
Scene Quality Rating Study. In the second study. participants evaluated scene quality through immersive video walkthroughs. Each trial displayed a 360 simulated flythrough of a given scene, as illustrated in Fig. 37(b). Participants were then asked to assign a quality score (1-5) reflecting overall realism, asset contextual appropriateness, layout realism, and object diversity. The Likert-style rating scale ranged from 1 Very Poor to 5 Excellent. This setup allowed participants to assess not only static composition but also temporal and spatial coherence as the camera moved through the scene.
Study Deployment. Both studies were conducted with 32 undergraduate participants. To ensure fairness, all scenes and videos were shuffled randomly across participants, and instructions clarified the evaluation criteria prior to the study. This interface design ensured consistent, criterion-driven human judgments across both comparative and -ating-based evaluations.
43


Reviews:
[{"review_id": "VDTuXZZGIE", "reviewer": "Reviewer_Nwq7", "summary": "The paper introduces UrbanVerse, a data-driven system that automatically converts real-world city-tour videos into interactive, physics-aware urban simulations. The framework comprises:\n\n- UrbanVerse-100K, a large-scale annotated database of 100k+ metric-scale 3D urban assets with semantic and physical attributes.\n\n- UrbanVerse-Gen, an automated pipeline that reconstructs scene layouts and lighting from uncalibrated videos and instantiates them into simulation-ready environments in Isaac Sim.\n\nThe authors demonstrate that policies trained on these automatically generated scenes follow scaling laws and transfer effectively to real-world navigation tasks—achieving up to +30% zero-shot sim-to-real improvement over prior baselines.", "questions": "- Can UrbanVerse handle dynamic agents (e.g., pedestrians, vehicles in motion), or is it restricted to static geometry?\n\n- How does scene reconstruction cope with occlusions and low-light conditions in real videos?\n\n- Would the system generalize to non-street urban contexts (e.g., parks, campuses, or indoor–outdoor transitions)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Novel real-to-sim paradigm: The approach of transforming crowd-sourced city-tour videos into physically interactive simulations is innovative and highly scalable.\n\n- Large, well-annotated asset database: The UrbanVerse-100K dataset provides a valuable and reusable resource with semantic and physical labels, addressing a major limitation of existing simulators.\n\n- Strong quantitative and qualitative results: The system shows high reconstruction fidelity (93% semantic accuracy, 1.4 m localization error) and impressive sim-to-real performance across multiple robot platforms.\n\n- Comprehensive experiments: The paper evaluates reconstruction quality, scaling effects (power-law validation), and real-world zero-shot transfer, providing a well-rounded empirical validation.\n\n- Open-sourcing commitment: Promising for community impact, given the open release of code, assets, and simulation scenes.", "weaknesses": "- **Overly complex pipeline with high compounding error:**\n    The overall data annotation and reconstruction pipeline involves numerous components—open-vocabulary detection, depth estimation, 3D lifting, GPT-4 labeling, geometry matching, appearance retrieval, and simulation instantiation. Each step introduces noise, and the compounded uncertainty across so many modules raises serious concerns about consistency and accuracy. While the qualitative examples look convincing, they may reflect cherry-picked successes rather than representative results at scale. \n\n- **Lack of failure case analysis:**\nFollowing the previous, the paper only shows successful examples but omits clear failure cases or error breakdowns. Understanding when and why the system fails—e.g., mis-segmentation, inaccurate depth estimation, wrong asset retrieval—would be critical to assessing robustness and diagnosing limitations. Including a few representative failure visualizations would make the results more transparent and credible.\n    \n- **Questionable large-scale data quality:**\n    The pipeline heavily relies on noisy, crowd-sourced internet videos, often with unstable camera motion, motion blur, or occlusions. Given such low-quality inputs and a long chain of automated processing, it is doubtful that most of the reconstructed scenes achieve the same level of fidelity as the few presented cases. The paper lacks quantitative or statistical analysis of **data noise, annotation error rates, or reconstruction failure modes**, which undermines confidence in the dataset’s overall reliability.\n    \n- **Limited scope of validation:**\n    Despite the title’s emphasis on “urban simulation,” the evaluation focuses only on **navigation** tasks, without exploring manipulation, traffic reasoning, or social interaction scenarios. It remains unclear whether UrbanVerse scenes are truly general-purpose or mainly tuned for simple navigation.\n    \n- **System integration over algorithmic innovation:**\n    The paper’s novelty lies mainly in combining existing large vision models (CLIP, SAM2, GPT-4.1, MASt3R, etc.) into a pipeline, rather than introducing new algorithms or learning frameworks. The contribution is thus primarily engineering-based.\n    \n- **Insufficient runtime and scalability analysis:**\n    The pipeline’s cost—both in compute and manual verification—is not clearly reported. Given the dependence on heavy models and large-scale video inputs, the claimed scalability may be impractical without significant resources.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wQU5HOKth5", "reviewer": "Reviewer_9ozp", "summary": "The paper introduces UrbanVerse, a real-to-sim pipeline that converts crowd-sourced city-tour videos into physics-aware, interactive urban scenes in Isaac Sim. It builds UrbanVerse-100K (100k+ metric-scaled, attribute-annotated assets) and UrbanVerse-Gen to lift semantics, layout, ground, and sky from uncalibrated videos, producing 160 training scenes plus a 20-scene benchmark; KITTI-360 reconstruction metrics, human studies, and scaling experiments show improved scene realism and power-law gains in mapless navigation. Policies trained on these scenes outperform baselines in simulation and zero-shot real-world tests (up to ~90% success on a quadruped) and complete a 337 m route with two interventions.", "questions": "In Tab. 1, why VGGT has lower quality reconstruction than MASt3R? This seems contridictory to common knowledge that VGGT generally outperformes MASt3R in different scenes.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The work shows extensive efforts for data curation to ensure a diverse and realistic dataset.\n2. The paper shows human evaluation on scene quality. This is the most reliable comparison in the absense of a ground truth scene layout. The result shows scenes in UrbanVerse generally have better quality than UrbanSim.\n3. Real world deployment experiments shows good sim-to-real performance of the RL-trained models on the UrbanVerse environment.", "weaknesses": "1. The work develops digital cousin from online videos, but does not provide valid reasons why previous 3DGS-based digital twin methods fail for the purpose. This motivation of the work is not fully consolidated.\n2. The data curation pipeline utilized LLM to annotate the physical attributes of the objects. However, this is not verified by human and prone to error. The authors should show evidence that these annotations are correct and convincing.\n3. The groud fitting step assumes a single ground plane. This fails to model uneven surfaces and curbs that are vital for the success of sidewalk navigation. These sides walk features can also be different for wheeled or legged robots.\n4. The number of the scenes in the dataset is limited to 160. This is quite small considering the large number of city-touring videos available online. The limitation can be verified in Fig. 7 where the navigation performance grows linearly with the nubmer of digital cousins. It is unknown where is the margin for the scaling law on the number of environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xyvTff1lNf", "reviewer": "Reviewer_ND3o", "summary": "This paper presents UrbanVerse, a data-driven real-to-sim system that automatically converts city-tour videos into physics-aware, interactive simulation environments for training urban embodied AI agents. The system comprises two main components: (1) UrbanVerse-100K,  a curated repository of 102,530 high-quality metric-scale urban object assets, and (2) UrbanVerse-Gen, an automated pipeline that extracts scene layouts from real-world videos and instantiates simulation scenes by retrieving matched assets. The authors construct 160 training scenes from city-tour videos spanning 24 countries and demonstrate that policies trained in these environments follow scaling power laws and achieve strong sim-to-real transfer performance in mapless urban navigation tasks, including a 337-meter real-world deployment with only two human interventions.", "questions": "q1: Please supplement the functional innovations of UrbanVerse compared to other simulators (CARLA, MetaUrban, UrbanSim), including: supported embodied intelligence tasks, rendering efficiency comparison, supported agent types, multi-agent interaction, and environmental dynamic disturbances.\nq2:Please describe the user-side pipeline of UrbanVerse. Based on the current content in the article, it is difficult to determine UrbanVerse's potential contributions to the community.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "1. UrbanVerse presents an effective framework that retrieves similar 3D assets through video reference and employs a mapping mechanism to construct street scenes that closely approximate real-world environments. This framework theoretically allow the system to recreate any scenario captured in city-tour videos.\n2. The author establishes a comprehensive evaluation framework with six reasonable metrics (Cat, Ast, Cov, Lay, Loc, and Div) that assess different aspects of reconstruction fidelity. Combined with human evaluation, this hybrid evaluation thoroughly validates the effectiveness of the proposed framework. \n3. The navigation experiments prove that scene diversity significantly enhances the generalization capability of navigation agents, which highlights the urbanverse's potential capability.\n4. The long-distance real-world deployment provides convincing evidence of UrbanVerse's excellent sim-to-real transfer capabilities.", "weaknesses": "1. UrbanVerse currently exhibits limited interactivity between agents and the environment. After reading the paper, I assume that the urbanverse only supports collision response now. The lack of dynamic environment interactions (pedestrians, moving cars, etc.) prevents UrbanVerse from serving as a general-purpose platform for diverse urban embodied AI research, limiting its utility to primarily perception-based navigation scenarios.\n\n2. The paper does not convincingly articulate why researchers should adopt UrbanVerse over these established alternatives beyond improved visual realism. Without demonstrating substantial functional advantages—such as superior sim-to-real transfer, computational efficiency, or unique interaction capabilities—the contribution appears incremental. The claim that UrbanVerse addresses limitations of procedural generation is weakened by the fact that existing simulators already achieve effective sim-to-real transfer for navigation tasks.\n\n3. Without clear documentation of how the community can leverage this work—whether through asset libraries, simulation APIs, or integration tools—the practical impact remains limited. The contribution of  this urban simulation is undermined by the absence of concrete plans for open-source release, developer documentation, or community engagement strategies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "adpAIYpoix", "reviewer": "Reviewer_eUF4", "summary": "The paper provides a new dataset of 100k assets for urban scene simulators, including physical properties, along with ~300 PBR materials for ground surfaces, and ~300 lighting skyboxes. The process for data filtering and annotation is documented as well.\n\nThe paper also proposes a real2sim process to convert ego-centric urban navigation videos into interactive simulators. This is done by processing videos to extract a scene graph, match the text and geometry of the scene objects to assets in the above database, and then place assets in corresponding locations in the scene. \n\nExperiments demonstrate the quality of this approach compared to a baseline model. Human preference scores for scenes (in diversity, coherence, and quality) favor the model over a baseline, nearing a ceiling set of scores from a set of human-crafted scenes. RL policy learning shows power law scaling both in the number of distinct scenes (from videos) and variations of scenes per video. Real-world evaluations add further rigor to the success of the sim2real transfer, including an extended navigation task with minimal human intervention for success.", "questions": "# questions\n- How does UrbanVerse scale?\n\t- How many VLM/LLM calls are needed?\n\t- How much wall time / compute?\n\t- How does UrbanVerse compare to UrbanSim in these dimensions?\n- How does this scale with input video length or scene size (in meters, number of entities/assets, and so on)?\n\t- Is there any ablation data on training based on different video lengths or scene dimensions?\n- lines 281-281: How sensitive are the outcomes to assigned parameters? How are they assigned?\n- line 476: What were the two interventions needed?\n- Table 2: Define the evaluation metrics. Some were not clear to me:\n\t- How does success rate differ to route completion? When can you complete the route without succeeding?\n\t- CT: Should the arrow be arrow down or up?\n\t- \"Collision times\" is ambiguous. It's not clear if this means \"(mean) time to collision\" vs \"number of times there was a collision\", for example.\n\t- The upward arrow suggests lower is better, implying the latter. But on that metric, PPO-UrbanVerse does quite poorly in Table 2 and for wheeled in Table 3. But well for quadruped? It's a bit confusing how to interpret this.\n\n\n# suggestions\n- Table 1: Separate \"Ast.\" as a clearly marked final column as it is evaluated on different data (and thus not quite continuous with the other metrics).\n- Figure 7: Would be nice to increase the scale to see where improvements asymptotically saturate. Given the layouts are limited to 32 environments it would suffice to increase the number of training cousins to show where success rate asymptotes (near 100%?).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "# originality\n- Combines a variety of techniques to produce high quality scenes. The novelty is not so much the problem or any individual component of the generation process, but the combination is a non-trivial addition given the strong results.\n\n# quality\n- Experiments show very strong sim2real results.\n- Provides a massive and usable asset dataset that has been cleaned up and validated.\n\n# clarity\n- The data preparation process is thoroughly explained, as is the generation process. The plan to open-source will further facilitate reproduction, adoption, and extension.\n- The narrative and steps were easy to follow.\n\n# significance\n- sim2real for urban scene navigation is directly valuable to the autonomous driving community. The assets will be of general value to downstream work on improved generation methods, with clear improvement over existing options and baselines (KITTI).", "weaknesses": "# originality\n- No single part of the approach differs from established practices. This is fine given the power of the contribution.\n\n# quality\n- It would help to provide data on the scaling and costs of the different methods. For example: how many LLM/VLM calls are needed? how much wall clock time to process a scene? how do these vary in the size of the scene or length of the video?\n\n# clarity\n- The sim2real evaluation metrics would benefit from definitions (see below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)