SimWorld-Robotics: Synthesizing Photorealistic and Dynamic Urban Environments for Multimodal Robot Navigation and Collaboration
Authors: Yan Zhuang
Jiawei Ren
Xiaokang Ye
Jianzhi Shen
Ruixuan Zhang
Tianai Yue
Muhammad Faayez
Xuhong He
Xiyan Zhang
Ziqiao Ma
Lianhui Qin
Zhiting Hu
Tianmin Shu
Venue: NeurIPS 2025 poster ()
Keywords: Embodied Simulator
Multimodal Instruction Following
Robot Navigation
Multi-Robot Collaboration
Communication
Primary area: applications

Abstract: Recent advances in foundation models have shown promising results in developing generalist robotics that can perform diverse tasks in open-ended scenarios given multimodal inputs. However, current work has been mainly focused on indoor, household scenarios. In this work, we present SimWorld-Robotics (SWR), a simulation platform for embodied AI in large-scale, photorealistic urban environments. Built on Unreal Engine 5, SWR procedurally generates unlimited photorealistic urban scenes populated with dynamic elements such as pedestrians and traffic systems, surpassing prior urban simulations in realism, complexity, and scalability. It also supports multi-robot control and communication. With these key features, we build two challenging robot benchmarks: (1) a multimodal instruction-following task, where a robot must follow vision-language navigation instructions to reach a destination in the presence of pedestrians and traffic; and (2) a multi-agent search task, where two robots must communicate to cooperatively locate and meet each other. Unlike existing benchmarks, these two new benchmarks comprehensively evaluate a wide range of critical robot capacities in realistic scenarios, including (1) multimodal instructions grounding, (2) 3D spatial reasoning in large environments, (3) safe, long-range navigation with people and traffic, (4) multi-robot collaboration, and (5) grounded communication. Our experimental results demonstrate that state-of-the-art models, including vision-language models (VLMs), struggle with our tasks, lacking robust perception, reasoning, and planning abilities necessary for urban environments.

Sim World-Robotics: Synthesizing Photorealistic and Dynamic Urban Environments for Multimodal Robot Navigation and Collaboration
Yan Zhuang! Jiawei Ren Xiaokang Ye Jianzhi Shen
Ruixuan Zhang'
Tianai Yue Muhammad Faayez' Xuhong He Xiyan Zhang*
Ziqiao Ma Lianhui Qin Zhiting Hu Tianmin Shu
University of Virginia UCC San Diego 3Johns Hopkins University
Caanegge Mellon University University of Michigan
Abstract
Recent advances foundation models have shown promising results developing generalist robotics that can perform diverse tasks in open-ended scenarios given multimodal inputs. However, current work has been mainly focused on indoor, household scenarios. In this work, present SimWorld- Robotics (SWR), a simulation platform for embodied AI in large-scale, photorealistic urban environments. Built on Unreal Engine SWR procedurally generates unlimited photorealistic urban scenes populated with dynamic elements such pedestrians and traffic systems, surpassing prior urban simulations in realism, complexity, and scalability. It also supports multi-robot control and communication. With these key features, we build two challenging robot benchmarks: (1) multimodal instruction-followin
task, where robot must
follow languael navigation instructions reach destination in the presence of pedestrians and traffic; and (2) multi-agent search task, where two robots must communicate cooperatively locate and meet each other. Unlike existing benchmarks, these two new benchmarks comprehensivel, evaluate wide range critical robot capacities instructions grounding, (2) 3D spatial reasoning in large environments, (3) safe, long-range navigation with people and traffic, (4) multi-robot collaboration, and (5) grounded communication. Our experimental results demonstrate that state- of-the-art art models, including vision-language models (VLMs), struggle with our tasks, lacking robust perception, reasoning, and planning abilities necessary for urban environments.
realistic scenarios, including (1) multimodal
Project website: SimWorld-Robotics
Introduction
There has been tremendous progress in engineering general- robotics that can follow human instructions and perform open-ended tasks [2], 128, ISS 27 [6]], thanks to the advances in robot foundation models. Training these models requires a large amount of data, much of which can be generated in high-fidelity embodied simulators, such Habitat 3
[[]]] RoboTHOR [ITA], TDW [[24], VirtualHome [[8], Virtual Community [61]] and BEHAVIOR [[Z]]. They can also be systematically evaluated in diverse scenarios created in these simulators. However, current embodied simulators for robotics have been focused on tabletop [65] 58, 341 22, 5 or household tasks [4H, 27. 26, 47, [6].
In this work, want study how create realistic and scalable embodied simulator for outdoor robotics tasks.
Equal contribution. Equal advising.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Diverse Building Assets
Diverse Detail Assets
Road
Generr abinn
Bulkding
Embodied
Diverse Agents
Goneration
Agents
Action Space
City Procedural Generation
Waypoint Generator Generator Traffic Manager
SimWorld-Robotic
Figure Overview of SirWd Robotics (SWR). Built upon Unreal Engine 5, SWR is a simulation platform for large-scale, photorealistic, and dynamic urban environments. It offers diverse high-fidelity building and object assets, supports embodied agents with rich action spaces, includes background traffic system powered by city-scale waypoint generation, and enables comprehensive city procedural generation.
Compared to indoor scenarios, robotics in outdoor environments, in particular, large urban environments, introduces additional challenges, such as (1) 3D perception, spatial reasoning and grounding in large environments; (2) safe navigation in dynamic scenes with people and traffic; (3)
long-range spatial memory; and (4) multi-agent collaboration and communication in task.
There have been urban simulators developed in recent years. However, to address the critical challenges faced by real-world roboties in urban environments, they lack the necessary realism, customizability, scalability, and versatility. For instance, well-known simulators such as AirSim [45] CARLA [I2] mainly focus on autonomous driving domains. While it supports some manual customization to the provided city environments, it does not support procedural city environment generation. It also does not support the flexible control of embodied agents (such as mobile robots or pedestrians) other than vehicles. More recent city simulators, such as MetaDrive [2%]. MetaUrban [αб]], significantly improve the scalability. However, the simulated environments still lack photorealism shown in Figure A
Therefore, we introduce HeeWWolldRoboiice (SWR), new embodied AI simulation platform for large-scale, photorealistic, and dynamic urban environments. As illustrated in Figure MI SWR offers diverse action spaces, waypoint-based background traffic system, and comprehensive procedural city generation pipeline to generate infinite cities. SWR also supports scalable data generation with fine- grained ground-truth annotations, enabling the training and evaluation of embodied agents at scale. Together, these features accelerate progress toward stronger embodied intelligence.
building and object assets, multiple types of embodied agents with rich
By leveraging SWR, we develop two novel benchmarks for robots in large, urban environments. Each benchmark evaluates crucial robot capabilities uniquely supported by the key features of SWR. As shown in g. the first is multimodal instruction following benchmark, SIMWORLD. MMNAV, for robot navigation, in which a robot must follow vision and language instructions to reach the target location. Unlike existing robot navigation benchmarks, we evaluate multiple robot capacities necessary for real-world urban navigation jointly, including robust 3D visual perception, grounding multimodal instructions to 3D environments, obstacle avoidance, following traffic rules, and walking around people in socially acceptable way. The second a multi-robot search benchmark, SIMWORID-MRS, in which two robots must cooperate to localize and meet each

---

[4] 210 PK pn]. There have SimWorld-Robotics Key Features
Festos
Lighting System
Weather System
Diverse Building Blocks
Pedestrian Behaviors
MetaUrban
AirSim
CARLA
MetaDrive
Figure 2: Simulator Comparison Top: Our simulator demonstrates key features including dynamic lighting (e-g.. sunrise), realistic weather (e.g., rain), diverse high-fidelity buildings, and rich pedestrian behaviors. Bottom: MetaUrban and MetaDrive support large-scale maps but suffer from poor rendering quality; AirSim and CARLA offer better visuals but have limited building diversity and high repetition. All existing simulators exhibit simplistic pedestrian behaviors limited to random walking, failing to reflect real urban dynamics.
other via physical navigation and verbal communication illustrated in E. To be successful in this benchmark, robots must be able to effectively communicate about each other's 's locations in a large, unfamiliar space and discuss joint plan, combined with their 3D spatial reasoning abilities. Our experimental results demonstrate that existing models, including state-of-the-arr vision-language models (VLMs), fail achieve meaningful success on our benchmarks. This highlights the gap in current foundation models for challenging, realistic robot tasks in urban environments.
To address this gap, introduce SimWorld-20K, large-scale dataset for benchmarking multimodal robot navigation in photo-realistic urban environments. The dataset contains 20K training steps sampled from 200 episodes, each averaging 500 in length, across 100 procedurally generated city environments with average area of 2 km Compared to MetaUrban [50], the most recent urban simulator supporting procedural city generation, SWR offers environments that are 100x larger in area and episodes that over 1.2x longer (MetaUrban: 410 meters per episode and 0.02 km on average). This enables evaluation of long-horizon, cal-world-scale navigation. After fine-tuning on Sim World-20K, Qwen achieves a non-zero success rate on the test set and outperforms SOTA proprietary models across several key metries.
In sum, our key contributions include: (1) new embodied AI simulator, Sim World-Robbotics (SWR), that supports the creation and simulation of photorealistic and dynamic urban environments with diverse embodied agents; (2) two novel benchmarks for single robot navigation and multi-robot search tasks that evaluate key robot capacities by leverage the key, unique features of our simulator; that enables long-horizon multimodal robot
(3) a large-scale training dataset, SimWorld-200 navigation across city-scale environments; (4) systematic evaluation of recent baseline models which identifies the significant limits of these models on the evaluated key capacities.
Related Work
Embodied Simulators for Urban Enyironments. Recent embodied simulators mostly focus on the indoor environment, mainly designed for houschold robots [ 52, 3X, HD
been outdoor simulators [122 85,, 56, ITa, 53, [3) that provide more open-ended tasks, facilitate more complicated dynamics, but face challenges in reality and multi-agent parallelism. CARLA[[2]] is flexible simulator dedicated to autonomous driving. AirSim [45]]] focuses on simulation for drone control. EmbodiedCity [IG]] provides an arena for embodied AI with realistic urban settings and enriched agent control. Grutopia [ offers a large-scale annotated scene dataset for general service robot tasks. AerialVIN [30]] serves as a highly photorealistic urban landscape for UAV navigation. MetaUrban [56) features wide range of micromobility tasks on the pedestrian, creating highly complex urban environment. However, the dynamic scene interaction of human agents and social navigation (e.g.. traffic rule following and pedestrian avoidance) has been much less explored.

---

Table 1:
Comparison of outdoor simulation platforms across key features. The Scenes section includes support for Procedural Generation (v: supported. x: not supported), and level of Photorealism. The Human section summarizes scene interaction capabilities Intera-)terratt) and the number of supported human Actions. Embodied Agents indicate whether robots (Rob.), humans (Hum.), and vehicles (Veh.) are supported. Multi-agent denotes support for asynchronous multi- agent control. Full comparisons with other types of simulators in Table i in Appendix Aw
Scones
Human
Embodied Agents
Simulator
Mulli-agent
Procedural
Scene-Interact. Interact.
CARLA IPC
SirSim
MetaUrben
Emhhxiiddiit [G]]
Grutopin [-]]
X
AerialVLN
X
X
X
X
UnrealZoo [X]
<
10
SWR (Ours)
V
26
Our simulator places the robot in a photorealistic, dynamically populated urban environment, evaluating not only multimodal instruction following but also the robots ability to adapt to real-time social cues and navigate amidst human agents a safe, rule-aware manner.
Instruction Following Benchmarks for Robot Navigation. Following complex instructions in the real world often requires understanding both language and vision cues in tandem. As summarized in Table H in Appendix B.ll prior benchmarks on vision-and-languag
navigation have tackled instruction following, but typically static indoor scenes or street panoramas using only textual descriptions(E]]]] For instance, Touchdown [8] explores city navigation via language alone in static street view setting, and R2R [2]] focuses on multi-step instructions in household tasks. These settings lack moving obstacles and additional visual guidance, making them only partial proxies for real-world urban navigation. In contrast, our multimodal robot navigation benchmark. SORLL MMNAV, challenges robot interpret and follow multimodal instructions (paired language instructions and visual hints) to reach the target location in a lcrge-scale, photorealistic, dynamic city environment.
Multi-Robot Collaboration Benchmarks. There have been many recent multi-robot collaboration benchmarks as summarized in Table EL but they do not evaluate multi-robot collaboration and communication for exploration and navigation large urban environments. RoCo [33] studies cooperative manipulation
tabletop tasks, without the need for environmental exploration. DOROTHIE [[2] introduces spoken dialogue for an ego vehicle, yet involves just one controllable agent. DriVLMe [20] simulates city-scale traffic with many vehicles, but lacks any verbal communication between robots. RobotSlang [ evaluates robot communication but is restricted to small environments. Where Are You? [[ frames localization two-party dialogue, but the robots do not physically navigate to meet each other through a large environment.
Sioooldd-Robotics
Built upon SimWorld [А SimWorld-Roboties (SWR) introduces key extensions including procedural city generation. a traffic system, and support for an additional embodied agent: the quadruped robot. We begin by describing how SWR procedurally generates diverse and scalable urban environments with varying specifications. We then introduce the embodied agents supported in these environmentsvehicles,, humans, and robotsand explain the logic behind asynchronously controlling multiple agents. Finally, we outline the rule-based system governing background pedestrians and traffic dynamics. Details of SWR can be found in Appendix A
3.1 Procedural City Generation
The Procedural City Generation pipeline SWR designed to synthesize realistic, structured urban environments from minimal specifications, supporting tasks such as autonomous driving, pedestrian navigation, and multi-agent simulations. As shown in Figure the pipeline follows four stages: road, building, street element, and traffic element generation. It begins by creating a road

---

Road Generation
Building Generation
Street Element Generation
Traffic Element Generation
metadeta "roads_number": 30,
metadata [{ guuldingssnumbber
metadaaa -( "selected_ detaiils.
=efffii_mttddtaa {
"pedetrainn max_vel"; 150.
"roads_length". 200,
(*B1%: "82": O.
["Tree", "Scooter',
"vehicle_mas sashicle_ma__veee mex_vel"" 500
"roads_width".
"distance, offset": 10
"area. width": "density": 0.13
vechtle__mxx_tteriing 0.15
Road Nodes Generation
Building Nodes Generation
Detail Nodes
Traffic Nodes Generation
Generation
Obsraaiks
Anea
TraffiAros
Procedural City Generation. SWR receives a user's specification and modularizes the process into road, building, details, and traffic elements generation.
Figure 3:
network through priority queue-based growth strategy that balances branching and depth, while ensuring plausible layouts via road-end attachment and intersection validation. Buildings are then placed along roads using collision-aware sampling and greedy gap-filling to maximize coverage and maintain uniformity. Next, contextual street elementstrees, cones, benches, and parked vehiclesare placed around buildings and sidewalks with basic accessibility constraints. Finally, dynamic alements-including
traffic elements- including vehicles and pedestrians- are integrated into the environment to support research on traffic dynamics, agent-based behavior modeling, and realistic navigation scenarios. detailed description of the city procedural generation pipeline is provided in Appendix A.G.
3.2 Embodied Agents
SWR supports three types of embodied agentshumans, vehicles, and robots. Unlike synchronous designs [38] where agents must wait for others to complete their actions, SWR allows asynchronous control, enabling each agent act independently. To better reflect real-world conditions and support diverse tasks, SWR also provides rich action spaces for different agents and flexible observation spaces. The creation of embodied agents in Appendix A.N.
56, Diitthtt focus on a specific type of embodied agent-such autonomous vehicles, robots, or humans and mainly support control over that particular agent type, our simulator simultaneously supports all three major categories. This unified design enables the development of broader and more diverse embodied AI tasks within a single environment. SWR, we have included all three types of embodied agents. For the robots, have two kinds: the scooter and the quadruped robot.
Types of Embodied Agents. Unlike previous simulators
realistically model scenarios where multiple agents act independently and simultaneously, SWR uses asynchronous multi-agent control framework. Each agent receives its observation from a centralized buffer and can submit an action only when marked available. The buffer updates at fixed interval (default: 0.01 seconds), checking for actions from available agents and updating their availability status. Valid actions are executed concurrently, after which all agents become unavailable until their actions complete. Once finished, agents are marked as available again and receive updated observations. The control pipeline is illustrated in Figure in Appendix A
Asynchronous Multi-agent Control.
Observation Space. SWR provides three primary types of visual observations: RGB images, depth images, and semantic segmentation masks, as illustrated in Figure & in Appendix A44. In addition to these, SWR also offers ground-truth language descriptions and 3D bounding boxes for objects present in the environment.
5

---

Bubrosk Type: Oftentationnll gromnn
Type Turning imfrrection
Language Instrution
Visual Hert
Languago Instruction
Visut Heet
sag rid Fespett
Hoopla Jenias alove
KONSACtION The
supportod
phlera wor 19g1t
Type: Reach
Langunge Instuution
Vacal Fine
and mooet
Lnctyra reo oround
son lige basdings Aht prier pi
rou nghtweh uage
ENtINes NATOORNG
Figure Illustration of a multimodal robot navigation task.
acceleration, braking, and steering. Each action is continuous within a set range, allowing flexible controle.g. higher acceleration leads faster speeds, and larger steering values result in sharper turns. For robot control, actions include continuous translation (forward, backward. left, right) and free-angle rotation, enabling flexible movement and orientation, as illustrated in Figure & in Appendix A.S. Human agents can perform navigation actions (movement, turning) and interaction actions relevant to urban scenarios, grouped into: (1) human-object (e.g.. pick up/drop off objects, sit/stand), (2) human-vehicks (e.g., drive, enter/exit, open/close trunk), and (3) numan-human (e.g., wave, argue, point).
Action Space. SWR supports three types of continuous vehicle control:
3.3 Pedestrian and Traffic Simulation
The traffic simulation in SWR creates dynamic urban scenarios by orchestrating vehicle and pedestrian movement across a generated city map. It supports route assignment, intersection control, and pedestrian flow simulation, running on a fixed-time update loop for consistent real-time updates. Vehicle motion is governed by a feedback-basee model using a PID controller, with empirically tuned parameters for realistic acceleration, braking, and turning dynamics [2I]]. Pedestrians follow a lightweight model, adjusting orientation incrementally toward their goals based on angular differences. To simulate realistic patterns. SWR uses probabilistic routing strategy at intersections. where agents select paths based on predefined probabilities. This stochastic behavior introduces natural variability and enhances scene diversity. Details of predestrain and traffic simulation can be found in Appendix A.
4 Multimodal Robot Navigation Benchmark
We propose a novel multimodal robot navigation benchmark, SIMWORLD-MMNAN where a robot must follow multimodal instructionspaired language and visual hintsto reach target in a large-scale, photorealistic, dynamic city environment (Figure ED) This requires grounding verbal and visual references in the 3D environment based on the robots observations, while adapting to real-world complexities like traffic and pedestrians. SWR enables scalable evaluation of such tasks through (1) procedurally generated instructions with annotated, richly detailed city layouts, (2) photorealistic rendering of visual hints, and (3) simulation of pedestrians and traffic. We highlight core challenges of dynamic grounding: aligning language with visual targets, navigating around moving entities, and obeying environmental rulesall within an open- world urban setting that surpasses prior benchmarks. can be found in Appendix C.
Details of
4.1 Setup
Task Definition. Each task consists of series of multimodal instructions I1... that the robot must interpret and execute. As illustrated in Figure A. each instruction Jn includes a natural language instruction and a visual hint illustrating what the robot can expect to see after reaching the goal described by the language instruction. These two modalities together guide the robot toward a specific location and/or orientation. Each instruction is associated with ground-truth goal ga. At

---

Table 2: Experimental results on the WORLD- benchmark (easy task set). The numbers in parentheses indicate the improvement after finetuning.
Models
SR%t
Subtask SR% 1
Distance Progress % 1
Proprietary Models
GPT-4o
0
33.07
15.60
Gemini 2.5 Flash
0
37.06
31.29
Reasoning Models
GPT-o3
5.0
42.50
38.43
GPT-o3-prro
8.3
46.35
39.46
Open-sonne sourced Models
Qwen 2.5 7B
0
16,86
7.82
QwenVL 2.5 72B
0
23,80
17.50
Gemma 27B
15.36
6.83
InterVL 78B
0
18.31
9.34
Fine-tuned Models QwenVL 2.5 7
4.0 (+4.0)
52.45 (+35.59)
53.63 (+45.81)
Hybrid Baselines HybridGPT
32.53
27.24
RL-based Baselines VLA-RL
0
28.37
22.79
any given timestep, only one instruction is presented the robot, which must successfully complete the current instruction (i.e., reaching 9k)) before receiving the next instruction 4++1.
In real-world urban navigation, instructions typically fall into of four categories: Orientation Alignment, Move Along the Road, Turn at the Intersection, and Reach Destination. As shown in Figure HL complete sequential instruction-followe following task is formed by chaining together multiple different types of instructions. Details of procedural task generation can be found in Appendix CK
To closely mirror real-world navigation challenges, the environments simulated incorporate both statie obstacles (e.g., buildings, barriers) and dynamic agents (e.g., pedestrians, vehicles) whose trajectories are not known in advance. Task difficulty is determined by the complexity of the environment, and we define two levels: an easy task includes no obstacles (static objects outside of buildings, pedestrians or vehicles), and hard task adds both static object obstacles and dynamic pedestrians and vehicles.
Obsevation and Action Space. Apart from the subtask description and expected visual hint, the robot has access the egocentric RGB image, segmentation image, and depth image. Ground truth orientation is provided, assuming a built-in compass in the robot. The robot's action space includes moving in four directions, as well as turning. can also stay still and confirm task completion. SWR also supports active perception by providing different viewpoint images.
Sim World-20k. Based on SWR, we construct the Soold--d- dataset, including 100 training maps with average area of 2 km and 200 oracle trajectories generated by A* [[С] for 200 training tasks, whose average length is greater than 2.5 km. Each trajectory contains over 100 steps, forming training dataset of 20K steps. Details of it can be found in Appendix C.
Statisties. For both training and evaluation, we synthesize 100 distinct worlds. With them, we create 200 easy tasks and 200 hard tasks. Each task has 2-4 instructions. On average, each task requires traveling 500 meters over 250 steps. To ensure generalization, 33% of the buildings the testing set are exclusive and unseen during training.
Evaluation Metrics. We evaluate navigation performance in the easy setting through three key metrics: (1) Success Rate (SR) [M]] measures the percentage of goal arrivals, (2) Subtask SR [46] tracks the ratio of completed subtasks, (3) Distance Progress [SID] evaluates instruction-folloving by the relative reduction in distance to the goal. The hard evaluation setting, which introduces
comprehensive traffic system and pedestrians, requires three additional safety metrics: (1) Static
Collision counts collision between the robot and immovable objects like buildings and trees, (2)

---

Table Experimental results on the SiMWORLD-MMNAN benchmark (hard task set).
Models
% Stat. Coll. Dyn. Coll.) Red Light Viol.; Subtask SR %1 Distance Progress %
GPT-4o
1.92
10.37
3.02
34.38
24.83
Gemini 2.5 Flash
3.21
4.29
7.875
32.29
29.87
Qwen 72B
5.0
11.73
2.86
23.86
21.97
Dynamic Collision counts collisions with moving elements, such as pedestrians and vehicles, (3)
Traffic Light Violation records the number of times the robot fails to adhere to traffic signal regulations.
Table 4: Most common failure modes in SIMWORLD-MMNAY.
Subtask
Failure Mode
Frequency (%
Misestimate the distance the intersection
53.33
Moving to Intersection
Fail to detect intersection Misidentify the reference landmark
28.33
18.33
Misinterpret turning pattern
42.86
Turning
Misunderstand history status summary Fail to detect upfront buildings
42.86
14.29
Fail to match the landmark in different perspective Stop early to face the landmark Fail align the landmark
60.00
Reaching Destination
30,00
10.00
Baselines. Inspired by the recent success of large lasion-la models (VLMs) on navigation
tasks [54]], we evaluate multiple recent VLMs as backbones using ReAct [57]] including GPT-4o, GPT-o3, GPT--33pro [3G], Gemini 2.5 Flash [49], Qwen-VI. VI. 2.5 [3]. Gemma 3 [50] and Intern VI.
[9]. Additionally, we finetune Qwen V1.2.55 2..2.---B on SimWorld-20k We also test hybrid baseline,
HybridGPT, where GPT-4o is used as a high-level decision maker and A [[8] is used as a low-level
motion planner. For RL baselines, we train multimodal policy model, VLA-RL, with DeBERTa-
v3 [T*]] as language encoding and DINOv2 [BZ] visual encoding, following VLN-CE [24]]. We
include more baseline implementation details in Appendix (K
4.2 Results
Zero-shot VLMs. As shown in Table EL among zero-shot ReAct models, Gemini 2.5 Flash achieves the highest progress score. GPT-4o exhibits mismatch between its distance progress and subtask completion rate, primarily due to its inability to detect termination conditions, often overshooting the goal and yielding zero distance progress. VWe2.V..2.57792 ranks highest among open-source models, while Qwen VL2.5-7B performs comparably to significantly larger models.
Finetuned Models. After fine-tuning, QwenVL2..57B shows substantial improvements across all metrics and is the only model to achieve a non-zero full task success rate. However, the absolute success rate remains low, partly because the training set is grounded oracle action traces, which limits robustness. Incorporating reinforcement learning or corrective demonstrations could further enhance performance.
zero-shot non-reasoning models score zero SR, highlighting fundamental capability gap. Case studies in Appendix E. suggest that these models often fail in task completion due to insufficient instruction grounding or inability to handle long-horizon dependencies. However, the results for reasoning models indicate that improved reasoning abilities boost performance. In our experiment, the reasoning models show improved depth estimation and destination alignment, which further demonstrates the importance of visual and spatial reasoning in our benchmark.
Reasoning Models.
Other Baselines. The hybridGPT handles local turning more stably than zero-shot GPT-4o, but lacks fine-grained control, making overall performance more sensitive to GPT-4o's first-attempt accuracy. The RL baseline, VLA-RL, fails to outperform zero-shot LLMs, indicating the difficulty of our benchmark, where sparse reward signals and visually complex spatial reasoning pose challenges for conventional vision encoders.

---

At Initial Status:
Final Goal:
Landimark memony main robor's
Meet up the city environment
aa currentlyy white building with red flyg
Fallwe Raners
You should kaep moving forward
Main Robot
Maln Robott
Man Putoot's
Current View
Figure 5: Illustration of multi-robot search task.
Hard Setting. We further evaluated realistic obstacle avoidance and traffic rule obedience on models that performed relatively well on the easy setting. As Table B shows, Gemini 2.5 Flash performs better on avoiding pedestrians and vehicles; however, its red light violation count is higher, not due failure stop at red lights, but because the agent often freezes after detecting red signal, even when already within the intersection. This indicates that there is still room for improvement in pragmatic reasoning and safety alignment under real-world conditions. Reasoning models, due to their inference latency, are not suitable for this setting of real-time real-time traffic avoidance.
Ablation Test. We construct an ablation test using GPT-4o the backbone, and we find that the explicit ReAct framework and segmentation provide the most significant marginal improvements among all the components. Details can be found in Table I in Appendix C.G.
Failure Analysis. We summarize typical failure modes of VLMs as follows, with specific qualitative examples detailed in Appendix DIll (I) Visual Grounding. The grounding of VLMs hinder their performance. They fail to recognize the intersection, which is vital in our setting. The relatively low perspective of the robot dog add to the difficulties.
(2) Spatial and Embodied Reasoning, VLMs do not yet have good 3D spatial reasoning capacity. They cannot robustly estiamte how close the obstacle. (3) Pragmatic Thinking. To emulate real-world
robot is to certain intersection textual navigation, our instructions use high-level phrases like "turn right at the intersection" without detailing the specific turning and crossing actions required. Most models fail to interpret such ambiguity correctly. (4) Memory and Planning. VLMs sometimes fail to adapt the information in memory the current situation. The frequency for each common failure mode is shown in Table
5 Multi-Robot Search Benchmark
In many real-world urban applications like search and rescue, multiple robots must collaborate. In large, unfamiliar environments, this requires localizing one another and meeting convenient location. Such multi-robot search is foundational for effective collaboration, but presents unique challenges: (i) each robot has partial, egocentric perception, (ii) the environment dynamic and safety-critical, and (iii) coordination depends grounded natural-language dialogue. As shown in Table gl (Appendix BB.2), prior benchmarks only address parts of this problem. We introduce --MS benchmark to fill this gap.
5.1
Setup
Task Definition. There are two robots, a main robot and a follower robot. The main robot has explored the city, so it has the memory of map and images of landmarks (typically over 20 landmarks) the city. However, the follower robot is new to the city and does not have such information. Neither robot knows the other's location. Their goal is to meet each other as soon as possible by physical navigation as well as verbal communication. when at least one of the robots confirms that it can see the other robot in its egocentric view.
task is considered successful

---

Table 5: Experiment results on SIMWORLD-MRS benchmark.
Models
Method
CSR
Task Progress% ?
Proprietary Models
GPT-4o
Oracle Planner
76.90
Gemini 2.5 Flash
Oracle Planner
54.55
75.84
GPT-4o
RoCo
33.33
22.93
Open-sourced Models
QwenVL 2.5 72B
RoCo
11.11
35.94
Both robots have similar observation and action spaces as in the Sim HimWord--MMaa benchmark. Additionally, both robots can send natural language messages to each other. Each robot can send confirmation signal whenever it believes that it has seen the other rohot.
We provide more details of the benchmark in Appendix D
Statisties. For evaluation in the multi-robot search benchmark, we construct 100 unique urban environments, each covering an area of 2.5 km 2 each environment, distinguishable landmarks are selecteddistributeed across all the city blocksas the main robot's memory of the city. On average, the initial distance between the two robots spawning locations is 576 m, requiring 287 steps for an oracle planner to complete the task.
Evaluation Metries. We assess multi-agent navigation performance through two principal metries: (1) Collaborative Success Rate (CSR) [3]]] measures the percentage of tasks completed successfully before meeting; (2) Task Progress averages, across robots, the fraction of their shortest- path distance that is covered by the end of an episode.
Baselines. Following the collaboration paradigm of RoCo [[X]) enable robots to discuss joint plan for concrete rendezvous behavior using a VLM. The follower robot will first describe its location using language for the main robot to localize it. Afterwards,
robots will confirm a plan, where the main robot will describe paths to a meeting location for the follower robot to follow, while the main robot will plan its own path to reach the meeting location. In SIMWORLE MRS, we first evaluated GPT-4o and Gemini 2.5 Flash with oracal planner and then picked the best performing VLM (GPT-4o) and paired with RoCo. We include more implementation details the Appendix D
5.2 Results
Table B summarizes baseline results on our multi-robot search task, with specific qualitative examples detailed Appendix Ex GPT-4o with the oracle planner achieves the highest CSR (52%) and task progress (68.44%) by combining precise landmark localization with optimal A planning, showing upper-bound performance with full map access. The RoCo policy lets the follower describe its view, which the map-aware robot localizes via VLM-based retrieval. This one-shot communication enables concurrent movement and more realistic coordination. However, converting rendezvous plans into language introduces ambiguity and exccution noise, often causing path deviations. Without iterative replanning, grounding or control errors can't t be corrected. Consequently, GPT-4o under RoCo achieves only 33.33% CSR and 22.93% task progresssignfficanty lower than the oracle baseline.
6 Conclusion
We have created novel embodied AI simulator, SimWorld-Robotics (SWR), for synthesizing photorealistic and dynamic urban environments. It can procedurally generate infinite photorealistic urban environments. Additionally, can populate the environments with pedestrians, vehicles, and robots. By leveraging these features, we have built two new robot benchmarks. One focuses on multimodal robot navigation (SIMWORLD-A -MMAA))),,, and the other evaluates multi-robot collaboration in searching tasks (SIMWORLD-MEL limits in strong VL.M-aseed baselines. Our evaluation also demonstrated the value of finetuning VLMs on large-scale training sets synthesized in our simulator.
Our experimental results reveal significant
10

---

Limitations and Future Work. Our current simulator focuses only on outdoor environments. The action space of the human agents, though more diverse than prior simulators, is still limited. In the future, we intend to scale up the action space by leveraging recent human body motion generation models. We also plan to incorporate indoor scenes into SWR.
References
[1] Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, and Amir R. Zamir. On evaluation of embodied navigation agents, 2018. URL / 06757
[2] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. Vision-andd-anguaage navigation: Interpreting visually- grounded navigation instructions in real environments. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3674-3683, 2018.
[3] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. 2025. URL
13923.
[4] Shurjo Banerjee, Jesse Thomason, and Jason J. Corso. The robotslang benchmark: Dialog- guided robot localization and navigation, 2020. URL
12639.
[5] Abhijat Biswas, Allan Wang, Gustavo Silvera, Aaron Steinfeld, and Henny Admoni. Socnavbench: A grounded simulation testing framework for evaluating social navigation, 2021. URL
00047.
[6] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. 2016. URL https://arxw..rr//ab//
[7] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias NieSSner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments, 2017. URL https://arxiv orgg/bs/1709.06t 06158
[8] Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. Touchdown: Natural language navigation and spatial reasoning in visual street environments, 2020. URL
nttps://arx...or//////111 12354.
[9] Zhe Chen, Weiyun Wang, Yue Cao. Yangzhou Liu, Zhangwei Gao, Erfei Cui. Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuchui Wang. Qingyun Li, Yimin Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang. Conghui He, Botian Shi, Xingcheng Zhang, Han Lv, Yi Wang, Wenqi Shao, Pei Chu, Zhongying Tu, Tong He, Zhiyong Wu, Huipeng Deng. Jiaye Ge, Kai Chen, Kaipeng Zhang, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. Expanding performance boundaries of open- source multimodal models with model, data, and test-time scaling, 2025. URL https://arxiv.
org/abs/2412 Faaabs/2411.0555 052/1.
[10] Matt Deitke, Winson Han, Alvaro Herrasti, Aniruddha Kembhavi, Eric Kolve, Roozbeh Mottaghi, Jordi Salvador, Dustin Schwenk, Eli VanderBilt, Matthew Wallingford, Luca Weihs, Mark Yatskar, and Ali Farhadi. Robothor: An open simulation-to-real embodied ai platform, 2020. URL
nttps://arxiv. www.arwwww.g.....aaaoooooe..o.rw www.or/g///b//eeee /btp///..///////////
[11] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms, 2023. URL https://arxxw.org/abb////
14314.
[12] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In Proceedings of the 1st Annual Conference on Robot Learning, pages 1-16, 2017.
11

---

[13] Yue Fan, Winson Chen, Tongzhou Jiang, Chun Zhou, Yi Zhang. and Xin Eric Wang. Aerial vision-and -ana-llodddddg navigation, 2023. URL ftttps://arxw..org/abb////// 12219
[14] Chuang Gan, Jeremy Schwartz, Seth Alter, Damian Mrowca, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, Megumi Sano, Kuno Kim, Elias Wang. Michael Lingelbach, Aidan Curtis, Kevin Feigelis, Daniel M. Bear, Dan Gutfreund, David Cox, Antonio Torralba. James J. DiCarlo, Joshua B. Tenenbaum, Josh H. McDermott, and Daniel L. K. Yamins. Threedworld: A platform for interactive multi-modal physical simulation, 2021. URL https://arxiv. wwtrrkiv.orggbbs/////e 04954.
[15] Chuang Gan, Siyuan Zhou, Jeremy Schwartz, Seth Alter, Abhishek Bhandwaldar, Dan Gutfireund Daniel L. K. Yamins, James J DiCarlo, Josh McDermott, Antonio Torralba, and Joshua B. Tenenbaum. The threedworld transport challenge: A visually guided task mndtmoonnn planning benchmark for physically realistic embodied ai, 2021. URL https//arxxw.org/aa5///
14025
[16] Chen Gao, Baining Zhao, Weichen Zhang, Jinzhu Mao, Jun Zhang, Zhiheng Zheng, Fanhang Man, Jianjie Fang, Zile Zhou, Jinqiang Cui, Xinlei Chen, and Yong Li. Embodicdcity: A benchmark platform for embodied agent in real-world city environment, 2024. URL https://arxiv. wrwp//wwworg/bsss/..//r 2410.09504.
[17] Meera Hahn, Jacob Krantz, Dhruv Batra, Devi Parikh, James M. Rehg, Stefan Lee, and Peter Anderson. Where are you? localization from embodied dialog, 2021. URL https://arxıv.org abs/2011.08277.
[18] Peter Hart, Nils Nilsson, and Bertram Raphael. formal basis for the heuristic determination of minimum cost paths. IEEE Transactions Systems Science and Cybernetics, 1968. doi: 10. 10.1109/tscc.19 196..9001...66... URL https: org/ 10. 1109/tssc. 1968 300136.
[19] Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra- style pre-training with gradient-disentangled embedding sharing, 2023. URL https://arxiv.orgg
abs/2T11. 09543.
[20] Yidong Huang, Jacob Sansom, Ziqiao Ma, Felix Gervits, and Joyce Chai. Drivlme: Enhancing Ilm-based autonomous driving agents with embodied and social experiences, 2024. URL https:
wwwxii.orggabs/////. atrrg//bb//24.... 03008.
[21] Harshit Jain and Priyal Babel. A comprehensive survey of pid and pure pursuit control algorithms for autonomous vehicle navigation, 2024. URL https://arxxv..om orgg www../org/bbb///////// abs//409./22.... 09848.
[22] Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts, 2023, URL attps://arxivv
[23] Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, Aniruddha Kembhavi, Abhinav Gupta, and Ali Farhadi. Ai2-thor: An interactive 3d environment for visual ai, 2022. URL https://arxiv.
brg/abs/1/12.05//
[24] Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, and Stefan Lee. Beyond the nav- graph: -ad---ng navigation in continuous environments, 2020. URL https://arxıv. mrg/abs/2004. 02857.
[25] Linh Kästner, Volodymyir Shcherbyna, Huajian Zeng, Tuan Anh Le, Maximilian Ho-Kyoung Schreff, Halid Osmaev, Nam Truong Tran, Diego Diaz, Jan Golebiowski, Harold Soh, and Jens Lambrecht. Arena 3.0: Advancing social navigation in collaborative and highly dynamic environments, 2024. URL htttp::/aarxiv.or//abt 2255/2466..0000 2406.00837
[26] Chengshu Li, Fei Xia, Roberto Martín-Martinn Michael Lingelbach, Sanjana Srivastava, Bokui Shen, Kent Vainio, Cem Gokmen, Gokul Dharan, Tanish Jain, Andrey Kurenkov, C. Karen Liu, Hyowon Gweon, Jiajun Wu, Li Fei-Fei, and Silvio Savarese. igibson 2.0: Object-centric simulation for robot learning of everyday houschold tasks, 2021. URL https: / /awxiv..vvv. wwt./r/////a/ 03272.
12

---

[27] Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín- Martín, Chen Wang, Gabrael Levine, Michael Lingelbach. Jiankai Sun, Mona Anvari, Minjune Hwang, Manasi Sharma, Arman Aydin, Dhruva Bansal, Samuel Hunter, Yyu-Young& Kim, Alan Lou, Caleb Matthews, Ivan Villa-Renteria, Jerry Huayang Tang, Claire Tang, Fei Xia, Silvio Savarese, Hyowon Gweon, Karen Liu, Jiajun Wu, and Li Fei-Fei. Behavior- 1k: A benchmark for embodied with 1,000 everyday activities and realistic simulation. In Karen Liu, Dana Kulic, and Jeff Ichnowski, editors, Proceedings of The 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 80-93, PMLR, 14-18 Dee 2023. URL https: pppoceedings. mlr. press/v205/ http/i222aaahh.. .htll
[28] Heng Li, Minghan Li, Zhi-Qi Cheng, Yifei Dong, Yuxuan Zhou, Jan He, Qi Dai, Teruko Mitamura, and Alexander G. Hauptmann. Human-aware vision-and- language navigation: Bridging simulation to reality with dynamic human interactions. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 119411-119442. Curran Associates, Inc., 2024. URL https::/ neurips cc/ paper fiappppri1lss//ppper/222/4 Ler_saann__Becchnrss____eeee
[29] Quanyi Li, Zhenghao Peng, Lan Feng, Qihang Zhang. Zhenghai Xue, and Bolei Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning, 2022.
URL
[30] Shubo Liu, Hongsheng Zhang, Yuankai Qi, Peng Wang. Yaning Zhang, and Qi Wu. AerialvIn: aaoon-andlannguaee navigation for uavs, 2023. URL
http://aaxxiv..g/bbbbss// abrr///bb//33/.... 06735.
[31] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-comppttiivee environments. In NeurlPS, 2017.
[32] Ziqiao Ma, Ben VanDerPloeg, Cristian-Paut Bara, Huang Yidong, Eui-In Kim, Felix Gervits, Matthew Marge, and Joyce Chai. Dorothie: Spoken dialogue for handling unexpected situations in interactive autonomous driving agents, 2022. URL https://arxiv org abwp/2//2210. 125TT]
[33] Zhao Mandi, Shreeya Jain, and Shuran Song. Roco: Dialectic multi-robot collaboration with large language models, 2023. URL
[34] Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard, Calvin: A benchmark for language-conditiongg policy learning for long-horizon robot manipulation tasks, 2022. URL http:///axxiv.org/bbs////// 032227
[35] Tongzhou Mu, Zhan Ling, Fanbo Xiang, Derek Yang, Xuanlin Li, Stone Tao, Zhiao Huang, Zhiwei Jia, and Hao Su. Maniskill: Generalizable manipulation skill benchmark with large-scale demonstrations, 2021. URL http:///axxx...r/.bb///////11
[36] OpenAI. Gpt-4o system card, 2024. URL https://arxix org/ horg/bs//244 22015226661122666
[37] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po-Yao Huang, Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2024. URL
[38] Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs, 2018. URL https://arxiv.
org/abs/1805.0d 07011.
[39] Xavier Puig, Tianmin Shu. Shuang Li, Zilin Wang. Yuan-Hong Liao, Joshua B. Tenenbaum, Sanja Fidler, and Antonio Torralba. Watch-and-helpy A challenge for social perception and human- ai collaboration, 2021. URL
http:///.ggaa//////01/ 09890,
[40] Xavier Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Tsung- Yee Yang, Ruslan Partsey, Ruta Desai, Alexander William Clegg, Michal Hlavac, So Yeon Min, Vladimír Vondru, Theophile Gervet, Vincent- Pierre Berges, John M. Turner, Oleksandr Maksymets, Zsolt Kira, Mrinal
13

---

Kalakrishnan, Jitendra Malik, Devendra Singh Chaplot, Unnat Jain, Dhruv Batra, Akshara Rai, and Roozbeh Mottagii. Habitat 3.0: A co-habitat for humans, avatars and robots, 2023. URL https:
//arxiv.org www.:org///ao www.orggg///.... abs/231/2331.. 13724.
[41] Claudia Pérez-D D' Arpino, Can Liu, Patrick Goebel, Roberto Martin-Marttnn and Silvio Savarese. Robot navigation in constrained pedestrian environments using reinforcement learning, 2020. URL
08608.
[42] Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang, Chunhua Shen, and Anton van den Hengel. Reverie: Remote embodied visual referring expression in real indoor environments, 2020. URL http::////axxvvoog//abbss./
10151.
[43] Weichao Qiu, Fangwei Zhong, Yi Zhang, Siyuan Qiao, Zihao Xiao, Tae Soo Kim, and Yizhou Wang. Unrealev: Virtual worlds for computer vision. In Proceedings of the 25th ACM International Conference on Multimedia, MM '17, pages 1221-1224, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 978 1450349062. doi: 10.1145/312226631129333 URL https:
7/do1.org/100 10. 7145/3123336666 3129396.
[44] Jiawei Ren, Yan Zhuang, Xiaokang Ye, Lingjun Mao, Xuhong He, Jianzhi Shen, Mrinaal Dogra, Yiming Liang, Ruixuan Zhang, Tianai Yue, Yiqing Yang, Eric Liu, Ryan Wu, Kevin Benavente. Rajiv Mandya Nagaraju. Muhammad Faayez. Xiyan Zhang, Dhruv Vivek Sharma, Xianrui Zhong, Ziqiao Ma, Tianmin Shu, Zhiting Hu, and Lianhui Qin. Simworld: An open- ended realistic simulator for autonomous agents in physical and social worlds, 2025. URL
https://arxtw.orr//bb////
[45] Shital Shah, Debadeepta Dey, Chris Lovett, and Ashish Kapoor. Airsim: and physical simulation for autonomous vehicles, 2017. URL https://arxxv.org/aah 85065.
visual
[46] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[47] Sanjana Srivastava, Chengshu Li, Michael Lingelbach, Roberto Martín-Martín, Fei Xia, Kent Vainio, Zheng Lian, Cem Gokmen, Shyamal Buch, C. Karen Liu, Silvio Savarese, Hyowon Gweon, Jiajun Wu, and Li Fei-Fei. Behavior: Benchmark for everyday houschold activities in virtual, interactive, and ecological environments, 2021. URL nttps://arxx orgg abs/2/08 03332.
[48] Andrew Szot, Alex Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir Vondrus, Sameer Dharur, Franziska Meier, Wojeiech Galuba, Angel Chang. Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, and Dhruv Batra. Habitat 2.0: Training home assistants to rearrange their habitat, 2022. URL
14405.
[49] Gemini Team. Gemini: A family of highly capable multimodal models, 2025. URL https: 7/arxiv wrtp/b/////./....// 11865.
[50] Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej. Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Ftienne Pot, Ivo Penchev, Gaäl Liu, Francesco Visin, Kathleen Kenealy, Lucas Beyer, Xiaohai Zhai, Anton Tsitsulin, Robert Busa-Fekete, Alex Feng, Noveen Sachdeva, Benjamin Coleman, Yi Gao, Basil Mustafa, lain Barr, Emilio Parisotto, David Tian, Matan Eyal, Colin Cherry, Jan-Thorsten Peter, Danila Sinopalnikov, Surya Bhupatiraju, Rishabh Agarwal, Mehran Kazemi, Dan Malkin, Ravin Kumar, David Vilar, Idan Brusilovsky, Jiaming Luo, Andreas Steiner, Abe Friesen, Abhanshu Sharma, Abheesht Sharma, Adi Mayrav Gilady, Adrian Goedeckemeyer, Alaa Saade, Alex Feng, Alexander Kolesnikov, Alexei Bendebury, Alvin Abdagic. Amit Vadi, András György, André Susano Pinto, Anil Das, Ankur Bapna, Antoine Miech, Antoine Yang. Antonia Paterson, Ashish Shenoy, Ayan Chakrabarti, Bilal Piot, Bo Wu, Bobak Shahriari, Bryce Petrini, Charlie Chen, Charline Le Lan, Christopher A. Choquette-Choo. CJ Carey, Cormac Brick, Daniel Deutsch, Danielle Fisenbud, Dee Cattle, Derek Cheng, Dimitris Paparas, Divyashree Shivakumar
14

---

Sreepathiballi, Doug Reid, Dustin Tran, Dustin Zelle, Eric Noland, Erwin Huizenga, Eugene Kharitonov, Frederick Liu, Gagik Amirkhanyan, Glenn Cameron, Hadi Hashemi, Hanna Klimezak- Pluciska, Harman Singh, Harsh Mehta, Harshal Tushar Lehri, Hussein Hazimeh, Ian Ballantyne, Idan Szpektor, Ivan Nardini, Jean Pouget-Abadie, Jetha Chan, Joe Stanton, John Wieting, Jonathan Lai, Jordi Orbay, Joseph Fernandez, Josh Newlan, Ju yeong Ji, Jyotinder Singh, Kat Black, Kathy Yu, Kevin Hui, Kiran Vodrahalli, Klaus Greff, Linhai Qiu, Marcella Valentine, Marina Coelho, Marvin Ritter, Matt Hoffman, Matthew Watson, Mayank Chaturvedi, Michael Moynihan, Min Ma, Nabila Babar, Natasha Noy, Nathan Byrd, Nick Roy, Nikola Momchev, Nilay Chauhan, Noveen Sachdeva, Oskar Bunyan, Pankil Botarda, Paul Caron, Paul Kishan Rubenstein, Phil Philipp Schmid, Pier Giuseppe Sessa, Pingmei Xu, Piotr Stanczyk, Pouya Tafti, Rakesh Shivanna, Renjie Wu, Renke Pan, Reza Rokni, Rob Willoughby, Rohith Vallu, Ryan Mullins, Sammy Jerome, Sara Smoot, Sertan Girgin, Shariq lqbal, Shashir Reddy, Shruti Sheth, Siim Pöder, Sijal Bhatnagar, Sindhu Raghuram Panyam, Sivan Higer, Susan Zhang, Tianqi Liu, Trevor Yacovone, Tyler Liechty, Uday Kalra, Utku Evei, Vedant Misra, Vincent Roseberry, Vlad Feinberg, Vlad Kolesnikov, Woohyun Han, Woosuk Kwon, Xi Chen, Yinlam Chow, Yuvein Zhu, Zichuan Wei, Zoltan Egyed. Victor Cotruta, Minh Giang, Phoebe Kirk, Anand Rao, Kat Black, Nabila Babar, Jessica Lo, Erica Moreira. Luiz Gustavo Martins, Omar Sanseviero, Lucas Gonzalez, Zach Gleicher, Tris Warkentin, Vahab Mirrokni, Evan Senter, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, Yossi Matias, D. Sculley, Slav Petrov, Noah Fiedel. Noam Shazeer, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Jean -aa-tttttstt Alayrac, Rohan Anil. Dmitry, Lepikhin, Sebastian Borgeaud, Olivier Bachem, Armand Joulin, Alck Andreev, Cassidy Hardin, Robert Dadashi, and Léonard Hussenot. Gemma 3 technical report, 2025. 19786.
Culliton,
URL
[51] Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. Vision-and-dialog navigation, 2019. URL https://arxivv
04957
[52] Nathan Tsoi, Alec Xiang, Peter Yu, Samuel S. Sohn, Greg Schwartz, Subashri Ramesh, Mohamed Hussein, Anjali W. Gupta, Mubbasir Kapadia, and Marynel Vázquez. Sean 2.0: Formalizing and generating social situations for robot navigation. IEEE Robotics and Automation Letters, pages 1-8, 2022. doi: 10. N0.110//RAA.022.31996
[53] Hanqing Wang, Jiahe Chen, Wensi Huang, Qingwei Ben, Tai Wang, Boyu Mi, Tao Huang, Siheng Zhao, Yilun Chen, Sizhe Yang. Peizhou Cao, Wenye Yu, Zichao Ye, Jialun Li, Junfeng Long. Zirui Wang, Huiling Wang, Ying Zhao, Zhongying Tu, Yu Qiao, Dahua Lin, and Jiangmiao Pang. Grutopia: Dream general robots in a city at scale, 2024. URL https://arxixl org/abs/
2407 10941
[54] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's perception of the world any resolution, 2024. URL https://arxv..or///a////a/ 12191.
[55] Edwin B. Wilson. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(155):209-212. 1927. ISSN 01621459, 1537274X. URL http://www.sstr.orrrr
[56] Wayne Wu, Honglin He, Jack He, Yiran Wang, Chenda Duan, Zhizheng Liu, Quanyi Li. and Bolei Zhou. Metaurban: An embodied ai simulation platform for urban micromobility, 2024. URL
https://arxa..or/aabs///e
08725.
[57] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023. URL https://arxıv.
org/abs/2210.03b2
[58] Yifan Yin, Zhengtao Han, Shivam Aarya, Jianxin Wang, Shuhang Xu, Jiawei Peng, Angtian Wang, Alan Yuille, and Tianmin Shu. Partinstruct: Part-level instruction following for fine-grained robot manipulation, 2025. URL
15

---

[59] Shengqiang Zhang, Philipp Wicke, Lütfi Kerem Şenel. Luis Figueredo, Abdeldjallil Naceri, Sami Haddadin, Barbara Plank, and Hinrich Schütze. Lohoravens: long-horizon language- conditioned benchmark for robotic tabletop manipulation, 2023. URL /ttps://arxiv.
2310.12020.
[60] Fangwei Zhong, Kui Wu, Churan Wang, Hao Chen, Hai Ci. Zhoujun Li, and Yizhou Wang. Unrealzoo: Enriching photo-realistic virtual worlds for embodied ai. In Proceedings of the JEEE/CVF Interaational Conference on Computer Vision (ICCV), pages 5769-5779, October 2025.
[61] Qinhong Zhou, Hongxin Zhang, Xiangye Lin, Zheyuan Zhang, Yutian Chen, Wenjun Liu, Zunzhe Zhang, Sunli Chen, Lixing Fang. Qiushi Lyu, Xinyu Sun, Jincheng Yang. Zeyuan Wang, Bao Chi Dang, Zhehuan Chen, Daksha Ladia, Jiageng Liu, and Chuang Gan. Virtual community: An open world for humans, robots, and society, 2025. http::/tp:/rrrt wwt..orgg/ass//0..1122 14893.
16

---

A Sim World-Robotics Details
A.1 Assets
Building and Detail Assets We utilize wide range of high-fidelity buildings and street-level details, all sourced from Unreal Engines official marketplace, Fab.com. All assets are used in compliance with their respective licenses and terms of use. The ground-truth attribution for building assets are made into word cloud, shown in Figure f
detailing vertical
landmark glass modern
awning
two outside grid
g
elements
level
signages
symmetrical
lower ornate
facade
prominent
entrances
accents
gold
shutters
hand
tinted
tall awnings
near
design
colorful
wall
three
floor
covered
long billboards
gray
medium
around
signage
decorative
hanging curved grand
placed base
steel
upper
arched
large
doors
roller
red
high
residential
windows
horizontal
bins several
signs
metal
concrete
panels
dark
ground
stone
white
reflective
stacked
green
commercial
rectangular
apartment
floors
yellow
uniform
black
boxes
brown
pink
office
brick
low
frames
blue
beige
light
framed
multiple
columns
entrance
small
sleek
billboard
striped
characters
front
handcart
door
architectural
pattern
framing
Figure 6: Building Attribution SWR Each building asset SWR paired with ground-truth
description initially generated by language model and later verified and corrected by human in our team.
Embodied Agent Assets Quadruped robot models are sourced from CGTrader and integrated into SWR in compliance with the platform's 's licensing terms. Vehicles are adapted from the official Unreal Engine package Ci tySampleVehicle with customized blueprint logic to support the specific requirements of our simulation. Human agents are created using Unreal Engines MetaHuman framework, providing diverse, realistic character models.
A.2 Full Comparison of SWR with Prior Simulators
Table 6: Comparison of simulation platforms across key features. The Scenes section includes environment Type (U: Urban, I: Indoor), support for Procedural Generation (V: supported, x: supported), and level of Photorealism. The Human section summarizes scene interaction capabilities
and the number of supported human Actions. Embodied Agents indicate whether robots (Rob.), humans (Hum.), and vehicles (Veh.) are supported. AMC denotes support for asynchronous multi-agen agent control.
Scenes
Human
Embodied Agents
Simulator
AMC
Type
Procedural Geucration
Pbotorcalistic
Scene-Interact.
Actions
Veh.
Matterportdd [M]
Sean2.0 (522)
Arena3.0
AI2THOR [0]]
SocNavBeechh
X
X
X
Habitat 3.0
VirualHame 2.0 [5]
V
X
X
BEHAVIOR [X
X
X
X
CARLA [TC
X
X
AirSin [X
X
MetaUrben 24C
X
EmhodiedCity
Gratopia
ALN
(Ours)
17

---

A.3 Asynchronous Multi-agent Control
Update Observation
Send Actions
Agent its observation
RoboScape
Agent I's Model
Agent Action
Stay still
Agent j's observation
Centralized Buffer
actions
Agent J's Model
Agent j's Action
Reversing
Receive Observation
Agent k's observation
Agent kts Model
Agent k's Action
Stay still
Figure 7: Asynchronous multi-agent control. To ensure each agent independently selects and executes actions based on its local observation and policy model, centralized buffer serves as the interface between agents and the environment, receiving actions from all agents and updating their observations based on the resulting environmental changes.
The control pipeline is illustrated in Figure where each agentregardless of embodiment (robot, vehicle, or policy model. These actions asynchronously sent to centralized buffer, which mediates the interaction between agents and the environment by updating each agents observation based on the environments response to all executed actions.
perceives its local observation and selects an action using its
A.4 Observation Space
As illustrated in Figure Ra, the robot receives multimodal observations at each step, including RGB images, semantic segmentation maps, and depth maps, enabling a rich understanding of its surrounding environment.
a RGB
Segmentation
Depth
Action Space.
The robot operates discrete action space comprising:
Translation: Move forward, backward, left, right
Rotation:
Turn left or right
Idle: Remain stationary for 2 seconds
View Adjustment: Look up or down
Figure 8: Multimodal observation and discrete action space. The robot perceives the environment via RGB, segmentation, and depth modalities. (b) Its action space includes translation, rotation, idling, and view adjustment.
18

---

A.5
Robot Action Space
The robot exccutes actions within discrete control space composed of directional movements, rotations, idle behavior, and vertical view adjustments, as illustrated in Figure Bb.
A.6 Procedural City Generation
The Procedural City Generation pipeline is a central component of the SWR simulator, designed to generate realistic and structured urban environments from minimal input specifications. This pipeline offers a highly modular and extensible architecture that supports the creation of diverse cityscapes, enabling a wide range of embodied AI tasks such as autonomous driving, pedestrian navigation, and multi-agent simulations.
As illustrated in Figure C. the city generation process is organized into four sequential stages: road generation, building generation, street element generation, and traffic element generation. Each stage progressively adds layers of realism and complexity to the simulated environment.
The process begins with the creation of road network, which serves as the backbone of the city. Roads are generated through an initiation phase and a tree-like growth process that balances depth and branching using priority queue. Mechanisms such as road-end attachment and intersection checking ensure a coherent and plausible layout.
Road Generation:
Building Generation: Once roads are established, buildings are procedurally placed along road segments. For each side of the road, candidate positions are sampled while checking for space availability and avoiding collisions. A greedy strategy is used to fill remaining gaps near road ends, maximizing spatial utilization and maintaining visual uniformity.
Street Element Generation: Smaller environmental elements such as trees, road cones, benches, and parked vehicles are generated around buildings and alongside roads. These details are categorized and placed based on contextual zoneseither surrounding buildings or within designated sidewalk areas. While collisions with other objects are not strictly enforced for performance reasons, the placement respects basic accessibility constraints.
Traffic Element Generation:
The final stage involves populating the city with dynamic actors such as cars, pedestrians, and agents. These elements bring life to the simulation and interact with the static environment, enabling research in traffic flow, behavioral modeling, and agent-based navigation.
Internally, the pipeline utilizes dedicated managers for roads, buildings, and elements. Each manager maintains spatial data using both lists and quadtree structuresenabling efficient queries, spatial indexing, and collision checks. Procedural rules and constraints guide item generation to ensure the city is functionally consistent and visually appealing.
By separating generation into clearly defined stages and maintaining rule-driven architecture, this pipeline provides a robust foundation for scalable and customizable city simulation within SWR.
A.7 Background Traffic System
In SWR, both pedestrians and vehicles are controlled using rule-based traffic system, as illustrated in Figure E. A waypoint system is constructed over the entire procedurally generated city, consisting of two types of waypoints: road waypoints and intersection waypoints. At each intersection, four intersection waypoints are sampled, one at each corner. For every road segment connecting two intersections, road waypoints are sampled at 17-meter intervals, linking the intersection waypoints at both ends. Prior to traffic simulation, we sample sequence of connected waypoints for each pedestrian and vehicle. During simulation, each agent follows assigned path. When an agent reaches an intersection waypoint, it checks the traffic light status: if the light is green and the remaining duration exceeds 15 seconds, the agent proceeds; otherwise, it waits until the signal turns green, ensuring safe and realistic traffic behavior.
19

---

Figure 9: Traffic system in SWR. Agents follow sampled waypoints and obey traffic lights at intersections, proceeding only it's green signal.
A.8
Customization SWR
The benchmark tasks included in the paper intended studies demonstrate the functionalities and utilities of SWR. SWR was explicitly designed to support user-friendly customization of embodied agents, environments and tasks.
A.8.1 New Embodied Agents
To add new robot type in SWR, one can leverage the existing Python API of SWR to conveniently customize action spaceseither continuous or discreteas well as observation spaces such as RGB, depth, semantic segmentation images. The required additional work (I) obtaining new robot asset, typically from the Unreal Engine Marketplace; (2) defining the robot's actions using Unreal's Blueprint system; (3) integrating these actions with Python API to enable high-level control; and (4) attaching our camera components to support the desired observation space.
A.8.2
New Environments
Users can generate diverse and realistic urban layouts through our Python API by providing simple metadata inputs (e.g., number of streets, street length, object categories, and their spatial distributions, such as 10% trees, 5% tables and chairs). This allows researchers to create arbitrarily large and varied city environments.
A.8.3 New Tasks
Defining tasks in SWR is similar to standard Gym environments [6], as the Python API of SWR follows the same format for agent control (including pedestrians and vehicles). Users can spawn different types of agents, customize observation spaces (e.g.. RGB, depth. or semantic segmentation) and action spaces (continuous or discrete actions), and program new goal definitions (c.g.. language instructions. target images, or spatial objectives). Additionally, while the current pedestrians follow rule-based logic, users can override behaviors to simulate complex or rare cases. For instance, one can simulate jaywalking pedestrian by scripting few agents to cross during red light, while placing robot the intersection evaluate its reaction.
A.9 SWR Technical and Operational Details
Underlying simulation engine. Our simulation is built UE5(Unreal Engine 5), leveraging its native Chaos physics pipeline: each object is assigned an appropriate collision mesh, and at each fixed simulation tick performs discretetime integration of Newtons equationsresolving forces, collisions, and joint constraints via its iterative solver.
Python API and runtime environment. On top of UE5, weve implemented a dedicated Python layer that communicates with the engine through an updated UnrealCVbased TCP server [H33], where highlevel commands issued in Python are forwarded over TCP straight into the UE5 runtime. On top of this API, we provide standard Gym interface so that researchers can plug and benchmark any baseline with minimal effort. We will distribute a Windows executable and Singularity container
20

---

for Ubuntu and macOS so that users can run SWR out of the box without local compilation; all binaries, container images, and the Gym wrapper will be opensoureed upon publication.
Human-in-the-loop interface. We support a human-in-the-loop interface through which a human operator mouse and keyboard control the robot; all trajectories are recorded automatically as expert demonstrations for downstream training.
A.10 SWR Computational Requirements
To ensure wide accessibility, SWR supports adjustable rendering resolutions, allowing deployment across both high-end servers and modest laptops.
Recommended Setup
CPU: Intel Core 17-12700H or AMD Ryzen 9 5900HS
GPU: NVIDIA RTX 3070 or GPU with more than 6 GB
RAM: 32 GB
Minimum Setup (60 FPS for SWR)
CPU: Intel Core 17-11300H AMD Ryzen 9 4800H
GPU: NVIDIA RTX 2060 (notebook)
RAM: GB
A.11 Running Efficiency of SWR
We evaluated all baselines on a headless machine with an AMD EPYC 9534 CPU, L40S GPU and 64 GB RAM. We can run 2 instances in parallel with a fixed 60 fps. Here's the table when stress-test the runtime performance with different settings.
Table 7: Runtime Performance Stress Test
Resolution
Rendering Quality
GPU Utils (%)
CPU Utills (%)
RAM (MB)
low 640 x 360
30.04
16.41
561.5
720 600
low
29.72
17.73
596.4
1280 720
low
26.5
18.56
734.81
640 x 360
high high high
30.03
16.59
564.04
720
29.61
17.47
1280 720
27.44
19.08
738.6
B Benchmark Comparison
B.1 Comparing Sim Simoordd-MNNav with Prior Vision-Language Navigation Benchmarks
Table 8: Comparison of instruction following benchmarks for navigation.
Benchmark
Env
Agent
Route
Type
Num
Type
Act Space
Gen
Acts
Num
CVDN 5T
Indoor, Static
90
Camera
Graph- based
Manual
7,415
REVERIE 2
Indoor, Static
90
Camera Graph-based
Manual
5
7,000
TouchDown
Outdoor, Static
Camera Graph-
Procedural
9,326
ANDH [[]]]
Outdoor, Static
Drone
Manual
7
6,269
AerialVLN [EU]
Outdoor. Static
Drone
Manual
204
8.446
Sim World-MMNa (Ours) Outdoor, Dynamic
400
Robot
7
Procedural
100
400
21

---

Table Comparison of multi-robot collaboration and navigation benchmarks. Dynamic indicates whether pedestrians, vehicles, or other moving obstacles are present.
Benchmark
Eav
Agent
Comm.
Map
Objective
Type
Scale
Num
Type
Modality
Max Len Access
[SC]
Indoor. Static
Mobile
None
Collab, pick&place
DOROTHIE 12C
Outdoor, Dynamic
Road
1 lego car)
Vehicle
Spoken dig.
20 tok.
Full
Dialogue nay.
MetaUrbon
Outdoor, Dynamic
City
1+
Esscooterfoboo
None
Full
Micromobility
Dynamic
Vehicle
None
Cocp. driving
RohotSlang
Indoor, Static
Lab
23
Mohile base
Natural lang-
80 char
Fartial
Joint object seurch
Where Are You? [X
Outdoor, Static
City
(tourist-guide)
Pedestrian
Natural lang.
40 tok.
Split
Localizatin via dialog
SimWorld-MRR. (Ours) Outdoor, Dynamic City-scale
Robot
Natural-languut
128 char
Split
Rendezvous / meet-up
B.2 Comparing Sim World-MRS with Prior Multi-Robot Collaboration and Navigation Benchmarks
C SIMWORLD-MMNAY
In this section, present SimWorld-MMNNr, single-robot benchmark designed evaluate multimodal navigation in large-scale urban environments. We begin by describing our procedural task generation pipeline, which enables the creation of diverse and realistic navigation tasks. We then introduce the SWR-20k training dataset, detailing how we generate fine-grained supervision signals for multimodal learning. Next, we elaborate on the baseline models used in our experiments, including their implementation pipelines and prompting strategies. Finally, we provide an in in-depth analysis of notable failure cases observed during evaluation, such as incorrect 3D spatial reasoning and other representative errors.
C.1 Detailed Task Settings
Observation Space In addition to the multimodal instruction, the robot is equipped with compass that indicates its current facing direction. To facilitate navigation, the robot also receives its egocentric RGB image, segmentation image, and depth image, which together provide rich perception of the surrounding environment.
Action Space The robots action space consists of two categories: movement actions and task- related actions. In the movement category, the robot can choose from the following options: move forward, move backward, move left, move right, turn left (-90%), turn right (+-99), or stay still. In the task-related category, the robot can select the action evaluate, indicating that it believes the current task has been completed. the evaluation is correct, the robot receives the next subtasks language instruction and visual hint; otherwise, the episode terminates as failure. Our simulator allows active perception like looking up by providing egocentric observation from different fields of view.
C.2 Metric Detail
We evaluate Single-Agent Instruction Following using three metries: Success Rate (SR), Subtask Success Rate (SSR), and Distance Progress (DP).
Success Rate
This metric measures the proportion of navigation tasks in which the agent successfully reaches the final destination. A trial is counted as successful only if the agent stops near the correct landmark building and is oriented towards it at the end of the trajectory.
Subtask Success Rate This metric evaluates the proportion of correctly completed subtasks within navigation episode. Let N denote the total number of subtasks in given instruction, and n. the number of subtasks successfully completed by the agent. The Subtask Success Rate (SSR) for that episode is computed as:
The
SSR
(1)
We report the average SSR across all test episodes.
22

---

Distance Progress This metric quantifies how much closer the agent is to the goal at the end of the navigation compared the beginning. Let dg the initial distance the goal and dr the final distance. The Distance Progress (DP) is computed as:
do (lo-dr,0) do
DP = max
(2)
DP values are also averaged across all tasks to obtain the final score. All distances are computed using the Manhattan metric.
In the more challenging environment containing obstacles, pedestrians, and vehicles, we additionally measure three metrics to assess the robot's social navigation capabilities: the average number of static collisions (i.e., collisions with buildings or static obstacles), the average number of dynamic collisions (i.e., collisions with pedestrians vehicles), and the number of actions that violate traffic light rules during navigation.
C.3 Procedural Task Generation
Algorithm 1: Task Generation for
Input:
Navigation path / = [n1 .2.............................. n. orientations (01, .g.00,,,. Output: Ordered list of subtasks T {(41, Vi), (12, .
Initialize subtask list T + []:
V/2 < capture visual cue at R4 with orientation 01;
3 [ + generate Orientation Alignment instruction using compass heading 0 and nearby
landmark;
Add (I1, V1) to T:
for i 4- 2 to k do
0, * 0;-1 then
Vinve capture visual cue n;-1 with orientation 0;-1;
move + generate Move Along the Road instruction using a landmark near 74-1; Add (mmove, Vmove) to T:
10
Vum & capture visual cue at with new orientation 0;;
11
Jumn & generate Turn at Intersection instruction based on the relative angle between
0 and 0;:
12
Add (Tum: Vumn T:
13
Ypoal f- capture visual cue at rig with orientation Oz:
goaa generate Reach Destination instruction using goal landmark description;
15 Add (Igoal, Vooal) to T:
16 return T
To support diverse multimodal navigation scenarios, we construct the SIMWORLD-MMWN benchmark under two difficulty levels: easy (without obstacles) and hard (with obstacles such as vehicles and pedestrians). We generate 200 city maps using our procedural city generation pipeline, with 100 maps allocated to each setting. For the map under hard setting, streetside obstacles are additionally generated. The task generation process detailed in Algorithm IGG
On each generated map, we randomly sample a pair of points, denoted as Pannt and Ppoal- For each point, we locate the nearest landmark building and extract the front-door location, referred as Lysat and Lgoat respectively. The robot is spawned at Lytat and tasked with navigating to Lgaat-
We then the A algorithm to compute optimal path between Lstant and Lgoal over the city-wide waypoint graph.
This path is decomposed into sequence of subtasks that reflect the robots expected behavior along the route. These subtasks are:
Orientation Alignment Each episode begins with orientation alignment subtask. The robot provided with a target compass direction (e.g., North, South, East, West) and a nearby landmark description to assist alignment, such as: Face north. You will see modern building with light blue
23

---

glass on your left. An image is captured at the initial location with the correct orientation to serve as the visual cue for this step.
Move Along the Road As the robot follows the computed path, we iterate through consecutive waypoints. When the robot is expected travel straight between two intersections, we define move along the road subtask. Specifically, we identify prominent landmark near the intersection where the robot is expected turn, and use to generate a descriptive instruction, such as: Move along the road and stop at the intersection when you see a large building with glass panels and a light brick base on the opposite side. A visual cue is captured at the intersection with the robots current orientation to assist visual grounding.
Turning at Intersections When change in orientation is detected in the pathtypically at an intersectionwe insert a turn at the intersection subtask immediately following the previous move along the road step. Based on the relative orientation of the next waypoint, we determine whether the robot should turn left or right. A new visual cue is captured at the next node with the updated heading, and corresponding instruction is generated, such as: Turn left the intersection and you should see this view.
Repeat Navigation Steps The move along the road and turn at the intersection subtasks may repeat multiple times until the robot reaches the final waypoint along the planned path.
Reach Destination At the final step, we define find destination subtask. A detailed description of the goal building and its spatial relationship to the robots position is used to generate the final language instruction. A visual cue is captured with the robot facing the destination building to aid recognition.
This process completes one full navigation episode. For each map, generate two episodes by sampling different pairs of start and goal locations, resulting in a total of 400 navigation tasks200 under the easy setting and 200 under the hard setting.
C.4 Details of SWR-20k
The SWR-20k generated as training dataset using distinct set of maps, referred to as training maps, which differ from those used for evaluation. The primary difference lies in the building distribution: training maps include only 66% of the building types that appear in the testing maps. The remaining 34% of building types are held out exclusively for testing, ensuring a clear separation between seen and unseen environments and promoting better generalization.
We generated 100 training maps using only the selected 66% of building assets. On each map, two tasks are sampled, and the oracle trajectories are generated by A* algorithm, resulting 200 orcale trajectories. Each trajectory contains over 100 steps, forming training corpus of 20K steps.
At each step during training, the robot receives synthesized observation and a corresponding sub-instruction, and is required to predict the correct next action. To enhance the robot's ability to reason about the goal and environment, several intermediate reasoning targets. Specifically, the robot is trained to jointly predict: (1) the distance between the current observation and the target visual hint; (2) the orientation of the visual hint; and (3) the potential sequence of actions from the current location to the final goal. All of these supervisory signals are paired with ground-truth annotations and serve multi-task learning objectives, guiding the robot toward deeper understanding of spatial context and task intent.
supervise not only the action prediction but also
C.5 Baseline Detail
Zero-shot Pipeline To reduce the mapping complexity and prevent hallucinations, we decompose step-wise action prediction in multimodal instruction folllowin into two modules: a context-awe aware perception module and ReAct-based [[57]] action module as detailed in Figure II]. Throughout this process, the model autonomously maintains a working memory.
The Context-Aware Perception module receives the current observation, visual hint (i.e., the expected view after subtask completion), the language instruction, and the current memory state. We prompt rision-language model with the instruction and memory to generate task-relevant
24

---

Working Memory
Cont Perception
concrete building lett Looking high cosorese reskential building
Calnese characters.
lirrary
Orientation
More forward times, mow lefi, turn right
OBS Description
TAR Description
ReAct
Instraction
Proceed
meacinng high brow
cunerte residential building with lange bilbuard
fustrrnn Chiness
Reasening
Updsted Memory
Action
your left, stop huilding
description
Move Forward
yet, suggesting moving
Move Right
Move Left
Look
Figure 10: Zero-shot single agent multimodal instruction following pipeline
description. This description incorporates information from both the current observation and the target view, and is explicitly encouraged to compare the images in order to assess whether the subtask has been completed.
The ReAct module operates purely in the text domain. It receives as input the perception-generatee description, the original instruction, the memory, the robots current orientation, and textualized history of past actions. ReAct is expected to first perform reasoning, then update the memory module accordingly, and finally select the next action for the robot.
We utilize segmentation images solely auxiliary visual signals. Given the varying levels of training and understanding of depth images across different models, and considering that interpreting depth information is not the primary focus of our task, we exclude depth images from all inputs eliminate potential confounding factors. In our experiments, we allow model to generate sequence of actions at each step. This design improves execution efficiency during long straight-path phases and serves as test of the model's short-horizon planning capabilities.
Content-Aware Perception System Prompt in Zero-Shot Single Agent Instruction Following
You are perception module of a navigation robot in 3D environment. The ultimate goal is to place yourself right in front of a par particular building.
You will given:
A history summary.
Your vision description at the last step.
single egocentric image.
The exact expected view you will see you complete the current subtask.
segmentation image. The segmentation of the entire view. Green trees, purple buildings, yellow sidewalks/croossallks, black = driveways.
The subtask you are working on.
The current cardinal direction.
Instructions:
First, describe the observation in detail, focusing on the color, texture, attachment, etc. of buildings. Focus potential landmarks (if needed), obstacles, intersections. Reason about useful details actions.
25

---

Include details like whether the agent is walking straight on sidewalk and where the driveway is.
If the upper parts of the entire observation are filled with buildings, the agent is not walking straight along sidewalk.
Judge the distance by the vertical position of the object infer how many forward steps can be taken or how close intersections are.
-Finally describe the expected view, also attentive to the buildings. Match the expected view with current view determine if the subtask is completed.
When aligning with the last landmarks:
When the subtask asks you to face a building, first ensure proximity to the landmark not just visibility in the strip. Look for close-up details to ensure you have reached the right position.
You don't 't need to mention the orientation, because will be given.
Intersection:
When approaching intersections, actively look for the expected Landmark.
Keep in mind: landmark might not be visible at the intersection due to limited field view.
If you cannot see the landmark, only use the expected view as reference. Turning once alone does not guarantee completion of the turning subtask. Always verify against the expected view.
Output Format:
You must return a JSON object like:
("Reason": "The detailed description and your reasoning about what detail is useful.", "Description"; "A useful summary of the observation")
ReAct System Prompt in Zero-Shot Single Agent Instruction Following
You are navigation robot in a 3D environment. The ultimate goal is to place yourself next to particular building.
You will given:
list actions that have already been taken (action history).
The action sequence you took last step.
A description of last step's observation. You can use it to compare with the current
observation.
surmary of the history hint for future planning.
current status. You can use and update the summary as
The current subtask you are working on.
Current orientation of the robot.
detailed description of the agent's current perception. Then description of the
exact expected view you will see once you complete the current subtask.
Valid actions:
-1: bubtask_compleds
If you believe the current subtask is completed, the action sequence should be [-1].
0: Move_forward Move meters forward in the direction the robot is facing.
1: Rotate_left Rotate 90F to the left.
2: Rotate_right Rotate 90F to the right.
Move_left Move meters left, without rotating.
4: Move_right Move meters right, without rotating.
Instructions:
First, analyze the current visual observation, the instruction, current situation, and history, reason about how to update history summary
next
action.
Next update and resummarize the history and current status. If you have aligned to the current landmark, update the landmark to the next one.
Finally, decide the next action steps based the previous analysis.
26

---

Alignment:
You have the cardinal direction to help you align at the beginning of the task.
The last Alignment:
When handl ing the "face the building" subtask, you must close enough turn to face the building to complete the subtask.
If cannot see the building after rotating, it means you are not close enough.
Intersection:
While reaching intersections, actively look for the expected landmark. Once it's spotted, update your history so the next intersection is the one.
Keep in mind: the landmark might not be visible at the intersection due to limited field of viewuse the expected view your reference.
Turning once alone does not guarantee completion the turning subtask. Always verify against the expected view.
Important Rules:
Make sure you are oriented along the sidewalk when following "move forward" commands
You can plan by outputting variable-length action sequences. For example, [e, 0, 0,
0, 0] if the path is clear.
Shorter sequences if obstaclles/nntessetioon ahead.
you only see sky and road on one side, means you are at the map boundary. Rotate to face buildings.
- The subtask is done only when the current view matches the expected view.
Remember:
you believe the subtask completed, output [-1]. Remind yourself in the history that you are starting to do the next subtask.
You must always output at least one action. If lost, try rotating.
Output Format:
You must ALWAYS return a JSON exactly like: [ ("Reason": "Your reason", "Summary": "New summary of history and current status", Actions": [list of integers]]
Table 10: Key hyperparameters used during finetuning
Hyperparameter
Value
Training epochs
2
Batch size
32 (4x8)
Optimizer
AdamW
Learning rate
2e-4
Weight decay
0.01
LR Scheduler
Constant (with warmup)
Warmup ratio
0.03
Gradient Clipping
0.3
LoRA rank
8
LoRA alpha
16
LoRA dropout
0.05
Finetuning During finetuning, the model receives as input the current observation, the target image representing the expected view upon subtask completion, the instruction, the current orientation, and the ground-truth action history. Based on these inputs, the model is trained to predict the estimated distance, the anticipated final orientation, and the remaining action sequence as form of CoT planning, followed by the prediction of next action. We finetune the Qwen2.5- VL--BB-Instruct model using LoRA [[TI] applied to both the language model head and the merging projection layer. The loss is computed solely on the tokens generated by the decoder. used four A100 GPUs with 80GB VRAM each for finetuning. The hyperparameters can be found in Table III
27

---

Hybrid Method The hybrid system uses GPT-4o as high-level decision maker determining whether to continue straight or turn at intersections. It then uses A * as low-level path planner to generate and execute movement commands.
RL Method The policy is first pretrained through behavioral cloning on expert demonstrations and then finetuned with PPO. Training is conducted on two NVIDIA L40S GPUs, each running two parallel instances of SWR.
System Prompts in Finetuned Single Agent Instruction Following
You are navigation robot in 3D environment.
You will given:
An current egocentric image.
An expected view image (what you should see when the subtask is completed).
A textual description of the subtask.
Your current cardinal direction (e.g., "North").
history of previously taken actions.
You must:
1. Determine whether the current subtask is already completed by comparing the current
and expected views.
2. Deduce the expected orientation when the subtask is completed.
3. Deduce the distance from the current position to the expected position.
4. Plan the remaining actions to complete the subtask based current and expected views.
If the subtask not completed, output the action you will take in this step. the subtask is completed, output -1.
Valid actions:
-1: Subtask_ Subtask_completed
Move_forward Move meters forward the direction the robot facing.
1: Rotate_left Rotate 90F to the left.
2: Rotate_right Rotate 90F to the right.
Output Format:
Only return a JSON object like:
Eixpected__Oienttatee "The Orientation", "Remaining Distancce" "The Distance", RemaininggActions"" "Textual Plan of the Actions", "Next_Action"; integer)
C.6 More Quantitative Results
Ablation Here we provide the ablation study of our ReAct baseline with GPT-4o as base model,
tested on 50-task subset of SimWorld-MMNNn Table [ shows.
Table 11: Ablation study with key components.
Configuration
Explicit Reason
Separate
Depth
Segment
Strip
Subtask Success Rate (%)
Perceive/Act
Our setting Merged call
34,38
33.54
w/ depth
33.39
w/o explicit ReAct
32.21
w/o segmentation
31,90
w/ stripping
31.22
Our setting requires the model to explicitly reason before acting through multi-turn interaction: first describes the observation, then reasons and decides the action. The input includes the observation and its segmentation mask, without depth stripped visual parts.
The perception-action framework simplifies the mapping process and reduces hallucinations. While the model possesses certain implicit depth estimation capacity, directly adding depth images
28

---

yields marginal gain and can even introduce noise if the colormap is misaligned. The explicit ReAct framework notably stabilizes the reasoning process and mitigates hallucinations in complex intersections. Although GPT-4o itself lacks sufficient training on intersection-heavy scenes, the ground-truth segmentation image helps alleviate this limitation. Finally, stripping the observation into vertical chunks increases visual matching difficulty, leading to degraded performance in our simplified setting.
Statistical Significance We also provide the 95% CI of our main results, as Table 12 shows. For success rate and subtask success rate, we use binomial proportion confidence interval [5.S] and display both lower and upper bound in the table.
Table 12: Experimental results on the Sim World-MMNal benchmark (easy task set) with confidence intervals
Models
SR&
Subtask SR% +
Distance Progress
Proprietary Models
GPT-4o
0 [0, 3.85]
33.07 [24.71, 43.24]
15.60 (16.97) 31,29 (+8.11))
Gemini 2.5 Flash
0 [0. 4.32]
37.06 [28.09. 48.27]
Open-sounced Models
QwenVL 2.5 7B
[0, 4.14)
16.86 [10.49, 25.96]
7.82 (#:2.45)
QwenVT. 2.5 72B
[0, 3.89)
23.80 [17.60, 34.84]
17.50 (#6.11)
Gemma 27B
[0, 3.85]
15.36 (9.70, 24.19]
6.83 (#55.44)
InterVLL 3
[0, 4.28)
18.31 (11.79, 28.11]
(1.4.04)
Fine-tuned Models
QwenVL2.55 yz
[1.08, 13.22]
52.45 (39.52, 65.95)
53.63 (=:10.88)
D SIMWORLD-MRS
To address the limitations of existing benchmarks in multi-robot search, we propose Sorld- MRS, new benchmark designed to evaluate collaboration. localization, and communication among multiple robots in large-scale, photo-realistic urban environments. Sim Worlddd-RRS simulates realistic challenges such as partial observability, dynamic environments, and natural language coordination. In the following subsections, we detail the procedural task generation, baseline implementations, prompting strategies, and provide case studies to illustrate key behaviors like localization and communication.
D.1 Detailed Task Settings
Observation Space The observation space for the follower robot closely mirrors that of the single- agent setting, with the key difference being that its instructions are exclusively derived from inter- agent communication. In contrast, the guide robot has access to the complete map and landmark information of the simulated city, enabling it to generate an oracle path plan and identify specific landmarks for rendezvous.
Action Space Compared to the single-agent setting, the follower robot is additionally capable of initiating communication, while the guide robot is responsible for route planning and conveying instructions to the follower. Both robots are also able to pause and check whether the other is within their field of view, facilitating coordinated rendezvous.
D.2 Metric Detail
We report two metries for evaluating performance on the SimWorld-MRS task: Collaborative Success Rate (CSR) and Task Progress (TP).
Collaborative Success Rate This metric estimates the probability that the two robots successfully meet up across different maps. A meet-up is considered successful if at least one robot executes the check_ tastttasssppppeee complete action and detects the other robot dog its observation via ground-truth segmentation.
29

---

Task Progress This metric quantifies the relative reduction in distance between the two robots by the end of the task. Let Do denote the initial distance between the two agents, and Dr the distance when the task terminates. The Task Progress (TP) is defined as:
Do Dr
TP max
00
(3)
Do
D.3 Procedural Task Generation
To construct the Sim World-RMMRS benchmark, we procedurally generate 100 unique city maps, each covering a large-scale urban environment. For each map, we instantiate one multi-robot search task, resulting in total of 100 evaluation tasks. The task generation process follows Algorithm I..
Algorithm Task Generation for SimWordd-MRS
Input: City map with m streets; sample n landmark buildings street
Output: main robot's memory M, main robot spawn Smain- following robot spawn Sfollow // Phase 1: Build Main Robots Memory
Initialize memory set M <- []:
for i 1 to do Sample n landmark buildings on street i: {L11, Liz, Lin);
for j <- to do
Fij < get front-door location of Lij:
Oij compute orientation facing toward Lyy:
Tij & capture visual cue at Fiy with orientation Ozj:
Add Fig) memory set M:
// Phase Sample Initial Robot Positions
Obtain a set of valid robot spawn points S; Randomly sample two distinct spawn points from S: Smain: Stollow
return (M, Smain: Sfollow)
Specifically, we first sample n landmark buildings along each the m streets in the city and collect their front-door images with aligned orientations. These image-location pairs serve as the main robot's memory of the city. Next, we sample two distinct and valid spawn locations as the starting positions for the main robot and the following robot. This ensures spatial diversity and supports realistic localization and communication challenges.
To evaluate the accuracy of the final meetup, we utilize the observations captured by both robots at the end of the multi-agent navigation process. Given that our system includes access to ground-truth segmentation, we determine whether the other robot appears within the field of view by inspecting the segmentation output. The presence of the counterpart robot in the observation is treated as evidence of a successful meetup.
D.4 Baseline Detail
Baseline Oracle Planner In the multi-robot oracle setting, baseline communication pipeline consists of three key components: the follower robot's description of the current building, the main robot's retrieval of the building, and the oracle path planning reach the follower.
The description module takes the current egocentric observation as input and feeds it into VLM using templated prompt. This prompt is specifically designed to guide the model towards generating goal-oriented and informative descriptions the current scene, ensuring alignment with task requirements.
The memory retrieval module receives the generated textual description and a pre-collected landmark image dataset as input. It then uses the VLM to identify the landmark image that best matches the description. To achieve this, we implement tournament-style elimination process: images are compared in pairs based on their semantic and visual alignment with the description, and in each round, the less relevant image is discarded. This process continues iteratively until a single most relevant image remains. To mitigate hallucinations and enhance retrieval stability, we employ
30

---

system prompt that encourages precise and comparative reasoning. The output of this module is the position of the landmark image judged by the VLM be the most semantically and visually consistent with the description.
The oracle path planning module receives input the current location of the main robot and the estimated location of the follower robot, as determined by the memory retrieval module. Using full access the global city map, the module computes the shortest collision-free path between the two locations via the A algorithm. The resulting path is converted into sequence of discrete navigation actions, such as moving forward or turning at intersections. These actions serve as an oracle reference and can be directly issued to the follower robot or translated into natural language instructions for communication purposes.
During the main robots execution of the oracle path, the follower robot actively rotates at each step and uses the VLM to detect whether the other robot dog appears in its field of view.
beige and building is medium height. with approximately five floors...
Description
"Locate the follower
*Path Planning the meet-up point
Instruction 1 (Turn to face west Walk forward until you see dark brown, stone and glass huilding)
"Fail to spot the guide Description 2 The facade is dark brown and the building is tall...
"Locate the follower
Main
*Path Planning for the new meet-up point Instruction face south forward ontil see an intersection with...]
Follower
Figure 11: Example communication for ROCO baseline
Baseline ROCO The ROCO-based [33] setting extends the oracle setup by introducing collaborative planning and communication between two robots. After the two agents communicate and confirm the followers current location in terms of a landmark building, the main robot uses access to the ground-trui truth map to identify the landmark closest to the midpoint between both agents. then performs path planning to compute the optimal trajectories for both agents to converge at this intermediate landmark. The main robot transforms the computed path for the follower into natural language instructions, which are communicated to the follower robot. The follower then performs multimodal instruction following to navigate toward the meeting point and rendezvous with the main robot. An example for communication in the ROCO baseline can be seen in Figure Iل]
The description, memory retrieval, and path planning modules mirror those in the oracle baseline. The instruction generation module translates the planned path into natural language commands using predefined set of sentence templates. These instructions designed to guide the follower robot step-by-step without access to the full map. Each instruction encapsulates or more discrete navigation actions, such moving forward until reaching visible landmark or turning at an intersection.
compact and landmark- -aware,
The communication module governs when to initiate new dialogue round. After executing received instruction, the follower robot monitors its state and triggers a new communication cycle under the condition: if it believes it has completed the instruction but, after rotating to search, does not observe the main robot. Upon reactivation, the follower generates a new scene description, enabling the main robot to re-localize its position and update the rendezvous plan accordingly.
31

---

The cntruuctionfolllviiii module remains largely consistent with the single-agent setting, with the primary difference being that the visual hint is longer provided.
Prompting for Multi-agent Baselines We adopt modular prompting strategy that aligns with our multi-agent architecture. Specifically, we design two system promptsone for generating egocentric scene descriptions (used by the follower robot), and another for conducting comparative landmark retrieval (used by the main robot). These prompts are implemented via VLM with image and text modality support.
The follower robot observes its local environment and produces textual description aimed helping the main robot identify its current location. The prompt guides the VLM to emphasize salient and matchable visual features such color, height, materials, signage, and local context.
Follower Description Prompt
You are an expert building-description assistant.
In 200 words, describe the building so another person could match photos of it.
**Start the first sentence with the facade's MAIN COLOR HEIGHT.
Cover these attributes as comma- semma-tee phrases:
main color dominant facade color
height low medium tall (number of floors)
primary materials e.g., brick, concrete, glass, steel window grid / pattern shape and arrangement of windows ground-floor layout doors, arches, glazing style signage text exact words visible; say "no signage" if none sidewalk objects posts, trees, benches, etc.
distinctive features murals, balconies, arches, etc. neighbor immediate surrounding context (e.g., adjacent buildings or empty lots)
Keep it factual and avoid subjective opinions. Return single descriptive paragraph in natural language.
After receiving a description from the follower robot, the main robot attempts to match it against its landmark memory using VLM-bbsed comparative prompt. The model is required to select the image (A or B) that better matches the textual input, focusing on distinctive visual features.
Main Robot Retrieval Prompt
TEXT Natural-language description of the target building.
Two FULL-FACADE candidate photos are shown: (first) and B (second)
Decide which matches TEXT better.
Guidelines:
Give strong weight to facade color, signs, materials, and distinctive features. If the facade color clearly mismatches, that candidate must lose. You must compare both options and select the better one.
Reply only with A or B.
E
Qualitative Examples
We provide qualitative examples to illustrate the common failure modes of VLMs.
32

---

visible Observation if robot turns right
Ideat Thought
The current vision descriiption metches the
intersection while the high yellow residential building my right. should turn right to see that building is right.
visible
Ideal Action
Tum Right
Current Instruction
Robot's current observation
VLM Thought
Continue straight stop intersection
with high yellow residential building
The current vision description matches expectad view intersection while the high yellow residential
featuring striped awning over small
entrance on your right
Current Visual Hint
my right. should keep moving forward.
VLM Action
Moving Forward
Analysis
VLM lacks active search
Figure 12: Qualitative result lack of active perception
E.1 -iMWorrd-MMNNa
Active Perception The VLM lacks initiative in active perception. Figure shows, at intersections, the robots field of view often limited, and the landmark referenced in the instruction may not be visible from the current perspective. As a result, the robot does not recognize the location the intended intersection, even when the visual hint has already been matched. Ideally, the robot should rotate to check its surroundings for the missing landmark. However, the VLM tends to continue moving forward, waiting for the landmark appear directly in front of the robot rather than actively seeking it through lateral exploration.
Current Observation
Visual Hint
Perception Description
The view displays wide road with light gray building featuring brown accents and dark brown columns. The building is close, with visible broken glass windows. indicating the intersection is near.
the right,
VLM Action
Subtask Complete
Figure 13: Qualitative result lack distance grounding
Spatial Reasoning The VLM exhibits limitations reasoning about spatial relationships, particularly in estimating distance, maintaining spatial continuity, and interpreting alternate perspectives. In one failure case, the current observation partially resembles the visual hint, leading the VLM prematurely assume arrival the intersection, despite the robot still being far from the crosswalk, as Figure IG shows. This example also indicates that the recognition of intersection lacks robustness and relies heavily referential landmark buildings.
In another instance, as Figure [4] shows, during an orientation alignment task, the robot initially facing a landmark. After turning right, a characteristic part of the landmark disappears from view.
33

---

Current Instruction
Step 1 (has already seen the hospital)
Location
Observation
VLM Thought at Step 2 The current vision description matches expected the intersection while hospital with red 'Central Hospital' letters not my right.
Face north. should see a red-brick hospital with red 'Central Hospitall
CENTRAL HOSPITAL
letters above entrance canopy
supported by gray pillars
right
Current Visual Hint
Step 2 (after turning left)
should keep turning...
Location
Observation
VLM Action
Turn Left
Analysis
VLM lacks spatial reasoning
Figure 14: Qualitative result lack of embodied reasoning
Given a working memory, an embodied agent would robustly infer that it has aligned accordingly. However, the VLM fails make this inference, indicating lack of embodied spatial understanding.
Current Observation
Visual Hint
Perception Description
The target building with light gray stone detailing and gold-framed entrance doors has not yet appeared the left. The current view shows large building the left, and the path is clear ahead.
dark windows on
VLM Action
Move Forward
Figure 15: Qualitative result lack of perspective-adaptiee matching
These limitations also manifest when matching buildings from different perspectives. The target building provided as a frontal image, but during navigation, only a side view may be visible. The perception module often fails to associate the side and frontal views and provides insufficient information, causing the robot overlook the destination, as Figure shows.
Even when a correct match is made, the VLM may still fail to reorient. For example, given the thought, "Currently, am on wide street facing east with the high brown concrete building and billboard on my left. The billboard is not directly in front of me yet, indicating that I need to continue moving forward until it is," the model chooses proceed rather than rotate, failing to reason from an embodied perspective.
Pragmatic Reasoning The VLM also struggles with interpreting pragmatic intent in natural instructions. When given the instruction "turn right at the intersection," often treats the subtask complete after single turning action. In real-world scenarios, such turn typically involves multiple steps, like moving forward, turning, and possibly crossing the street. The VLMs overly literal interpretation leads partial execution and deviation from the intended path.
E.2 A Success Case of Finetuned Baseline
As Figure shows, finetuning facilitates When the agent is relatively close to the subtask target, the model is able to accurately infer the remaining distance and the corresponding sequence of actions, while also correctly predicting the final orientation. This enables a strong foundation for chain-of-thouge (CoT) planning and improves the reliability of next-step action prediction. Furthermore, finetuned model demonstrates, for the first time, the ability to reason about intermediate stepssuch as the need to cross the streetin order to align with the target image. This allows the model to successfully complete the illustrated navigation task.
spatial reasoning capabilities.
turning scenarios, the
34

---

Step
Instraction
Turn to face your left, you will gray concrete
residential building with vertical yellow signboard on the facade.
Output
Expected
Exppeceed_Orieenattioo Orientation: West
Remaining, Distance:
Remaiining, Actions: Turn left, Subtask completed
Action:
Step 5
Instraction
Continue straight and stop at the intersection where see the low gray concrete residential building with two rusted metal doors and green on your
Oberrvakk
Target Vieee
Ouiput
Expected Sopeeceed_Orientttionn Orientation: West
Remaining,
Remaining Actions: Move forward for Subtask completed
Next Action:
Step 10
Instruction
Turn the intersection.
Treget
Oalpul
Expected Orientation: North
Remaining. Distance: 34
Remaining Actions: Yurn right, Move forward for times, Subtaek completed
Next_Action:
Step 63
reaching high gray commercial building with large glass-framed entrances with brown frames your right, stop and face that building.
Proceed
Turget
Ourpat
Expected Expected__Orientatiion Orientation: East
Remaining, Distance:
Remaining Actions: Turn right, Subtask completed
Figure 16: Qualitative result key-step VLM outputs from the finetuned model successfully completing the task
However, finetuning also exhibits certain limitations. First, when the target image corresponds to a location far from the agent's current observation, the lack meaningful visual overlap makes difficult for the model to reason about progress. Second, the model is finetuned solely on ground- truth action sequences and lacks robustness to error correction. As a result, if a mistake occurs during navigation, the model struggles to recover, which partially explains the still limited overall success rate.
E.3 -inWordd-MRS
Salient Feature Ignorance In several failure cases, the follower robot generates descriptions that miss highly distinctive features such store signs, murals, logos. Instead, the VLM focuses on general elements like a modern building with glass windows, which are insufficient for precise localization. This results in ambiguous matches and large localization errors during memory retrieval by the main robot.
Failure of Instruction Following The issues observed in single-agent instruction following often persist in this setting and are further exacerbated by the absence of a visual hint, making accurate instruction execution more challenging. Although repeated communication can partially correct navigation drift, the task may still fail if the follower agent stops non-landmark building, as the main robot will be unable to localize it for subsequent rendezvous planning.
35

---

Lack of Awareness While executing instructions, the follower robot often struggles to determine whether it has become lost. Even when the subtask has not been completed, and the action history indicates that the robot has already moved forward for extended sequence, the model tends continue moving forward until the task termination conditions are met. As result, the robot fails to recognize its deviation in time to trigger communication for goal correction, ultimately leading to out-of-bounde behavior or exceeding the maximum allowed number of steps.
Code Availability Statement
The implementation of our system builds codebase developed through multi-institutional collaboration, with components that are difficult to anonymize due to dependency structures and prior repository history. To maintain the integrity of the double-blind review process, have withheld the release of the code at this stage. We are committed to open science and will publicly release the complete codebase, along with detailed documentation and instructions for reproduction, upon acceptance.
G LLM Usage Statement
We utilize several large multimodal models (VLMs) as core components our study, including GPT-4o, Gemini 2.5 Flash, Gemini 2.0 Flash, Owen2.55-VL-72B-Innsttt InternV and Gemma3-27b-it. These models are evaluated within our simulator-based benchmarks to investigate their embodied navigation capabilities and multi-agent communication performance. The LLMs are responsible for interpreting visual-linguitic instructions, reasoning about spatial environments, and generating actions or dialogue.
Dwen2.5-VV-77--Inssuu
Furthermore, we fine-tune effectiveness of task-specifi supervision. Since the models play a central role in both methodology and experimental analysis, and significantly influence the reported results, we declare their usage as integral to the core of this research.
-L-.----LLB---sstrc on our proposed training dataset assess the
36

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract summarizes the key features of our work, focusing on what differentiates our simulator from existing onesthe main focus of this paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discussed limitations in the conclusion.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results.
37

---

Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We will open-source source our gym environment, the agent's prompt, backend, and model. Please check the details of our two benchmarks and the baselines implementation in Appendix I and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data provided not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can he accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome to describe the particular way they provide for reproducibility. closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
the case of
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We will open-source our code as mentioned in Appendix Band the details of baseline implementation can be found in Appendix C. and Appendix D4
38

---

Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines httpss///nnps.ccppbbbb/ ct
guides/CodeSubmii swt.suuuiiisiioooocico... for more details.
While encourage the release of code and data, understand that this might not be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central the contribution (e.g., for
open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.
1ooooooly for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have specified that 33 percent of buildings in the training set are excluded from testing environment.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
Each experiment is run twice to account for variability, and results are reported accordingly. While limited in sample size, this provides initial estimate of consistency.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, statistical significance tests, least for the experiments that support the main
claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
39

---

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates). error bars are reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification:
We use headless machine with an AMD EPYC 9534 CPU, L40S GPU, 64GB
RAM
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurs: http:///llee/hhiccssii..........
Answer: [Yes]
Justification:
We follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: While we do not explicitly discuss societal impacts, we believe our simulator and benchmark positively contribute to research in embodied AI, and do not foresee any negative societal consequences from this work.
Guidelines:
The answer NA means that there is societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
legitimate to point out that
40

---

The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification:
Our work has no misuse risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use the model, for example by requiring that users adhere to usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes. All assets used in work were purchased from the official Unreal Engine Marketplace (fab.com), and we fully comply with their licensing terms. No assets were used for any unauthorized or additional 3D asset generation.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. wwt.odde.cmmaaa cww/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification: Yes. We introduce a 20K training dataset to support languaee navigation tasks, and detail of it can be found in appendix eZ)
41

---

Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Question:
Answer: [NA]
Justification:
We not involve any research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We no involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, non- standard component of the core methods in this research? Note that the LIM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: We provide a detailed explanation in the experimental section on how large language models (LLMs) and vision-language models (VLMs) used to conduct experiments, which form a core component in our benchmark. Details can be found in appendix G
42


Reviews:
[{"review_id": "cwfzYqX9bI", "reviewer": "Reviewer_S7EF", "summary": "The paper presents RoboScape, a novel simulation platform for embodied AI in large-scale, photorealistic urban environments. The platform is built on Unreal Engine 5 and can procedurally generate unlimited photorealistic urban scenes with dynamic elements such as pedestrians and traffic systems. The authors also introduce two new benchmarks: ROBOSCAPE-MMNAV for multimodal robot navigation and ROBOSCAPE-MRS for multi-robot search tasks. The paper demonstrates that current state-of-the-art models, including vision-language models (VLMs), struggle with these tasks, highlighting the gap in current foundation models for urban robot tasks.", "questions": "1. How does RoboScape handle the integration of new AI models or algorithms into its simulation framework?\n2. What are the specific computational requirements and limitations for running RoboScape on different hardware configurations?", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n1. RoboScape is a novel simulator that addresses the lack of realism, scalability, and versatility in existing urban simulators.\n2. The platform offers high-fidelity rendering and dynamic elements like pedestrians and traffic, making it suitable for realistic robot training and evaluation.\n3. The two new benchmarks provide a comprehensive evaluation of robot capabilities in realistic scenarios.\n\n\n### Weaknesses\n1. Although more diverse than previous simulators, the action space for human agents is still limited.", "was_revised": true, "final_justification": "The authors’ response was clear and convincing, addressing the issues I had pointed out. I believe the paper is stronger than I initially assessed, and I will increase my score.", "rebuttal": "We appreciate the reviewer for recognizing our contributions of “a novel simulator that addresses the lack of realism, scalability, and versatility in existing urban simulators” and “two new benchmarks provide a comprehensive evaluation of robot capabilities in realistic scenarios.” We address your questions and concerns below.\n## Q1: Integration of New AI Models\nCurrently, RoboScape uses pre-defined 3D assets, primarily obtained from Unreal Engine Marketplace (FAB), which we further customize for integration. However, the framework allows easy integration of generative methods such as DreamFusion to synthesize 3D assets from text, and SMPL-X-based text-to-body motion generation models to generate human motion sequences from natural language descriptions. We are actively developing these extensions to further enhance RoboScape’s ability to support diverse AI research.\n\n## Q2: Specific Computational Requirements\nTo ensure wide accessibility, RoboScape supports adjustable rendering resolutions, allowing deployment across both high-end servers and modest laptops. \n#### Recommended Setup\n- **CPU**: Intel Core i7‑12700H or AMD Ryzen 9 5900HS  \n- **GPU**: NVIDIA RTX 3070 or GPU with more than 6 GB  \n- **RAM**: 32 GB  \n\n#### Minimum Setup (60 FPS for RoboScape)\n- **CPU**: Intel Core i7‑11300H or AMD Ryzen 9 4800H  \n- **GPU**: NVIDIA RTX 2060 (notebook)  \n- **RAM**: 16 GB  \n\n## W1: Limited Action Space for Human Agent\nWe agree that the action space can be further expanded. By defining poses, it is convenient to add more actions within our framework. As an initial effort, we have added 5 new types of human actions commonly observed in daily life: (1) picking up, carrying, and dropping objects; (2) sitting down and standing up; (3) entering and exiting vehicles; (4) opening and closing containers (e.g., boxes, cabinets, car trunks); and (5) engaging in social interactions (e.g., chatting, discussing, arguing). \n\nWe achieved this by using pre-defined human body motion for each type and adjusting the poses using inverse kinematics for human-object interactions (e.g., adjusting hand poses to grasp objects). We have also implemented a flexible API to enable these actions to be triggered not only by rule-based planners but also by other models such as LLMs, enabling more diverse and context-aware human behaviors. In ongoing work, we are exploring the use of generative models such as text-to-body motion for producing realistic human actions beyond the predefined space. This will allow RoboScape to support richer and more human-like behavior generation in the future."}, {"review_id": "LT2mXz7lHB", "reviewer": "Reviewer_bmod", "summary": "The paper introduces a potentially valuable outdoor simulator RoboScape for robotics with support for high-fidelity assets and multi-agent cooperation. The experimental results demonstrate that state-of-the-art vision-language models (VLMs) lack robust perception, reasoning, and planning abilities necessary for urban environments.", "questions": "1. As the simulator constitutes a core contribution of the paper and serves as the foundation for various tasks, the manuscript lacks sufficient discussion of critical aspects such as the runtime environment, hardware performance, rendering capability, API adaptability, simulation type (e.g., PP, OS, or RBF), and the availability of a human-in-the-loop (HITL) interface. These dimensions are essential benchmarks for assessing the simulator’s effectiveness. The authors are encouraged to provide a detailed specification of these components and offer a comparative analysis against existing simulators.\n\n2. Although the paper compares weather conditions, lighting, building structures, and pedestrian behavior with other simulators, it remains unclear how these features influence performance in VLN tasks. The presented scenarios are insufficient to clearly demonstrate the simulator’s unique contributions to the field.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The paper presents an outdoor simulator designed for robotics tasks, featuring high-fidelity building and object assets. It is of significance to the field.\n\n2. The simulator is reported to support both single-agent and multi-agent collaborative operations.\n\nWeakness:\n\n1. Key aspects such as simulation realism, agent interactions, and task definitions are insufficiently addressed, raising concerns about the simulator’s usability and generalizability. \n\n2. The paper would benefit from a more rigorous exposition of design choices, empirical validation, and clearer articulation of its unique value compared to existing platforms.\n\n3. The selection of baselines and planning methods appears underdeveloped. The authors only present fine-tuning results on QwenVL2.5 7B and do not propose any new baseline or planning method tailored to this specific simulator. Furthermore, the study lacks sufficient ablation experiments that could provide insights into the effectiveness of different components.", "was_revised": true, "final_justification": "Thank you for the rebuttal. While some limitations remain, I feel that my main concerns have been adequately addressed by the authors. As a result, I am willing to raise my score by 1 point.", "rebuttal": "We are happy that the reviewer considered our work “significant to the field,” and we address your questions and concerns below.\n## Q1: RoboScape Details\nThanks for the suggestion. We further clarify the details of RoboScape below and will include them in the revised Section 3 and Appendix A. \n- **Underlying simulation engine**. Our simulation is built on Unreal Engine 5, leveraging its native Chaos physics pipeline: each object is assigned an appropriate collision mesh, and at each fixed simulation tick performs discrete‑time integration of Newton’s equations—resolving forces, collisions, and joint constraints via its iterative solver. \n- **Python API and runtime environment**. On top of UE5, we’ve implemented a dedicated Python layer that communicates with the engine through an updated UnrealCV‑based TCP server, where high‑level commands issued in Python are forwarded over TCP straight into the UE5 runtime.  On top of this API, we provide a standard Gym interface so that researchers can plug in and benchmark any baseline with minimal effort. We will distribute a Windows executable and a Singularity container for Ubuntu and macOS so that users can run RoboScape out of the box without local compilation; all binaries, container images, and the Gym wrapper will be open‑sourced upon publication. \n- **Hardware requirements and simulation/rendering performance**. To ensure wide accessibility, RoboScape supports adjustable rendering resolutions, allowing deployment across both high-end servers and modest laptops. \n\n#### Recommended Setup\n- **CPU**: Intel Core i7‑12700H or AMD Ryzen 9 5900HS  \n- **GPU**: NVIDIA RTX 3070 or GPU with more than 6 GB  \n- **RAM**: 32 GB  \n\n#### Minimum Setup (60 FPS for RoboScape)\n- **CPU**: Intel Core i7‑11300H or AMD Ryzen 9 4800H  \n- **GPU**: NVIDIA RTX 2060 (notebook)  \n- **RAM**: 16 GB  \n\nWe evaluated all baselines on a headless machine with an AMD EPYC 9534 CPU, L40S GPU, 64 GB RAM, and 720×600 resolution. We can run 2 instances in parallel with a fixed 60 fps. Here’s the table when we stress-test the runtime performance with different settings.\n| Resolution | Rendering Quality | GPU Utils (%) | CPU Utils (%) | RAM (MB) |\n|------------|-------------------|----------------|----------------|----------------|\n| 640x360    | low               | 30.04         | 16.41           | 561.56         |\n| 720x600    | low               | 29.72         | 17.73            | 596.4         |\n| 1280x720   | low               | 26.5         | 18.56            | 734.81        |\n| 640x360    | high              | 30.03          |16.59            | 564.04         |\n| 720x600    | high              | 29.61          | 17.47            | 589.3        |\n| 1280x720   | high              | 27.44          | 19.08            | 738.6        |\n\n\n- **Human‑in‑the‑loop interface**. We support a human‑in‑the‑loop interface through which a human operator uses a mouse and keyboard to control the robot; all trajectories are recorded automatically as expert demonstrations for downstream training.\n## Q2: Baselines Evaluation under Different Settings\nTo clarify, our ROBOSCAPE‑MMNAV benchmark includes two distinct task settings to isolate how different conditions affect baseline performance in Table 2 and Table 3 at line 196. In the easy setting, all static obstacles (trees, benches, fire hydrants) and dynamic agents (traffic, pedestrians) are removed, so the challenge reduces to navigation in static environments. In the hard setting, we introduce both static obstacles and dynamic obstacles under sunny conditions. While baseline models achieve similar overall success rates and progress scores in both settings, they are unable to effectively avoid static and dynamic obstacles, consistently fail to yield to pedestrians, and do not adhere to basic traffic rules. None of the baselines could reliably complete the task under sunny setting and we could further anticipate that under more adverse weather or lighting condition—overcast skies, rain, or added visual occlusions—their performance would degrade even more.\n## W3: Lack of New Methods & Ablation Experiment\nThe goal of this work is the development of a highly customizable, photorealistic, city-scale simulation platform designed to provide the embodied AI community with training and testing environments that closely mirror real-world scenarios (Line 44). We further introduce two new benchmarks as case studies to illustrate RoboScape’s functionalities and versatility. Developing new methods is out of the scope of this work. We believe that our platform can be a valuable resource for the community to develop new methods in future work.\n\nWe conducted new ablation experiments to assess the contribution of each input modality, using the same GPT‑4o backbone.  Our basic setting includes a ground truth segmentation input, explicit ReAct prompt framework [2] and separate perception-action calls which simplify the mapping. While none of the implementations successfully reach the destination, we observe that removing the GT segmentation image, ablating the ReAct framework, or merging the perception-action calls into a single inference would decrease the performance of the baseline. Furthermore, the depth image and stripped vision input hurdle the performance.\n\nWe will include the following results in the revised Section 4.2 using an additional table.\n\nPerformance summary (subtask success rate):\n- **GPT‑4o**: 34.38%\n- **Merged perception-action call**:33.54%\n- **w/ depth**: 33.39%\n- **w/o explicit ReAct framework**: 32.21%\n- **w/o GT segmentation**: 31.90%\n- **w/ stripping**: 31.22%\n\nAnalysis:\n- The perception-action framework simplifies the mapping process and reduces hallucinations.\n- The model already has a certain depth prediction capacity. Adding depth images does not help significantly and may even introduce noise if the colormap is not well aligned.\n- ReAct framework does help reduce hallucination and stabilize the policy in complex scenarios\n- GPT-4o lacks sufficient training on intersection-related scenarios, but the GT segmentation image helps mitigate this limitation.\n- Although shown to be promising in [2], stripping the observation into vertical chunks increases the difficulty of image matching, which is crucial to our current simple settings.\n\n## References\n\n[1] ReAct: Synergizing Reasoning and Acting in Language Models  \n[2] NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models"}, {"review_id": "BAW6l3KDId", "reviewer": "Reviewer_kyYy", "summary": "This paper proposes a new simulation platform RoboScape for embodied AI tasks in large-scale urban environments. Based on the simulator, it proposes two new benchmarks: multimodal robot navigation and multi-robot search and benchmarks the performance of state-of-the-art vision language models.", "questions": "1. What is the running efficiency of the simulator (e.g. memory cost, fps)? Does it support running multiple instances in parallel in headless mode for efficient RL training?\n2. Why the authors only evaluate the performance of VLMs on the proposed benchmarks without also comparing with task-specific models or trained RL policies?\n3. Does the simulator support user-friendly customization of new tasks and generating new scenarios? The current two tasks introduced in the paper can only represent a small fraction of tasks that agents can perform in urban settings.", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. Based on an advanced graphics engine, the proposed simulator has better photorealism than existing urban simulators.\n2. It has more realistic pedestrian simulation with different behaviors.\n3. The proposed benchmarks are practical in the real world while less studied in the urban embodied settings.\n\nWeaknesses:\n1. The paper only benchmarks the performance of general VLMs without evaluating the performance of some task-specific models like robot navigation models or RL-based policies. Given the very low success rate of VLMs on these tasks due to lack of spatial understanding, it is questionable whether these benchmarks are truly useful without at least some reasonable baselines. \n2. It seems there's only one type of embodied urban robot in the simulator (the robot dog).  However, many other types of robots could appear in the urban setting like delivery robot, autonomous scooter, autonomous wheelchair, etc. \n3. Following 2, different types of robots could also have different dynamics and low-level control, while the simulator only models the action space with continuous translation and free rotation. Some of the actions may not be achievable by certain types of robots. This would lead to large sim2real gap if the simulator only models very high-level action.\n4. The authors also mention the RoboScape-20K training set as their main contribution, but it is not explained in the main paper with only some vague explanations in the supplementary.", "was_revised": true, "final_justification": "Based on the rebuttal and discussions, I have increased my score by 1 point to reflect the authors’ efforts in addressing my feedback on adding new RL/hybrid-based baselines. However, I still feel like the current status of the simulation framework could be further improved for better user experience and incline towards rejection.", "rebuttal": "We are happy that the reviewer considered our work “has better photorealism than existing urban simulators” and the two case studies “are practical in the real world while less studied in the urban embodied settings”. We address your questions and concerns below.\n\n## Q1: Running Efficiency of RoboScape\nRoboScape supports running multiple instances in parallel in headless mode for efficient RL training.\n\nWe evaluated all baselines on a headless machine with an AMD EPYC 9534 CPU, L40S GPU, 64 GB RAM, and 720×600 resolution. We can run 2 instances in parallel with a fixed 60 fps. Here’s the table when we stress-test the runtime performance with different settings.\n\n| Resolution | Rendering Quality | GPU Utils (%) | CPU Utils (%) | RAM (MB) |\n|------------|-------------------|----------------|----------------|----------------|\n| 640x360    | low               | 30.04         | 16.41           | 561.56         |\n| 720x600    | low               | 29.72         | 17.73            | 596.4         |\n| 1280x720   | low               | 26.5         | 18.56            | 734.81        |\n| 640x360    | high              | 30.03          |16.59            | 564.04         |\n| 720x600    | high              | 29.61          | 17.47            | 589.3        |\n| 1280x720   | high              | 27.44          | 19.08            | 738.6        |\n\n## Q2 & W1: Selection of Baselines\nWe appreciate the reviewer’s suggestion regarding the inclusion of task-specific models such as robot navigation systems or RL-based policies. We have conducted preliminary experiments with standard RL-based baselines (PPO). However, these approaches performed poorly in our proposed case studies, likely due to the complexity of instruction-driven tasks and the need for long-horizon reasoning. Designing high-performing policies for these tasks would require significantly more sophisticated architectures and task-specific tuning—an important direction for future work, but one that is orthogonal to the primary contribution of our paper. Our primary contribution lies in the introduction of RoboScape, an open-source and extensible simulation platform that supports customizable, diverse scenarios for evaluating embodied agents.\n\nWe have conducted the following additional experiments and will include them in the revised Table 2:\n- **Hybrid Baseline**: We have also evaluated the performance of a hybrid planner where a VLM, GPT-4o,  serves as the high‑level decision maker—deciding whether to continue straight down a street until the next intersection or to turn at that intersection—while A* is used as a low‑level path planner to generate and execute the movement commands. The subtask success rate is 32.53%.\n- **RL Baseline**:  We utilize two NVIDIA L40S GPUs, each running two parallel instances of RoboScape. Following the setting of VLN-CE [1], the agent is powered by a multimodal policy that combines DeBERTa-v3 for instruction encoding and DINOv2 for visual perception. We first pretrain the policy through imitation learning (specifically, behavioral cloning) on expert demonstrations, and then further finetune it using PPO. Rollouts are collected concurrently across all environments. The current RL baseline achieves a 28.37% subtask success rate, highlighting the challenging nature of our tasks. We believe RoboScape provides a strong foundation for future research on more advanced RL methods, by offering parallelizable, photorealistic, large-scale, and dynamic city environments, along with the ability to collect and incorporate human demonstrations that naturally include errors and corrective behaviors.\n## Q3: Customization\nHigh customizability is a core feature of RoboScape. We will outline how to create new tasks, scenarios, and embodied agents in Appendix A.4 to support flexible benchmark extension. The two benchmark tasks included in the paper are intended as case studies to demonstrate the functionalities and utilities of RoboScape. RoboScape was explicitly designed to support user-friendly customization of both tasks and environments.\n- **New scenarios**. Users can generate diverse and realistic urban layouts through our Python API by providing simple metadata inputs (e.g., number of streets, street length, object categories, and their spatial distributions, such as “10% trees, 5% tables and chairs”). This allows researchers to create arbitrarily large and varied city environments.\n- **New tasks**. Defining tasks in RoboScape is similar to standard Gym environments, as the Python API of RoboScape follows the same format for agent control (including pedestrians and vehicles). Users can spawn different types of agents, customize observation spaces (e.g., RGB, depth, or semantic segmentation) and action spaces (continuous or discrete actions), and program new goal definitions (e.g., language instructions, target images, or spatial objectives). Additionally, while the current pedestrians follow rule-based logic, users can override behaviors to simulate complex or rare cases. For instance, one can simulate a jaywalking pedestrian by scripting a few agents to cross during a red light, while placing a robot at the intersection to evaluate its reaction. \n## W2: New Embodied Robot\nWe fully agree that many other types of embodied agents (e.g., delivery robots, wheelchairs, humanoids, or drones) are relevant in urban settings. In RoboScape, we have included an additional embodied agent—a scooter—that can be controlled using velocity and angular velocity commands through our Python API, and it will be included in the revised Section 3.2. \n\nTo add a new robot type in RoboScape, one can leverage the existing Python API of RoboScape to conveniently customize action spaces—either continuous or discrete—as well as observation spaces such as RGB, depth, or semantic segmentation images. The required additional work (1) obtaining a new robot asset, typically from the Unreal Engine Marketplace; (2) defining the robot's actions using Unreal's Blueprint system; (3) integrating these actions with our Python API to enable high-level control; and (4) attaching our camera components to support the desired observation space.\n## W3: Action Space in RoboScape\nCurrent action space in RoboScape is consistent with other urban simulators like CARLA and MetaUrban, which adopt a similar high-level action space. Additionally, RoboScape is built on Unreal Engine, which supports integration with ROS2, allowing users to implement low-level dynamics and control if needed. This makes it feasible to reduce the sim2real gap by customizing robot-specific control pipelines.\n## W4: Details of Dataset RoboScape-20K\nThanks for the suggestion. We will move the details of RoboScape-20K from Appendix D.4 to the main paper. To further clarify, RoboScape-20K is a training set consisting of 20,000 robot navigation cases for RoboScape-MMNav. The training environments are entirely disjoint from the evaluation environments, and 33% of the building assets used during evaluation are intentionally excluded from the training set to ensure a clean generalization split.\n\nFor each training case, the input is the robot’s current observation, and the label is the correct next action to take. Since our refined pipeline includes a reasoning module, we additionally provide optional intermediate supervision signals—such as the correct facing direction and the ground-truth distance to the goal—to encourage chain-of-thought learning. These auxiliary labels are optional and can be ignored depending on the training paradigm and model architecture.\n## Paper Formatting\nWe will merge the appendix and supplementary materials into a single, complete appendix in the final version.\n\n## References:\n[1] Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments"}, {"review_id": "uYGmJ0uciF", "reviewer": "Reviewer_4XVJ", "summary": "This paper introduces RoboScape, a new embodied AI simulation platform for large-scale, photorealistic, dynamic urban environments built on Unreal Engine. RoboScape supports procedurally generated cities with vehicles, pedestrians, and multiple embodied agents controlled asynchronously. They introduce RoboScape-20K, a dataset of 20,000 steps over 200 episodes in 100 procedurally generated cities. The authors also introduce two benchmarks: 1) ROBOSCAPE-MMNAV: Single-robot, multimodal (vision + language) navigation in city-scale settings with traffic and pedestrians. 2) ROBOSCAPE-MRS: Multi-robot collaborative search requiring communication. Experiments show that current vision-language models perform poorly on these tasks without fine-tuning, exposing critical gaps in instruction grounding, spatial reasoning, and planning. Fine-tuning improves performance but remains limited.", "questions": "- Do you plan to add real-world sensory noise or domain randomization to enable sim-to-real transfer?  \n- Will you support continuous robot control as most controller take continuous reference input?  \n- Why were only VLM-based baselines included? Did you try classical planners?  \n- What are the compute requirements for procedural generation and simulation?  \n- Do you plan to release the full simulator and benchmark code for public use?", "limitations": "- The entire study is in a synthetic environment. There is no transfer to real robots or validation under real-world sensory noise. This limits immediate practical applicability.  \n- Although the simulator is built for photorealistic, dynamic urban settings, the robot action space remains discretized (move/rotate in steps). Continuous low-level control is not explored.  \n- The RoboScape-20K dataset uses expert trajectories without modeling errors or recovery. This limits robustness and generalization, as shown in the poor success rates even after fine-tuning.  \n- All baselines are vision-language model (VLM)-based. There is no comparison to classical planning, imitation learning, or hybrid approaches that might perform better.  \n- The traffic system is rule-based, and pedestrian behaviors remain relatively simple.\n- While the environments are procedurally generated at city scale (2 km²), the average navigation task length (500 m) is modest, and it’s unclear how well the simulation supports extremely long-horizon or many-agent scenarios in practice.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n- An urban simulators pocessing photorealism, procedural generation, agent diversity, and multi-agent control.\n- Robust engineering—procedural road/building/traffic generation, rich action/observation spaces, asynchronous multi-agent control.  \n- New benchmark MRS: multi-robot communication and search in large urban environments is very unique and meaningful.\n- RoboScape-20K provides large-scale training data with diverse city layouts.\n- Multiple VLM baselines (zero-shot and fine-tuned), realistic failure analyses.\n- Could become a standard evaluation platform for multimodal robot navigation research.\n\n---\n\n## Weaknesses\n- Entirely simulation-based—no real-world transfer or deployment results.  \n- Discrete robot action space despite claims of realism; continuous control not explored.  \n- Training data relies on oracle traces; no learning of error correction or reinforcement learning shown.  \n- Baselines are all VLM-based; no comparison to classical planners or hybrid methods.  \n- Traffic are rule-based, while in real world it is much more complex.\n- Scalability claims not fully stress-tested (episodes still ~500m on average).", "was_revised": true, "final_justification": "### Final Justification\n\n- **Resolved:**  \n  - **Sim-to-real considerations:** Authors clarified plans to inject realistic sensor noise and designed RoboScape with sim-to-real transfer in mind.  \n  - **Action space:** The platform already supports continuous action spaces, and integration with ROS2 is possible.  \n  - **Baselines:** Additional experiments with RL and hybrid methods were provided. Results confirm the difficulty of tasks and demonstrate that RoboScape can benchmark future methods.  \n  - **Compute requirements:** Authors provided both minimum and recommended specs, showing that the platform is accessible on modest hardware.  \n  - **Code release:** Authors commit to releasing binaries, Gym wrapper, executables, and containers to ensure broad usability.  \n  - **Scalability:** Authors clarified that RoboScape supports procedural generation of larger urban environments and significantly longer episodes.  \n\n- **Better to have:**  \n  - **Performance of baselines:** While RL and hybrid baselines were added, their relatively low performance highlights the challenge but leaves open how well current methods can adapt without heavy tuning.  \n  - **Traffic simulation realism:** Current rule-based traffic models are tailored to case studies; more sophisticated behavior remains future work.  \n  - **Dataset diversity:** Training data is currently expert-only; broader demonstrations (including noisy or human data) remain to be explored.  \n\n- **Weighting:**  \n  - The rebuttal successfully addressed core feasibility concerns (action space, compute requirements, scalability, code release).  \n  - Remaining issues (traffic realism, dataset diversity) are secondary and can be addressed by the community once the platform is released.  \n  - The main contribution—a large-scale, photorealistic, extensible simulation platform with meaningful benchmarks—is strong and unique.  \n\n**Overall:** The rebuttal improved confidence in the paper’s impact. Despite some remaining limitations, the platform and benchmarks represent a valuable community resource. I recommend to accept but keep my score.", "rebuttal": "We appreciate the reviewer for recognizing our contributions of “a new embodied AI simulation platform for large-scale, photorealistic, dynamic urban environments”, “very unique and meaningful” benchmarks. We address your questions and concerns below.\n\n## Q1&W1&L1: Sim-to-real\nYes, we fully agree that enabling sim-to-real transfer is crucial for embodied AI research. Although our current experiments focus on simulation, RoboScape was designed with sim-to-real considerations in mind. We plan to inject realistic sensory noise, such as image blurring, depth corruption, and camera jitter.\n\n## Q2&W2&L2: Continuous robot action space\nYes, we will. While the current benchmarks use a discrete action space for simplicity and clarity in the two case studies, RoboScape supports a continuous action space through velocity or acceleration commands. Additionally, RoboScape is built on Unreal Engine, which supports integration with ROS2, allowing users to implement low-level dynamics and control if needed.\n\n## Q3&W4&L4: Selection of baselines\nWe did conduct preliminary experiments with standard RL-based baselines (e.g., PPO) and a hybrid baseline. However, these approaches performed poorly in our proposed case studies, likely due to the complexity of instruction-driven tasks and the need for long-horizon reasoning. Our focus is on developing a flexible and extensible simulation platform. While important, designing high-performing policies for these tasks would require significantly more sophisticated architectures and task-specific tuning beyond the current state-of-the-art, which is out of the scope and orthogonal to the primary contribution of our paper.\n\nHaving said that, we have conducted additional experiments during rebuttal to include several non-VLM baselines and will include them in the revised Table 2:\n- **Hybrid Baseline**: We have evaluated the performance of a hybrid planner where a VLM serves as the high‑level decision maker—deciding whether to continue straight down a street until the next intersection or to turn at that intersection—while A* is used as a low‑level path planner to generate and execute the movement commands. The subtask success rate is 32.53%.\n- **RL Baseline**:  We utilize two NVIDIA L40S GPUs, each running two parallel instances of RoboScape. Following the setting of VLN-CE [1], the agent is powered by a multimodal policy that combines DeBERTa-v3 for instruction encoding and DINOv2 for visual perception. We first pretrain the policy through imitation learning (specifically, behavioral cloning) on expert demonstrations, and then further finetune it using PPO. Rollouts are collected concurrently across all environments. The current RL baseline achieves a 28.37% subtask success rate, highlighting the challenging nature of our tasks. We believe RoboScape provides a strong foundation for future research on more advanced RL methods, by offering parallelizable, photorealistic, large-scale, and dynamic city environments, along with the ability to collect and incorporate human demonstrations that naturally include errors and corrective behaviors.\n\n## Q4: Compute requirements\nTo ensure wide accessibility, RoboScape supports adjustable rendering resolutions, allowing deployment across both high-end servers and modest laptops. \n#### Recommended Setup\n- **CPU**: Intel Core i7‑12700H or AMD Ryzen 9 5900HS  \n- **GPU**: NVIDIA RTX 3070 or GPU with more than 6 GB  \n- **RAM**: 32 GB  \n\n#### Minimum Setup (60 FPS for RoboScape)\n- **CPU**: Intel Core i7‑11300H or AMD Ryzen 9 4800H  \n- **GPU**: NVIDIA RTX 2060 (notebook)  \n- **RAM**: 16 GB  \n\n## Q5: Release the full simulator and benchmark code\nYes, we plan to release the full simulator and benchmark code.  Upon publication, we will open-source all binaries, the Gym wrapper, and provide both a Windows executable and a Singularity container for Ubuntu and macOS, allowing users to run RoboScape out of the box without compilation.\n\n## W3&L3: RoboScape-20K Dataset\nIndeed, current training data only consists of expert (oracle) trajectories. However, we emphasize that RoboScape is fully compatible with reinforcement learning, which could potentially address the limit as shown in the additional experiments. Furthermore, RoboScape provides a human interface for collecting trajectories from human users, which naturally include suboptimal and error-correcting behaviors. This enables future work to explore learning from human corrections or noisy demonstrations. \n\n## W5&L5: Rule-based traffic simulation\nAll pedestrians and vehicles in RoboScape are independently controllable and support asynchronous control. By applying more sophisticated control policies to each pedestrian, RoboScape is capable of supporting highly realistic traffic simulations. Currently, the traffic system is tailored to support our two case studies, primarily to demonstrate RoboScape’s capabilities as a flexible and extensible platform.\n\n## W6&L6: Scalability\nWhile the average episode length in our current experiments is around 500 meters, this setting already reveals significant performance degradation across baselines, highlighting the challenge even at this scale. Importantly, RoboScape is built with scalability in mind: it supports procedural generation of much larger urban environments and can accommodate significantly longer episodes. This design ensures that researchers can easily stress-test navigation agents in more complex, long-horizon tasks, which we plan to explore in future benchmarks.\n\n## References:\n[1] Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments"}]

Author rebuttal:

Meta-review: The paper presents a simulation platform for embodied AI in large-scale urban environments. The paper further introduces two benchmarks for multi-modal instruction following for navigating, and multi-robot search. Experimental results demonstrate shortcomings of state-of-the-art VLMs.

The reviewers appreciated the introduction of a high-fidelity simulator and highlighted the rich action and observation space, the meaningfulness of introduced benchmarks, and the analysis of benchmark results. Concerns were raised on the selection of baselines, which amounted to be only VLM-based instead of being task-specific, and only one baseline being fine-tuned for the task at hand. Further concerns regarded missing technical details, missing ablation experiments, and limitations in the available types of robots, and the restriction of the training set to oracle traces. The rebuttal and discussion added additional baselines and experimental results, technical details, and clarified remaining concerns. While it shows that an impressive amount of work went into the development of the simulator, it seems that more time could have been spend on the implementation of baselines and polishing the presentation. The additions made in the rebuttal clearly strengthened the paper here. Although a few things could still be improved (e.g. dedicated baselines developed for the benchmarks, built on findings from the zero-shot VLM baselines), they are not preventing a recommendation to accept the paper.
Decision: Accept (poster)