SimWorld: An Open-ended Simulator for Agents in Physical and Social Worlds
Authors: Xiaokang Ye
Jiawei Ren
Yan Zhuang
Xuhong He
Yiming Liang
Yiqing Yang
Mrinaal Dogra
Xianrui Zhong
Eric Liu
Kevin Benavente
Rajiv Mandya Nagaraju
Dhruv Vivek Sharma
Ziqiao Ma
Tianmin Shu
Zhiting Hu
Lianhui Qin
Venue: NeurIPS 2025 spotlight ()
Keywords: Multi-Agent Simulator
Real World Agent
Social Simulation
Physical Simulation
Primary area: applications

Abstract: While LLM/VLM-powered AI agents have advanced rapidly in math, coding, and computer use, their applications in complex physical and social environments remain challenging. Building agents that can survive and thrive in the real world (e.g., by autonomously earning income) requires massive-scale interaction, reasoning, training, and evaluation across diverse scenarios. However, existing world simulators for such development fall short: they often rely on limited hand-crafted environments, simulate simplified game-like physics and social rules, and lack native support for LLM/VLM agents. We introduce SimWorld, a new simulator built on Unreal Engine 5, designed for developing and evaluating LLM/VLM agents in rich, real-world-like settings. SimWorld offers three core capabilities: (1) realistic, open-ended world simulation, including accurate physical and social dynamics and language-driven procedural environment generation; (2) rich interface for LLM/VLM agents, with multi-modal world inputs/feedback and open-vocabulary action outputs at varying levels of abstraction; and (3) diverse physical and social reasoning scenarios that are easily customizable by users. We demonstrate SimWorld by deploying frontier LLM agents (e.g., Gemini-2.5-Flash, Claude-3.5, GPT-4o, and DeepSeek-Prover-V2) on both short-horizon navigation tasks requiring grounded re-planning, and long-horizon multi-agent food delivery tasks involving strategic cooperation and competition. The results reveal distinct reasoning patterns and limitations across models. We open-source SimWorld and hope it becomes a foundational platform for advancing real-world agent intelligence across disciplines. Please refer to the project website for the most up-to-date information: http://simworld.org/.

SIMWORLD: An Open-ended Simulator for Agents in Physical and Social Worlds
Xiaokang Yel
Jiawei Ren
Yan Zhuang?
Xuhong He*
Yiming Liang*
Yiqing Yang
Xianrui Zhong'
Mrinaal Dogra'
Eric Liu
Kevin Benavente'
Rajiv Nagaraju
Dhruv Sharma!
Ziqiao Ma
Tianmin Shu
Zhiting Hulf
Lianhui Qin
UCSD UVA
JHU 4CMU Purdue Polyy UIc USC
UMich
Eqqua contribution.
"qqual advising.
Roalistic, Open-ended
Rich Intorlace for LLMVLM Agents
Diverse Physical and Social Reasoning Scenarios
World Generation
Cargo
$20
Human
Floxible Agent Controliers
Meettpontt
Soctal Reesoring Seanang
Realstic Phyrical Scbnarg
Open Vocabulary
Social Norms
Economic Behavior
Share?
Spead
Procedura
Language
Efficiency
Generation
Control
Mutti Foodback
Social Interaction
Obtest Interaation
Figure WORLD (1) simulates open-ended environments with realistic physics and social rules, (2) provides native support for M.M/VVM agents to deploy and interact in the simulated worlds, and (3) facilitates the construction of diverse scenarios for Al agent evaluation and development.
Abstract
While LLM/VL.M Al agents have advanced rapidly in math, coding, and computer use, their applications complex physical and social environments remain challenging. Building agents that survive and thrive in the real world (e.g.. by autonomously earning income) requires massive-scale interaction, rea- soning, training, and evaluation across diverse scenarios. However, existing world simulators for such development fall short: they often rely on limited hand-crafted environments, simulate simplified game-like physics and social rules, and lack native support for LLMM/VLM agents. We introduce SORROR.. a new simulator built on Unreal Engine 5, designed for developing and evaluating LLM/VLM agents in rich, real-world-like settings. SIMWORLD offers three core capabili- ties: (1) realistic, open-ended world simulation, including accurate physical and social dynamics and language-driven procedural environment generation; (2) rich interface for LLM/VLM agents, with multi-modal world inputs/feedbbac and open- vocabulary action outputs at varying levels of abstraction; and (3) diverse physical and social reasoning scenarios that are easily customizable by users. We demon- strate SIMWORLD by deploying frontier LLM agents (e.g., 2...iii2..--aaa Claude-3.5 5, GPT-4o, and DeepSeek- SrepSeek-Prover-Oe gation tasks requiring grounded re-planning, and long-horizon multi-agent food- delivery tasks involving strategic cooperation and competition. The results reveal distinct reasoning patterns and limitations across models. We open-source SIM- WORLD and hope it becomes a foundational platform for advancing real-world
on both short-horizon navi-
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

agent intelligence across disciplines. Please refer to the project website for the most up-to-date information: https://smmworld.org
Introduction
Large language models (LLMs) have emerged as powerful foundations for intelligent agents, demon- strating strong reasoning capabilities, particularly in domains such as mathematics, coding (e.g., RI, o3), and digital tool use (e.g., web navigation). However, these domains are relatively clean and well-structured, offering clear feedback, unlike the noisy, dynamic nature of the physical and social world where real-world agents are ultimately expected to operate. In practice, such agents must interact with rich, unpredictable environments such as navigating cities, engaging with people, and even making strategic decisions to carn living [1]].
To advance agent development, recent efforts have explored simulation environments that offer rich interactive experiences for training and evaluation (Table DT However, game-like platforms such as Minecraft [2 50 32
29 28 and Pokémon [16 provide accessible setups for embodied inter- action but lack realistic physical dynamics and social structures, limiting real-world generalization.
Domain-speciin simulators such as CARLA [10] and A12-THOR [23 target areas like autonomous driving and household robotics but are limited to narrow task scopes or static environments. Social sandboxes [8] such as Virtual Village [36], simulate interpersonal interactions in scripted, small- scale communities, but lack the open-endedness and scalability required for modeling richer social complexity. Morcover, many of these environments do not support natural language interfaces for goal setting, planning, and control, limiting their compatibility with LLM-based agents.
To meet these growing demands, we present SIMWORLD, a platform designed to support the development and evaluation of LLM (and VLM) agents in complex, dynamic, and interactive environments. SIMWORLD is grounded in three core design principles (Figure D
1) Realistic, Open-Ended World Generation. SIMWORLD advances simulation by integrating two key capabilities: realism in physical and social dynamics, and open-ended, language- world generation. On the realism front, SIMWORLD generates hierarchical, city-scale 3D environments grounded in physical laws (e.g.. gravity, momentum) and enriched with dynamic features such as lighting, weather, and pedestrian flow. It also embeds socially grounded behaviors- such as obeying traffic signals and maintaining personal space- directyy into agent logic to support realistic interactions. On the open-ended side, SIMWORI WORLD supports infinite environment expansion through procedural generation, enabling diverse road networks, building layouts, and urban configurations. Users or Al agents can modify scenes on-the-fly fly via natural language commands (e.g.. "add a tree next to the hospital") thanks to WORL''''ss LLM-based environment editing and asset generation modules, allowing for adaptive environment creation.
2) Rich Interface for LLM/VLM Agents. SIMWORLD connects language models with interactive environments by enabling LLM/VLM agents to perceive rich multimodal observations, including visual scenes, abstract layouts, and action feedback. Agents can respond with high-level natural language actions. For example, an agent may reason and generate an abstract action, "sit on the nearest chair, , which SIM WORLD translates into a sequence of low-level actions such as moving through waypoints and sitting down. After executing the actions, the simulator provides updated observations and feedback, allowing the agent to adjust its strategy. This closed-loop interaction supports open-ended, language-driven behaviors and empowers agents to perform long-horizon reasoning at a proper abstraction level.
3) Diverse Physical and Social Reasoning Scenarios. Building on the above physically/socially grounded environments and LLM/VLM agent interface, SIMWORLD natively supports systematic evaluation and training of agent reasoning in realistic settings. To show how these capabilities work in concert, we present two case studies spanning both physical and social reasoning. Case Study 1: Grounding Agent Physical Reasoning and Planning involves a usioonbbsed navigation task where agents interpret visual input, avoid obstacles, and follow traffic rules in real time. Case Study Strategic Multi-Agent Collaboration and Competition models an urban delivery economy in which agents bid, invest, and share orders while navigating dynamic environments. With different personas, budgets, and tools (e.g., scooters), agents develop diverse strategies shaped by their goals and shifting conditions (e.g., fluctuating prices). The task highlights complex decision decision-making and long-horizon planning, where cooperation and competition emerge naturally.

---

Table 1: Comparative Analysis of SIMWORLD and Existing Simulators across key dimensions: Open-ended World (procedural scene/asset generation, Social Realism (how close to real-world mechanies) Action Space (action abstraction level, open- vocabulary action space), Agent Type (types of controllable agents: Humanoid (Hum.), Robot, Drone or Vehicle (Veh.),, and Physies Engine (underlying simulation engine). means high-level actions (e.g., deliver, navigate to), and L means levelevel actions (e.g., "forward by 1 step"). See Appendix for more discussion of related work.
editing), Physical/-
Simulator
Open-ended World
Physical/Social Realism
Action Space
Agent Type
Physies Engine
Procedural Lang-Ctrl
Abstr. Open-Vocab
Minedojo 12
L
X
Hum.
Minecraft
Mindcraft
M0
H
Hum.
Minccraft
MetaUrhan
52
T.
X
Veh.
PyBuller
EmbodiedCity
1k]
x
L
X
Drone/
Unreal Engine
CARLA [M
X
640
T.
Veh.
UR. & Unity
GRUlopia
++
Hum Rooho
Isaac
OmniGihson
25
+4
HA
Robot
Omniverse
A12TTHOR
++
Unity
Habitat 3.0
X
++
X
Hum /Robot
Bullet
VinualCommunily
++
Hum /Rooo
Genesis
UnrealZooo 63
X
X
Hum Vel.
Unreal Eiggie
SIWWORLD
+++ MM..
Hum Rohhs//hho/V
Unreal Engine
We evaluate frontier LLMs as agents including GPT-4o, Claude-3. 3- Gemini-2.4e 5-Pro and others on the two challenging tasks described above in SIMWORLD. the navigation task (Case 1), GPT-4o and Cl Claude-3. 7-Sonnet show strong planning efficiency and high success rates, but consistently ignore red lights and fail to adjust their view to perceive traffic signals. This reveals disconnect between passive visual perception and active attention, the models often fail appropriately on what they observe. In the multi-agent delivery task (Case 2), agents must operate in a dynamic, competitive economy. Claude 5..---nnnee and DeepSeek- V3 earn the highest on low-value orders or spending all their money on
profits, but often behave erratically-overiidddne 5-Fassh scooters they never use. In contrast, Ge and conservative, stable strategies, trading peak performance for consistency. Personality traits also shape agent behavior: conscientious agents focus on task completion, while open agents explore but frequently lose money. These findings expose both the strengths and current limitations of L..M-based agents, while revealing rich, often unexpected behaviors that emerge from their interaction with complex environments in SIMWORLD.
Prover- follow more
2
The SIM WORLD Simulator
Realistic, open-ended, and natively LLM/VLM-compaaübl simulators are essential for developing agents complex physical and social environments. WORLD takes a step toward this goal with a two-tier architecture: Unreal Engine (UE) backend and fully integrated Python layer. The UE backend provides state-of-the-art 3D rendering. real-time physics simulation, and highly interactive, customizable environments. The Python layer bridges this with high-level functionalities, including procedural and language-de -diuue scene generation, rich input-output interfaces for LLM/VLM agents, and modular support for constructing diverse, customizable physical and social reasoning scenarios.
2.1 Realistic, Open-Ended World Simulation
Procedural and L.M--Cotrollabble Environment Generation. Previous simulators typically rely on a limited set of hand-crafted scenes (e.g., 15 in CARLA [10], 211 in Habitat 3.0
(39)
SIMWORLD develops a procedural generation system (37 Appendix A.I.T capable of producing diverse, randomized urban and finc-grained elements like street furniture-cnabling enabling effectively infinite simulation scenarios. All parameters (e.g., city size, building density, vehicle and pedestrian count) are customizable, allowing users to generate varied and controllable environments with minimal manual effort
-nnlluunn-nnlludinng road networks, building layouts, dynamic traffic,
Beyond randomized procedural generation, SIMWORLD supports natural language- based scene editing, enabling dynamic world construction through open-ended instructions (Figure[]aa Users or agents can modify scenes on-the-fly with commands such as "add a red sports car next to the hospital near museum". SIMWORLD contains retrieval-augmeened L.Le- scene agent that grounds the command by querying the current environment's scene graph. The agent identifies the intended location using spatial anchors (c.g., "hospital") and contextual landmarks

---

Realistic World Generation
LLM Controliable Environment Edit
LLM/VLM Agent Interface
Mulliccde Obsorsston feetboch
Por the vemgiil
SmiWorld
Uides
Tolks
SirWorld
Headan
(a)
(D)
Figure 2: Realistic Open-ended Simulation and Rich Agent Interface. (a) SIMW WORL simulates environment with realistic physical and social mechanics (left) and enables language control for real-time environment editing (right) (Section 2.1 (b) SIMWORLD supports rich multi-modal inputs
and feedback, and accept open-vocabulary high-level actions, to facilitate complex LLM//LM agent reasoning at proper abstraction levels.
("museum"), ") retrieves matching asset from library, and inserts it accordingly. If suitable asset is unavailable, the agent invokes an off-the-shelf text-to-3D generation model [46 to synthesize a new object from the prompt ('red sports car"), converts it into compatible format, and integrates it into the environment. This approach enables semantically grounded, spatially coherent, and scalable world construction, laying the foundation for interactive and compositional simulation.
Realistic Physical and Social Simulation. Powered by UE-5, SIM ORLD provides accurate and continuous physical simulation. Unlike popular environments such as Minecraft, which rely on discrete, block-based mechanics without real gravity or momentum, SIMWoRL models real- world physical dynamics. Agents experience physical forces such as gravity, mass, and inertia: they may slide down slopes, lose balance while pushing heavy objects, or trip over uneven terrain. This realism is essential for supporting grounded, embodied interaction and physically plausible agent behavior.
Social realism is similarly emphasized. SIMWORLD simulates dynamic traffic systems with both vehicles and pedestrians, adhering to real-world norms like crosswalk rules, traffic lights, and personal space. This realism supports high-fidelity, reactive mobility and interaction within urban environments. Together, these physical and social layers provide a rich substrate for evaluating agent reasoning and behavior in real human- Mike scenarios. Complex multi agent collaboration and competition dynamics are further supported and described in later sections.
2.2 Rich Interface for LLM/VLM Agents
SIMWORLD designed to make the deployment of L.M//VLM agents straightforward and flexible by exposing rich, modular input-output interface (Figure 2b)). This design supports diverse forms of evaluation, reasoning, and training for agents operating in complex environments.
For an agent's 's inputs. SIMWORLE provides rich multi-modal sensory information and enviromment feedback that mimic what humans would perceive in the real world. Inputs include raw RGB images, depth maps, segmentation masks, and structured semantic scene graphs, enabling agents to perceive both pixel-level signals and symbolic abstractions. Additionally, agents receive environment feedback in response to their actions. For example. agent attempting to walk into a closed door would receive a collision event and an updated scene state, allowing it to revise plan. To further support agent navigation, STMWORLD implements a waypoint-based abstraction layer, similar as in the popular driving simulator CARLA [10] (Appendix(A.1.2) The waypoint system offers high-level reference points along streets and intersections. This enables agents to plan over meaningful spatial landmarks without needing to handle low-level locomotion mechanics. Collectively, these input modalities provide agent developers with a broad design space to tailor LLM/VLM agents based on their desired level of perception and abstraction.
For an agent's 's outputs. SIMWORLD moves beyond the restrictive, low-level action spaces of many existing simulators (e.g., "move forward by block" in Minecraft). Instead, SIMWORLD supports an open- vocabulayyy action interface that accepts natural language commands at varying levels of abstraction. For instance, an agent can issue high-level instruction such as "go to the nearest chair and sit down". A built-in local action planner interprets this instruction and decomposes

---

Case 1
Agent Physical Reasoning
Case 2 Multi-Agent Collaboration and Competition
Tank:
Carier State
Rearaeiig
Persuing
1011 107, to dutiuatr
Caver
20g
Profit
Reparement Aeid
Follwctttl
on Cade Lane
Making Investment
Obonaation:
Paared rur sarpores
Careldana aod woppaina
Currert State
Reaseeing Lor
Collaborating
de Goed
Action
Do arbio
Durrent State
Reasening
Competiting
Tarn aettiiht
Berry 300mch
Dhocou new wayport
Figure 3: Overview of Two Case Studies, showcasing the social-to-physical full stack simulation capability of SIMWORLD. Case lemphasizes physical reasoning, where agents aim to reach their destinations safely and efficiently. Case 2 focuses on social reasoning, requiring multi-agent col- laboration and competition in a real-world delivery scenario. Each agent is initialized with distinct personalities and internal states, and can act grow, thrive, and ultimately maximize their earnings.
it into sequence of low-level actions executable by the simulator. The planner consists of two components: an LLM-based parser (e.g., GPT-4o or any user-specified LLMs) that grounds and breaks down high-level intentions (e.g., "navigate" and "sit"), and an executor that handles actual environment interaction. Depending on the task, the executor can be a simple reliable rule-based system (e.g., following the shortest path to an object) VLM-based planner for more complex, perceptionnssnnitiiee scenarios (e.g.. avoiding dynamic pedestrians en route). Our ablation study (Section shows the local action planner substantially facilitates LLM planning.
This layered design enables LLM/VLM agents to focus on long-horizon reasoning and high-level decision-making while offloading low-level physical execution to the simulator. As result, SIM- WORLD bridges the gap between high-level cognition and low-level embodiment, making it a practical and extensible platform for building and evaluating intelligent agents.
Besides the above advantages, SIMWORLD offers tools for defining tasks, agent roles, rewards, and evaluation metrics. We showcase how to build analyses on SIMWORD in Sections 3 and (Figure
3) and the workflow for multiagent communication and physical world creation Appendix A.2
Case Study 1:
Grounding Agent Reasoning and Planning to Physical World
Formulation. Embodied reasoning and physical interaction are evaluated in SIMWORLD through vision-based urban navigation task. In this task, agents are required to reach a designated goal while cireumventing both static (e.g., trees, benches) and dynamic (e.g., pedestrians) obstacles. This necessitates multimodal perception and real-time decision decision-making making under uncertainties and unexpected events (34). Two distinct settings are considered: with and without a local action planner. The planner provides high-level A b-badd route without considering obstacles; agents follow this path waypoint- wuyyoint-by-waypoir by-wappoint while avoiding obstacles. In the setting without the planner, agents navigate towards the goal position with visual observations. Agents choose from discrete action set at each step (see below).
Average Distance Goal
Action Space. At each decision step. agent may
choose from the following actions: Do Nothing (re-
Distance to Goal
2000
Arrival threshold
main stationary). Step (move forward or backward for a fixed duration). Turn (rotate left or right by a
Gemini
4000
GPT-4
continuous angle). Change View (adjust pitch or field of view), and Change Next Waypoint (local action planner only, switch to an alternative waypoint).
6000
Claude
8000
10000
Benchmark Environment. Experiments are con- ducted with two virtual cities in SIMWORD WOLLD, featur- ing diverse sidewalk layouts and realistic obstacles. Fach sidewalk is automatically annotated with both coarse- and fine-grain grainndi waypoint graphs to support flexible control and obstacle placement. Static obstacles such street furniture and trees are randomly placed along fine-grained waypoints, creating
12000
20
80
100
Step
Figure 4: Step-wise average destination dis- tance across tasks for various models.
5

---

Table 2: Performance Comparison across Different Difficulty Levels with Static Obstacles. SR: Success Rate, CC: Collision Count, CC-S: Collision Count (Success Only), RVR: Red-light Violation Rate, STR: Stuck Rate, NDC: Normalized Decision Count. E: Easy, M: Medium, H: Hard. RVR is not 100% as agents may happen to cross during green lights. An ablation study with only red lights presented to the agent resulted in a 100% red-light violation rate, confirming the agent's s tendency to ignore red signals when encountered.
Baseline
SR
CCL
CC-S)
RVR.
STR
NDC.
M
M
H
E
M
Rule-based
66.73 63.33
40.0
1,80
4.14
4.00
1.55
4.74
4.67
0.00
100.00
100.00
100.00
N/A
N/A
N/A
GPT-4o-mini
60.00
20.00
16.67
5.97
11.13
5.33
4.50
7.33
6.00
80.00
50.00
16.77
32.00
3.98
2.28
4.25
GPT-4o
93.33
93.33
93.33
1.73
3.37
5.10
1.64
3.46
5.07
67.86
100.00
100.00
50.00
2.22
2.48
2.32
Claude-3,--Somee
86.67
96.67 93.33
2.00
4.13
4.63
1.81
4,10
4.50
46.43
100.00
0.00
100.00
2.07
2.07
2.12
Gemini-2.5-Pro
96.70 83.30 80.0
1.50
2.80 3,53
145
2.76
3.08
62.50
100.00
40.00 50.00
2.16
2.51
2.50
spatial constraints. Dynamic obstacles (pedestrians) follow predefined motion patterns across selected
sidewalks. Tasks are categorized by complexity: Easy (a single, straight segment), Medium (a single turn at an intersection), Hard (multiple turns across four segments), and Dynamic (pedestrian interference along straight path).
Baseline Agents. VLM agents (GPT-4o, Claude-3. Claude-3.7-Sonne 7SSonett, Gemini-2 2.t.iii..5-Prer 2-roro etc.) are com- pared against a rule-based baseline that strictly follows the planner-generated path. VLM agents provided with visual input, action history, and task metadata to inform action selection at each step. Further details are available in Appendix B
Metrics. Agent performance evaluated along three axes: task success, safety, and decision efficiency. We report the following metrics: Success Rate (SR): fraction of tasks completed without timing out, getting stuck, or deviating significantly; Collision Count (CC): total collisions with static dynamic obstacles; Collision Count in Success (CC-S): collisions in successful episodes, reflecting safety upon goal completion; Red-light Violation Rate (RVR): proportion of successful episodes with red-light violations; Stuck Rate (STR): fraction of failed episodes where the agent remains immobilized for over minutes; Normalized Decision Count (NDC): VLM decisions per waypoint in successful episodes; and Decision Steps in Success (DSS): average number of VLM decisions in successful episodes, indicating planning efficiency under decision constraints. Formal definitions and computation protocols are provided in Appendix
3.1
Main Results: Agent Planning With Local Action Planner
Static Obstacles. We evaluate three levels of task difficulty (Easy, Medium, Hard) under static obstacles with local action planner. Time limits are to 15 minutes for Easy and Medium tasks, and 25 minutes for Hard tasks. Four VLMs are evaluated: GPT-4o-ming GPT-4o, Claude-3. 3.aue-3.7-SSommee 7SSoneet. and 5- and summarized below.
Results are shown in Table with key insights discussed in Appendix B.7
Takeaway: Replanning with Static Obstacles
Model capability plays a decisive role in determining task success, yet current VLMs exhibit limited spatial reasoning and consistently fall short of functioning as truly embodied agents. Despite this, different models display distinct operational strengths, indicating varied but incomplete competencies across the board.
Dynamic Obstacles. To assess agents' dynamic obstacle avoidance, we test tasks with walking pedestrians. VLM agents are limited to 30 decision steps, while the rule-based baseline is given 15 minutes. Results and analysis follow. We summarize the takeaway below and leave experimental details in Appendix B.7
Takeaway: Replanning with Dynamic Obstacles
GPT-4o and Claude 3. 7-Sonntt excel in dynamic scenarios with high task success and efficient decision-makinn Gemini-2 12ami-2.560 5-Pro shows superior obstacle sensitivity, achieving the fewest collisions, but its cautious behavior leads to more frequent stalls and lower task success (Table

---

3.2 Ablation Study: Without Local Action Planner
To evaluate the robustness of VLM-driven agents under more challenging conditions, remove the local action planner and re-evaluate their performance on Medium-level tasks. In this setting, agents only provided with the final destination instead of pre-defined series of waypoints. They required to not only avoid collisions but also plan an efficient trajectory autonomously. All agents illustrates FilutreBlllssrrtes the progression of the average distance to the goal for each model in a single experimental run. We summarize the key findings below, with detailed experimental settings provided in Appendix BB8
restricted to a maximum of 90 decision steps per task. Figure Tigure4iilusse
Takeaway: Navigation without Local Action Planner
Gemini-2.3 5-Pro demonstrates superior autonomous planning and obstacle avoidance with- out local guidance, while Claude-3. 3.. adapt without pre-defined waypoints (Ta
7-Sonnet is efficient but less safe. GPT-4o fails to
4 Case Study 2: Strategic Multi-Agent Collaboration and Competition
Formulation. evaluate the social reasoning capabilities of foundation models realistic, open- world urban environments, we propose strategic multi-agent delivery task that involves both collaboration and competition. LLM-based agents are deployed as delivery personnel in a city-scale environment built using SIMWORLD. Their goal is to grow and thrive in this dynamic setting. To enhance realism and complexity, the environment incorporates several dynamic systems: (1) an
energy system, where agents must manage stamina and replenish it through consumables; (2) an economic system, where agents earn and spend currency on purchases such as scooters and drinks; (3) an order-sharing mechanism, enabling agents to collaborate by sharing delivery tasks and optimizing group performance. These components create a rich, interactive simulation environment for evaluating agents' decision-making, adaptability, and social reasoning complex urban scenarios.
Environment. All experiments are conducted on the same map generated by SIMWORLD. To ensure fair and efficient evaluation, graphical rendering is disabled during simulation. However, the physical simulation and evaluation threads remain active to preserve environment dynamics and task fidelity.
Action Space. The task features a two-tiered action space: high-level actions decided by LLM- agents and low-level actions executed by the SIMWORIE local action planner. each decision step, the agent may choose: Bid Order (offer a price to compete for a new order), Pick Up Order (navigate to the pick-up point), Deliver Order (complete delivery the destination), Share Order (publish the order for collaboration), Cancel Share Order (withdraw a previously shared order), Go to Meet-point (head to the coordination point for a shared order), Purchase Drinks (buy consumables to restore energy), Adjust Speed (cancel order or modify travel speed dynamically). Full action space shown in Table 7
Baseline Agents.
We evaluate multiple foundation models as the backbone of delivery agents, includ- ing Claude-3. 5-Soneet, DeepSeek- VV3, GPT-4o. Gemini-2. Gemini-2.5-Flashh. 5-alash, and QWQ. The ReAct (56) prompting framework is employed to explicitly separate reasoning and action selection. Further implementation details can be found in Appendix [
Metrics. Aligned with the agent's 's hierarchical decision-making, we design a three-level evaluation framework. Overall performance measured by total profit, while order success rate, energy efficiency, order sharing count, and investment count assess operational effectiveness. Additionally, we validate individual actions against simulation rules to ensure stability and evaluate the agent's instruction-folle following and rule compliance under complex constraints. More details are provided in Appendix C
4.1 Main Results
For each evaluation, we sample 20 agents controlled by the same language model, each running for 5000 simulation steps. At each step, an agent issues two API requests, averaging around 7000 tokens per request. Based on results in Table 3, empirical analysis over three simulation rounds reveals distinct operational behaviors across models. Key insights are discussed in Appendix C.5.1 and summarized below.
7

---

Table 3: Performance of Model-controlled Agents. Metrics are reported as mean (Avg) and standard deviation (Std) over three step simulations. Bold indicates the best Avg per column.
Model
Profit
Successful Orders
Energy Efficiency
Sharing Count
Investment Count
Avg
Std
Avg
Std
Avg
Std
Avg
Std
Avg
Std
DeepSeek-V3
69.48
16.77
2.10
0.47
0.34
0.07
2.33
0.47
8.00
3.00
Claude-3.5-Sone
69.07
20.69
2.73
1.10
0.54
0.20
11.33
8.39
9.00
3,46
GPT-4o
43.91
14.16
1.63
0.43
0.30
0.06
0.67
0.47
4.67
0.47
Gemini-2.5 2.5--hshh
42.42
3.10
2.10
0.17
0.17
0.04
2.67
1.25
2.00
2.00
28.72
12.04
1.53
0.58
0.11
0.03
0.67
0.47
0.67
1.00
Qwen3-32B
24.73
7.95
1.37
0.13
0.40
0.17
1.33
0.47
5.33
2.06
DeepSeek Prover-V V2
DeepSeek-Prover--2
21.66
7.18
0.67
0.14
0,42
0.03
7.33
8.39
1.00
Qw
17.31
4.07
0.87
0.20
0.41
0.20
0.33
0.47
3.33
2.52
GPT-4o-mini
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Takeaway: Model Performances in Multi-agent Tasks
Top-performing models like DeepSeek- V3 and Claude-3. 5-Sonnet achieved high average profits but exhibited significant variability, while Gemini-2 5-Flash showed more stable, though moderate, performance. mo-mio failed entirely across all metrics. Overall, the results highlight a trade-off between maximizing average performance and ensuring consistent, reliable behavior.
4.2
Ablation Study
To take a step deeper into multi-agent collaboration and competition, we conduct three ablation experiments. Model Competition, sample 24 agents and 12 models that each model controls two agents running the experiment for 1000 rounds. In this ablation experiment, we study how models make choice within highly competitive environment create much benefit as they can. Environment Configuration, we control two environment configurations, initial financial amount and global order number. For each configuration, we sample several stages from low to high to how agent's s behaviors are influenced by the global environment setting, Persona, we use the model with the best performance to control the agents with persona description in the prompts, we sample 20 agents and each persona is allocated to two agents, and we observe how different personas influence the agents' behaviors and decision strategies.
Model Competition. To intensify inter-agent competition, we constrain each agent to handle at most order time and set the environment's hunger rate to 0.9, ensuring high demand for delivery. In each experimental session, 24 agents are jointly controlled by 12 different models, where each model governs two agents. These agents actively bid for orders in shared environment with the goal of maximizing profit. Each session runs for 1000 simulation steps, and results are averaged across three random seeds.
As shown in Figure 5a models exhibit distinet bidding behaviors. Notably, Claude-3. 7-Sonnet, Fomini-2.5-Flast and Gemini-2.. (a.ii-2.0-Faa) O-Flash demonstrate broad bid price distributions, indicating a flexible bidding strategy. This flcxibility increases their chances winning orders when in competition with other models. In contrast, models such as LLaMA-4-Scout and LLaMA-3. 2. 11b tend to use narrower bidding ranges, which limits their competitiveness and results in lower win rates.
Figure Sb pblpppsentts the head-to- -o-aed competition outcomes. Deepseek Sereeekk-rrvve----- V2 and Qwen3-321 achieve the highest win rates against other models. This is primarily because they often bid lower prices, making their offers more likely to be accepted by the platform. Conversely, models like GPT-4o and LLaMA-3.. 2-11b tend to place higher bids, reducing their success rate despite frequent participation. Models such as QwQ-32B and CPT-4o -inii are less active overall, leading to fewer bids and lower order acquisition rates. This inactivity contributes to their diminished final profit, as shown in Table 3
Takeaway: Multi-Agent Competition
Models with flexible bidding strategies, like Claude-3, 7 7-Sonnet and Gemini- -2.5-Flashh achieve higher order win rates, while those with narrow or high bids, like LLaMA-3, 2- 11b and GPT-4o, underperform. Models that bid aggressively, such as Srepseek-Provers V2 and Qwen3-32B, dominate head-to-head competitions, whereas inactive models like GPT-4o-mini fail secure bids and profits. QwQ and GPT-4o-min show minimal bid- ding activity and weak task participation (Figure

---

Bid Distribution by Model
Head-to-Hee Results with
competit on between models
qwen3-32b
Claude 37 sonnet-0
gemini-2.5-flash
gemin-25 25--ash---
10
v3
gerinn 20-faahh-12
Was (1) 7 t0
Bama- 4- d-scout-an -4
deepseek-prover-v2
Model
cloude 3 5--conett-i
llama-3.2-11b
deepseek proveer-2
Uama-4-scout
deepteck-ud -1
got-4o
Mama 3.2
claude-3.7-sonnet
awen3
got-4o
apt-Ae-mini
100
gemini-20-flash
clacde gprne
Lame Treee dospeet
5 E-thoe
Bid Price
(a) Bid price distrihution by model. The violin plots illustrate the distribution of bid prices generated by each model. Red points indicate bids that were successfully accepted.
Oppoment Model
(b) WonsLoss Matrix of Model Competition.
Figure 5: Bidding Behavior and Evaluation Results. (a) Lower bid prices may increase the likelihood of being assigned an order, but often come at the cost of reduced profit margins. (b) Higher values in red represent more wins; lower values in blue indicate more frequent losses in pairs.
Environment Configuration. We further investigate how different environmental configurations impact agent behavior and overall performance. Specifically, we explore two key factors: the global order availability and the agents' initial financial endowment. For each factor, we conduct a series of controlled experiments to observe how variations affect agents' action distributions.
As shown in Figure 20a (Appendix), when the total number of available orders increases, agents tend to perform fewer pickup and delivery actions and instead choose the do nothing action more frequently. This suggests that in resource-rich environments, agents are more inclined to conserve energy and avoid unnecessary effort, opting to wait for optimal opportunities rather than actively pursue deliveries. Conversely, in low-resource settings, agents are more motivated to engage in delivery tasks to secure profits. Additionally, as resource abundance increases, agents demonstrate higher tendency to initiate and complete shared deliveries, likely as a means to reduce energy costs through collaboration.
Figure 20b (Appendix) illustrates the impact of agents' initial monetary resources. As initial capital increases, agents are less reliant on aggressive bidding and instead prioritize actions such as order pickup. When funds are limited, competition intensifies, leading to more frequent bidding behavior. Furthermore, with sufficient initial capital, agents are more willing to invest in infrastructure, such as purchasing a bike, which enhances their long-term delivery efficiency.
Takeaway: Resource and Decision-Makee Strategy
Order resource scarcity increases agent competitiveness and task urgency. Sufficient agent initial money leads to more relaxed, profit-insensitive behavior (Figure 20 in Appendix).
These observations suggest that agents are more com- petitive and task-driven in ronments. In contrast, resource-rich conditions reduce the urgency to complete tasks and generate immediate profits. Importantly, agents are also more likely to en- gage in actions that involve upfront costs but promise long-term benefits-ae sscefittsssuhheee as investment and shared de- piovre-provided they have the financial capacity and enough taken orders to do so.
Trabwa Win Rats/Action Proportin
envi-
Operness 0.02
038
6:30
0.70
0.44 0.04
051
Comscintauanee
0.65 664
066
0.08
0.4 028
0.42
Extrovenccon
0.25 0.3
015
0.37
0.00
0.0
053
Agressbieness 0.63 034 0.20 0.01 0:35 0.70 0.00
Neurstidim 0.08
0.11
0.19
0.04 0007 0.08
0.39
Bid Win Rate
Desivar Orde Count
Bay Beveraee Count
Netting Buy Bike Count
Bu Onter
Pix Up Order
Influence of Persona. Personality traits significantly affect the decision-making and performance of deliv- ery agents. As shown in Figure 6) agents with higher
Figure 6: Pearson correlation b/w Big Five personality traits and agent behaviors. Conscientiousness tend to exhibit lower frequency of bidding actions, a higher frequency of task-completion actions (e.g., picking up orders), and achieve higher bid win rate. This suggests that conscientious agents prioritize task completion over strategic competition. Agents with higher Agreeableness are less likely to remain inactive (i.e., perform "do nothing" actions) and tend to achieve higher bid win rates. Conversely, agents with lower Agreeableness display higher inactivity and narrower bidding price ranges, limiting their competitiveness. Interestingly. agents with higher Openness exhibit reduced engagement in delivery tasks, possibly because they explore competitive or unconventional bidding strategies that divert attention from task execution.
9

---

Takeaway: Impact of Persona in Multi-agent Interaction
Agent personalities shape strategic tendencies: conscientious agents prioritize task fulfillment, while openness and agrecableness modulate competitiveness and inactivity (Figure
5 Related Works
Simulations have played a crucial role in constructing environments for training and evaluating autonomous agents. Text-based simulators often emphasize social scenarios, such as human interac- tion [55], daily activities [36), and relational polarization [38]. Popular embodied simulators support broader range of applications, particularly in embodied AI research and 2D/3D scene synthesis (27] However, most embodied simulators remain constrained to either indoor household environments
(e.g., A12-HOR [23], Habitat [39]. iGibson
[24] outdoor driving scenarios (e.g.. CARLA [10),
MetaDrive (2)) or natural scenes (c.g., AirSim [43]) These simulators [10, 39 24 43
(3 [3] 63 of-
ten rely a limited number of manually crafted scenes, which hinders scalability and diversity. Some platforms, such as MetaUrban [52). MetaDrive (26). A12-THOR 23 and Genesis [4) introduce
rule-based procedural generation to alleviate this issue. Nonetheless, existing embodied simulators typically lack support for dynamic multi-agent interactions in complex, outdoor environments.
Recent advancements have introduced large-scale, language-driven social simulators capable of modeling complex societal behaviors. OASIS (55) simulates up to one million LLM-powered agents interacting on social media platforms, capturing phenomena such as information diffusion, echo chambers, and polarization. Casevo [22] integrates thain-of-thought reasoning, attrival-augmeetee generation, and customizable memory mechanisms to simulate intricate social phenomena and rocision-makinge processes. MineLand [57] offers a multi-agent Minecraft environment where agents, driven by physiological needs and limited multimodal perception, engage in collective behaviors, fostering ecological and detailed simulations. Project sid (2 further advances this landscape by deploying large number of AI agents within Minecraft environment to explore the emergence of Al civilizations. VirtualCommunity [64] leverages Genesis to conduct community influence task in out-door, multi-agent scenes. These simulation platforms demonstrate agents' capabilities in forming complex social structures, economies, and governance systems, providing insights into large-scale societal simulations and agentic organizational intelligence.
While prior simulators have demonstrated success in specific domains such as indoor navigation or autonomous driving, none have been explicitly designed support dynamic, multi-agent interactions in outdoor, city-scale environments with both realistic rendering and physical simulation. Sim World fills this gap by offering scalable platform that enables multi-agent collaboration and competi- grounded interactions, and procedural generation of urban scenes- all essential for benchmarking advanced embodied agents.
tion,
6 Conclusion
We presented SIM WORLD, a novel simulator featuring realistic, open-ended world simulation, rich interface for LLM/VLM agents and diverse physical and social reasoning scenarios. We deployed two case studies that highlight key differences in agent performance across models in both single agent physical reasoning and multi- multi-agent social reasoning. These findings underscore the importance of embodied evaluation and open-ended task design in advancing the frontier of Al agent research.
Regarding limitations and future work, although HIMWORL.D has demonstrated strong advantages in simulating multi-agent systems within realistic environments rich in physical and visual signals, its deployment for large-scale training, particularly for efficient online reinforcement learning, still leaves room for optimization. We are also excited to build more advanced autonomous agents that can survive and thrive in simulated and real physical worlds, with sophisticated reasoning abilities
14
world models 53 20 memory
199 emotions, and other capabilities.
We hope that SIMWORLD will serve as a powerful open-source platform for the research community and continue to evolve through collective contributions. Please refer to the project website for the latest updates: https://simworld.orgg
10

---

References
[1] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang. Kuang- Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do as can, not i say: Grounding language in robotic affordances, 2022. URL https://arxiv.. org/ abs aww.aba2200 .
[2] Altera AL, Andrew Ahn, Nic Becker, Stephanie Carroll, Nico Christie, Manuel Cortes, Arda Demirei, Melissa Du, Frankie Li, Shuying Luo, Peter Y Wang. Mathew Willows, Feitong Yang, and Guangyu Robert Yang. Project sid: Many-agent simulations toward ai civilization, 2024. URL.
sstps://arxiv.org/aae httgg/bb//411..... 00014
[3] Anthropic. Claude's extended thinking, 2025. URL https://www. http://ppw..athrpp....cooo com/ wtwpcc//iiiibbl-eeteeeeeeeeeeeee
Accessed: 02--005-4.
[4] Genesis Authors, Genesis: A universal and generative physics engine for robotics and beyond, December 2024.
[5] Matthew Chang, Gunjan Chhablani, Alexander Clegg, Mikael Dallaire Cote, Ruta Desai, Michal Hlavac, Vladimir Karashchuk, Jacob Krantz, Roozbeh Mottaghi, Priyam Parashar, et al. Partnr: A benchmark for planning and reasoning in embodied multi-agent tasks. arXiv preprint arXiv:2411 1,00081, 2024.
[6] Junzhe Chen, Xuming Hu, Shuodi Liu, Shiyu Huang. Wei-Wei Tu, Zhaofeng He, and Lijie Wen. LImarena: Assessing capabilities of large language models in dynamic multi-agent environments. arXiv preprint arXiv:2402. 16499, 2024.
[7] Thibault Le Sellier De Chezelles, Maxime Gasse, Alexandre Drouin, Massimo Caccia, Léo Boisvert, Megh Thakkar, Tom Marty, Rim Assouel, Sahar Omidi Shayegan, Lawrence Keunho Jang, Xing Han LU, Ori Yoran, Dehan Kong, Frank F. Xu, Siva Reddy, Quentin Cappart, Graham Neubig, Ruslan Salakhutdinov, Nicolas Chapados, and Alexandre Lacoste. The browsergym ecosystem for web agent research, 2025. URL httph::t:///ttt arxiv org/ /a//a/b//2/11 2////4112.0000 05467
[8] Jac-Woo Choi, Youngwoo Yoon, Hyobin Ong, Jachong Kim, and Minsu Jang. Lota- bench: Benchmarking language-oriented task planners for embodied agents. arXiv preprint 20Xi:22402.88178 2024.
[9] Edsger W. Dijkstra. A note two problems in connexion with graphs. Numerische Mathematik, 1(1):269-271, 1959.
[10] Alexey Dosovitskiy, German Ros, Felipe Codevilla. Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In Proceedings of the 1st Annual Conference on Robot Learning, pages 1-16, 2017.
[11] Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang. Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model, 2023. https:// /ttp:////axv.v/.. org/ abs/ 2203. 03378
[12] Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang. Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Information Processing Systems. 35: 18343-18362, 2022.
[13] Chen Gao, Baining Zhao, Weichen Zhang, Jinzhu Mao, Jun Zhang, Zhiheng Zheng, Fanhang Man, Jianjie Fang, Zile Zhou, Jinqiang Cui, et al. Embodiedeity: benchmark platform for embodied agent in worldworld city environment. arXiv preprint 2rXiv:2410.09604 2024.
11

---

[14] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, al. Deepscek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
[15] Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world webagent with planning, long context understanding, and program synthesis, 2024. URL (RRps:ts:://art org/ 2207/ 12856
[16] Anthony Ha. Google's gemini has beaten pokémon blue (with a lit- tle help). TechCrunch, 2025.
URL https://techcrre com 2tt///2225000000 googles- gemini- hass- wew-bbetten--oee blue- aduuewith--ee wwthwwitthaaa-e lith---lttlee-eeepp Accessed: 2025-05- 14.
[17] Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.1499 14992, 2023.
[18] Hongliang He, Wenlin Yao, Kaixin Ma. Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. WebVoyager: Building an end-to-end web agent with large multimodal models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pages 6864 6890. Bangkok, Thailand, August 2024, Association for Computational Linguistics. doi: 1/211665////2244.clllll org/ acl www.ooon..333
371. httthhtss///ccaaaa
[19] Matthew Ho. Chen Si. Zhaoxiang Feng, Fangxu Yu, Yichi Yang, Zhijian Liu, Zhiting Hu, and Lianhui Qin. Arememo: Abstract reasoning composition with lifelong Ilm memory. arXiv preprint XXiv:2509.04439, 2025.
[20] Zhiting Hu and Tianmin Shu. Language models, agent models, and world models: The law for machine reasoning and planning. arXiv preprini arXiv:2312.052300 2023.
[21] Dong Huang, Jie M Zhang, Michael Luck, Qingwen Bu, Yuhao Qing, and Heming Cui. Agentcoder: Multi-agent-based code generation with iterative testing and optimisation. arXiv preprint arXiv:2312.13011 2023.
[22] Zexun Jiang, Yafang Shi, Maoxu Li, Hongjiang Xiao, Yunxiao Qin, Qinglan Wei, Ye Wang, and Yuan Zhang. Casevo: A cognitive agents and social evolution simulator, 2024. URL https
7/arxiv. sarxiv.org/ae www.:oggg////aaeee.ee.o httg//as//4411.999// 19498
[23] Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luea Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, et al. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint
2017.
[24] Chengshu Li, Fei Xia, Roberto Martin, Michael Lingelbach, Sanjana Srivastava, Bokui Shen, Kent Vainio, Cem Gokmen, Gokul Dharan, Tanish Jain, al. igibson 2.0: Object-centric simulation for robot learning of everyday household tasks. arXiv preprint (oX.x:2208.032777 2021.
Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín Martín, Chen Wang, Gabrael Levine, Wensi Ai, Benjamin Martinez, al. Behavior- lk: A human- centered, embodied ai benchmark with 1,000 everyday activities and realistic simulation. arXiv preprint N0VV:2403300227, 2024.
[25]
[26] Quanyi Li, Zhenghao Peng, Lan Feng, Qihang Zhang, Zhenghai Xue, and Bolci Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. IEEE transactions on pattern analysis and machine intelligence, 45(3):3461-3475, 2022.
[27] Xinqing Li, Ruiqi Song, Qingyu Xie, Ye Wu, Nanxin Zeng, and Yunfeng Ai. Simworld: A unified benchmark for simulator-conditioned scene generation via world model. arXiv preprint arXiv:2503.1 1 3952, 2025.
[28] Zaijing Li, Yuquan Xie, Rui Shao, Gongwei Chen, Dongmei Jiang, and Liqiang Nie. Optimus-2: Multimodal minecraft agent with goal-observation-ace conditioned policy. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 9039-9049, 2025.
12

---

[29] Shunyu Liu, Yaoru Li, Kongcheng Zhang, Zhenyu Cui, Wenkai Fang, Yuxuan Zheng, Tongya Zheng, and Mingli Song. Odyssey: Empowering minecraft agents with open-world skills, arXiv preprint arXiv:2407. 15325, 2024.
[30] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai. Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating Ilms as agents. arXiv preprint arXiv:23008. 03688, 2023.
[31] Xiao Liu, Tianjie Zhang, Yu Gu. Iat Long long, Yifan Xu, Xixuan Song, Shudan Zhang, Hanyu Lai, Xinyi Liu, Hanlin Zhao, et al. Visualagentbench: Towards large multimodal models as visual foundation agents. arXiv preprint 06.322,
[32] Qian Long, Zhi Li, Ran Gong, Ying Nian Wu, Demetri Terzopoulos, and Xiaofeng Gao. Teamcraft: A benchmark for multi--odal multi-agent systems in minecraft, 2024. URL https //axxiv.org/bblll wttg/ba/////tte 05255
7/arxiv.
[33] Jianlan Luo, Charles Xu, Fangchen Liu, Liam Tan, Zipeng Lin, Jeffrey Wu, Pieter Abbeel, and Sergey Levine. Fmb: functional manipulation benchmark for generalizable robotic learning. The International Journal of Robotics Research, 44(4): 2025.
[34] Ziqiao Ma, Benjamin VanDerPloeg, Cristian-Paul Bara, Yidong Huang, Eui-In Kim, Felix Gervits, Matthew Marge, and Joyce Chai. Dorothie: Spoken dialogue for handling unexpected situations interactive autonomous driving agents. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4800-4822, 2022.
[35] Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. In The Twelfth International Conference on Learning Representations, 2023.
[36] Joon Sung Park, Joseph O' O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pages 1-22, 2023.
[37] phiresky. procedural-cities.
2024.
[38] Jinghua Piao, Yuwei Yan, Jun Zhang, Nian Li, Junbo Yan, Xiaochong Lan, Zhihong Lu, Zhiheng Zheng, Jing Yi Wang, Di Zhou, et al. Agentsociety: Large-scale simulation of llm-driven generative agents advances understanding of human behaviors and society. arXiv preprint arXiv:2502.088 086911, 2025.
[39] Xavier Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Tsung- Yen Yang, Ruslan Partsey, Ruta Desai, Alexander William Clegg, Michal Hlavac, So Yeon Min, et al. Habitat 3.0: A co-habitat for humans, avatars and robots. arXiv preprint arXiv:2310.13724, 2023.
[40] Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint (ixv:2307.1668)) 2023.
[41] Weichao Qiu, Fangwei Zhong, Yi Zhang, Siyuan Qiao, Zihao Xiao, Tae Soo Kim, and Yizhou Wang. Unrealex: Virtual worlds for computer vision. In Proceedings of the 25th ACM International Conference on Multimedia, MM '17 page 1221-1224, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 978 450349002. doi: 10.1 N..1145/3100 11113311226631129339 URL https
//doi org 10. 11111111////166111 3129396
[42] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL httph:ttpp/::/tt org/abs/2103 00020
[43] Shital Shah, Debadeepta Dey, Chris Lovett, and Ashish Kapoor. Airsim: High-fidelity visual and physical simulation for autonomous vehicles. In Field and Service Robotics: Results of the i 11th International Conference, pages 621-635. Springer, 2018.
13

---

[44] Chan Hee Song. Jiaman Wu, Clayton Washington, Brian M. Sadler, Wei-Lun Chao, and Yu Su. Lim-planner: Fhw-shot grounded planning for embodied agents with large language models, 2023.
URI.htpss:
org/ abs/ 2212. 04088
[45] Xinshuai Song, Weixing Chen, Yang Liu, Weikai Chen, Guanbin Li, and Liang Lin. Towards long-horizon vision-language navigation: Platform, benchmark and method. In Pruceedings of the Computer Vision and Patlern Recognition Conference, pages 12078-12088 2025.
[46] Tencent Hunyuan3D Team. Hunyuan3d 2.0: Scaling diffusion models for high resolution textured 3d assets generation, 2025.
[47] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023.
[48] Hanqing Wang. Jiahe Chen, Wensi Huang, Qingwei Ben, Tai Wang, Boyu Mi, Tao Huang, Siheng Zhao, Yilun Chen, Sizhe Yang, et al. Grutopia: Dream general robots in city at scale. arXiv preprint arXiv:2407.10943, 2024.
[49] Yu Wang. Yifan Gao, Xiusi Chen, Haoming Jiang, Shiyang Li, Jingfeng Yang, Qingyu Yin, Zheng Li, Xian Li, Bing Yin, et al. Memoryllm: Towards self-updatable large language models. arXiv preprint arXiv:2402.04624, 2024.
[50] Isadora White, Kolby Nottingham, Ayush Maniar, Max Robinson, Hansen Lillemark, Mchul Maheshwari, Lianhui Qin, and Prithviraj Ammanabrolu. Collaborating action by action: A multi- agent IIm framework for embodied reasoning. arXiv preprint arXiv:2504. 2025.
[51] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. Autogen: Enabling next-gen Ilm applications via multi-agent conversation. arXiv preprint Xiv: 08188,088555..000 2023.
[52] Wayne Wu, Honglin He, Jack He, Yiran Wang, Chenda Duan, Zhizheng Liu, Quanyi Li, and Bolei Zhou. Metaurban: An embodied ai simulation platform for urban micromobility. International Conference on Learning Representation, 2025.
[53] Eric Xing, Mingkai Deng. Jinyu Hou, and Zhiting Hu. Critiques of world models. arXiv preprint 269,, 2025.
[54] Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang. Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, et al. Embodiedbench: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents. arXiv preprint arXiv:2502. 095600 2025.
[55] Ziyi Yang, Zaibin Zhang, Zirui Zheng, Yuxian Jiang, Ziyue Gan, Zhiyu Wang, Zijian Ling, Jinsong Chen, Martz Ma, Bowen Dong. Prateek Gupta, Shuyue Hu, Zhenfei Yin, Guohao Li, Xu Jia, Lijum Wang. Bernard Ghanem, Huchuan Lu, Chaochao Lu, Wanli Ouyang, Yu Qiao, Philip Torr, and Jing Shao. Oasis: Open agent social interaction simulations with one million agents, 2025. URL
https://arxiv. org/ abs 2411. 11581
[56] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting language models, 2023. URL itttps//////
org/ htg/abs/221000 2bb22220003669
[57] Xianhao Yu, Jiaqi Fu, Renjia Deng, and Wenjuan Han. Mineland: Simulating large-scale multi-agent interactions with limited multimodal senses and physical needs. arXiv preprint arXiv:2403. 2024.
[58] Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B. Tenenbaum, Tianmin Shu, and Chuang Gan. Building cooperative embodied agents modularly with large language models, 2024. URL /tttps://aaxi...... org/ abs/ 2307 02485
[59] Shiduo Zhang. Zhe Xu, Peiju Liu, Xiaopeng Yu, Yuan Li, Qinghui Gao, Zhaoye Fei, Zhangyue Yin, Zuxuan Wu, Yu-Gang Jiang, et al. Vlabench: large-scale benchmark for language-conditinn robotics manipulation with morghooizon reasoning tasks. arXiv preprint arXiv:2412. 18194, 2024.
14

---

[60] Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v(ision) is a generalist web agent, if grounded. 2024. URL URL htRp::ts::///axa org/ abs abs/2401.01611
[61] Boyuan Zheng, Michael Fatemi, Xiaolong Jin, Zora Zhiruo Wang, Apurva Gandhi, Yueqi Song, Yu Gu, Jayanth Srinivasa, Gaowen Liu, Graham Neubig, and Yu Su. Skillweaver: Web agents can ielf-ve by discovering and honing skills, 2025. URL htttsst/taattiiii wwwpiv.org/bbb//50000 07079
[62] Kaizhi Zheng, Xiaotong Chen, Odest Chadwicke Jenkins, and Xin Wang. VImbench: A compositional benchmark for vision-and- anni-n-an---aguaaee manipulation. Advances in Neural Information Processing Systems, 35:665-678, 2022.
[63] Fangwei Zhong, Kui Wu, Churan Wang, Hao Chen, Hai Ci, Zhoujun Li, and Yizhou Wang. Unrealzoo: Enriching photo-realistic virtual worlds for embodied ai. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5769-5777 5779, 2025.
[64] Qinhong Zhou, Hongxin Zhang, Xiangye Lin, Zheyuan Zhang, Yutian Chen, Wenjun Liu, Zunzhe Zhang, Sunli Chen, Lixing Fang, Qiushi Lyu, Xinyu Sun, Jincheng Yang, Zeyuan Wang, Bao Chi Dang, Zhehuan Chen, Daksha Ladia, Jiageng Liu, and Chuang Gan. Virtual community: An open world for humans, robots, and society, 2025. URL htttss://ariiis http:////aavvvv.oaaa///////// www.or/a///...aa///// 14893
[65] Kunlun Zhu, Hongyi Du, Zhaochen Hong, Xiaocheng Yang, Shuyi Guo, Zhe Wang, Zhenhailong Wang, Cheng Qian, Xiangru Tang, Heng Ji, et al. Multiagenthench: Evaluating the collaboration and competition of Ilm agents. arXiv preprint 25X/1:2503.0/935, 2025.
15

---

A
Sim World Simulator
A.1 Python Package
A.1.1 Layout Generation
The pipeline of procedural generation pipe is shown in Figure 7 which support generating road, building and detail elements in scene graph of city. As Algorithm shows, we use the quad-tree data structure manage the hierarchical structure of scene graph which enables us to support space retrieval function in the language control environment module and support generated different tasks easily.
We have three generation stages
Algorithm 1: Layout Generation Algorithm with QuadTree
Input: Configuration parameters
Output: Final merged QuadTree representing the city layout Function RoadMapGenerat on (config):
Initialize road network grid;
Create a new QuadTTee for the road map;
Place primary roads and secondary roads; Smooth intersections;
Insert roads into the QuadTree; return roadQuadTree
8 Function BuildingGenerat in (config, roadQuad'rree:
Create QuadTTre for the buildings; Partition land into plots based on road network;
10
Generate building types, shapes, heights;
12
Insert building geometries into the QuadTree; return QuildingQuadTree
Function SerralGGeneration (config, roadQuadTree, buildingQuadTree):
15
Create a new QuadTree for the details; For each building, add facade details (windows, doors, decorations); For each road, add lanes, crosswalks, signals;
17
Add environmental elements (trees, benches, lights); Insert details into the QuadTrre:; return detailQuadTrree
19
Function MergeQuadTrees (roadQuadTree, buildingQudTree, detailQuadTree): Initialize new QuadTree for the complete city;
22
Merge roadQuadTree into city QuadTree; Merge buildingQuade Quad'reee into city QuadTree;
24
Merge detailQuad into city QuadTree; return finalQuadTree
26
27 Main Procedure:
roadQuadTree +- Ro.dlappGenerattiocccmmm)))
building t- Build BuildingGeneratioon (config, roadQuadTree):
detailQuadTree f- DetaiiGeneration (config, roudQuadTree, buildingQuadTree):
30
finalQuadTree +- MergeQuadTrees (roadQuadTree building detuilQuadTree);
32 return finlQuaTrree
A.1.2 Waypoint System
To facilitate robust agent navigation and path planning within the simulated environment, SIMWORLD incorporates customizable waypoint system (Figure that serves as the foundational representation of navigable space. This system initially constructed using coarse-grained waypoint skeleton derived from the output of the Layout Generation pipeline. Specifically, the pipeline produces high-level geometric features of the environment, such as road centerlines and intersection points, which are then used to define the topological structure of the navigation graph.
16

---

1. Road Generation
2. B Building Generation
3. Element Generation
Figure 7: Generation Pipeline for Static World Initialization. The process consists of four stages: (I) road network generation, (2) procedural building placement and (3) insertion of street-level elements.
Building upon this base structure, we procedurally generate additional navigable elements, including traffic lanes, sidewalks, and crosswalks. These elements are instantiated by applying spatial offsets relative to the underlying road geometry, enabling the delineation of separate navigation channels for different agent types (e.g., vehicles vs. pedestrians). The result an initial set of discrete waypoints that capture the essential connectivity of the environment. At this stage, all waypoints are categorized as "intersection" waypoints, reflecting their origin from key structural nodes the road network.
To improve the granularity and realism of agent motion, we further enrich the navigation graph by interpolating fine- ane-grained waypoints between each pair of adjacent intersection waypoints. These interpolated waypoints are labeled as "normal" waypoints and serve to densify the navigation graph, thereby allowing agents to traverse smoother and more continuous paths. The density of these fineggraindd waypoints is configurable through parameters such as interpolation step size and spatial offset magnitude, which can be tuned to suit different simulation scenarios.
This hierarchical waypoint system provides a flexible trade-off between simulation performance and trajectory fidelity. By adjusting the level of waypoint granularity, users can optimize for either computational efficiency or motion realism, depending on the specific demands of downstream tasks such autonomous navigation, traffic simulation, or human-agent interaction modeling.
A.1.3 Traffic System
The traffic system is a core component of our simulator, responsible for simulating dynamic road usage by both vehicles and pedestrians. It enables the representation of realistic traffic flow, including vehicle generation, path planning, intersection control, and pedestrian behavior. By managing road interactions and traffic signals, this system supports complex urban scenarios such congestion, pedestrian crossings, and traffic light coordination, providing a critical foundation for evaluating urban infrastructure and mobility policies.
The traffic system operates on layout graph provided by the Layout Generation, designed to support traffic simulation for any city layout. Based on the coarse-graiie gaanse-r waypoints and detailed traffic routes, including traffic lanes, sidewalks, and crosswalks, provided by Waypoint System, we sample vehicles, pedestrians and traffic signals in the city. Three managers handle populating the traffic elements:
Samples positions along the traffic lanes to spawn vehicles, assigning them predefined dynamically generated routes through the network.
Managee:
PedestrianManager Places pedestrians along sidewalks and controls their movement patterns, including crossing decisions at intersections.
IntersectionManngger Identifies intersections within the route network and installs traffic signal agents that control the right-of-way based on configurable timing policies adaptive logic.
Together, these components form a fully functional traffic simulation pipeline, allowing the virtual city to support realistic and reactive mobility behavior.
17

---

Road Netack
Connoterr
PD Consoler
Teate Netwex
Camwarer
Vinidd Monager
Prevetron Marager
Irdersection Nenaaee
Lare
Bidwwak
Crosseatt
Pedestrion
Intersation
Signal
(a) Architecture of Traffic System.
Position: (-1550, -10000)
Position:
(-1550, 100000)
Direction: (-0.39, 0.92)
Direction:
(-0.39, 0.92)
Next Waypoint: f-1990. -10000)
Next Waypoint:
(-1990, -9300)
Action History:
Action History:
Step forward seconds)
forward seconds) Change waypoint (1990 -9300
Next Action:
Next Action:
Change waypoint (-1990, -9300)
Turn right (30 degrees)
Reasoning:
Reasoning:
Change waypoint (-1990 9300)
There's pedestrian ahead, blocking my path to the next waypoint. need to replan. Turning hitting streatlight. I'll reroute to right instead.
My next waypoint slightly ahead to the right, but I'm already facing nearly that directionn. I'll about 30 degrees the right to align with
Turn right (30 degrees)
(b) Waypoint System and Local Action Planner. The waypoint system provides spatial guidance by specifying the positions and orientations of intermediate navigation subgoals. Based on this information, agents can perform high-level reasoning plan their behavior. A local action planner operates conjunction with the waypoint system, generating low-level action suggestions such as path following obstacle avoidance that help agents navigate between waypoints effectively.
Figure 8: Overview of Traffic and Waypoint Systems.
As shown in Figure Sa the traffic system adopts a modular architecture centered around top-level TrafficControlle which coordinates the behavior of three specialized managers: VehicleManngeer, IntersectionManngeer, and PedestrianManager.
To enable real-time simulation, the system integrates with Unreal Engine through the Communicator which provides bidirectional communication between the traffic logic and the simulation environment. This allows the system to send control signals and receive state updates from virtual actors within the engine.
A.1.4 Local Action Planner
Recent research on AI agents incorporate action planning modules into their agent's architecture. In our work, we integrate such module into the simulator itself, providing useful interface for LLM agents to use and assist the agents' high-level hong-horizon reasoning. LLM-Planner [44
18

---

uses visual information of the environment for decomposing high level plan into low level actions. CoELA [58] uses a rule-based only executor for execution. Compare to their similar module, ours are more comprehensive in that we include rule-based and LLM executor; we receive both information in the form of natural language as well as in the form of visual; our local action planner module also extendable to corresponding with our extendable simulator. Our local action planner module is devided into two part, parser and executor. Parser simply receive the high-level plan from user LLM and parse the high-level plan into low-level executable actions, while executor is responsible of executing the parsed low-level actions one by one given the environmnet information in the simulator. Considering the variety of needs from different researchers, planner executor module using two different ways: rule- based and visual based. For perceiving the enviromment of the simulator: rule-based local action planner will receive abstract city layout information, and visual-based local action planner will capture the visual information from the environment directly, which allows the module be adapted to different visual language models and further improve the comprehensiveness. For non-atomic action like navigation, we integrate a route planner based the Dijkstra's 's algorithm[9) for rule-based local action planner, which provides ground-truth solutions for navigation tasks. This route planner serves as a supporting tool for the local action planner module, ensuring that natural language descriptions of activities can be reliably converted into executable low-level action sequences. For visual-based local action planner on the other hand, we simply feed the visual information backbone visual language model and let the model decide what to do next to test the capability of visual language model.
implement local action
For example, when local action planner receives user plan such "go to the nearest chair and sit down," " it decomposes the instruction into an action list: navigate, agent_sit_ _ii__st_down. For the navigate action, the rule-based mode first computes the shortest path from the agent's current position to the nearest chair, which results in a sequence of navigation actions such as novigate(O, 1), navigate(I, [1, 10), and navigate (10, 10), where (10, 10) is the location of the chair. Once the agent reaches the chair, local action planner proceeds to the next action, agent_si sit__ssttdoow,, and completes execution as the action list becomes empty.
A.1.5 LLM-based Environment Editing
SIMWORLD supports real-time, language-driven environment editing, enabling the creation of open- ended, generative worlds through natural language. Users interact via a Python interface, issuing instructions such as "add a tree next to the hospital" or "place a red sports car near the fountain." To handle such commands, we introduce dual-stage pipeline: (1) asset retrieval and placement, and (2) text-to-3D asset generation.
For asset retrieval and placement, given instruction like "Add a bench near the cafe; I can see museum and a skyscraper nearby," an LLM first parses the target asset, spatial anchor (c.g.. "the cafe"), and contextual landmarks ("museum" and "skyscraper"), Candidate reference assets are located via semantic search on the scene graph and ranked by comparing the described surroundings with actual neighbors using embedding- based similarity. The top-ranked reference point determines the target location. The desired asset is then retrieved from the library using CLII--ased] 42 matching and placed accordingly.
no suitable asset is found the database, we fall back to a text-to-3D generation module, Given a prompt such as "a red sports car," the system generates a new asset using generative model, converts it into a usable format, and integrates it into the scene in real time.
This approach allows for flexible, scalable environment editing that is both semantically grounded and spatially coherent coherent-laying the foundation for open-ent open-ended simulation and interactive world modeling.
A.2
Communicator and Seneraios Creation
A.2.1
Communicator
Inspiring by UnrealCV[41] [1], the Communicator module serves as the bridge between Python-based simulation logic and Unreal Engine's real-time 3D environment. Implemented in both Python and C++, uses UnrealCV to establish a TCP connection, enabling asynchronous communication between the two sides.
19

---

We extended the basic communication protocol by defining a custom set of commands for scene control, actor manipulation, and data querying. These commands are encoded JSON format and passed through the TCP channel. For instance, Python can instruct UE to spawn an actor specific location, retrieve the current pose of pedestrian, or trigger motion events on robot.
The Communicator plays a central role in the simulation loop:
Python uses it to update the virtual world, while UE reports back environmental feedback visual state. This design allows us to decouple logic computation from rendering, achieving both flexibility and modularity.
A.2.2 Task Suite Workflow
(a) Subset Collected Assets.
Avg Profit
Avg. Success Rate
i Avg.
Avg Invalid Action Rate
Aug. Delay Time
Evaluator
Order Shared Rate
State
Action Hastory
CO
np. herovard
Platform
loft Speodl 100
Mex horovard
Task State Machine
Logging History
State & Action History
Outour dest Actiety Desovpooe Activity and sp Order
Prompt You erea desiery man the city Your go morimo your profle
LLM
cisigge, thee num Artise pirt sptorder))
codan vdcibe Hattry
Planner
Simulator
Action Checker
Mutt-modal Observations
Simulated Scone
Datasct
(b) Workflow for Constructing Task Suite. We implement delivery task representative example within the SIMWORLD simulator. The full task suite includes five key components: (1) Dataset, Simulator, (3)
LLM, (4) Platform, and (5) Evaluator.
Figure 9: SIMWORLD Assets and Task Suite Workflow.
20

---

Action
Category
Description
Move Forward
Navigation
Keep moving in the current direction
Step Forward/Backward
Navigation
Step forward/backward for fixed time
Rotate
Navigation
Turn the body to face a new direction
Stop
Navigation
Stop moving
Look Up/Down
Observation
Adjust the gaze upward/downwwrd by a degree
Focus
Observation
Adjust the field of view
Pick Up
Object Interaction
Grasp and lift an object
Drop Off
Object Interaction
Release a held object at the target location
Sit Down
Object Interaction
Transition to a seated position
Stand Up
Object Interaction
Rise from seated position
Open Door
Object Interaction
Interact open door into vehicle Leave a vehicle
Enter Car
Object Interaction
Exit Car
Object Interaction
Carry Heavy Object
Object Interaction
Transport heavy object Place heavy object the ground Control and ride a
Put Down Heavy Object
Object Interaction
Ride Scooter
Object Interaction
Have Conversation
Social Action
Exchange verbal communication
Point Direction
Social Action
Gesture to indicate direction
Wave Hand
Social Action
Signal or greet with a hand wave Engage in dialogue explanation Express disagreement using gestures Perform rhythmic movement with the body
Discuss
Social Action
Argue with Body Language
Social Action
Dance
Social Action
Make Phone Call
Social Action
Simulate or engage in phone conversation
Table 4: Comprehensive List of Actions with Categories and Descriptions.
A.3
Unreal Engine Integration
A.3.1
Action Space
Table
presents the complete set of low-level actions supported by SIMWORLD.
A.3.2
Assets
Our simulator provides rich collection of city-scale assets, designed to support realistic and diverse urban simulations. These assets include buildings, trees, street furniture, vehicles, pedestrians, and robots (Figure 9a). All assets are sourced from the Unreal Engine Marketplace to ensure high visual fidelity and performance.
Below a selection of the assets currently available our simulator:
Buildings: A variety of architectural styles, including residential, commercial, and industrial structures.
Trees:
Multiple tree species with seasonal variations enhance environmental realism.
Street Furniture:
Items such as benches, streetlights, mailboxes, and trash bins to add detail and
immersion.
Vehicles: A range of vehicles including cars, buses, trucks, and bicycles, each with accurate scale and animations.
Pedestrians: Human characters with diverse appearances and animations to simulate crowd behavior.
Robots:
Humanoid robots for testing autonomous navigation and interaction.
These assets collectively enable the creation complex, dynamic, and realistic city scenes for simulation, visualization, and research purposes.
In addition to the curated asset library, we provide an Asset Generation Pipeline that leverages Text-to-3D models[46] to generate 3D assets directly from natural language descriptions. This tool streamlines the content creation process by translating user prompts into usable Unreal Engine assets, thereby significantly lowering the barrier to customizing city environments.
21

---

B
Case Study Details
B.1 Formulation
To evaluate the embodied reasoning and physical interaction capabilities of agents within SIMWORLD we introduce vision-based navigation task in urban environments. Agents are tasked with reaching a specified destination while avoiding both static obstacles (c.g., trees, benches) and dynamic agents (e.g.. pedestrians), necessitating strong spatial reasoning and multimodal perception under uncertainty. The task requires agents to process real-time visual input and make sequential decisions to avoid collisions, respect traffic rules, and reach the goal efficiently.
To analyze the interplay between high-level planning and low-level reactive control, we consider two settings: with and without a Local Action Planner. In the Local Action Planner setting, agents guided by a precomputed global route generated via A search, which omits obstacle information. Agents must navigate from one waypoint to the next along this route while handling local interactions. In the planner-fiee setting, agents receive only the goal position and are expected to autonomously plan and execute safe trajectory.
B.2
Action Space
Each agent may choose from the following actions at each decision step:
Do Nothing: Remain stationary.
Step: Move forward or backward for a fixed duration.
Turn: Rotate left or right by any angle (continuous range).
Change Next Waypoint: (Local Action Planner only) Switch to a different candidate waypoint.
B.3 City Map and Task Generation
Coarse-grained Map: Each sidewalk is divided into three waypoints (start, midpoint, end),
forming navigation graph to generate tasks.
Fine-grained Map: Each sidewalk is sampled with 70 evenly spaced waypoints. These serve both as candidate motion targets and obstacle anchor points.
Obstacles: Static objects (trees, trash bins, etc.) are randomly placed on fine-grained waypoints.
Dynamic pedestrians follow sidewalk paths at moderate speed with randomized starting positions.
Figure 10 illusttrates the task generation pipeline. We begin with basic road network provided by Sim World, which includes sidewalks, crosswalks, and intersection nodes. To construct the coarse- grained navigation graph, we interpolate a midpoint for each sidewalk segment, yielding a simplified map structure suitable for high-level planning.
Tasks generated by randomly sampling start and goal nodes from this graph, with constraints on the number of segments between them to control difficulty. For fine-grained navigation, we densify the original sidewalk geometry by interpolating evenly spaced waypoints. Each sidewalk is divided into four lateral lanes, each containing 17 waypoints, while each crosswalk contains 8 waypoints.
To simulate realistic urban clutter, place static obstacles such trees and street furniture along these waypoints. Trees are sampled on the innermost lane (nearest to buildings) of each sidewalk, while other street elements (e.g.. benches, trash bins) are distributed across the remaining three lanes. Importantly, for each group of four laterally aligned waypoints (i.e., across the four sidewalk lanes at the same longitudinal position), at least one is guaranteed remain obstacle-free. This ensures that there is always a traversable option at every step along the sidewalk.
We sample 40 tasks spanning the four difficulty levels:
Easy (10 tasks), Medium (10 tasks), Hard (10 tasks), Dynamic (10 tasks). Each task is defined by a sequence of waypoints T; [W1, W2,.., Wn).
B.4 Baseline Agent Protocols
Rule-based Agent: Follows Local Action Planner waypoints using a fixed stepping strategy. Obey the traffic rules and does not observe visual input or adapt to obstacles.
22

---

Task
Coarse-grained Map
Precompute Route
Road Network
ine ine- -grainedd Map
Figure 10: Task Generation Pipeline for Case Study 1.
MLLM Agents:
current image, last-step image, past 3 actions and feedback, current navigation goal, relative distance and angle to current navigation goal.
Input:
Output: action selected from the predefined space. Perception is simulated via RGB screenshots with 90 horizontal FOV.
B.5 Metries
Throughout experiment, we evaluate agent performance using following six metrics, which together capture task success, safety compliance, and decision efficiency:
Success Rate (SR): The percentage of navigation tasks in which the agent successfully reaches its destination without getting stuck, timing out, deviating significantly from the designated route. This metric reflects the overall effectiveness the agent's decision-making and path-following capabilities.
N Nauceess
SR
Nioual
where Naucess is the number of successfully completed tasks, and Nistl is the total number of tasks.
Collision Count (CC): The total number of collisions that occur between the agent and either static obstacles (e.g., trees, trash bins) or dynamic entities (e.g., pedestrians). A lower collision count indicates better obstacle avoidance and situational awareness. Collisions are recorded for all episodes, regardless success or failure.
New
CC
Ca
where e; is the number of collisions in episode
i.
23

---

Collision Count (Success Only) (CC-S): The number of collisions occurring only within success- ful episodes, which reflects if the agent really avoids the obstacles. Since the CC possibly low the agent get stuck at the very beginning. This metric helps assess how safely the agent completes tasks when does reach the goal.
CC-S =
Σ iES
where is the set of successful episode indices.
Red-light Violation Rate (RVR): The fraction of successful episodes in which the agent crosses an intersection during red traffic signal, violating traffic rules. This metric evaluates the agent's s ability to obey traffic signals and is computed only over successful episodes to ensure fair assessment under comparable task completion.
Eies"
RVR
[S]
where 7; = a red-light violation occurred in episode and otherwise, and |S] the total number of successful episodes.
Stuck Rate (STR): The percentage failed episodes which the agent remains unable to reach the destination due either physical immobilization (e.g., blocked by obstacles) ineffective decision-making (e.g., repeatedly turning or repositioning without forward movement) over a period of minutes. STR captures both hard failures (being physically stuck) and soft failures (behavioral indecision), reflecting challenges in planning and low-level control.
Nstuck
STR
Nisited
where Nisted Nuval Suncess and N is the number of failed episodes classified as stuck. Normalized Decision Count (NDC): The ratio between the total number of decision-making by the large model and the total number of fine fine-grained waypoints provided by the Local Action Planner, computed over successful episodes. NDC measures the agent's decision efficiency, with lower value indicating more economical use of the model.
d
NDC
S
where de is the number of model decisions made in episode i, and w; is the number of fine-grained waypoints along the successful path. [S] the total number of successful episodes.
Decision Steps in Success (DSS): The average number of decision-making steps taken by the large model in episodes that were successfully completed. DSS measures the typical number of decisions required to achieve a successful outcome, with a lower value potentially indicating more efficient or direct Decision-making in successful scenarios.
Σd DSS
[S] iES
where d, the number model decisions made successful episode i, and |S] is the total number of successful episodes.
Together, these metrics provide a comprehensive evaluation of an agent's 's navigation performance, safety behavior, and decision decision-making efficiency in complex urban environments.
B.6
Prompt Design
B.6.1
With Local Action Planner
System Prompt Navigation Task with Local Action Planner
SYSTEM_ PRONPT **B
You are an enbodied agent in a 3D sinulation environnent, where the unit is contimeters
navigate your current position specified destination eafely and efficiently. to the destination provided as list waypointe. You only need to go to the next waypoint
24

---

if it's not blocked. Otherwise, you should choose nee next vaypoint. For this task, please only output paraable jaon string inside brackets. Please start your anszer with and end your ansser with }. Don't include any notes explanations
output joon string.
Figure 11: Examples System Prompt of Navigation Task with Local Action Planner.
User Prompt of Navigation Task with Local Action Planner
USER PROMPT
Currently you
(ourrent_ppe [aurreet_position) and your direction (current_ direction ). Your final destination is (target.positionl Your next xaypoint is next waypeint),, among the poseible next waypoints: wwwssibee_eet..vappppe waypoint is seelative_distance:: relative angle to the next waypoint is (relative_angle:... :.) degrees saypoint
). The
away from you. The
negative
your is to your right). Your walking speed is 200 ca/s.
waypoint
pesitive
You are given two images:
Previous viex (1 step ago): shows what you sav before your last decision.
Current vier: shove vhat you
Use the teo inages to understand the changes in your aurroundings and help you better decision.
have following {action faction_historry)
history
recent at botton):
Before naking your decision, you should consider the history actions. Avoid saypoint multiple times choosing nez waypoint, you should nav try noving touard it unless it's truly blocked. Avoid turning
cheosing
your last action
multiple tines rew. should try to move forvard a listle bit before turning around.
You must:
Avoid obstacles and pedestrians on the street.
Obey traffic lights when intersections: stop green.
lights, proceed
previous vieual observations Only adjust your walking direction when it deviates more than 15 degrees from the next vaypoint.
both current
your aurroundinge.
Avoid selecting new waypoint repeatedly or turning around too trequently.
Never xalk forvard continuously for more than 5 seconds; 1-5 seconds is acceptable.
Think step by step reason about your
If you are facing your next vaypoint and it's not blocked, you should move forvard
seconda).
If
not facing face the next waypoint. your vaypoint possible next vaypoints.
next saypoint roughly, should turn around
you should
vaypoint and change your next vaypoint step
you get stuck, you can backvard for a short tine (1-5 seconds).
intersection, you should atep and wait for padeatrian light to turn green before moving.
you
If you need to change your vies, you can look up or set your field of view.
Now it's tine for you to nake decision. You have the following options: nothing.
1:
Stap. Must apecify duration (nax sec) and diraction (0 forvard, 1 backvard).
Turn. Specify angle (0-180 degrees) and direction (clockwiee true/false).
3: Look up. Specify book_up_angl (-90 to 90 degrees). field
Specify (0-180 degrees).
5: Choose new vaypoint. Nust specify one from the possible next vaypoints.
Output aingle jaan object string, shose kaya are 'choica', 'duration > direction', "angle', "clockriee", anggle,, 'fov', "new_kaypoiat? .*.
Figure 12: Examples User Prompt of Navigation Task with Local Action Planner.
25

---

B.6.2 Without Local Action Planner
System Prompt Navigation Task without Local Action Planner
SYSTEM_ PRONPT .*.
You are an enbodied agent in a 3D simulation environment, where the unit centineters (ct), You good navigating in task is to navigate fron your current pasitian to a spacified destination safely and efficiently. string inside brackets. Please start your snever with { end end your snswer with string.
environnent. Your
please only output parsable jaon
include any
explenations with output json
Figure 13: Examples System Prompt of Navigation Task without Local Action Planner.
User Prompt of Navigation Task without Local Action Planner
SRER_P nnP
You are currently at (ourrent_ppo _ourrent_poeitioa) and your direction
dirrectioon ). Your final destination (target_position) The destination is approrinately seelative_distance :.2) en avay, and the relative angle to it degrees (negative
your left, positive your right). Your walking epeed is 200 cm/s.
frelative_ wawpiieeeangleeeeee
You
two images:
Previous viex (1 step ago): what you sax before your last decision.
Current vier: shat
Use these images to understand how your anvironnent ia changing and nake smart decisions.
You have the following action history (most recent at the botton): (action_bieton) bistory)
Before naking your decision, you should consider the history actions.
The relative distance and angle only give you a rough idea of where the directly toward destination checking the surroundings. Carefully plan your path instead.
destination
You must:
Avoid obstacles on the sidevalk.
step buildings, or hit obatacles on your vay. current previous Never salk forvard continuously for more than 5 seconds; 1-5 seconds is acceptable.
Keep valking
aidevalk.
readuay, crash
observatione assess your surroundinge.
You are required to make a decision for your next action. You have the following options:
0: Do nothing.
Step. Must specify
direction forward,
backvard).
Turn. Specify angle (0-180 degreas) direction (clockvise true/false).
Output a aingle jaen object string, whose keys are 'choice', 'duration', direction', 'angle', "elocknise", "nes_uaypoint",
Figure 14: Examples User Prompt of Navigation Task without Local Action Planner.
B.7
Main Results: Agent Planning With Local Action Planner
Static Obstacle. We evaluate three levels of task difficulty (Easy, Medium, Hard) under static obsta- cles with a local action planner. Time limits are set to minutes for Easy and Medium tasks, and minutes for Hard tasks. Four MILMs are evaluated: GPT-4o-mini, GPT-4o, 3ondde3.7-Sonmee and Gemini-2.. P-Poo insights discussed below:
Results are shown Table 2 with key
26

---

100-
Percentage of Action Types
100
Percentage Action Types
100
Percentage of Action Types
Parcentage of Actoons (%)
(a=0.8)
Parcentaga of Artions 1%)
Gemini-2.5pp
Invalid (0-0.4)
80
GPT-4o
Claude sonnet
60-
60
60
Percentage
40-
40-
20
20
20
1um
Stap
Re-planDo Nothing
0-
1mm
Step
Re-planDo Nothing
Tum
Stop
Do Nothing
(a) Comparison of model action types in static obstacle environ- ments (success cases analysis).
(b) Comparison of model ac- tion types in dynamic obstacle (pedestrian) environments (suc- cess cases analysis).
(c) Comparison of model action types without local action planner (all cases analysis).
Figure 15:
Distribution of Action Types for Navigation Models in Environments with/without Obstacles and Different Obstacle Types.
Model capability decisive factor in task success. State-of-the-art models such GPT-4o, Claude-3. 3.7--nnet,, and Gemini-2 5-roo consistently outperform the rule-based baseline across all three difficulty levels. These advanced MLLMs achieve higher SR, along with lower CC and CC-S, indicating their ability to understand the environment and avoid obstacles more effectively. In contrast, GPT -4--ni performs even worse than the rule-basee baseline, with lower SR and higher CC and CC-S, suggesting that it struggles to follow task rules and may be prone to hallucinations or unreasonable actions.
MLLMs demonstrate limited spatial reasoning. Although larger MLLMs attain higher SR across all tasks, do not observe corresponding drop in CC CC-S. This implies that while these models attempt to avoid obstacles, they often fail-frquently grazing or barely touching the edges of objects. This behavior suggests that current MLLMs possess strong visual recognition abilities but lack robust understanding of spatial relationships and depth.
Different models exhibit distinct strengths. Our analysis reveals that individual models have spe- cialized advantages. For instance, GPT-4o and Claude-3. 3. 7-Snnetttt show higher STR, indicating that its failures more often due to getting stuck rather than poor decisions, suggesting better decision-making but weaker spatial adaptability. In contrast, Gemini 2..i-2..5PPP 5-Pro shows a lower STR, implying that it struggles more with planning and timely decision-making. These trends are consistent with their respective SR, CC, and CC-S values: GPT-4o and Claude-3. -.aaude3.7--onnee excel in making quick and precise decisions, while Gemini-2.5-25 [Salshows that Gemini-2, 5..iii.2.5-Prr.. 5-Pro performs "choose a new waypoint" more than other models, which further explains that Gemini- 5-Pro are more sensitive to obstacle avoidance. Additionally, lower NDC supports these findings by showing which models complete tasks using fewer steps, reflecting planning efficiency.
shows better spatial comprehension. Figure
MLLMs fail to function as truly embodied agents. Although MLLM-driven agents do not achieve 100% RVR in the table, additional experiments the Hard tasks where all pedestrian signals were kept rev-reveal a more fundamental issue: agents crossed streets despite the red lights. To correctly perceive the signal state, agent must look up and focus the signal. Upon detecting a red light, it should repeatedly choose the do nothing action until the signal turns green. However, experimental results show that none of the agents chose adjust their view, and very few opted do nothing, indicating that MLLMs do not recognize their ability to adjust visual sensors. Instead, they react only based on the input they are given. This suggests that LLM-based agents require substantial additional components to behave like real embodied agents the LLM alone is not sufficient.
Dynamic Obstacles. Table presents the performance of different models dynamic navigation tasks. GPT-40 and C1 Claude-3 3.77-Sonee demonstrate superior performance, achieving higher SR and lower NDC. This suggests that these models are capable of making faster and more decisive actions, enabling them to complete navigation tasks efficiently in dynamic environments. In contrast, Gemini-2.8 2.ror experiences a significant drop in SR and exhibits higher NDC values-ccmpart to those of GPT-4o-mint while also recording the lowest values in both CC and CC-S among all baselines.
Figure I5g reveals that Gemini-22 -2.5 5-Pro frequently switches waypoints and tends to choose valid ones, suggesting stronger spatial sensitivity. However, this caution often leads to hesitation or
27

---

stalls near moving obstacles, highlighting a trade-off between spatial adaptability and robust task completion.
Table 5: Performance Comparison under Dynamic Obstacles. SR:
Average
Success Rate, CC: Collision Count, CC-S: Collision Count (Success Only), NDC: Normalized Decision Count. Human: Human Collision
Distoncc nc Goal
2932
Arrival throshold
Garrini
Count, Obj: Object Collision Count.
0030
0032
10020
Baseline
SKt
CC
CCSL
NDC
12000
03
Step
Human
Human
Figure 16: Step- Average Destination Distance across
GPT-4o-mini
26.67
1.37
3.80
5.17
1.12
2.75
3.88
2.43
GPT-4o 86.67
1.13
2.63
3.77
1.04
2.54
3.58
2.30
Tasks, GPT-40 often exhibits early stagnation, making limited goal progress, while Claude-3. 7-Sonnet shows fluctuating effectiveness. stands out by demonstrating the most stable and continuous reduction in goal distance, highlighting its superior planning and navigation capabilities without local action planner.
Claude
93.33
2.57
3.57
1.04
2.36
3.39
Gemini
56.67
0.33
1.60
1.33
0.35
1.18
1.53
2.77
Table 6: Performance Without Local Action Planner. SR: Success
Rate, CC: Collision Count, CC-S: Collision Count (Success Only), DSS: Decision Steps in Success. Bldg: Building Collision Count,
Gemini-2.5-PP
Obj: Object Collision Count.
Baseline
CC
CC-SL
DSS
Bldg
Obj
Bldg
Obj
GPT-4o
0.00
4,50
0.00
4.50
N/A
N/A
N/A
NIA
58.71
Gemini
60.00
15.83
0.40
16.23
10.44
0.50
10.94
68.72
B.8 Ablation Study: Without Local Action Planner
To evaluate the robustness of MLLM-driven agents under more challenging conditions, we remove local action planner and re-evaluate their performance on Medium-level tasks. In this setting, agents are only provided with the final destination instead of pre-defined series of waypoints. They are required to not only avoid collisions but also plan an efficient trajectory autonomously. All agents restricted to maximum of 90 decision steps per task.
We compare three state-of -htttrrrtttt MILMs: GPT-4o, Claude-2 3. 7-Sonnet, and Gemini 1.r.i--..-PP-1 P-ror As shown in Table GP fails to complete any task successfully (SR 0), while Claude-3, 3. 7-Soneet achieves a 56.67% SR and it also achieves the lowest DSS of 58.71, indicating more efficient decision-making Gemini-2 2. 5-Pro outperforms both with the highest SR (60.0%) and the lowest overall CC (16.23), including significantly fewer building and object collisions.
Figure iolltuststtes the distance-to-goal progression for each model from single experimental run. The behavioral patterns underlying these distance trends are further elucidated by Figure [5c] which shows GPT-4o and C1 7-Sonnet frequently engaging pro-progeesssve turning actions, whereas Gemini-2.55 5-Pro predominantly maintain forward movement.
B.9 Chain of Thought Example
This section, showcase a representative chain of thought from Gemini-2.5 2.5-Prro as it navigated Easy level task. This demonstration provided to give readers a transparent view the model's deliberative process when prompted to "think step-by-step. The following transcript captures the VLM's sequential reasoning, from understanding the prompt to formulating its next action.
Example Chain of Thought from Gemini-2 2-.5-P 5-Pro on Successful Easy Task
Proupt:
currently Turn around
Action:
Reasoning: : relative angle
vaypoint 144.77 degrees, which greater than 15 degrees. So need turn to face the next vaypoint. The path to the waypoint is clear based on the inages.
28

---

y--4800.0) Prompt: You currently
Action: Step forvard
Reasoning:
facing path la clear.
waypoint (relative angle 0.77 degrees)
last action was turn, nos I ahould move forvard. The distance the vaypoint 948.68 68 cn, and ny apeed is 200 ca/s, so can reach it in 4.74 seconds, which is within the alloved duration.
Proupt:
currently
Action: Turn around
Reasoning:
angle to next vaypoint absolute value of this angle (16.89) is greater than the 15-degree threshold, need to adjust ny direction. The negative sign indicates the will
degrees.
waypoint is ny left, ahovs no obstacles directly in the path to the waypoint. My last action was stepping forvard,
counter conter-clockviss. clockrise. The current viev
is repatitive.
Prompt: You are currently at
Action:
Step forvard
Reasoning: I am facing the next waypoiat (relative angle -0.89 degrees). The path clear. Ny last action turning, should forward
The distance to the saypoint is 896.46 cm, and ny apead ia 200 cn/s, so the 896.46
time
4.48
Proupt: You are currently at Step forvard
Action:
Reasoning:
The path to the next waypoiat is clear and an facing it. The small (-2.55 degress), forvard to get closer to the vaypoint.
relative angle
neaded. I will step
Proupt: You are currently at
Action: Step forvard
Reasoning:
the waypoint clear, and the relative angle is small (-7.56 degrees). Therefore, will step forward tovards the waypoint. The time reach xaypoint 200 em/a approxinately seconds, which is vithin the allosed duration.
The
Proupt: You are currently at
Action: Turn around
Reasoning:
target vaypoint Vector y=-4800.0) shich selected in ny previous acticn. The relative angle to this waypoint is significantly greater indicating I an facing almost directly oppoeite to the vaypoint. The informstion blocking ny current path. turning 179.06 degrees to the right (clockvise saypoint
179.06 degresa.
15-degres threshold,
current
immediately
me.
align nyself The tree currently front will then bohind According the guidelines, if the last action was cheesing nez xaypeint, should the first step. action does not violate rules sgainet repeated waypoint selection or turns, my
as relativa angle positive),
attenpt
Turning
history CHOOSE_MATPOIXT TURN, CHOOSE_VAYPOINT, and nov TURK.
Prompt: You currently
Action: Step forvard
Ressoning: currently facing degrees, which is xell vithin the 15- de-reg with relative angle of 0.06
vaypoint (Vector(I--188)
ries shors Ny last action sas turn, and prior to that I chose nau waypoint and turned, make progress and avoid repeated turns, should nov nove that
tolerance.
path
eaypoint
clear.
foruard. The
vaypoint 118.72 away. The instructions step duration of 1-5 seconda is acceptable. Therefore, will choose to this
step forvard
speed 200 en, vhich will overshoot the current saypoint. This is acceptable constraint,
given
system provide
aftervarda.
Proupt:
currently
Action:
Turn around
Ressoning: : relative angle greater than 15 degrees. Therefore, I need to turn to face the waypoint. Since angle is positive,
vaypoint
degrees,
right (clockvise).
Proupt: You are currently at
Action:
Choose vaypoint
Ressoning: The current next waypoint Vector (1--10000 y--3900.0 is blocked by choosing nev waypoint Vector( x=-1250.0, y=-3900.0), which is to the right of the current blocked path. Viaual inspaction suggesta this path night be clearer.
visible
current
29

---

Prompt: You currently
Action:
Turn around
Reasoning: Ky action gas choosing saypoint. current waypoiat Vector (x=-1250.0, -4-3390.0).. The relative angle this kaypoint is -15.62 degrees. Since the absolute value the relative angle (15.62 degrees) is greater than the de-degree threshold, need to The negative angle indicates the vaypoint is to ny left, so will turn cllckr-ccockkiise by obstacles in that direction.
face the waypoint.
degreaa.
innadiate
Proupt: You currently at
Action:
Turn around
vaypoint. The vaypoint Reasoning: The last y=-3900.0) 0). The relative angle to this saypoint is 16.64 Vector (x- -18-000000,
choosing
degreea,
15-degree threshold turning. waypoint is to right, so I need to tura clockwise. The path to the current vies.
greater
xaypoint appears clear
Proupt: You are currently at forvard
Action:
Reasoning: I am facing the next waypoint (ralative angle 0.64 dagraes) and it is blocked. My
turn, before waypoint. Therefore, I should now move towards the current waypoint. The distance is 928.38 en speed is 200 em/s, so it would take about 4.64 seconds reach it. will step forward for 4.5 seconds.
chose
Prompt: You currently
Action: Choose vaypoint
Ressoning: current waypoint
y--3900.0) 0) visibly blocked by construction barrier. I an choosing a new waypoint, Vector (x --1850.0, y--3900.0) shich is to the left of the current blocked waypolnt and appears offer clear path around the barrier based the current viev.
Proupt: You are currently at
Action: Choose vaypoint
Reasoning: Ky last action was choosing new vaypoint. The current next waypoint Vector (x--100008 y--3900.01 is 19.41 degrees to my right, The current shovs barrier
right, which likely blocks this vaypoint. Therefore, will choose nes waypoint that is to the left of the current Vector 2300.0, y=-3900.0).
Proupt: currently
Action: Step forvard
Reasoning:
facing next waypoint (relative angle 0.78 degraes), appears clear based the current and previous vas a turn, praceded by choosing a nes waypoint. According to the rules, vaypoint.
last action
should nox
step forward seconds to get closer to the vaypoint, which is 1046.26 en away and ny ca/s.
speed
Proupt: You are currently at
Action:
Turn around
Reasoning: My last action vas choosing a nav vaypoint. The current waypoint ia angle waypoint is degrees, vhich is greater than 15 degrees. Therefore, need to turn left to face the xaypoint.
blocked.
Proupt: You are currently at
Action: Choose vaypoint
Reasoning: The current vaypoint Vector (x=-1850.00 y=-3900.0) is blocked by construction
visible
Therefore,
vaypoint
needs to be chosen. (xctor(==1650..1 3--3900.00 0) is selected as it is to (+19.3 degrees),
barrier, requires progreasea tovards the final deatination. The right side appeara more open based on the image.
right
Proupt: You are currently at
Action:
Reasoning: Ky last action vas choosing vaypoint. The relative angle to the -19.36 degresa, is greater than degress. Therefore, need to turn face the waypoint. The obstacle in the viex is distant and does not innedistely block the path for turning.
current vaypoint
Proupt: You are currently at
Action:
Reasoning: The last action gas choosing nes vaypoint. The current saypoint is not blocked on the viaual information. relative angle to the next
30

---

vaypoint -1662 degrees, which greater than the 15-deeree threshold for turning. Therefore, I will left by 16.62 next vaypoint.
with the acceptable range.
Proupt: You currently at
Action: Step forvard
Reasoning: The
waypoint 441.28 away and the relative angle -1.47 degrees, which is small. The path appears clear from the current and The action stapping tovards the vaypoint is appropriate. The calculated duration to reach the
previous
continuing
xaypoint
seconda,
Proupt: You are currently at
Action: Turn around
Raasoning:
The relative angle to the next vaypaint is 47.34 degrees, which is degrees. Therefore, need to face the waypoint. The waypoint is to my right, so will turn clockniss.
greater
Proupt: You currently at
Action: Step forvard
Reasoning: last action
facing the waypoint (relative angle 0.34 degraea). The path to the vaypoint appears in
tavards vaypoint.
waypoint is 939.89 en. At speed of 200 ca/s, it would take about 4.7 seconds. I will atep forvard for 4.5 seconds, which is within the 1-5 second limit and will close the vaypoint.
previous
Prompt: You currently
Action: Step forvard
Ressoning:
relative angle within the de-reg tolerance, so I roughly facing it. The current view shows clear path to the waypolnt. My last action was stepping forvard, should continue noving towards the vaypoint. The distance zaypoint is 1347.76 cm. Stepping forvard for 4.0 secends at 200 cn/s will unypoint without exceading the
vaypoint
degrees,
the
800 bringing closer maximun step duration of seconds.
Figure 17: Example Chain of Thought from Gemini-2 12.mi--25550 2-r--ro on a Successful Easy Level Task.
31

---

C Case Study Details
C.1
Problem Formulation
In Case Study 2, we formalize the scenario as sequential decision decision-making problem within multi- agent environment. Specifically, we focus on evaluating the decision-making capabilities of Delivery Agents. Customers and Stores are managed by a unified rule-based manager, while each Delivery Agents autonomously decides actions maximize profit.
C.1.1
Agent Definition
Let the set of Delivery Agent be represented as D = 2D. D_n), each operating indepen- dently yet concurrently the shared environment.
C.1.2 Decision Variables
At each discrete timestep f, an agent D_i chooses from the following set of actions:
1. Accept or reject available orders.
2. Plan routing along waypoints defined by provided map structure.
3. Purchase transportation equipment to enhance delivery efficiency, subject to available financial resources.
Adjust travel speed v_l
[v rr_min, v_max] to balance energy consumption and delivery speed.
C.1.3 Constraints
Each Delivery Agent D_i operates under the following constraints:
Agent D_i has an energy level E't 2 O. Movement at higher speeds consumes more energy, and energy can be replenished by purchasing beverages at supply points. Financial Constraint: Each agent starts with an initial amount of money M 0. Actions such as
Energy Constraint:
purchasing transportation equipment or beverages require spending money. The agent's financial
status at time / is represented by M_/ 2 0.
Spatial Constraint: Movements must follow a predetermined waypoint map G (V,E), where V is the set of waypoints, and represents feasible paths.
C.1.4 Objective Function
The primary objective for each Delivery Agent D_i is to maximize total money M M_T accumulated by the end of simulation period T:
max M -
(I)
where:
R_ represents revenue earned at timestep / from successful deliveries.
C_I represents incurred timestep I, including equipment purchase costs, energy replenish- ment costs, and penalties from delayed or failed deliveries.
C.1.5 Agent Dynamics
Agent dynamics timestep are governed by:
Eft+1 E f(v_t)++ AE repleniishh
MJ+1 M
RJ- C_t,
X_++1 WaypointUppaate(___ppr
where:
f((_t) denotes energy consumption function of travel speed.
32

---

AE replenish is the energy replenished through beverages at supply points.
X denotes the agent's position timestep updated based on actions a_t t following the waypoint system.
This formulation offers comprehensive framework for evaluating long-horizon decision-making strategies of Delivery Agents within the multi-agent environment of SIMWORLDD
C.1.6 Metries Formulation
Aligned with our hierarchical evaluation framework, define the following metrics for each agent Dj operating over timesteps:
High-level: Total Profit
P; (T) M,(0),
where (t) is the agent's 's monetary balance at time
Mid-level: Operational Effectiveness
O
Order Success Rate:
SR,
(5)
1Opid
Odelay
Delay Rate:
DR,
(6)
O
Energy Efficiency:
EEg
D es
Order-Sharing Number:
SH, 1 Ophared_ssuee succ
(8)
Investment Number: IN; = buy_ bike bilee_suee
11
where
Obid is the set of orders on which D, bid,
Osucc S Obid are those bids that led to successful deliveries,
(t) is energy consumed by D, at timestep I,
Odelay Oquce are deliveries completed after the deadline,
Ophared_ denotes successfully shared orders.
buyy_bike bike_suce demotes successfully buy bike actions.
Low-level: Instruction and Rule Compliance
I IDR, D,"
Tia Dia
IDR
Va (bid, pickup, deliver, beverage, shared, meet, cancel, speed, nothing),
where
D; is the total number of actions taken by D.,
I; is the total number of those actions deemed invalid,
Di,a and Ita are the counts of attempts and invalid attempts for action a.
For each metric E (P, ADS, DR, EE, SH, IDR, IDR,). we report group-level aggregates:
m
m,s
median(m),
(m)
(m
m
1=1
These statistics are written to our CSV outputs for each model grouping. And we show the part of metric in the Table
33

---

C.2 Action Space
The low-level action space comprises fine-grained executable commands in Unreal Engine, such as move forward, rotate, pick up, and drop off. In contrast, high-level actions are determined by the LLM.
Table 7: Hierarchical Action Space Design in Case Study 2. High-level actions are given to language models to make decision, which correspond to strategic decisions, while low-level actions are only exposed to local action planner module to execute concrete movements and interactions.
Action Level
Action Name
Description
Invocation Method
order on platform compete other Genenations Navigate to the pick-up point of onder
price
Generation
Pick Up Order
Model Generation
Deliver Order
Navigate to the delivery point and complete the order
Model Generation
Share Order
Publish order for multi-medel published
cooperation
Model Generation
High-Level Actions
Cancel
Generation
Gā to Meet-point
Navigate to the meet point for the shared order
Midel Generition
Purchase Scooter
Buy and use scooter
Model Generation
Purchase
Buy
restore energy
Adjust Speed
Adjust travel speed
Model Generition
Move Forward
Basic movement action
Local Planner
Stop moving
Local Planner
Low-Level
Adjust facing direction
Change Spoed
Adjust walking speed
Local Planner
Drive Scooter
Control scooter for movement
Local Planner
C.3 Prompt Design
System Prompt of Long-Horizon Delivery Task
SYSTEM_ P*n
You face the folloving considerations:
Accepting orders along your
increase earninga afficiancy by reducing empty travel time and grouping deliveries.
Risks includs unpredictable
conditions, inaccurate time estimates,
potential delays leading to penaltisa and lover ratings.
Late deliveries incur negative ratinge and nonetary penalties, affecting your overall perfornance.
4. For each order, evaluste its fit to your current route, delivery time promise quality.
balance efficiency 5. You can buy bike for <PRICE_ _F_EEEE>>>>>> dollare to increase speed and reduce energy consumption.
You change your speed (<DELIVERY MAN_TAAAANNNNNNAAAAAAA MIN_NII_SPEED- to
SAEEED You may share picked meating point; both delivery man share aarnings and panalties if late. This bring both you 8. You can cancel a shared order if no one sccepts it after long wait. That means you decide which
DELIVERY _AM_MAX SPEED> ca/s); higher speeds consune more energy.
order with another delivery
specifying
noney
your energy.
de it by yourself, rather than waiting for too long lead to penalty.
9. If lov on energy, go to a supply point to buy beverage (cost < <DELIVERY MAN_RECOVER_EAEEE >
COST_0F OF BEVERAGE>> dollars, anergy).
You
finish order time to noney, delivery it.
Impertant:
All your decisions are based on your persona and the considerations. Provide only the final action JSOK, the JSOK format vould ba givan each
round of request. Do not add explanations or repeat given examples. city, try compete with then or cooperate vith then according to the your persons and the situation.
There are other delivery nen
If you can not nake the nost money at the end of gane, you will die. So try the rules to make the noney.
your best
Figure 18: Examples System Prompt of Long-horizon Delivery Task.
34

---

second User Prompt of Hong-Horizon Delivery Task
User prompt to output the reasoning REASONING_ [SER_PROMPT san
You are now at <POSITION> in city, where the unit ia en. buy Your position is always at or near node in the graph and you can move from one edge
beverage
recover your energy.
mode your poasible next waypoints are: <POSSIBLE, MEXBTE NAYPOINT?
made only
You have already picked up the folloving order the format Order:
<PICKED_UP URDDERR>
already
the following ordere
[irder):::;
<ORDERS>
you
new
platforn
[Order]:
<ORDERS_TO TO_BID And you could notice the following notification, which is the platforn wanns you know:
<HOTIFICATION>
each order, you
attributes
making decision:
_ax_saale_rrie: the naximun price of the order. nin_sale.price: minimun price cuatomer.position the position of the customer.
order.
ssore_porition: the position of the store.
has_picked_up:
picked up by you.
petinsted_time the estimated time to deliver the order. is_shared: shether
thared.
neeting_point: the necting point of the shared order
spent_tine: the time that haa passed since the order waa opened. energy long you
Note that you
There two ways to walking and driving. You lose energy as long as you nove. Walking is your default vay
drive you have enough money. Your current walking speed is <SPEED> ca/s. Faster kalking speed will consume more energy Currently, have <HONEY> dollars and <ENERGY> energy.
you
buy
your moving vay <PHYSICAL,STATE>>
You have the following history of your actions: <HISTORY>
each tuple
hietory your position,
iten is your action.
top botton, the botton.
history sorted by
recent
To improve efficiency, avoid visiting the sane position nore than once unless necessary and avoid repeating the action except Deliver order.
you nead make deciaion. You have the folloving options:
for an bid for order the platforn. The bid price should be betzeen the nin_ssle_price and mas sas_ssle_preeee.eeseeeeeeeeeesse price of the order.
platforn.
only
Pick up order. You pick up order from your orders. You only deliver
pick up an order that has not belanged to you.
Deliver
order from your by
the order that has not been picked
Buy baverage.
your energy 189, you buy beverage to recover energy. The price of beverage start from <COST_0F SOST_OF_BEVERRR BEVERAGE dollars and vill go up 5% every tine you buy a beverage. A beverage recovers RECOVER ENERGY AMDUNT> energy.
5. Open shared order. You can apan a shared ordar fron your orders so another shared order. This but you need to split the sarnings with another delivery man. You can only share an order that has been picked up by you. You need to decide the They will
delivery
join
your
and energy,
neeting point and go there to sait for another delivery finich the rest of the order. Xote that if the shared order is delivered late, both of you will
negative 6. Go to the nesting point. If you are joining shared order, you need to go to neeting point and
ratings and panalties.
delivery man.
7. Cancel shared order. If shared ordar has not been accepted another delivery nan for a long tine, you can cancel it and do it by yourself. change speed (range from 100 to 250 cn/s). bike
8. Change valking speed, You
valking speed
valking
Buy bike. You
your speed. bike PRICE_ _F,, UF_BIKE> dollars. But it will increase your apeed and decrease your energy coneumpticn.
Make reasoning about your next action based on the above information. For each action, you have the folloving tips:
35

---

Explain your reasoning using one sentence with clear instructions json format Example reasoning not repeat):
('reasoning': 'I have enough noney to buy bike, 80 will buy a bike n)w.")
Nov, provide only the action Jsok with keys: reasoning. e**
# User to output the action CEPION_USER_POO PROMPT Your reasoning: <REASOHING>
you need respond your
Jaon
that you only
choose one of the folloving actions:
Bid for order. Beapend with the index of the order vant bid for and the bid price.
Pick up order. Respend with the index of the order you pick up. the next vaypoint and the targat point.
Deliver
index of order you deliver, the next vaypoint and the target point.
Reapond
4. Buy beverage. Just respond with your choice.
shared order. Reepond with the index of the order vant to share and the nesting point.
Go
nesting point. Respond with the index of the shared order you to join and the next saypoint. Cancel
shared you want Change valking speed. Reapond with the new valking speed.
Reapond with index
cancel.
Buy bike. Just reepond with choice.
Provide only action JSON with keya: choice, inder, target_point, poiting.point, neyt_xaypoint, neu_apeed.
Example action (do not repeat):
('cooicel; : Lindex': 1, 'sarget_point' (200, 200), 'meeting_poiat (300, 300), 'next_xaypoint (400, 460), "nev_speed! ! 2003 .*.
Figure 19: Examples User Prompt of Long-horizon Delivery Task.
C.4 Dataset Details
C.4.1 Simulated Environment Generation
Maps created via multi-stage procedural pipeline (used in both variants):
1. Road-Grid Skeleton:
define an N x road network (N E (3..... .......8)) 8)), centered at (0, 0), with
uniform segment length and spacing.
2. Element Population: place buildings programmatically along each road; add trees and street lights for visual realism.
3. Data Screening: export scene elements, then remove non-restaurant nodes (e.g., hospitals, schools) to prepare for real-world alignment.
C.4.2 Real-World Dataset (LLMDeliveryReal--o0))
This variant aligns simulated blocks with actual restaurant data:
1. Data Acquisition: fetch Manhattan restaurant listings (latitudes 40.758-40.770 longitudes -73.995--33 -73.970) via Google Maps API; clean and convert to geospatial points.
2. Neighborhood Assignment: build polygonal blocks from street intersections; assign each restau- rant via point-in-polygon tests.
36

---

3. Normalization: map real streets/avenues to grid indices .. 1. For each restaurant, compute relative position (a, 5) E [0, 1J2 within its block and obtain
(Tnorm Muorm) (Zo + Mo 9).
4. Building Coordinate Scaling: convert original building coordinates (Forig: Morig)
E
-- offset, offset]² to [0, - 1]2 via
Torig offset IN ynorm
Morig + offset
(N 1).
Unorm
2offset
2offset
5. Matching: compute pairwise Euclidean distances in normalized space; match each building to its nearest restaurant within threshold 0.25, resolving conflicts by closest-pair selection.
6. Metadata Injection: embed a match flag and restaurant metadata (ID, name, category, geo-coords) into each building node.
C.4.3 EeeeduurallyGGenrrate Dataset ILLMDeliveryRaaddom
This variant omits real-world alignment and instead samples agent and store positions randomly:
Generation Type: "random procedural generation."
Agent Placement: ManiveryManager instantiates fixed counts of stores, customers and delivery agents at uniformly random building locations.
C.4.4 File Structure
Each map folder contains:
metadata. diffieulty, entity counts, generation type, file paths. positions. 2D coordinates of stores, customers, delivery agents.
progen_world.r json: full scene graph with normalized coords and match flags.
roads.json road segment list with endpoints.
city eity_visualizatin png: full-view rendering.
an_visualization png: delivery-focusee schematic.
C.5 Results
C.5.1 Main Results
DeepSeek-V3 V3 (69.475 + 16.772) and Claude-3. 5-S-oneet-Sttttt (69.068 8 + 20.685) achieved the highest
mean profits, with Claude-3. 5-Sonnet also leading mean successful orders (2.733 + 1.102) and energy efficiency (0.5411 + 0.1981). Notably, these superior average outcomes were associated with substantial performance variability, as reflected in their respective standard deviations.
Conversely, 5-t.iii..5-FFaa. while attaining moderate mean profit of 42.423. exhibited markedly more consistent profit generation with standard deviation of 103103, and also demon- strated stability in successful orders (2.100 + 0.173). Extreme variability was evident in specific metries for certain models; for instance, sharing counts for Deepseek-Prover-Ae Prover-V2 (7.333 8.386) and Cl aude-3. 5-Sonnet (11.333 + 8.386) showed standard deviations exceeding their means, indicating highly unpredictable behavior in this aspect.
The GPT-4o-ming model consistently yielded zero values across all metrics (0.000 + 0.000), suggest- ing it does truly understand the goals make reasonable decisions based on the given instructions and context information.
Higher investment strategies, such those adopted by DeepSeek-V3 (8.000 = 3.000) and Claude-3. Sonnee (9.000 3.464), generally correlated with greater mean profit achievement but also with increased outcome volatility. These findings underscore prevalent trade-off between optimizing for peak average performance metrics and ensuring consistent, predictable agent behavior, critical consideration for robust deployment in dynamic environments.
C.5.2 Environment Configuration
We further investigate how different environmental configurations impact agent behavior and overall performance. Specifically, explore two key factors: the global order availability and the agents'
37

---

initial financial endowment. For each factor, we conduct a series of controlled experiments to observe how variations affect agents' action distributions.
As shown in Figure 20a when the total number of available orders increases, agents tend to perform fewer pickup and delivery actions and instead choose the do nothing action more frequently. This suggests that in resource-rich environments, agents are more inclined to conserve energy and avoid unnecessary effort, opting to wait for optimal opportunities rather than actively pursue deliveries. Conversely, in low-resource settings, agents are more motivated to engage in delivery tasks to secure profits. Additionally, as resource abundance increases, agents demonstrate higher tendency to initiate and complete shared deliveries, likely as a means to reduce energy costs through collaboration.
Figure 20b illustrates the impact of agents' initial monetary resources. As initial capital increases, agents are less reliant on aggressive bidding and instead prioritize actions such as order pickup. When funds limited, competition intensifies, leading with sufficient initial capital, agents are more willing to invest in infrastructure such as purchasing a which enhances their long-term delivery efficiency.
frequent bidding behavior. Furthermore,
ORDER
BID_ORDER
DELIVER_ORDER
OPEN_SHARED_ORDS ORRDER
WAANGE_WALKNN_SPED
Main Actions
Auxiliary Actions
Main Actions
Auxiliary Actions
Count
Count
Count
Count
080
4
0
8
Total Order Quantity
Total Order Quantity
Agent Initial Money
Agent Initial Money
(a) Effect of Order Quantity
(b) Effect of Initial Money
Figure 20: Action Distribution across Environmental Settings. (a) Shows the change of effect of
global order quantity on agent behavior. (b) Shows the effect of initial money on action selection.
C.5.3 Influence of Persona.
Personality traits sigmificantly affect the decision makiin and performance of delivery agents. As shown in Figure 21, agents with higher Conscientiousness tend to exhibit lower frequency of bidding actions, a higher frequency of task-completion actions (e.g., picking up orders), and achieve higher bid win rate. This suggests that conscientious agents prioritize task completion strategic competition. Agents with higher Agreeableness are less likely to remain inactive (i.e., perform "do nothing" actions) and tend to achieve higher bid win rates. Conversely, agents with lower agreeableness display higher inactivity and narrower bidding price ranges, limiting their competitiveness. Interestingly, agents with higher Openness exhibit reduced engagement in delivery tasks, possibly because they explore competitive or unconventional bidding strategies that divert. attention from task execution.
C.6 Chain of Thought Example
This section, we showcase a representative chain of thought from Claude-3. 5-0ee.3.5-Sommm 5-Soneet. Which shows the model's 's reasoning process under different cireumstances.
Example Chain of Thought from Claude 3. 5-Snnet on different actions
Action: Do Kothing
Reasoning:
Casel- ne*
Since have no orders to deliver and no new orders to bid on, I'1l vait at my current position conserve energy and unit for orders appear.
C
deliver money buy anything. so should new orders from the platforn.
38

---

Conscientiousness vs Bid Order
Agreeableness vs Do Nothing
Openness Vs Deliver Order
-0.64
-0.00
Deliver Order
-0700 RMSE 0.035
Bid Order
RMSE 0.096
Do Nothing
RMSE 0.016
Conscientiousness Conscientiousness Bid Win Rate
Agreeableness Agreeableness vs Bid Win Rate
Openness
Conscientiousness Pick Up Order
Bid Win Rate
Win Rate
Pick Up Order
r= 0.65
r= 0.63
r= 0.66 RMSE 0.076
RMSE 0.017
RMSE 0.017
Conscientiousness
Agreeableness
Conscientiousness
Figure 21: Linear Regression Reveals Strong Correlations between Behaviors and Persona Traits The results demonstrate that agent behaviors are tightly aligned with their corresponding persona attributes, highlighting the effectiveness of trait-driven behavior modeling.
Action: Bidding
Reasoning:
Casei=""rSine 't any money yet are good near me, will bid 142.0 for Order www.ttoee..ossttiiii....o.o
(x=-18300.8) y=-1700. 0) sirre_poeition-ectooe (2--1170.0 0, y-1700.0)) since it's close to my current position and has good profit margin.
Since have money and active orders, should bid Order (max, palc.price -146. 577503676688, min, sal.sall_ppp pricee-rrc--1 (tr.ee1111..22223355555355 which starts near my location Vector (x=38300.0, y=-1700.0) naximize efficiency.
Came
Since have nо orders
money, I'1l bid Order which delivers between -58300, -1700 and -58300, 1700 since it's closest ny current location and has a short estimated time of 68 seconds.
Case
As have dollars and see other delivery men bidding 130-140 for the first for Order which high price and earning money.
should close position
Case5=ner
competitor bid 130, higher price of 135 for Order (custoner position-Vector 183000300.0 0, y --1700.0), ttoro_ooittionn-cctorr (x-1700.0 y=-21700.0) 0)) increase my chances of winning.
slready
Action: Change Speed
Reasoning:
Casei=une
are orders on, should change my valking speed 200 en/s to conserve energy while still maintaining reasonable mobility.
orders
.*B
I have order to pick Vector -1700. 6, y=1700_00 so need to move through Vector (x=21700.0, y=1700.00 to get there, but 1o noving back
39

---

and forth inefficiently. so should change my valking speed to 200 cn/s anergy while maintaining progress.
Action: Up
Reasoning:
Casel-"nr
Since already bid for two orders and have high conpetition, should towards Vector (x-18300.0, y=1700.0) to get claser to my assigned order 's location Vactor (x#38300.0, y=-1700.0).
Case2="Hr
I need to go to Vector (2--1880000 y=-1700.01 0) to coupete with other delivery men for orders there since there multiple orders that
Case3-"ne
at the adge of the nap -38300.0, y-1700.0) 0) to be closer the center where orders concentrated.
money, so will tosards Vactor
Caset="ue
I need to move towards Vector (x-38300.0 0,, 0- to pick order at the store since have already won bid ror the delivery Vector -21700.0, y=-21700.0)
CaseS="ur
store (-58300, 20 I'll move towards Vector (-41700.0, 1700.0) to pick the first order.
pick up
Action: Daliver Ordar
Reasoning:
Casel="ns
I will to Vector 0, y-1700.0 siace it's the path to the store of .*.
ay current order at Vector (x- -170000000 y-1700.0).
"ase2=""r
I need to deliver order at Vector (x---8300.0 0, y--1700.01 since have slready position, will Vector (x
picked up -.68300.0, y=-1700.0).
Case3="u*
(ector(x--1700 0,, y--1700.00 location, 20 I'll move to Vector (x-21700.0, y=-1700.0) which is on the path customer.
Cased="ur
I should nove to Vector (x=-18300.0 0,, y=-1700.D0 D)) to deliver the order since it noney quickly.
Action:
Buy Beverage
Reasoning:
Casel="nr
edge map energy multiple nearby ordere, should buy beverage to recover energy before taking on more deliveries.
.*.
Case2-"nr
energy getting
long delivery abead, should buy a beverage to recover energy before beading to the store location.
and
.*.
I have picked up order but is getting 1ou, should beverage firet to ensure can complete the delivery eafely.
.*.
Action: Buy
Reasoning:
40

---

I have several orders at ny current location but need to pick then up from distant
should buy bike firat increase efficiency since have enough noney
e*a
Action: Shared Order
Reasoning:
Casel="ur
DeliveryHan nead deliver ny
orders
anergy, ,
ahould share an order to reduce ny sorkload and conserve energy.
DeliveryHan 4: Since DeliveryNan haa opened shared order with neeting point at Vector 0, y=-1700.0) ().00), should nove to Vector and join the shared delivery.
--1700.0)
DeliveryMan
neeting point Vector (x=1700.. y=-1700.0) and DeliveryHan & wants to join shared order, I should wait here to neet hin
DeliveryHan 4: Since an at the neeting point and have lov energy but not anough money beverage, should unit here get my share fron the ahared ordars to earn monay first.
DeliveryHsn 1:
don't have enough money energy to take new orders, and firat shared order has been vaiting too long, so I ahould cancel the shared order Order wttp./aal..piiie-i...........e
-114.17753286299996 -12221111--111-17711118811111111 472221168311, min_ssle_pric y=-1700.0),
(x=-18300.0 (x=-11188000 y=-21700.0)
vtore (x=170000, f0-0000000
and do it nyself.
Figure 22: Example Chain of Thought from Claude-3. 3...-3...000 S-Sonnet on a Delivery Task.
41

---

D Related Work
Minecraft and Web-based Agents Recent advancements have led to the development of intelligent
agents utilizing open-ended platforms such as Minecraft and the Web. Minecraft offers a procedurally generated, sandbox environment that supports long-horizon planning, multi-agent collaboration, and complex tool use. A series of benchmark efforts have emerged in this setting: Voyager [47] enables lifelong learning and autonomous skill acquisition through LLM-powered exploration; Team- Craft [2] evaluates collaborative agent behaviors grounded in multimodal perception; Odyssey [29] incorporates comprchensive skill library with fine-tuned LLaMA-3 model to enhance agent capa- bilities; and Optimus-2 (8 demonstrates hierarchical control using MLLMs for complex mission execution.
Parallel research has extended to web-based agents, where the environment entails realistic interaction patterns, long-term planning, and diverse user-facing tasks. Web [88, for example, builds an end-to-end system using multimodal models to support browsing, form-filling, and question answering. Complementary frameworks such as BrowserGym 77, GPT-4V Web Agent [60], Skill- Weaver [6]] and Keal-World WebAgent [I3] further push the boundary by enabling seff-improvement, long-context reasoning, and programmatic action synthesis.
Building on these foundational efforts, we introduce SIMWORLD and the Delivery Benchmark to explore new frontier in agent research: LLM-driven multi-agent behavior in large-scale, dynamic, urban environments. Unlike prior setups, our benchmark places agents in photorealistic, procedu- rally generated city and tasks them with fulfilling complex delivery jobs over extended time horizons. To meet the demands of realism and open-ended interaction, we incorporate a hierarchical control framework, persistent memory modules, in-context reasoning feedback, and personality-driven be- havioral diversity. Together, these components enable immersive and coordinated agent performance, offering robust testbed for studying language- conditione reasoning, cooperation, and adaptation in real-world inspired settings.
Physical and Social Simulator for AI Agents Simulations have played crucial rolc in con- structing environments for training and evaluating autonomous agents. Text-based simulators often emphasize social scenarios, such as human interaction [55], daily activities (36), and relational polarization (38]. Popular embodied simulators support a broader range of applications, particularly in embodied Al research and 2D/3D scene synthesis [27]. However, most embodied simulators remain constrained to either indoor household environments (e.g.. AI2-THOR (23), Habitat (39),
iGibson (24) or outdoor driving scenarios (e.g.. CARLA [10, MetaDrive (26) or natural scenes
(e.g., AirSim 4k ). Most of these simulators 110 39 24
43
48 13 often rely on a limited num-
ber of manually crafted scenes, which hinders scalability and diversity. Some platforms, such as MetaUrban [52), MetaDrive 26 and A12-THOR [23), introduce rule-based procedural generation to
alleviate this issue. Nonetheless, existing embodied simulators typically lack support for dynamic multi-agent interactions in complex, outdoor environments.
Recent advancements have introduced large-scale, language-driven social simulators capable of modeling complex societal behaviors. OASIS (55]
simulates up to one million LLM L..M-pooered agents interacting on social media platforms, capturing phenomena such as information diffusion, echo chambers, and polarization. Casevo 22 integrates ohain-of-thhuuhh reasoning,
-uggeettd generation, and customizable memory mechanisms to simulate intricate social phenomena and decision-making processes. MinelLand (57] offers a multi-agent Minecraft environment where agents, driven by physiological needs and limited multimodal perception, engage in collective behaviors, fostering ecological and detailed simulations. Project SID [2] further advances this landscape by deploying over 1,000 autonomous AI agents within a Minecraft environment to explore the emergence of Al civilizations. Utilizing the PIANO (Parallel Information Aggregation via Neural Orchestration) architecture, agents interact in real-time, developing specialized roles, adhering to and modifying collective rules, and engaging in cultural and religious transmission. These simulations demonstrate agents' capabilities in forming complex social structures, economies, and governance systems, providing insights into large-scale societal simulations and agentic organizational intelligence.
In summary, while prior simulators have demonstrated success in specific domains such as indoor navigation or autonomous driving, none have been explicitly designed to support dynamic, multi- agent interactions in outdoor, city-scale environments. SIMWORLD fills this gap by offering scalable
42

---

platform that enables multi collaboration and competition, language groounddd interactions, and procedural generation of urban scenes- all essential for benchmarking advanced embodied agents.
With the rise of foundation models, wide range of agent frameworks have been developed that leverage large language models (LLMs) 65] or multimodal large language models (MLLMs) [4 31 35 to perform complex tasks. Numerous benchmarks have emerged evaluate LLM--bsed agents' capabilities across domains such as decision-making [3), tool 40), scientific reasoning[5]] and software Development(311 MLLM-based agents, common benchmarks often involve embodied tasks such as vision-language navigation (VL robotic manipulation [59 33 62], and interactive household activities 22 5]. And some benchmarks focus on playing games, especially Minecraft(57]. However, most of these benchmarks present relatively short-horizon goal- nil-direetedd tasks with limited action spaces- often consisting of just few discrete options. Agents are typically instructed to win game or complete predefined task [54] 31 33), which oversimplifies real-world reasoning and coordination challenges. Moreover, most settings ignore the complexity introduced by multiple agents operating in the same environment [54 31 30 40 [8 where cooperation, competition, negotiation, and resource contention are vital aspects of intelligent behavior.
Benchmarks for Foundation Model Agents
For
In contrast to benchmarks that focus short-horizon, single-agent tasks with limited action spaces, our proposed Delivery Benchmark introduces a dynamic, multi-agent urban simulation where agents must compete, cooperate, and plan over extended time horizons. The open nature of the environment and the need for real-time resource management provide a more realistic and rigorous test bed to evaluate reasoning, coordination, and adaptation in agents based on LLM and MLLM.
43

---

E Conclusion
In this work, we introduce SIMWORLD, a next-generation simulator designed to bridge the gap between high-level cognitive reasoning and low-level embodied execution for LLM/VLLM agents. SIMWORLD uniquely combines realistic physical dynamics, socially grounded interactions, and sanguage-controlle
procedural generation to create open-ended, urban-scale environments. Through two comprehensive case sudies-urban physical navigation and multi-agent social delivery economy-w we demonstrate the platform's 's capability to support complex physical and social reasoning. long-horizon planning, and strategic interaction under diverse environmental conditions.
Our evaluations reveal that while frontier LLMs exhibit promising planning and interaction abilities, they also manifest limitations in perceptual grounding, rulc-following, and consistent miriiinnmmaking.... Importandl, SIM WORLD enables fine-grained analysis of agent behavior under varying resource conditions, competitive pressures, and personality traits, offering new insights into the nature of agent adaptability and emergent strategies.
We believe WORLD lays the foundation for more holistic and rigorous development of embodied intelligence. By supporting scalable simulation, rich agent interfaces, and systematic evaluation, opens up research opportunities across AI, robotics, social simulation, and multi-agent systems. We release SIMWORL.D as an open platform and invite the community to use, extend, and build upon toward the goal of general-purposs intelligent agents capable of thriving in real-world physical and social environments.
44

---

Broader Impacts
SimWorld aims establish new paradigm for simulating realistic, interactive, and open-ended environments that support large language model (LLM)-driven agents. Its design bridges the gap between traditional simulators and the growing need for multimodal, embodied reasoning and decision-making capabilities. The broader impacts of SimWorld span multiple domains:
Advancing Al Safety and Alignment. SimWorld offers controlled yet complex testbed for studying agent behavior, ethics, and alignment in real-world inspired scenarios. By supporting language-based control and social decision-making it allows researchers systematically evaluate how agents interpret ambiguous instructions, negotiate trade-offs, and adhere to human values - key to advancing safe and aligned Al systems.
Democratizing Embodied AT Research. Built on top of Unreal Engine and designed with modular extensibility, Sim World lowers the entry barrier for researchers and educators working on embodied agents, human-Al interaction, and reinforcement learning. It enables experimentation with low-cost models while supporting high-fidelity simulation, which makes embodied Al research more accessible and inclusive.
Benchmarking and Transparency. SimWorld includes standardized multi-agent tasks and bench- mark protocols, allowing fair comparison across models and strategies. This promotes reproducibility and transparency in a field where rapid progress has often outpaced rigorous evaluation.
Applicability. By simulating city-scale logistics, physical interaction, and social coordination tasks, SimWorld directly contributes to domains such as autonomous driving, ser- vice robotics, and digital urban planning. These applications have significant societal impact, improving safety, accessibility, and efficiency in public infrastructure.
Risks and Mitigations. As with any simulation platform, there is risk of overfitting models to simplified environments. We mitigate this by supporting real-world data integration and promoting diverse task settings. Moreover, the misuse of Sim World in developing deceptive manipulative agents is a concern; we encourage the community to adopt and share responsible usage practices, and we plan release only non-malicious agent baselines by default.
In summary, Sim World provides a generalizable, ethical, and heward-looking testbed that supports the next generation of interactive Al systems. Its broader impact lies in enabling the research community to move beyond static benchmarks toward dynamic, real-world-relevan evaluation and development.
45

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract describes the key features of our work and include the reason why our work distinguish with other simulators, which is the key point of this paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer:
[Yes]
Justification:
We discussed limitations of the work on Appendix F
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
46

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We open source our code, configuration and the prompt we used make main experimental results reproducible. However, because we using third-party API server to conduct our experiment, it is normal have some error. But the error would not affects our claims. Please refer to Appendix B and Appendix C for comprehensive details of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
47

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We have included example prompt in the supplementary material. And open source our code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ httppp.cc/publlcguuuceeee Codeeubmii ssi criionPooiccy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We include all information needed in Section Section Appendix and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The paper reports statistically meaningful results by providing mean and standard deviation over multiple runs. Please see Section 3, Section 4 Appendix and
Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
48

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see Appendix B and Appendix
C for the setup of experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
We follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impact in Appendix
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
49

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work has misuse risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, cited the original owners of assets in works and these assets follow the provided licences.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
50

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, batasets.papersstt paperswi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We introduced our SIMWORLD simulator in this paper. Please refer to Section and Appendix
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/mooel part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not involve any research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do no involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
51

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: We describe in detail how we deploy LLMs powered agents in our work in Appendix
experiment section. Please refer to Section 2 Section 3 Section
A Appendix B
and Appendix
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy htap:://eerrips.cco/ooffeerrrree for what should or should be described.
52


Reviews:
[{"review_id": "9iHdKIIFGT", "reviewer": "Reviewer_XFbv", "summary": "This paper presents SimWorld, a simulated 3D world with rich physical and social interfaces that supports research on multi-agent reasoning and planning. This simulator has two main components: a UE5-based, visually realistic world generation pipeline and a multi-modal agent interface. The former utilizes pre-collected assets and off-the-shelf text-to-3D generation models to establish environment editing. The latter provides both low-level and semantic visual observation and applies llm-based parser to handle and execute free-form text actions. Physical interaction, like collision, is also included in the simulation framework, based on the UE5 engine. The paper finally makes two benchmarks, i.e., the vision-based urban navigation task and the strategic multi-agent delivery task, and provides experimental results for different baselines, which show the applications of the designed simulated platform.", "questions": "1. There is a \"sit\" action presented in Figure 2-b, but it is not involved in any action space of the study cases as described in this paper. I wonder if such complicated low-level action is implemented in the simulator, and how many interaction-full actions are involved in the simulator. Besides, how does the action translate into skeleton animation? Details about the local action executor remain totally unclear.\n2. The visual modality does not contribute to the results of the study case 2, the delivery task, as the authors involved lots of blind LLM models to perform this task and achieve competitive results (DeepSeek V3). The necessity of building a visually realistic 3D environment for such a text-based task is confusing.", "limitations": "See weaknesses.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. The idea of building a visually realistic 3D environment for studying multi-agent cooperative and social behavior is interesting.\n2. Utilizing the powerful UE5 engine to perform simulation is a reasonable choice, but the diversity of the built scenes is restricted by the number of pre-collected scene assets, which may be expensive and labor-intensive.\n\n**Weaknesses**\n\nAs a simulated world, the main contribution of this work should be providing the community with a new toolbox for studying agent-scene and agent-agent or agent-human interaction that can not be modeled under plain text. However, the proposed SimWord does not meet this requirement for several reasons: Firstly, the environment edition pipeline is not novel, as many studies have already been involved in their simulator construction pipeline [1]. Secondly, the main design in this simulator has no essential difference from the previous method.\n1. **Interaction** between the agent and objects in the simulator is limited. The generated and selected assets typically act as obstacles, which restrict the complexity and realism of embodied research.\n2. **Action space** in the two study cases provided with the proposed simulator is also limited:\n- In case 1, the agents are only able to take simple navigation actions to change their state in the environment, but they can not affect the environment by removing an obstacle or opening a door, for example. \n- In case 2, the provided action space is highly correlated with the delivery task but not general for other social tasks, for example, performing human-agent cooperation to clean the street by sharing messages and taking action to wipe the floor. It is not feasible for a user to extend other social tasks, which is a crucial mission for a general simulator.\n3. **Scenes**. Although the authors claim the proposed simulator to be Realistic and Open-Ended, my opinion is opposed to it. The scenes presented in the paper seem highly homogenized as provided in the CitySample package in UE5 engine, and the LLM-powered scene generation pipeline does not benefit from any new buildings/objects out of the preloaded asset bag. This poses a severe problem that the simulator may differ a lot from the real-life application scenario. \n\n[1]. Genesis: A Generative and Universal Physics Engine for Robotics and Beyond.", "was_revised": true, "final_justification": "As all my concerns have been satisfactorily addressed, I recommend acceptance.", "rebuttal": "We thank the reviewer for recognizing *“building a visually realistic 3D environment for studying multi-agent cooperative and social behavior is interesting”* and *“the powerful UE5 engine to perform simulation is a reasonable choice.”* Below, we address the concerns regarding the diversity, novelty, interaction capacity, and case study setup of SimWorld.\n\n## Novelty and Contribution of SimWorld\nWe appreciate the opportunity to reemphasize the novelty of SimWorld. As noted in Lines 33–42 of the main text, and further discussed in Table 1 and Appendix D (Related Works), we have carefully compared SimWorld with existing simulators.  While prior works have proposed simulation pipelines, SimWorld is distinguished by its support for realistic, open-ended world simulation with LLM/VLM-based agents across diverse physical and social reasoning scenarios. Specifically, it introduces three key contributions (detailed in Lines 46–76): \n- **Realism**: Compared to existing simulators (see Table 1 and Appendix D), SimWorld delivers significantly more visually, physically, and socially realistic environments. This level of realism is critical for evaluating VLM-based agents in complex tasks such as object interaction, social competition, and beyond.\n- **LLM/VLM Integration**: Unlike existing simulators built on UE5 (e.g., CARLA [1]), which typically lack standardized interfaces for language model integration, SimWorld offers ready-to-use APIs and middleware that support reasoning, planning, and action execution in a unified framework.\n- **Unified Physical and Social Simulation**: SimWorld bridges the gap between low-level physical environments (e.g., Habitat 3.0 [2], AI2-Thor [3]) and text-based social simulators (e.g., AgentSociety [4], Oasis [5]) by supporting both embodied interaction and multi-agent social behavior through rich agent interfaces.\n\n### Relation to Genesis\nWe would like to clarify the distinction between SimWorld and Genesis. To our knowledge, Genesis is still under active development and lacks a publicly available technical report or preprint detailing its architecture and limitations. Genesis and SimWorld differ significantly in scope and purpose. **Genesis functions primarily as a physics and graphics engine, serving a role similar to that of UE5 in SimWorld, rather than as a platform for agent-centric simulation**.  \n\nCompared to UE5, Genesis places greater emphasis on physical simulation, with *limited* support for multi-agent interaction and agent action modeling. In contrast, UE5, upon which SimWorld is built, is a more mature simulation engine with extensive community support and robust capabilities for agent integration.\n\nMoreover, SimWorld adds a higher-level abstraction layer on top of UE5, enabling diverse agents, including LLM-, VLM-, and rule-based agents, to operate in both physical and social contexts. While it may be possible to extend Genesis to support such functionalities, doing so would require substantial engineering effort. SimWorld, by design, includes built-in components for agent modeling, task orchestration, and evaluation—capabilities that go beyond what Genesis currently offers.\n\n## W1/Q1.1: Limited Interaction.\n\nWe appreciate the reviewer’s comment and would like to clarify that SimWorld currently supports *10* atomic object interactions, which is more than those available in many other popular simulators like AI2-THOR [3] (*6 atomic object interactions*) and MineDojo [6] (*7*), as detailed in Appendix Table 4.  Examples include animation-driven actions such as entering a car, object manipulation, and \"sit\" as referred to by the reviewer. The action set **can be easily expanded** by importing additional assets or animations from the UE marketplace.\n\n\n## Q1.2: Implementation Details of (Inter)actions. \n\nWe kindly refer the reviewer to Appendix A.1.4 (Local Action Planner) and Appendix Figure 8 (b) for detailed descriptions. For example, when the planner receives a user instruction such as “go to the nearest chair and sit down,” the Local Action Planner decomposes it into a sequence of primitive actions, e.g., `navigate`, `agent_sit_down`, each with associated parameters. These primitives are implemented as callable APIs, each linked to specific skeleton animations from the agent’s Unreal Shader. This ensures that action execution is explicitly grounded in animation and tool use by the LMs.\n\n## W2: Limited Action Space for two Study Cases.\nSimWorld supports a broad range of environment interaction, including navigation, object manipulation, and communication, as listed in Appendix Table 4. In two case studies, we *intentionally* limited the action space to ensure clarity and experimental control.\n- **Case Study 1** focuses primarily on navigation actions, but SimWorld is fully capable of supporting more complex interactions such as object manipulation.\n- **Case Study 2** showcases high-level actions, which are automatically decomposed into sequences of low-level actions. Users can freely define new high-level actions by composing existing primitives, enabling more complex and general social tasks, such as collaborative cleaning or object handover. \n\n## W3: Diversity of Assets and Scenes\nTo clarify, **SimWorld is not limited to any asset resource**. Our asset library is built from multiple resources, including but not limited to “CitySample”. Our platform supports importing third-party 3D assets, including those from external datasets and generative models, allowing users to expand the asset library as needed.\nMore importantly, our scene generation pipeline focuses on composing diverse spatial layouts and interaction-rich scenarios using available assets. This approach enables rich open-endedness at the level of agent interaction, spatial variability, and task complexity. We believe this modular and extensible design offers a practical and scalable path toward realistic simulation, especially for benchmarking generalist agents across dynamic and socially grounded tasks.\n\n## Q2: Clarification on Case Study 2 (Delivery Task)\nThe delivery task supports both visual and headless (rendering-off) modes. We chose to evaluate LLM agent performance in headless mode for two reasons: (1) The physical simulation remains fully functional. Waypoints, dynamics, and constraints such as speed and distance are still enforced. (2) Given the current limitations of VLMs in visual perception, long-horizon tasks tend to accumulate significant perception errors. Since Case Study 1 already evaluates agents' visual and physical reasoning, our goal here is to decouple visual perception from long-horizon planning and isolate the planning capability of the LLM. This makes our experiment controlled for analytical purposes.\n\n[1]CARLA: An Open Urban Driving Simulator  \n[2]Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots  \n[3]AI2-THOR: An Interactive 3D Environment for Visual AI  \n[4]AgentSociety: Large-Scale Simulation of LLM-Driven Generative Agents Advances Understanding of Human Behaviors and Society  \n[5]OASIS: Open Agent Social Interaction Simulations with One Million Agents  \n[6]MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge"}, {"review_id": "ZJ0Gwpsrw6", "reviewer": "Reviewer_kTRD", "summary": "The paper introduces SimWorld, a new simulator built on Unreal Engine 5, designed for the development and evaluation of Large Language Model (LLM) and Vision-Language Model (VLM) agents in complex physical and social environments.\n\nThe authors argue that existing simulators often lack realism, rely on limited hand-crafted environments, have simplified physics and social rules, and lack native support for LLM/VLM agents. SimWorld aims to address these shortcomings by offering three core capabilities: (1) realistic, open-ended world simulation with accurate physical and social dynamics and language-driven procedural environment generation; (2) a rich interface for LLM/VLM agents, featuring multi-modal inputs/feedback and open-vocabulary actions at various abstraction levels; and (3) diverse, easily customizable physical and social reasoning scenarios.\n\nThe paper demonstrates SimWorld by deploying frontier LLM agents on short-horizon navigation and long-horizon multi-agent food-delivery tasks, revealing distinct reasoning patterns and limitations across different models. The authors intend to open-source SimWorld to serve as a foundational platform for advancing real-world agent intelligence.", "questions": "The paper notes graphical rendering was disabled in one case study for efficiency. Can you provide more specific metrics on performance, such as average simulation step time, maximum number of concurrent agents tested, or memory footprint under different configurations (e.g., with/without rendering, varying numbers of agents, city complexity)? What are the anticipated bottlenecks for scaling up simulations?", "limitations": "No, the authors don't adequately address the limitations.\n\nIn the checklist, the authors say that they address the limitations in Appendix E Conclusion, but it is not the case.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\nThe paper identifies and addresses a clear need for developing advanced AI agents capable of real-world interaction.\n\nI think that the paper is generally well-written and structured. Figures 1, 2, and 3 effectively illustrate the core concepts and case studies. The \"Takeaway\" summaries for experimental results are helpful for quick comprehension.\n\n## Weaknesses\n\nWhile the paper mentions simulating social norms like obeying traffic signals and maintaining personal space, and the multi-agent delivery task involves economic and sharing mechanisms, the complexity and generalizability of the \"social rules\" could be further elaborated. How easily can new, more nuanced social interactions or cultural norms be defined and integrated?\n\nHigh-fidelity simulation with UE5, especially with multiple agents and LLM/VLM processing, can be computationally expensive. The paper mentions disabling graphical rendering for one case study to ensure efficiency. While understandable, more detailed discussion or metrics on the computational resources required, scalability limits, or typical performance (e.g., simulation steps per second under certain conditions) would be beneficial.\n\n## Typo\n\nIn caption of Figure 3, there is a typo: \"Case 1emphasizes\".", "was_revised": true, "final_justification": "The authors' rebuttal was helpful and addressed my questions, it confirms my initial assessment of the paper's contributions and limitations. Therefore, I will maintain my current rating.", "rebuttal": "We truly appreciate the reviewer's recognition that *“The paper identifies and addresses a clear need for developing advanced AI agents capable of real-world interaction,”* which lies at the heart of our motivation. We are also grateful that the reviewer found *\"the paper is generally well-written and structured\"* and that *\"Figures 1, 2, and 3 effectively illustrate the core concepts and case studies.\"* It is especially rewarding to know that *\"The 'Takeaway' summaries for experimental results are helpful for quick comprehension,\"* as we designed them with clarity in mind. \n\n## W1: Integration and Generalizability of Social Rules\nSimWorld supports flexible definitions of social rules. These are typically encoded as high-level behavior protocols composed of sequences of low-level actions (see *Appendix Table 4* for supported atomic actions). Examples from the paper include:\n\n- **Navigation tasks**: Agents obey traffic signals, yield at intersections, and maintain personal space.\n- **Delivery tasks**: Agents engage in structured social behaviors such as bidding for delivery orders, order sharing among agents, and strategic investments (e.g., purchasing scooters for speed gains). These mechanisms simulate competitive and cooperative dynamics that mirror real-world economic interactions.\n\nTo support generalization, SimWorld does not constrain any specific social rules. Users are allowed to inherit SimWorld’s Python APIs to enable such extensions without engine-level modifications. For instance:\n\n- In **bike-sharing systems**, we can add rules for agents to compete for limited mobility resources while adhering to fairness policies, by writing some Python scripts.\n- In **waste-sorting scenarios**, agents must correctly classify and dispose of different types of trash based on visual cues, constrained by environmental rules that are defined by users.\n\n## W2/Q1: Computational Performance and Scalability\nFirst, we’d like to note that most existing simulators do not support multi-agent scenarios(e.g. MetaUrban[2], AI2-THOR[3]), and platforms like Habitat 3.0[1] only support up to two agents. In contrast, SimWorld can simulate up to 50 agents in real time (>60 fps) on a consumer-grade machine (e.g. RTX 4070 SUPER, Intel i7-14700F and 32GB RAM). With further optimization of the communication mechanism between Python and Unreal Engine (which we see as future work), the number of supported agents can further be increased toward the upper limit imposed by CPU performance.\nTo provide a clearer picture of SimWorld’s performance and requirements, we summarize the typical configuration below: \n- GPU: 6GB or more (8GB+ recommended).\n- RAM: 32GB or more is recommended for efficient and stable multi-agent simulation.\n- Disk: current SimWorld assets require ~50GB.\nWe are also conducting additional experiments to further evaluate SimWorld’s scalability and will include detailed results in the final version. All tests were conducted on a Windows desktop with 64GB RAM, Intel ii7-14700F (2.10 GHz), Windows 11 OS, and an NVIDIA GeForce RTX 4070 SUPER, using random seed 42 and a resolution of 1280×720.  \n\n### Memory footprint (in GB). \nWe evaluate peak system memory usage (RAM) under three rendering configurations: **Render** (standard real-time viewport rendering), **NullRHI** (disables graphics rendering but retains scene data and buffers), and **RenderOffscreen** (renders to an offscreen buffer without displaying frames). These settings reflect common use cases, from full visualization to headless simulation for efficiency. Since the static city assets (e.g., roads and buildings) are the primary contributors to memory consumption, we vary the number of roads (randomly select buildings along the roads) to assess how scene complexity affects memory usage.\n\n| # Roads | Render | NullRHI | RenderOffscreen |\n|--------:|--------:|--------:|----------------:|\n| 20      | 5.1     | 12.7    | 5.8             |\n| 50      | 12.9    | 19.2    | 13.5            |\n| 80      | 16.6    | 22.3    | 17.3            |\n| 100     | 20.4    | 22.6    | 20.9            |\n\nRender consumes the least memory, as real-time rendering leverages GPU resources efficiently and discards non-essential CPU-side buffers. In contrast, NullRHI, despite disabling visual rendering, retains all scene data and internal rendering pipeline structures in memory (e.g., mesh buffers, material data), leading to significantly higher memory usage. RenderOffscreen strikes a middle ground, allocating rendering buffers without displaying frames but still benefiting from certain optimizations in the pipeline.\n\n\n### Simulation step time and agents. \nFor simulation step time, SimWorld supports both synchronous and asynchronous modes. In **synchronous mode**,  the Python client explicitly controls the simulation timing: at each step, it sends a tick command to the UE server and waits until the server completes the simulation update. In **asynchronous mode**, the UE server runs continuously at its own frame rate, while the Python client can retrieve data at any time.\nTo evaluate scalability, we tested different numbers of humanoid agents on a blank map to eliminate other sources of overhead. As shown below, increasing the agent count leads to lower FPS and higher rendering latency. The current bottleneck primarily stems from the communication mechanism between Python and Unreal Engine, which can be optimized in future versions (e.g., through batched communication). The ultimate upper bound on agent count is determined by CPU performance, especially for complex behaviors and camera shooting. Supporting hundreds of agents may require multi-CPU or distributed setups to maintain real-time performance.\n\n| # Agents | FPS |\n|---------:|----:|\n| 0        | 360 |\n| 100      | 79  |\n| 500      | 12  |\n| 1000     | 8   |\n\n### Scalability factors:\n- GPU: Affects rendering speed and image-based model inference.\n- CPU: Affects tick rate and agent logic execution.\n- Memory: Limits the number and complexity of loaded assets.\n- Network latency: Affects communication between Python and UE.\n\n## L1: Clarification on Limitations Section\nThank you for the catch! Our original Appendix E was accidentally commented out. We will revise the final version to explicitly discuss the following limitations:\n- Constraints on scaling to hundreds of agents in large-scale cities. Scaling simulations to hundreds of agents in complex urban environments is currently constrained by the UnrealCV plugin that we used for communication. Optimizations at both the engine and system architecture levels are needed to address this challenge. \n- Limitations of the current prebuilt action space and asset library. While our current scale of assets and action library has been rich enough compared with other simulators, the number of actions and assets is still constrained by budget. There are thousands of additional actions and assets available on the marketplace and can be purchased to easily extend the set. Besides, future work will also incorporate on‑the‑fly motion generation to support more dynamic and realistic interactions and explore 3D concept learning to produce higher-quality, interactive, and more adaptable assets.\n\n[1]Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots  \n[2]MetaUrban: An Embodied AI Simulation Platform for Urban Micromobility  \n[3]AI2-THOR: An Interactive 3D Environment for Visual AI"}, {"review_id": "JmSIz4IkiU", "reviewer": "Reviewer_VvQg", "summary": "The authors introduce SimWorld, realistic physics-based simulation environment based on the Unreal engine that allows for setting up environments via natural language instructions (instead of having to specify the environment in some specification language like python).  Compared to other currently available simulation environments, SimWorld presents a superset of features, included open-ended simulations (although there was no particular evidence of the open-endedness).  The fact that the authors evaluated various frontier models in two experimental settings and presented results and demo videos shows that the simulation is ready for rich agent interactions, in particular, in more complex social environments (that are nevertheless based in physical realism).  The authors promise to make the simulation available open-source which would be valuable resource for the research community.", "questions": "Please make the link to the source repo available", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths: SimWorld combines diverse features from other simulation environments, thus allowing for comprehensive simulation studies all the way from bottom-up physics to high-level social interactions.  The demonstrated interfaces to external agents are excellent, although I would have liked to see non-gen AI agents connected to the simulation as well (e.g., robotic architectures).\n\nWeaknesses: I would have like to see evidence for the openendedness the authors claim the simulation has; the submission should have had a link to the repo that the authors claim to make available", "was_revised": true, "final_justification": "I have no more updates based on the authors' feedback.", "rebuttal": "We thank the reviewer for recognizing SimWorld’s *“comprehensive simulation studies from bottom-up physics to high-level social interactions”* and *“demonstrated interfaces to external agents are excellent”*. We appreciate the constructive feedback regarding evidence for open-endedness and the availability of the repository. \n\n## W1: Evidence for Open-Endedness\n\nWe hope to convey the open-endedness of SimWorld as clearly as possible, while NeurIPS 2025 does not permit external links in rebuttal. Alternatively, we have included rich-media evidence in the Supplementary Materials to illustrate key aspects of our system:\n- **bird'seye-generation.mp4**. Shows the process of city generation.\n- **case_study1.mp4**. Shows the physical simulation scenario in case study 1. \n- **case_study2.mp4**. Shows the social reasoning scene in case study 2.\n\nTo meet the file size requirements, we have compressed the videos, which may slightly reduce visual quality. Additionally, we provide the complete Python source code for both study cases and the SimWorld framework to ensure transparency and reproducibility.\n\n## Q1: Source Code Availability\nWe confirm that the full SimWorld codebase will be released publicly upon publication. The repository link will be included in the final camera-ready version, and we have prepared comprehensive documentation to support community adoption. \n\n## Formatting Comment: Related Work and Conclusion in Appendix\nThank you for pointing this out. We will move both sections into the main body of the paper in the final version to comply with formatting standards."}, {"review_id": "3XCiUfNJjb", "reviewer": "Reviewer_TMee", "summary": "The paper introduces SIMWORLD, a novel simulation platform designed for developing and evaluating large language model (LLM) and vision-language model (VLM) agents in complex, dynamic environments. Built on Unreal Engine 5, SIMWORLD offers realistic physical and social interactions, enabling agents to navigate urban settings, engage in social reasoning, and autonomously perform tasks like earning income. The authors demonstrate the capabilities of SIMWORLD through case studies involving navigation and multi-agent delivery tasks, revealing distinct reasoning patterns across various LLM models.", "questions": "1. How do the authors plan to address potential biases in agent behavior influenced by the training data of the LLMs?\n2. Are there plans to extend SimWorld to incorporate more diverse physical interactions to enhance realism?", "limitations": "1. The performance of SIMWORLD heavily relies on the capabilities of the underlying LLMs, which may limit the generalizability of findings if future models exhibit different behaviors.\n2. While the simulator aims for realism, there may be inherent trade-offs between accurately modeling complex human behaviors and maintaining computational efficiency.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The integration of realistic physics and social dynamics into a procedural generation framework represents a significant advancement in simulator design for AI agents.\n2. The paper thoroughly evaluates multiple frontier LLMs in diverse scenarios, providing insights into their strengths and weaknesses in real-world-like tasks.\n3. The authors plan to open-source SIMWORLD, promoting accessibility and further research in agent intelligence.\n4. The interface is rich. The multi-modal input and output capabilities facilitate complex interactions, allowing agents to process visual and semantic information effectively.\n5. The ability for users to modify scenarios through natural language input enhances the platform's flexibility and usability.\n\nWeaknesses:\n1. While the paper presents two case studies, additional scenarios could provide a more comprehensive evaluation of SIMWORLD’s capabilities. For example, the physical aspects of the environment is not fully presented. The authors present navigation as one of the phyiscal reasonign task? However, it cannot fully reveal the agents' understanding of fundamental physical concepts such as gravity, collision, and object permanence, etc.\n2. The discussion on agent behavior lacks depth in terms of analyzing the underlying decision-making processes and how personality traits influence performance.", "was_revised": true, "final_justification": "As the authors have addressed my concerns. I will keep the acceptance vote.", "rebuttal": "We sincerely thank the reviewers for their positive evaluation of our work and for highlighting the strengths of SimWorld, including its *“integration of realistic physics and social dynamics”*, *“thorough evaluation of multiple frontier LLMs in diverse scenarios”*, and *“plan to open-source”*. We appreciate the recognition of the platform’s *“rich interface”* and *“ability for users to modify scenarios through natural language”*. We address the main concerns below:\n\n## W1: Diversity of Physical Reasoning Scenarios\nWe’d like to clarify that SimWorld supports a rich range of low-level physical actions, as detailed in Appendix Table 4. We emphasize that the primary contribution of SimWorld lies in providing **a simulation platform designed for developing and evaluating LLM/VLM agents in complex, real-world-like environments**. We believe that the community can use our simulator to create controlled physical understanding benchmarks, e.g., for rigid-body dynamics, object manipulation, destructible meshes, and tasks requiring understanding of gravity, collisions, and object permanence.\n\n## W2: Discussion on Agent Behavior and Personality Traits.\nWe kindly refer to Appendix C.5.3 (Influence of Persona) and the examples of diverse agent behaviors and chain-of-thought reasoning in Appendix C (Figures 21 and 22), which illustrate how personality traits shape agent strategies. In summary: **Conscientious** agents prioritize task completion over competition, **Agreeable** agents are more active and competitive, while **Open** agents tend to explore unconventional strategies, potentially at the expense of task execution efficiency. \n\n## Q1: Potential bias of LLMs\nTo clarify, SimWorld is designed as a simulation and evaluation platform, not as a solution to mitigate biases inherent in pretrained LLMs. While we do not focus on addressing biases from the underlying LLMs, our goal is to provide a controlled, customizable environment where communities can systematically test and analyze agent behavior—including potential biases—across diverse tasks and contexts (physical and social scenarios). \n\n## Q2: Plans to Extend Diverse Physical Interactions\nWe would like to clarify that UE5 natively supports more diverse physical interactions (including rigid-body dynamics and destructible objects) and can be easily integrated into our work. In future work, we plan to extend SimWorld with richer physical interaction capabilities to further enhance realism and support more complex agent behaviors.\n\n## L1: Limited Generalization\n\nWe clarify that the two study cases in this paper are used to show the potential for building physical and social reasoning scenarios in SimWorld. Our goal is to build a powerful testbed for the evaluation of state-of-the-art LLMs/VLMs. \n\n\n## L2: Trade-offs between Accurately Modeling and Computational efficiency.\n\nSimWorld can simulate up to 50 agents in real time (>60 fps) on a consumer-grade machine (e.g. RTX 4070 SUPER, Intel i7-14700F and 32GB RAM). With further optimization of the communication mechanism between Python and Unreal Engine (which we see as future work), the number of supported agents can further be increased toward the upper limit imposed by CPU performance.\n\nTo provide a clearer picture of SimWorld’s performance and requirements, we summarize the typical configuration below: \n- GPU: 6GB or more (8GB+ recommended).  \n- RAM: 32GB or more is recommended for efficient and stable multi-agent simulation.  \n- Disk: current SimWorld assets require ~50GB.  \n\nWe are also conducting additional experiments to further evaluate SimWorld’s scalability and will include detailed results in the final version. All tests were conducted on a Windows desktop with 64GB RAM, Intel ii7-14700F (2.10 GHz), Windows 11 OS, and an NVIDIA GeForce RTX 4070 SUPER, using random seed 42 and a resolution of 1280×720.  \n\n### Memory footprint (in GB). \nWe evaluate peak system memory usage (RAM) under three rendering configurations: **Render** (standard real-time viewport rendering), **NullRHI** (disables graphics rendering but retains scene data and buffers), and **RenderOffscreen** (renders to an offscreen buffer without displaying frames). These settings reflect common use cases, from full visualization to headless simulation for efficiency. Since the static city assets (e.g., roads and buildings) are the primary contributors to memory consumption, we vary the number of roads (randomly select buildings along the roads) to assess how scene complexity affects memory usage.\n\n| # Roads | Render | NullRHI | RenderOffscreen |\n|--------:|--------:|--------:|----------------:|\n| 20      | 5.1     | 12.7    | 5.8             |\n| 50      | 12.9    | 19.2    | 13.5            |\n| 80      | 16.6    | 22.3    | 17.3            |\n| 100     | 20.4    | 22.6    | 20.9            |\n\nRender consumes the least memory, as real-time rendering leverages GPU resources efficiently and discards non-essential CPU-side buffers. In contrast, NullRHI, despite disabling visual rendering, retains all scene data and internal rendering pipeline structures in memory (e.g., mesh buffers, material data), leading to significantly higher memory usage. RenderOffscreen strikes a middle ground, allocating rendering buffers without displaying frames but still benefiting from certain optimizations in the pipeline.\n\n### Scalability. \nTo evaluate scalability, we tested different numbers of humanoid agents on a blank map to eliminate other sources of overhead. As shown below, increasing the agent count leads to lower FPS and higher rendering latency. The current bottleneck primarily stems from the communication mechanism between Python and Unreal Engine, which can be optimized in future versions (e.g., through batched communication). The ultimate upper bound on agent count is determined by CPU performance, especially for complex behaviors and camera shooting. Supporting hundreds of agents may require multi-CPU or distributed setups to maintain real-time performance.\n\n| # Agents | FPS |\n|---------:|----:|\n| 0        | 360 |\n| 100      | 79  |\n| 500      | 12  |\n| 1000     | 8   |"}]

Author rebuttal:

Meta-review: The paper introduces SimWorld, a novel simulation platform built on Unreal Engine 5, designed for developing and evaluating LLM/VLM agents in realistic, open-ended physical and social environments. It addresses limitations in existing simulators by offering: (1) realistic world simulation with accurate physical and social dynamics and language-driven procedural generation; (2) a rich multi-modal interface for LLM/VLM agents; and (3) customizable scenarios for diverse reasoning tasks. The authors demonstrate SimWorld’s capabilities through two case studies: a short-horizon navigation task and a long-horizon multi-agent food delivery task, evaluating frontier LLMs (e.g., Gemini-2.5-Flash, Claude-3.5, GPT-4o, DeepSeek-Prover-V2). Results highlight distinct reasoning patterns and limitations across models, underscoring SimWorld’s potential as a testbed for real-world agent intelligence.

This paper highlights its contribution on robust evaluation pipeline and its commitment to open-sourcing. It supports procedural generation and customizable scenarios via Python APIs.

Reviewers mention several weakness:

Scenario Diversity: Limited to two case studies; more physical reasoning tasks (e.g., gravity, collisions) could strengthen evaluation.

Scalability: High-fidelity simulations are computationally intensive; scaling to hundreds of agents is challenging.

Social Rules: Needs clearer elaboration on defining nuanced social norms.

During rebuttal, authors clarified SimWorld’s distinction from Genesis/VirtualCommunity (PS, would be good to also discuss with recent woks about LLM agent for scene creation), detailed 10 atomic interactions, provided performance metrics (50 agents at >60 FPS), and confirmed flexible social rule definitions. They addressed formatting issues and committed to open-sourcing. All concerns were resolved, with two reviewers strongly supporting acceptance and two maintaining borderline ratings.
Decision: Accept (spotlight)