CREW-Wildfire: Benchmarking Agentic Multi-Agent Collaborations at Scale
Authors: Jonathan Hyun
Nicholas R Waytowich
Boyuan Chen
Venue: Accepted by TMLR ()

Abstract: Despite rapid progress in large language model (LLM)-based multi-agent systems, current benchmarks fall short in evaluating their scalability, robustness, and coordination capabilities in complex, dynamic, real-world tasks. Existing environments typically focus on small-scale, fully observable, or low-complexity domains, limiting their utility for developing and assessing next-generation multi-agent Agentic AI frameworks. We introduce CREW-Wildfire, an open-source benchmark designed to close this gap. Built atop the human-AI teaming CREW simulation platform, CREW-Wildfire offers procedurally generated wildfire response scenarios featuring large maps, heterogeneous agents, partial observability, stochastic dynamics, and long-horizon planning objectives. The environment supports both low-level control and high-level natural language interactions through modular Perception and Execution modules. We implement and evaluate several state-of-the-art LLM-based multi-agent Agentic AI frameworks, uncovering significant performance gaps that highlight the unsolved challenges in large-scale coordination, communication, spatial reasoning, and long-horizon planning under uncertainty. By providing more realistic complexity, scalable architecture, and behavioral evaluation metrics, CREW-Wildfire establishes a critical foundation for advancing research in scalable multi-agent Agentic intelligence. All code, environments, data, and baselines will be released to support future research in this emerging domain.

Published in Transactions on Machine Learning Research (12/2025)
CREW-Wildfire: Benchaarking Agentic Multi-Agent Collab- orations at Scale
Jonathan Hyun', Nicholas R Waytowich", Boyuan Chen'
Duke University, 2 AAmm Research Laboratory
Reviewed on OpenReview:
Abstract
Despite rapid progress in large language model (LLM)-based multi-agent systems, current benchmarks fall short in evaluating their scalability, robustness, and coordination capabilities in complex, dynamic, real-world tasks. Existing environments typically focus on small-scale, fully observable, or low-complexity domains, limiting their utility for developing and assessing next-generation multi-agent Agentic AI frameworks. We introduce CREW- BLEEW-WLDFFIRE an open-source benchmark designed to close this gap. Built atop the human-AI tcaming CREW simulation platform, CREW-WIIE WILDFIRE offers procedurally generated wildfire response sce- narios featuring large maps, heterogeneous agents, partial observability, stochastic dynamics, and long-horizon planning objectives. The environment supports both low-level control and high-level natural language interactions through modular PERCEPTION and EXECUTION modules. We implement and evaluate several slate-oftthe-arl L.M--aased multi-agent. Agen- tic A1 frameworks, uncovering significant performance gaps that highlight the unsolved challenges in large-scale coordination, communication, spatial reasoning, and long-horizon planning under uncertainty. By providing more realistic complexity, scalable architecture, and behavioral evaluation metrics, CREW-WILDFIR establishes critical foundation for advancing research in scalable multi-agent Agentic intelligence. All code, environments, data, and baselines will be released to support future research in this emerging domain.
1 Introduction
Coordinating multiple agents to solve complex, dynamic, and high-stakes tasks a long-standing challenge in artificial intelligence (AI) (OroojlooyJadiid & Hajjiezhad 2021 Ismail et al. 2018 Buçoniu al.) 2010
Real-world scenarios such disaster response (Drew] 2021 Kashyap et al.) 2025), autonomous infrastructure
maintenance (Ismail et al.) 2018 Krnjaic al. 2024). workflow optimization (Krnjaic al. 2024
planetary exploration (Huang et al. 2020 Qin et al.) 2025b(a) often require teams of agents with diverse capabilities to operate under uncertainty, partial observability, and real-time constraints. Among these, wildfire response (Seraj et al.) 2021 Siedler 2025) stands out as a prototypical example: a setting that demands scalable collaboration, heterogeneous roles, and adaptive decision-making over extended time horizons. As interest grows gencralist and open-ended AI systems (Ahn al., 2022). there strong neod to evaluate whether today's 's Agentic AI [rameworks can meet the demands of such real-world complexity.
Traditionally, multi-agent systems have been studied through the lens of multi-agent reinforcement learning (MARL) (Buşoniu et al. 2010). decentralized control (Zhang et al. 2018), or classical robotics coordination Perez el al. 2018). These approaches have yielded impressive capabilities in domains such as warehouse logistics (Knjacc et al. 2024), robotic swarms (Huang et al., 2020), and cooperative manipulation (Chen et al, 2022). However, they typically roly on rigid communication protocols, domain-specific policics, or centralizod planners JardojlooyJadiid & Hajinezhad, (2021) that limit generalization and flexibility. Morcover, these systems often struggle to scale in the number of agents or in the complexity of their operating environments

---

Published in Transactions on Machine Learning Research (12/2025)
Procedurally Generated Large Scale Multi Agent Environments
LLM Ready Communication Framework
Diverse Helerogeneous Agent Tasks
Affi
S Searching for survivers!
G Fire Spotted
Figure 1: CREW-WILDFIRE features procedurally generated environments, LLM-compatible multi-agent framework, and heterogencous agents designed to evaluate Agentic collaborations at scale.
due to their inherent limitations in communication, commonly done through exchanging gradients (Buşoniu ct al., 2010), observations (Focrster et al., 2016), or intentions (Kim al., 2020). It becomes particularly chal challenging when coordination must emerge from high-level reasoning over extended periods of time rather than low-level behaviors.
Recent advances in large language models (LLMs) have opened new frontier in Agentic AI (Wang et al. 2024 Zhang el al. 2024a Park et al. 2023 LLM-based agents can communicate in natural languages, per perform in-context reasoning, and dynamically coordinate using structured dialogue or shared goals. Early frameworks have demonstrated promising behaviors such as delegation, consensus-building and role assignment, often in open- endee tasks like programming (Qian al., 2024h, board games (Chen et al.) (024a). or virtual social environments (Kaiya et al., 2023 However, these demonstrations largely operate in limited settings: they assume fully observable worlds, involve small number of agents, and lack real-time embodiment environmental complexity. As a result, it remains unclear whether existing multi-agent LLM frameworks can scale to realistic, physically grounded, long- horizon problems that demand both strategic coordination and low-level exccution.
In this work, we introduce CREW- WIILD-R (Fig. 1 an open-source benchmark specifically designed evaluate Agentic multi-agent LLM systems under conditions of real-world scale and complexity. Built atop the human-AI tcaming CREW Zhang et al. (2024c d) simulation framework, CREW-WILDFIR features procedurally generated wildfire environments with heterogeneous agents (e.g., drones, helicopters, bulldozers, and firefighters), partial observability, stochastic environments, and complex objectives such as civilian rescue, fire detection and suppression. It provides flexible observation and action spaces, ranging from low-level vectors to natural language commands, and includes built-in PERCEPTION and EXECUTION modules to interface with both low-level control primitives and T.L.-based agents. To support rigorous evaluation, we propose suite of task environments and behavioral goals, enabling both quantitative and qualitative assessment of multi-agent LLM frameworks on scalability, coordination, adaptability, and communication.
We implement and benchmark several recent LLM-based multi-agent Agentic frameworks in CREW-WILDFIRE Our results reveal significant performance gaps and highlight open challenges. Our experiments show that while these systems exhibit, emergent collaboration in simple lasks, they often fail to generalize to environments that require precise real-time coordination, spatial understanding, plan adaptation under uncertainty, and objective prioritization. By surfacing these limitations and offering a standardized evaluation platform, BREW-WILDFFES provides a critical foundation for accelerating progress in large-scale Agentic intelligence.
2

---

Published in Transactions on Machine Learning Research (12/2025)
Table Feature comparison across different multi-agent embodied control environments.
Euvironment
Maximumn Agunta
Heterogenous Aggezia
Low-Level Actions
Low-Lovd Observations
Partial Obsurvaiiity
Complex Long- Hurison Tioks
Stochastic Dymamioo
Enviroementt
Goncrative
Came
(Chil diacndas high-leod text)
(Qule dislgguns, desergetioe)
Virtuiloomn
Uug o ar 2013
x
x
(Diecneta artion tensces)
[Iaage)
PactingZoo Tarry 3025
x
x
X
V
(iierreta contimnne cction tessors)
[Loue-hood state vectors)
CUISINEW
x
(Only deseription)
high- toxt)
PARTNR Clang
(2 agumts)
(Coatisnmm artion tensoes)
[Iuayes
BREW-WRRISS (ours)
2000
(1 agynte)
[Diecnets continnonn centom tensors)
(Imagee, ASCIL lonsloel statos;
In summary, our main contributions are:
A new open-source benchmark for ovaluating LLM-based multi-agent Agentic systems in procedurally generated, physically grounded, and high-stakes disaster response environments.
A suite of sub-environments and behavioral goals, enabling fine-grained evaluation of collaboration, spatial reasoning, task delegation, and plan adaptation at scale.
Comprehensive benchmarking of state-of-the-art frameworks, revealing performance gaps and providing open challenges for future research in scalable coordination and Agentic AI.
2 Related Works
Benchmarks for Multi-Agent Embodied Environments. A wide range of simulation environments have been developed to benchmark multi-agent systems, spanning symbolic games, strategy simulations, and embodied tasks. Hanabi Bard al. (2020) has served a canonical benchmark for emergent communication and theory of mind under partial observability, but it is entirely symbolic and turn-based, lacking physical embodiment or dynamic interaction. StarCraft II Vinyals et al. (2017), in contrast, provides rich spatial dynamics and partial observability and has served as challenging environment for micromanagement and tactical coordination. However, its use in multi-agent evaluation typically emphasizes unit-level control and scripted tasks, rather than open-ended planning in heterogeneous teams.
Environments such as Overcooked Carroll et al. (2019) PettingZoo Terry et al. [2021), Visual Hide and Seek Chen al.] (2020) 2021); Ji et al. (2025) introduce complex behaviors and embodiments but are limited scale, environmental realism, and agent diversity. More recent efforts like MineDojo Fan al. (2022) and VirtualHome Puig et al. (2018) explore open-ended tasks and embodiment, but primarily focus single-agent
settings or symbolic role-play scenarios.
Several simulation frameworks have also been adapted for real-world domains, including traffic control and disaster response. Notably, platforms like FireCommander [Sraaj et al. (2021) and Hivex Siedler (0255 simulate wildfire scenarios using multi-agent systems for resource management and disaster relief. While domain- relevant and complex, these environments often lack the architectural support for largo-scale coordination and LLM-based agents. They do not support. flexible agent communication protocols and high agent counts, making them unsuitable for evaluating the scalability of emerging Agentic AI frameworks.
BREW-WILDFIE bridges these gaps by offering fully open-source, embodied, and highly scalable environ- ment centered on evaluating and benchmarking LLM-based multi-agent Agentic AI frameworks. It supports heterogeneous agents, complex and dynamic terrain, and realistic objectives under partial observability and stochastic conditions. By combining these features with support for low-level control and high-level language reasoning, CREW- WILDFIEE provides unique testbed for evaluating large-scale collaboration, perception, and planning in Agentic multi-agent systems.
3

---

Published in Transactions on Machine Learning Research (12/2025)
Overview of CREW Wildfire
(a) Procedurally Generated Environment
(b) LLM- Ready Agentic Framework
State
Agent State
50, $1;
position
Observations
Perception Module 5
Text Perception
extracted inf formation
Task
Team Structure
Memory
PnV
minimap
Past
Agent
Communication Framework
History
messages
messages
heterogenous agents
text action
Action
Execution Module
Primitive Library
0o, 01,....
Figure 2: An overview of the CREW Wildfire Framework. (a) Environments are procedurally generated through a combination of Perlin noise textures modeling vegelation, elevation, moisture, settlement, and wind vector maps. (b) CREW- WILDFIRE supports template infrastructure generalized from existing LLM agent frameworks, including PERCEPTION and EXECUTION modules, built for casy implementation of multi-agent Agentic frameworks.
Agentic AI and Multi-Agent Language-Based Frameworks. The rise of LLMs has catalyzed a new wave of research in language-enabled Agentic AI. Methods such as CAMEL Li et al. (222) Chat Qian et al. (2024). Lyfe Agents Kaiya et al. (2023), and Generative Agents Park al. 2023 demonstrate that LLMs can coordinate in multi-agent settings through natural language communication, role assignment, and iterative planning. These systems have shown promising emergent behaviors such delegation, consensus formation, and memory sharing. However, the vast majority of these demonstrations are under relatively constrained environments purely symbolic domains, such as collaborative software development, dialogue-based games, single-agent tasks, or simplified simulations. They lack embodied complexity, team size scalability, continuous dynamics, and low-level control interfaces found in real-world systems.
CREW-WILDFIRG complements and extends these prior efforts by providing a scalable, grounded, and embodied benchmark specifically designed for LLM-based multi-agent tasks. includes PERCEPTION and EXECUTION modules that enable language-based agents to interpret sensory information and generate executable actions, thereby supporting both high-level planning and low-level interaction. Furthermore, CREW-WH.DFIRF introduces st structured sub-environments and behavioral goals to facilitate rigorous evalua- tion of Agentic competencies such as task allocation, spatial reasoning, observation sharing, and adaptive caaniing--aabbblittie that are underexplored in current benchmarks.
3
The CREW-Wildfire Environment
3.1 Preliminaries
One natural path in designing a benchmark for large-scale LLM-based multi-agent systems is to adapt existing multi-agent reinforcement lcarning environments, such as StarCraft II or FircCommandor, to support LLM-based agents. While this approach would enable direct usage of prior MARL benchmarks, it constrains the integration of flexible communication structures and high-level Agentic reasoning capabilities of language
4

---

Published in Transactions on Machine Learning Research (12/2025)
Cellular Automata Fire Propagation Model
Figure 3: The Cellular Automata Fire Propagation Model
models. Another strategy is to extend current multi-agent LLM frameworks to handle larger, heterogeneous teams. However, these frameworks are typically built around simplified domains, and scaling agent count. alone does not inherently introduce the strategic complexity or environmental diversity neoded challenge scalable Agentic intelligence.
Therefore, chose to build CREW- from the ground up. This approach allowed to make principled design decisions tailored to evaluating Agentic intelligence at scale, including the development of heterogeneous agents, real-time coordination challenges, and long-horizon planning in dynamic environments.
CREW Platform. CREW-WILDFIRE is built on the CREW platform Zhang et al. (024c). a scalable human-Al tearning simulation infrastructure using the Unity engine. Unity provides a visually rich and customizable development environment, while CREW provides flexible modules for rapid prototyping of complex game dynamics and Al-ready programming interfaces.
3.2 Environment Design
Procedural Map Generation. distinctive feature of map capability. Each scenario procedurally generated using Perlin noise to produce continuous terrain features (i.e., elevation, wind, moisture) and discrete land types (i.e., forest, brush, rock, water). Human settlements also randomly placed controlled by seeds. These variables infiuence fire spread, visibility, and mission-critical decisions, ensuring each run presents unique and uncertain challenges. Visual illustrations of such procedure is shown in Appendix A.1 and Fig.
its scalable generative
Wildfire Simulation. We simulate fire dynamics using an advanced cellular automata model (Fig, that incorporates slope, wind, vegetation type, moisture, and terrain fcatures. This model creates realistic and unpredictable fire spread, forcing agents to adapt and exploit environmental features such as firebreaks. Specifically, we model the local fire spread probability Pspread from a source cell to neighboring target cell as:
moisture
T
Pspread (slope)
moisture constant.
|
A
where the slope factor fslopee accounts for uphill versus downhill fire spread dynamics:
kislope
if slope < 0,
(slope)
2e bsllopeppp
11'
Assope
if slope 2 0.
Here, t denotes the wind vector and AE is the direction vector pointing from the source cell to the target coll. The magnitude A represents the distance between cells: 1 for adjacent cells, /2 for diagonal cells. The slope factor (slopee scales the influence based on terrain steepness, increasing spread probability uphill and decreasing it downhill. The moisture ratio continuous, but ignition outcomes are binary (a cell either ignites or does not based on comparing Pspread against threshold). The moisture ratio attenuates the likelihood of ignition in wetter regions. The dot product modulates the alignment of wind and spread direction, increasing
5

---

Published in Transactions on Machine Learning Research (12/2025)
Firefighter
Bulldozer
Drone
Helicopter
Speed: Normal
Speed: PM
Speed: Fast observation Range: Cells
Speed: Veryy Fast
observation Range: Cells
Observation Range: Colls
Observation Range: Cells
Abilities
Abilities
Abilities
Abilities
Forward
Move Forward
Nove Forward
Rottte degrees
Rotate degrees
direction (x,y)
Rotate degrees
degrees
direction (x,y)
Cut tree of current cell
Move direction of (x,y)
Nove in direction (x,y)
Pick Firefighter
Spray
angle
Flow
Destay Firefighters
Spray water at target location
Refill
Raise Flow
Rafill water
Pick up civilian within range
Figure 4: The Four Heterogenous Agent Types: Firefighters, Bulldozers, Drones, and Helicopters
fire spread probability when aligned and decreasing it when opposing. The propagation speed is determined by the game update frequency, which is user-configurable. This computed value Pspread is then compared against a threshold to determine whether the target cell ignites in the next simulation step.
3.3 Agent Design
We include four heterogeneous agent types (Fig. abstracted from real wildfire operations:
Firefighters are generalist agents capable of cutting trees to create firelines, spraying water to extinguish fires or reduce flammability, and rescuing civilians.
Bulldozers quickly clear flammable vegetation. However, they have limited speed and cannot directly extinguish or rescue civilians, requiring support from firefighter agents.
Drones perform wide-area recomnaissance but lack physical intervention capabilitics.
Helicopters transport agents and civilians and deliver water. With large enough maps, helicopters play indispensable role in transporting firefighters distances too far to travel foot.
These roles are designed to: 1) simplify but sufficiently represent real-world fire crew specializations, 2) span diverse skill sets that require cooperative execution, and 3) necessitate non-trivial coordination strategies due to inter-agent dependencies.
These agent types have complementary capabilities requiring coordination: 1) helicopters can transport firefighters over long distances but cannot rescue civilians independently, requiring helicopter-firofihhee coordination, 2) drones can spot fires from afar while ground teams cannot, but drones cannot cut trees extinguish fires, necessitating drone-ground coordination, 3) bulldozers can cut firebreaks efficiently but cannot rescue civilians, requiring coordination with firefighters.
CREW-WI.DPIRE supports a large range of observation and action spaces, encompassing both low- and high-level control: Observations support direct mini-map encoding, third-person agent image data, textual descriptions of the observable enviroument, or ground-truth agent data vectors. Actions support 3- dimensional discrete or continuous action tensor, textual descriptions of an action, or high-level primitives involving multi-step actions.
3.3.1 Perception and Execution Modules
Our PERCEPTION and EXECUTION modules are frequently found in and designed to be compatible with modern LLM-based multi-agent Agentic Al frameworks.
Perception module CREW-WILDFIRE provides various types of observations, such as ground touud truth data tensors and raw images. However, LLMs still struggle with processing high-dimensional sensor data such

---

Published in Transactions on Machine Learning Research (12/2025)
images, and VLMs struggle with reasoning capabilities. Therefore, many milti-agent-LLMM frameworks employ PERCEPTION module that serves to bridge this gap. Our PERCEPTION module is made up of two components that first transform the raw tensor data into ASCII representations and then into high-level text summaries. The prompts for the PERCEPTION module can be found at Appendix A.2 along with some examples. We have observed that our PERCEPTION module can provide accurale and reliable observational information to the LLM agents, so that the agentic LLM framework can focus on team planning.
Execution module Consistent with related work (Zhang al. 2024b), we found that while LLMs succeed at high-level control, low-level control remains challenging for LLMs of today. To address this limitation, many LLM-based agentic frameworks employ "exccution modules" that serve as middle layer between high-level planning and low-level control. Given that our core environment requires precise, discrete and continuous control, we developed our own built-in EXECUTION module to support these preexisting LLM-based frameworks while leaving additional flexibility for low-level control.
Our EXECUTION module uses LLMs to translate natural language commands into series of one or more executable action codes. These action codes give all the necessary information to activate the associated low-level primitive functions (e.g., cutting tree, spray water at location, pick up civilian) within the simulation environment to control the agents. For information on these primitive functions, please refer to Appendix A.3.
We also provide the prompts of the EXECUTION in Appendix A.4
3.4 Pillars of CREW-Wildfire Design
Beyond the individual features described, CREW-Wildfiee strength lies in the combination of its properties, which together create unique and challenging benchmark aimed at tackling the inherent challenges that large and complex cooperative task, such as wildfire fighting, creates. We define the following pillars essential to create complex yet flexible enviromment that facilitates high-level cooperation: 1) High scalability: CREW-WILDPRRFCCM can support large number of agents (2000+ agents and 1 million++ cell maps on a desktop with 16GB GPU + 16GB RAM + 16GB GPU. Computational performance data up to 2000 agents and 1 million cells can be found in Appendix A616) ); 2) Task complexity: Due to the map randomness, fire simulation, and multiple map-dependent objectives, CREW-WILDEE
does not always have a clear best solution that fits every case. As the task progresses, dynamics may change, which requires continual plan revision and adaptation; 3) Heterogeneous agents: Role heterogeneity demands coordination through complementary capabilities; 4) Flexible observations and actions: CREW-WILDFIRE WILDFIRE accommodates wide range of learning paradigms via modular observation and action layers.
These strengths make CREW-WILDFIN a particularly challenging and comprehensive benchmark, where success would demonstrate a framework's ability to handle large-scale cooperation, partial observability, and long-term strategy all at once.
4 CREW-Wildfire Benchmarking Suite
Robust benchmarking in complex environments requires more than tracking cumulative rewards. It demands tools to diagnose where algorithm is conceptually challenged (as revealed through behavioral competency analysis in Section 5.3), not just how well it scores. To address this, CREW-WILDFIRB Benchmarking Suite provides infrastructure for systematic evaluation, testing, and behavioral analysis.
Procedurally Generated Task Levels. We define 12 distinct task levels (Table 2. four of which have both small and large size variants, resulting in 16 total benchmark configurations. Levels vary in team composition, map size, objectives, and difficulty. We measure the performance via task success, damage minimization, and agent/civillian safety. We also assign high-level behavioral goals to each level to evaluate a variety of high-level behaviors.
Behavioral Goals. We designed 7 behavioral competencies, ranging from simple task designation complex objective prioritization. Since each level is marked with behavior goals, we can them to pinpoint.
7

---

Published in Transactions on Machine Learning Research (12/2025)
Table Twelve Benchmark Levels (with size variants for four levels), F: Firefighters, B: Bulldozers,
D: Drones, H: Helicopters
Name
Objective
Agents
Map Size
Max Score
Behaviors
Cut Trees
Scoring Fuaction: Trees cut in labeled cella/lines
Spare (small)
Cut all troes in laheled cells
3 F
30
18
TD
Spaze (bage)
Cut all in labeled cells
F
Lines (small)
Cut all the labeled lines of trees
2 F.
30
30
TD. AC
Lines (large)
all labeled lines
105
TD, AC
Scout Fire Scoring Function: Drepes over the fire, max of two
(small) (large)
Scout comfirm
the map
3 D 5 D
100
TD, SR, OS
Soout and confirm fire within the map
250
2
TD, SR, OS
Transport Firefighters
Scoring Furctioun: Firefighters at laggel Jocasions
(small)
Transport all Erefighters target location
6.. 12
100
6
AC, SR, RC AC, SR, RC
(large)
Transpors all firefighters to a target location
250
12
Reseue Civillans Seoring Function: Civillans target location
Known Location (stzal)
Rescue all tiviliaus lo a Laggel location
3.F
40
3
TD. SR. PA SR, TD. SR. OS, PA
Known Location (large)
all divilians to target location
3.F
Search and Rescue
Locale and recue all civiliaus
largel location
5 F. D.D.D 2,
100
5
Transport
Locate rescue civilians target
TD, AC,
Suppress Fire Sooring Finction:
Trees Destroged 20x Agents Lost
Extingaish
Extinguish the
a known location with water
8 F 5 F, F B.2
60
N/A
TD, SR. PA
Conlain
Contain the fire at known location wishout water
60
N/A
TD, AC, SR, PA
Locale and Suppress
Suppres the fire al an unknown lomation
N/A
TD, AC, OS, SR, PA
Locase Deploy Suppress
Suppress the fize
unknown lonation
F. 2.
150
N/A
TD, AC, SR, RC,
Full Environment Scoring Fuaction:
Trees Destroyed + 20xAgents Lost Civillans Lost
Full Favironment
Locate and Suppress the fine while reseuing eivilians
H
N/A
emerging collaborative behaviors. For instance, the Cut Trees levels require Task Designation for agents to split up the trees to cut.
The behavioral goals include: 1) Task Designation (TD): the ability to divide tasks among agents; 2) Agent Capitalization (AC): the ability to recognize and capitalize the strengths and weaknesses in heterogeneous teams; 3) Spatial Reasoning (SR): The ability to reason and plan accordingly with spatial information; 4) Observation Sharing (OS): the ability to communicate useful observations when necessary; 5) Realtime Coordination (RC): the ability to communicate and rely on other agents to perform synchronized tasks; 6) Plan Adaptation (PA): the ability to adapt and revise plans; 7) Objective Prioritization (OP): the ability to rank competing goals contextually, More information available in Appendix g
5
Benchmark Experiments
5.1
Experiment Setup
Our goal is to assess how well current LLM-based multi-agent Agentic frameworks scale and generalize in realistic, embodied, and partially observable environments. We selected four representative frameworks from recent literature that span approaches ranging from fully decentralized to hybrid centralized-ddeenntraaieed systems:
CAMON Wu et al. assignments and global updates. Leadership transfers occur through agent-initiated interactions.
(004)) employs a hybrid coordination scheme with a dynamic leader who issues task
COELA Zhang et al. 2024bb is decentralized structure where each agent independently proposes information, evaluates the necessity of communication, and performs actions based on its generated plans.
Embodied Guo et al. 2024 follows a decentralized structure with no leader agent. Agents alternate communication rounds, allowing message broadcasting or targeted exchanges, followed by independent action planning.
HMAS-2 2 Chen et al. (2024b) adopts hybrid coordination scheme, using a centralized planner to propose actions refined through distributed agent feedback until consensus is reached.
8

---

Published in Transactions on Machine Learning Research (12/2025)
Table 3: Scores across All Baseline on Twelve Levels (mean + standard deviation).
Max Score
CAMON
COELA
Embodied
HMAS-2
Do Nothing
Cut Trees: Sparse (Smell)
18
18,00:10.00
14.00=4.18
14.60+2077
17.40=0..99
0.00=00.00
Out Trem: Sparse (Large)
75
72.00::4.00
$7.87=25.00
50.3319.00
13.03
0.00100.00
Cut Tree: Lines (Smzell
90
30.00:10.00
30.00=00.00
30.00::0.00
28.00:2..6 65
0.00110.00
Thves Lines (large)
94.33++7000
83.67-24.00
90-33+1.000
10.33-66000
0.00++0000
Scout Fire (Smal)
1.60=:0889
0.0020.00 00
0.8020.84
1.20:11.10
0.00200.00
Scout Fire (Large)
0.00:00000
0.0010.00
0.00:00.00
0.00110.00
Traceport Fizelighters (Small)
6.00=0.00
3.00: 00
1.601-4..00
1.10:55.00
0.00:00.00
Trarsport Firefighters (lang))
12
10.00++0.00
8.33+3.00
9.33+2.00
8.33+5.00
0.00+0.00
Reseso Civilians: Known Location (Smunll)
3.00==000
0.67+0.58
2.67+0.58
2.33+0058
0.00+0.00
Resene Civilians: Known Location (Large)
4.00=5.00
0.00:0000
4.33=3.00
4.00=22.00
0.00=0000
Rescue Civiliams: Search and Rescue
0.00110.00
0.00:10.00
0.00110.00
0.002.0.00
0.00100.00
Rescue Civiliaue: Search Rescue Transport
10
0.00:50.00
0.0620000
0.00:00.00
0.0020.00
0.00:10:00
Suppress Fire: Extiagrish
-588.331196
635.002189.66
-6363391198.67
424.33:1166.99
519.87:1144.91
Suppress Flie: Contain
736.67+200.04
7511-33+202.60
777-33+117744 422
7799000-190.00
1600677+144.05
Suppress Locate Suppress
1062.677225.61
1062.67=248.61
1002.67+-28.61
1062
1062.67=248.61
Suporess Fire: Locate + Transport Suppress
-129667336700 .07
379.677387..07
-729.67 .07
-720.674387.07
-729.672337.07 .07
Environment
46771.6743347.00
-5516.00_3388.81
-5539.3813381.00
33122.6713336.22
-5302.67.:333.00
Do Nothing is introduced as a naive basclinc where no agent intervencs in the environment.
All baselines are fully zero-shot, L.L.-bbased multi-agent. frameworks with no learning during inference. These implementations follow their original published designs without training or fine-tuning.
Evaluation Protocol. To ensure consistency across these frameworks, we implemented compatible PERCEPTION and EXECUTION modules to enable uniform testing.
We use GPT-4o as the underlying model, with temperature set to 0 for deterministic results and a single completion per decision step. Experiments were conducted on a laptop with a 3.0 GHz CPU, RTX 3060 GPU, and 16GB RAM. We ran between 3-10 random scods on all 16 level configurations for a total of 410 trajectories. Seeds and other APT call frequency, and input/output token usage. Results are in Tab
are located in A.19 We measured task success, task duration,
We used GPT-4o due to its superior performance on spatial reasoning and task planning among popular benchmarks at the time of experiments. Therefore, we chose to use it to benchmark all baselines with the same LLM backbone. Moreover, our focus not on language model capabilities in the baselines, but instead, on different multi-agent LLM planning architectures and workflows and their performance on complex and lcale environments. Finally, GPT-4o was also used by the original implementations in the baseline studies, hence our choice here for the benchmark. We leave the exploration of different LLM models as future work.
5.2 Results and Findings
Current frameworks handle simple tasks but falter at scale. Most frameworks, especially CAMON, performed well on Cut Trees tasks, showcasing effective task designation and asynchronous execution. In smaller maps, most frameworks successfuully divided the work, as seen in (Fig5). However, in larger instances, decentralized systems such as COELA and Embodied performed worse due to task overlap, with multiple agents redundantly cutting the same tree due the lack of shared global knowledge.
Coordination breaks down on complex and large-scale tasks. In tasks such Suppress Fire and Search and Rescue, performance across all frameworks was poor, often worse than the Do Nothing baseline due to penalties from agent loss. Centralized and hybrid systems like CAMON and HMAS-2 with leader agents struggled to assign roles effectively in complex scenarios, often issuing overlapping or redundant plans. While leader agents could successfully coordinate low-level tasks, such as cutting lines of trees (Tab. 8), they failed to perform the same level of task decomposition in complex scenarios. In Fig. Ga), multiple agents were directed cut the same tree without further decomposing the broader firebreak construction plan, showing that leader agents struggle to make complex multi-level tasks.
9

---

Published in Transactions on Machine Learning Research (12/2025)
Task: Cut all trees in cells: (10, 21), (19, 21), (17, 21), (25, 4), (15, 5), (10, 24)
Agent 1
[00211)
[7,21]
Agent 2
[0024]
[9.21]
CAMON
Agent
[1S,5]
[25,4]
Agent
Agent
[25,4]
D5.31
COELA
Agent 3
[19.21)
[7,21)
Agent 1
[15,5]
[5,A4]
Agent 2
[10.21]
[25.4]
Embodied
Agent
Agent
110.71)
D0.24
Agent
[19.2
[1.21)
HMAS
Agent 3
(5.55
[25,4]
Figure 5: Sample Multi-Agent Trajectorics for the Cut Trees: Sparse (small) level. Each highlightod section represents the correct tree being cut by that agent. Horizontal direction denotes time progression from left to right.
(a) Overlapping Actions
(b) Repetitive Messages and Broad Directions
Agent
agerts, maintain
creating *sre3reaks
Agent please down Location [25, 88] to assist crecting firebreak. This will help contain the fire more effectively
your current
melicsptees, pesoaze for resupsly operacions Stay alert and report any changes irmediately
Brsnes, updata
Time
Agent 2
Continuing
Agent 2, instead of noving to (21, 83), focus on cutting all create firebreak...
Locetion (25, 88) help
inresistely.
Agent 14
Agent please down location [25, 803 to assist in crecting firebreak. This will help contain the fire more effectively
your current
ALL ager ts, maintain focus on your tasks ... IDFones, peiorisize Finding water sources Inolicuptees are ready for zesusply operations Stay alert roport 2ny changes inmediately
Agent to LLeden
Agent 4, please cut down all trees at your current Location [25, 803 to assist creating firebreak. help contain
Agent
agerts, maintzin FocuS creating Fürmareaks Brones, update us any Witer sources Relicoptees, prepare for resupsly operacions report any changes trmediately
more effectively.
Figure 6: Common Failure Cases. (a) Overlapping actions in CAMON due to failed subtask decomposition by the leader during firebreak creation. (b) Repetitive and vague messaging in Embodied.
CORTNN
Input Tokens Per Timestep
CAPEN Inderred
Dutput Tokens Per Timestep
10090
API Calls Per Timestep
X0800
250000
10900
5900
Agent Count
Agent Count
Agent Count
Figure 7: API calls, input tokens, and output tokens per timestep across agent counts. These metrics show algorithm-sioe operations only; the CREW-Wildfre environment itself has zero API cost as it is entirely sell-contained.
Mcanwhile, docentralized frameworks such as COELA and Embodicd faced noise and repetition. As seen Fig. b), agents issued vague, overlapping instructions and failed converge on detailed strategy. Agents all wanted to take charge of the entire plan ending up in rigid loop without adaptation.
10

---

Published in Transactions on Machine Learning Research (12/2025)
Table Behavior Competency Scores (BCS) per algorithm. Objective Prioritization omitted from our competency analysis because no baseline encountered both the active fire front and endangered civilians within the full-envirrnment level across our recorded trajectories.
Behavior
CAMON
COELA
Embodied
HMAS-2
Task Designation (TD)
0.454
0.279
0.381
0.399
Agent Capitalization (AC)
0.474
0.391
0.435
0.418
Spatial Reasoning (SR)
0.368
0.156
0.301
0.301
Observation Sharing (OS)
0.180
0.067
0.124
0.153
Realtime Coordination (RC)
0.430
0.305
0.374
0.352
Plan Adaptation (PA)
0.239
0.093
0.219
0.209
Communication costs scale with agent count. As shown in Fig. 7 token usage scales with agent population. API calls and output tokens increase roughly linearly. However, input token usage in HMAS-2 grows quadratically, due to its shared observation vector that scales with the number of agents and partial observability. In fully observable environment with a small size, it is feasible provide each agent all the environment information. However, in large environments like ours (1000 1000 cells), this makes universal context sharing inefficient. These results highlight the need for scalable architectures, e-g., decentralized or hierarchical approaches, that avoid token blow-up through more structured information routing.
5.3 Behavioral Analysis
To better evaluate the competency of current multi-agent agentic systems beyond task completion, we introduce the Behavior-Comppe Competency Score (BCS), which captures the average normalized success across all tasks associated with given high-level behavioral goal. With BCS, we can extract more interpretable and quantifiable insights into how the structures of different architectures facilitate different behaviors and where, on a behavior level, they succeed or fail.
Step 1: Level Normalization. The range raw scores across different task levels is different due to different scoring functions. Therefore, we must first normalize our scores across all tasks types.
Finite (reward) tasks are tasks such Cut Trees, where the team receives positive scores, such for every correct tree cut. Open-ended (penalty) tasks are tasks such as Suppress Fire, where the team receives onen-ended negative scores, such as for every tree burned.
To normalize task f, we first find the bascline (Bt) or the worst possible score on a task (0 for finite tasks and the Do-Nothing score the penalty for losing all agents for open-ended lasks). Then we find the target (Tt) or the best possible score on task (max reward for finite tasks and 0 for open-ended tasks). Then, we normalize the algorithm's raw score (Sa.e) by subtracting the baseline and dividing by the range, Lastly, we logarithmically scale scores for open-ended tasks to amplify the smaller improvements the baseline score.
Sa, - Be
for finite (reward) tasks;
T/ Br
NS
log(
+
Su.e
T Bd
for open-ended (penalty) tasks.
log
Step Behavioral Aggregation. Since each benchmark task associated with a set of behavioral goals, each behavioral goal g also has set of tasks Tg that possess as behavior. Therefore, the BCSa,g for a given behavioral goal and algorithm is the mean of the normalized scores of that set..
BCSa,
NSa.t
[T
LE
11

---

Published in Transactions on Machine Learning Research (12/2025)
BCS Results and Findings Overall, shown in Tab. 4 and Fig. 10 current methods perform well at Realtime Coordination (RC), Task Designation (TD); and Agent Capitalization (AC), with CAMON particularly standing out across all behaviors. However, while all methods score comparably in behaviors such as Realtime Coordination (RC) and Agent Capitalization (AC), their performance varies much more in others, including Observation Sharing (OS), where the decentralized COELA scores significantly lower than the rest. Perhaps this implies that although dynamic communication may help planning and real-time coordination, periodic communication phases, such as those in Embodied and HMAS-2, are better suited for regular observation sharing.
The Plan Adaptation (PA) BCS also remains low across all basclines. PA is tested most extensively in Suppress Fire tasks, where fire shifts unpredictably. In these levels, instead of adapting when circumstances change, teams often reaffirm early decisions and plans, fixing agents to an outdated course of action. For instance, if fire lines breached an established boundary, most agents persisted in their outdated trajectorics, which often resulted in them being destroyed by the new danger.
5.4 Outlook
Our study reveals that while existing multi-agent LLM frameworks demonstrate promising behaviors simple, structured tasks, they struggle to generalize to complex, large-scale environments that demand dynamic coordination, role specialization, hierarchy, and real-time adaptation. These findings point to several critical directions for future research:
Scalable Architectures and Efficient Algorithms: More scalable communication architectures are needed to manage information flow as agent populations ions grow, such as hierarchical or attention-based routing. Our benchmark reveals efficiency challenges in current algorithms when scaled to complex tasks, inspiring algorithm development that prioritizes both scalability and computational efficiency.
Adaptíve Planning and Reasoning: Effective abstraction and modularity in task planning may enable agents to better decompose and assign sub-goals in evolving scenarios. Future systems should integrate uncertainly-aware reasoning and adaptive leadership to respond flexibly to noisy or incomplete observations.
Evaluation and Auman-AI Teaming: Developing systematic methods to measure communication efficiency, redundancy, and coordination-spocifie metrics for largo-scale multi-agent problems remains an open challenge. We will open-source all recorded observations, actions, and agent communications to enable deeper trajectory analysis. Additionally, while our environment supports low-level human control, studying human performance roquires investigating complex human teaming dynamics with 10+ agents in large-scale dynamic tasks, which we plan to explore in future work.
6 Limitations
We note several aspects of CREW-WILDEES that can be further improved. First, despite heavy referencing to real-world heterogeneous setups of multi-agent collaborations and coordination applications, our abstraction of agent roles is not yet one-to-one mapping of real-world wildfire complexity and agent ability, potentially limiting direct generalizability and deployment in real-world wildfire responses. The low-level control is also abstracted, excluding the real-world physics and actuator simulation in robotics simulators. We intentionally simplify these aspects for our initial developments to focus on the planning components. This design choice also allows controlled and repcatable benchmark efforts targoting current LLM-based Agentic frameworks. CREW-WILDFII still remains the state-of-the-arl scale and complexity in LLM-based multi-agent Agentic benchmarks. We plan to incorporate real-world embodied agents into CREW-WILDFIR in our future iterations.
Second, despile the large number of experiments across multiple random seeds, baselines, and task levels, more runs on different prompt variations can provide further insights on the subtle but important system designs. Our current experiments are limited by the relatively high costs associated with LLM token usage. For example, in algorithms where agents are given global-state vector (the combination of each agent's perceptions), the input token cost. for that step scales on the order of N2. With the cost of L.M-qqery
12

---

Published in Transactions on Machine Learning Research (12/2025)
becoming more affordable, CREW-WILDFIRE will provide promising evaluation platform to study more detailed algorithm designs.
Moreover, the above limitations are not a result of the CREW-WILDDIIEE as a benchmark, but rather the current algorithmic token inefficinccie that limit the extent of realistic scalability testing. The lack of baseline data with the full capabilities of CREW-Wildfre [2, 0000 agents and 1, 000, 000+ cell map size] due to the infeasibility resulting from the existing algorithm constraints, as indicated by our extensive quantitative and qualitative experiments. We include computational performance data up to 2000 agents and 1 million cells in Appendix A.16
7
Conclusion
We introduced CREW-WILDFIRE robustness of multi-agent systems powered by LLMs. Through procedurally generated environments including heterogeneous agents, partial observability, and long- ong-horizon coordination, the benchmark exposes the limitations of current. frameworks in handling complex, real-time, and high-agent-count tasks. Our experiments reveal critical performance gaps role assignment, communication efficiency, and adaptive planning, While there are limitations in our scope and baseline data, by releasing the environment, sub-lasks, and baseline implementations, we aim provide a foundation for the community to develop and compare next-generation Agentic systems. While the deployment of autonomous LLM agents presents positive opportunitics for decision-making efficiency, it also carries significant ethical risks, necessitating robust frameworks and monitoring via open-source oversight to prevent misuse and ensure accountability.
open-source benchmark designed to evaluate the scalability and
13

---

Published in Transactions on Machine Learning Research (12/2025)
References
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Copalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Tbarz, Brian Tchter, Alex Trpan, Fric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do can, not as i say: Grounding language in robotic affordances, 2022. URL https://arxiv: org/ abs//204..
Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch, Marc Lanctot, H Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, et al. The hanabi challenge: A new frontier for ai rescarch. Artificial Intelligence, 280:103216, 2020.
Lucian Buşoniu, Robert Babuška, and Bart De Schutter. Multi-agent reinforcement learning; An overview. Innovations in multi-agent systems and applications-1, pp. 183 283-221, 2010.
Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-ai coordination. Advances in neural information processing systems, 32, 2019.
Matthew Chang, Gunjan Chhablani, Alexander Clegg, Mikael Dallaire Cote, Ruta Desai, Michal Hlavac, Vladimir Karashchuk, Jacob Krantz, Roozbeh Mottaghi, Priyam Parashar, Siddharth Patki, Ishita Prasad, Xavier Puig, Akshara Rai, Ram Ramrakhya, Daniel Tran, Joanne Truong, John M. Turner, Eric Under- sander, and Tsung- Yen Yang. Partor: A benchmark for planning and reasoning in embodied multi-agent. tasks, 2024. URL
Boyuan Chen, Shuran Song, Hod Lipson, and Carl Vondrick. Visual hide and seek. In Artificial Life Conference Proceedings 32, pp. 645-655. MIT Press One Rogers Street, Cambridge, MA 02142-1209, USA journals-info ...., 2020.
Boyuan Chen, Yuhang Hu, Robert Kwiatkowski, Shuran Song, and Hod Lipson. Visual perspective taking for opponent behavior modeling. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 13678 13685. IEEE, 2021.
Junzhe Chen, Xuming Hu, Shuodi Liu, Shiyu Huang, Wei-Wei Tu, Zhaofeng He, and Lijie Wen. Llmarena: Assossing capabilities of large language models dynamic multi-agent environments, 2024a. URL
Yongchao Chen, Jacob Arkin, Yang Zhang, Nicholas Roy, and Chuchu Fan. Scalable multi-robot collaboration with large language models: Centralized or decentralized systems? In 2024 IEEE International Conference Robotics and Automotion (ICRA), pp. 4311 4317. IEEE, 2024b.
Yuanpei Chen, Tianhao Wu, Shengjie Wang, Xidong Feng, Jiechuan Jiang, Zongqing Lu, Stephen McAleer, Hao Dong, Long-Chm Zhu, and Yaodong Yang. Towards human-level bimanual dexterous manipulation with reinforcement learning. Advances in Neural Information Processing Systems, 35:5150-5166 2022.
Daniel S Drew. Multi-agent systems for search and rescue applications. Current Robotics Reports, 2:189-20 200, 2021.
Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Information Processing Systems, 35:18343 18362, 2022.
Jakob Focrster, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. Advances in neural information processing systems, 29, 2016.
14

---

Published in Transactions on Machine Learning Research (12/2025)
Ran Gong, Qiuyuan Huang, Xiaojian Ma, Yusuke Noda, Zane Durante, Zilong Zheng, Demetri Terzopoulos, Li Fei-Fei, Jianfeng Gao, and Hoi Vo. Mindagent: Emergent gaming interaction. In Findings of the Association for Computational Linguistics: NAACL 2024, pp. 3154 3183, 2024.
Xudong Guo, Kaixuan Huang, Jiale Liu, Wenhui Fan, Natalia Vélez, Qingyun Wu, Huazheng Wang, Thomas L. Griffiths, and Mengdi Wang. Embodied LLM agents learn to cooperate in organized teams. In Language Gamification NeurlPS 2024 Workshop, 2024. URL
Yixin Huang, Shufan Wu, Zhongcheng Mu, Xiangyu Long, Sunhao Chu, and Guohong Zhao. A multi-agent reinforcement learning method for swarm robots in space collaborative exploration. In 2020 6th international conference on control, aulomation and robolics (ICCAR), pp. 139-144. TEEE, 2020.
Zool Hilmi Ismail, Nohaidda Sariff, and E. Gorrostieta Hurtado. A survey and analysis of cooperative multi-agent robot systems: challenges and directions. Applications of Mobile Robots, 5:8-14, 2018.
Zhengran Ji, Lingyu Zhang, Paul Sajda, and Boyuan Chen. Enabling multi-robot collaboration from single-human guidance. In 2025 IEEE International Conference on Robolics and Aulomation (ICRA), pp. IEEE, 2025.
Zhao Kaiya, Michelangelo Naim, Jovana Kondic, Manuel Cortes, Jiaxin Ge, Shuying Luo, Guangyu Robert Yang, and Andrew Ahn. Lyfe agents: Generative agents for low-cost real-time social interactions. arXiv preprint arXiv:2310.00 02172, 2023.
Gautam Siddharth Kashyap, Deepkashi Mahajan, Orchid Chetia Phukan, Ankit Kumar, Alexander El Brownlee, and Jicchao Gao. From simulations to rcality: enhancing multi-robot exploration for urban search and rescue. Internalional Journal of Information Technology, pp. 1 1-12, 2025.
Woojun Kim, Jongeui Park, and Youngchul Sung. Communication in multi-agent reinforcement learning: Intention sharing. In International conference learning representations, 2020.
Aleksandar Krnjaic, Raul D Steleac, Jonathan D Thomas, Georgios Papoudakis, Lukas Schäfer, Andrew Wing Keung To, Kuan-Ho Lao, Mural Cubuklepe, Matthew Haley, Peter Börsting, et al. Scalable multi- agent reinforcement learning for warehouse logistics with robotic and human co-workers. In 2024 IEEE/RS) International Conference on Intelligent Robots and Systems (IROS), pp. 677 684. IEEE, 2024.
Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for' mind" exploration of large language model society. Advances in Neural Information Processing Systems, 36:51991-52008, 2023.
Afshin OroojlooyJaadid and Davood Hajinozhad. A review of cooperative multi-agent deep reinforcement learning, 2021. URL https://arxiv.. wwtrexivvv.rr////o httgg/bb//9900// 09966
Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pp. 22, 2023.
Javier Andreu Perez, Fani Deligianni, Daniele Ravi, Guang- Yang, et al. Artificial intelligence and robotics. arXiv preprint arXiv: 1003.1083, 147:2 44, 2018.
Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6494-8502, 2018.
Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, et al. Chatdev: Communcativee agents for software development. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Popers), pp. 15174 15186, 2024.
15

---

Published in Transactions on Machine Learning Research (12/2025)
Yinuo Qin, Richard T Lee, and Paul Sajda. Perception of an ai teammate in an embodied control task affects team performance, reflected in human teammates' behaviors and physiological responses. arXiv preprinl arXiv:2501.15392 2025a.
Yinuo Qin, Richard T. Lee, Weijia Zhang, Xiaoxiao Sun, and Paul Sajda. Physiologically informed pre- dictability of a teammate's future actions forecasts team performance. &Science, 28(5), May 2025b. ISSN 2589-0042. doi: 10.1016/j..sci.22255 200...202..1110 112429. URL
202.: 112429
Esmacil Seraj, Xiyang Wu, and Matthew Gombolay. Firccommander: An prtptttrrrrtiivveebbbobobbiiiaic multi-agent environment for heterogeneous robot leams, 2021. URL /ttpp////axxx..ooorr//////0....0
Philipp Dominic Siedler. Hivex: A high-impact environment suite for multi-agent. research (extended version). arXiv preprint wcXic:2501004180, 2025.
Jordan Terry, Benjamin Black, Nathaniel Grammel, Mario Jayakumar, Ananth Hari, Ryan Sullivan, Luis S Santos, Clemens Dieffendahl, Caroline Horsch, Rodrigo Perez-Vicente, et al. Pettingzoo: Gym for multi- agent reinforcement learning. Advances in Neural Information Processing Systems, 34:15032 15043, 2021.
Oriol Vinyals, Timo Ewalds, Sorgey Bartunov, Petko Georgiev, Alexander Sasha Vozhnevets, Michelle Yco, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schrittwieser, et al. Starcraft ii: A new challenge for reinforcement learning. arXiv preprint arXiv: 170. 04 782, 2017.
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. Transactions Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreevie: wttp:e/wwwiieeeeruunnnuiiii net/forum?id= ehfRiFOR3a
Pengying Wu, Yao Mu, Kangjie Zhou, Ji Ma, Junting Chen, and Chang Liu. Camon: Cooperative agents for multi-object. navigation with 1lm-based conversations. arXiv preprinl Xrx::22077.0000 2024.
Bin Zhang, Hangyu Mao, Jingqing Ruan, Ying Wen, Yang Li, Shao Zhang, Zhiwei Xu, Dapeng Li, Ziyue Tú, Rui Zhao, Lijuan Li, and Guoliang Fan. Controlling large language model-based agents for large-scale dacision-making: An actor-critic approach, 2024a. URL https: / org/ 2311. 13884.
Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B. Tenenbaum, Tianmin Shu, and Chuang Gan. Building cooperative embodied agents modularly with large language models. In The Twelfth International Conference on Leurning Representations, 2024b. URL /tttps::/ net/ Jorum?id=ExXJffqy00
Kaiqing Zhang, Zhuoran Yang, Han Liu, Tong Zhang, and Tamer Basar. Fully decentralized multi-agent reinforcement learning with networked agents. In International conference on machine learning, pp. 5872 5881 PMLR, 2018
Lingyu Zhang, Zhengran Ji, and Boyuan Chen. CREW: Pacilitating human-Al teaming research. Transactions on Machine Learning Research, 2024c. ISSN 2835-8856. URL https://openreviies www.:eeeffoffai. o
2RX
Lingyu Zhang, Zhengran Ji, Nicholas R Waytowich, and Boyuan Chen. GUIDE: Real-time human-shaped agents. Advances in Neurol Information Processing Systems, 38, 2024d.
16

---

Published in Transactions on Machine Learning Research (12/2025)
Appendix
A.1 Map Generation and Evaluation Process
Map Data
Environments
seed
Perlin Noise
Ground Type
Elevation
seed
Moisture
seed
level
Settlement
size
Level Generator
Wind
seed
CREW Framework
Agent Data
Agents
Game States
Task Evaluator
< U1 U2
Task
< U1 U2
Stask (Vo, U1, U1,...)
Score
[h]
Figure 8: Map generation and Evaluation process. Given the chosen level and level size, the Level Generator creates 5 distinct Perlin Noise textures for each generation seed: one for Ground Type, Flevation, Moisture, Settlement, and Wind X + Y vectors. After scaling or clamping them into appropriate maps, they are put together as the generated environments. The Task Evaluator then repeatedly checks their state vectors to return score depending on task type.
17

---

Published in Transactions on Machine Learning Research (12/2025)
A.2 Perception Module
The following prompt summarizes the ASCII represenlation.
Prompt for PERCEPTION Module
You are AGENT X, and your current location is POSITION, and thus your minimap view will be the range X [mo æ1), Y [yo - yıl, where #i = x rango/2 and #1 = x + range/2 (and similarly for y
coordinates), centering the minimap on the agent's 's location, with the top corner of the map being (0,0).
This is your minimap view:
[ASCII ENCODED MINIMAP]
Each cell is represented by a character corresponding to the type of terrain:
O: brush (no trees)
1: light forest (1 tree)
2: medium forest (2 trees)
3: dense forest (3 trees)
i: Ignited
f: On Fire
e: Extinguishing
x: Fully Extinguished
Water Source Cell (no trees)
B: building (no trees)
IGNORE ALL 9.", Those are unrevealed cells. They will reveal themselves when you get closer to them.
The cells in single quotations are wet cells. CC' cells are civilians.
The bolded cell is the current cell you are in. is a X cell at POSITION. There are other nearby agents at:
[OTHER AGENT LOCATIONS]
Your job is to process and understand your surroundings. Do not directly report explicit information from the minimap, but rather spatially understand your surroundings. Do not refer to character representations of the minimap, only what they actually represent. Report general obscrvations general direct Also report if there specific cells of interest, such fires, civilians, water, etc. If there are any, calculate their exact locations by explicitly counting cells. You should return detailed but concise text summary paragraph of all relevant information, including location, surroundings, and presence of important cells.
18

---

Published in Transactions on Machine Learning Research (12/2025)
1. For example, the following observations input:
[''oocaion': (33,93),
"trees": 2, "civilians": false, "status": burnable),
{ (llocation: (34,93),
"trees": 3, "civilians": false, "status": burnable),
{'location': (35,93),
"trees": 3, "civilians"; false, "status": burnable),
(397 more items)]
2. Is transformed into:
2:333333333222220333333
2, 3 2 2e2.,22.....222222,,,,2,,,.,,.,.
2, 2, 2223532222222222222222222.......
2, 223322222222222222333322.
222252222222111222225553333
2.222222222111:1122222222.
2223222211111:11122222222
02222a2211111111112223333...
00222i11111111111111111
[4221111111111111111111111
022222111111111111111111111
3,222211111112211111111111
1,1111111110111100000000000
011111111000000000000000000000.0
01111110000000000000000000000000
01111111100000000000000000000000
01111111110000000000000000000000
01111011e11111110000000000001111
1,1
Which summarized into:
I currently positioned at (43, 103) within light forest area. My immediate surroundings are predominantly light forest, with a mix of medium and dense forest areas the north and northwest. To the north and northwest, there patches of medium and dense forest, indicating polentially challenging terrain for movement and visibility. The southern and southeastern areas transition into brush and open spaces. Significantly, there are civilians present in the southern region of my view. One civilian is located at (43, 110), and three more are at (45, 110), (46, 109), and (48, 111). These civilians are in an area transitioning from light forest brush. There are no immediate signs of fire or ignited areas within my visible range.
19

---

Published in Transactions on Machine Learning Research (12/2025)
A.3 Primitives
Primitive functions serve simplified way to control our agents' low-level behavior within the environment. Although it is possible to control each agent's 's movement rotation manually at each step, these primitives allow agents to focus on high-lovel planning instcad of struggling with tedious low-level control. This is especially useful, since LLMs are proven to struggle with low-level control.
For example, the primitive 'move location' may involve multi-step sequence of movement and rotation actions to move the agent to the desired location.
Here is the list of all primitive methods agents can use within the enviromment:
Table 5: Primitives + Actions for All Agent Types
Primitive Name
Paramcters
Description
Firefighter Primitives
Move Location
<1, x coord, y coord>
Continually move in the direction of the loca- tion until it reaches there
Cut X Trees in current cell
2, of trees, 0>
Cut single tree X times Cut single tree until there are 0 trees left Pick up closest civilian
Cut All Trees in current cell
<3, 0, 0>
Pick Up Civilian
<4, 0, 0>
Drop Off Civilian
<5, 0, 0>
Unload carried civilian to current coll
Spray Water Cone toward target
<6, X coord, y coord>
Aim at target, spray water a wide cone Refill water supply if over a water source
Refill Water
<7, 0, 0>
Bulldozer Primitives
Drive to Location (No Cut)
Continually move the direction of the loca- <, x coord, y coord>
tion without the plow lowered until it reaches there
Drive to Location (Clear Path)
2, x coord, coord>
Continually move in the direction of the lo- cation with the plow lowered until it reaches there
Drone Primitives
Fly to Location
<, x coord, y coord>
Continually fly in the direction of the location until it. reaches there
Helicopter Primitives
Fly to Location
<1, x coord, y coord>
Continually fly in the direction of the location until it reaches there Load nearby firefighter agents Unload all carried firefighters to current cell Replenish water tank if over a water source Deploy one water payload at current location
Pick Up Firefighters
<2, 0, 0>
Drop Off Firefighters
<3, 0, 0>
Refill Water
<4, 0, 0>
Drop Water
<5, 0, 0>
20

---

Published in Transactions on Machine Learning Research (12/2025)
A.4 Execution Module
Prompt for EXECUTION Module
You are the controller of highly trained agent within a grid forest world. Your job is to convert a single text action into structured format for robotic control.
Here the action we want to perform: [ACTION]
Your job is to convert the action into executable format. Do not change the actions, just translate them.
This is the executable action format:
Actionf
int "type": type of action being performed
int "param 1": parameter 1 of action if applicable int "param 2": parameter 2 of action if applicable string "description". description of action
You have X distinct types of actions. You MUST choose one of them:
1. Move to any coordinate location in one step regardless of distance:
"type": 1
"param 1": x coordinate of location
"param 2": coordinate of location
"description": description of action
Example Action: [1, 500, 500, "move to coordinate location of (500, 500)"]
[OTHER PRIMITIVES]
21

---

Published in Transactions on Machine Learning Research (12/2025)
22

---

Published in Transactions on Machine Learning Research (12/2025)
A.5 Prompts of CAMON Generate Plan
Prompt for CAMON: Generate Plan
You are AGENT X, an AGENT-TYPE Agent, currently acting as the leader in a cooperative multi-agent robotic task. This your leam composition, (including yourself):
TEAM COMPOSITION
Your team's 's current task is:
CURRENT TASK
Your past actions were:
PAST ACTIONS
This is your chat history with agents in your team:
CHAT HISTORY
This is your team's (including you) collective observations, locations, current actions, and past actions of all agents.
GLOBAL DATA
Now your job is to provide the next best action for yourself, and OPTIONALLY: the next best action for any other agents. Remember, you are AGENT X an AGENT-TYPE. Agent, located at POSITION.
These are all the possible actions for each type of agent. This is comprehensive list, so the action MUST be one of these types. NO other responses allowed.
LIST OF AGENT ABILITIES ON TEAM
Provide your output in the following format:
<ae reasoning or calculationx</rrasnnnn>>
<action> 'MY NEXT ACTION"</action
OPTIONAL-for other agents:
AGENT ID-action>(AGENT ID'S NEXT <CTION ID-action>>
AGENT ID-message> (message AGENT ID)<AGENT ID-message>>
For example:
AGENT Daction>'actions AAGENT A-action>>
AGENT A-message>> 'acctin'' <AGENT A-message>
23

---

Published in Transactions on Machine Learning Research (12/2025)
A.6 Prompts of CAMON Propose Plan
Prompt for CAMON: Propose Plan
You are AGENT X, an embodied AGENT-TYPE agent within a X by forest grid world and part of a collaborative leam of Agents.
This is your team's composition (including yourself):
TEAM COMPOSITION
These are your current observations:
PERCEPTION
This is your team's overall task:
CURRENT TASK
Your past actions were:
ACTION HISTORY
This is your chat history with agents in your team:
CHAT HISTORY
Your job is to propose your next action. These are your possible actions:
AGENT ABILITIES
This comprehensive list, so your action MUST be one of these types. NO other responses are allowed.
Provide your output in the following format:
Srasoning>>ann reasoning or caleulationx<<<cassnnn>>> <action>> ''Y NEXT ACTION' </action>
24

---

Published in Transactions on Machine Learning Research (12/2025)
A.7 Prompts of CAMON Review Plan
Prompt for CAMON: Review Plan
You are AGENT X, currently acting as the leader in a cooperative multi-agent robotic task.
This is your team composition (including yourself);
TEAM COMPOSITION
Your team's 's current task is:
CURRENT TASK
This is your teams((includinn you) collective observations, locations, current actions, and past actions of all agents. Only you have all of this data.
GLOBAL DATA
Your teammate AGENT Y, AGENT-TYPE Agent, is proposing new action for itself:
PROPOSED ACTION
Your job is review this action and ACCEPT or REJECT it.
Then provide the next best action for AGENT Y, choosing better one if REJECT repeat- ing/rewriting the proposed one if ACCEPT. Also send message to AGENT Y describing your choice.
Additionally, you may announce information to other agents in your team with information. You may also choose to override actions for other agents as well. You must send a message to that agent if you do so. This interrupts their action, so only do this you want to change their current action.
These all the possible actions for each type of agent. This is comprehensive list, the action MUST be one of these types. NO other responses are allowed.
LIST OF AGENT ABILITIES ON TEAM
Provide your output in the following format: <reasoning> reasoning or aalculatiooss<crraooonnngg <ecisioon> ACCEPT OR REJECT </decision>> <action> AGENT Y's next action < /ction
<message>> message to AGENT Y< message>
OPTIONAL-for other agents:
<AGENT ID-action> NEXT ACTIO ID-action> (messsae AGENT ID-message>> (message AGENTID) ID-message>
For example: <AGENT A-action> acction </AGENT A-action>>
25

---

Published in Transactions on Machine Learning Research (12/2025)
A.8 Prompts of COELA: Propose Message
Prompt for COELA: Propose Message
You are the communicator module of Agent X, AGENT-TYPE Agent in a cooperative multi-agent robotic lask.
This is your team composition, including you:
TEAM COMPOSITION
Your team's 's task is: CURRENT TASK
Your status and observations:
PERCEPTION
Your chat history:
CHAT HISTORY
Your past actions:
ACTION HISTORY
Your job is to propose a message to send to the chat haat/roupphht.
Provide your output in the following format:
<reasoning>(ary reasoning or calculatiouss<<rrasnnnn>>>
<message>> MESSAGEE
Note: The generated message should be accurate, helpful, and brief. Do not generate repetitive messages
26

---

Published in Transactions on Machine Learning Research (12/2025)
A.9 Prompts of COELA: Choose Action
Prompt for COELA: Choose Action
You are Agent X, an AGENT-TYPE Agent in a cooperative multi-agent robotic task.
This is your team composition, including you:
TEAM COMPOSITION
Your team's 's task is: CURRENT TASK
Your status and observations:
PERCEPTION
Your chat history:
CHAT HISTORY
Your past actions:
ACTION HISTORY
Now your job is to provide the next best action for yourself. Remember, you are Agent X an AGENT-TYPF. Agent, located at POSITION.
These are all the possible actions for each lype of agent. This is a comprehensive list, so the action MUST be one of these types. NO other responses are allowed. Note that sending messages has cost think about the necessity of it.
[send message to groupchat] PPROPOSED MESSAGE OTHER ABILITIES
Provide your output in the following format:
<reasoning>(auy reasoning or caleulations)</rreonnnn>>>
<action>> 'MY NEXT ACTION' </actin>
Include 'SEND MESSAGE in all caps like so, if and only if your action is to send the message. For example:
<acDion>SEND MESSAGE pproposed message' < /action>>
27

---

Published in Transactions on Machine Learning Research (12/2025)
A.10 Prompts of Embodied: Generate Communications
Prompt for Embodied: Generate Communications
You are AGENT X, a AGENT-TYPE Agent in a cooperative multi-agent robotic task.
Given your shared goal, chat history, and your progress and previous actions, please generate list of short messages to members of your team in order to achieve the goal possible.
This is your team composition, including you:
TEAM COMPOSITION
Your team's task is: CURRENT TASK
Your status and observations:
PERCEPTION
Your past actions:
ACTION HISTORY
Your chats:
CHAT HISTORY
You may send messages to individual agents or a global channel. Think about the necessity of sending message. There are costs send messages. Provide your output in the following format. All names should be in all caps:
<reasoning>an (any reasoning calculationss<<reasnnnngg
RRCCPPENTT-MESSAE//EEEPRREE M MESSAGGF*/GLOOBATT
For Example:
AGENT memssgee< AGENT A
AGENT C>message</ AGENT C>
GLOBAL> GLOBAL>
28

---

Published in Transactions on Machine Learning Research (12/2025)
A.11 Prompts of Embodied: Generate Actions
Prompt for Embodied: Generate Action
You are AGENT X, an AGENT-TYPE Agent in cooperative multi-agent robotic task.
Your team's task is:
CURRENT TASK
Your status and obsorvations:
PERCEPTION
Your chat history:
CHAT HISTORY
Your past actions:
ACTION HISTORY
Now your job is to provide the next best action for yourself. Remember, you are AGENT X an AGENT-TYPE Agent, located POSITION.
These all the possible actions for each type of agent. This is comprehensive list, the action MUST be ONE and only ONE of these types. NO other responses are allowed.
AGENT ABILITIES
Provide your output in the following format:
<reasoning>(any reasoning caleulations)</cccaonnng>
<action> 'MY NEXT ACTION"</action
Make sure you include enough dotails in your action such as explicit target coordinate locations. For example:
<action> towards (500,500)</action)))
29

---

Published in Transactions on Machine Learning Research (12/2025)
A.12 Prompts of HMAS-2: Central Planner
Prompt for HMAS-2: Central Planner
You are central planner directing agents in a cooperative multi-agent robotic task.
Your team's task is:
CURRENT TASK
Your team's provious state action pairs at each step are:
STEP HISTORY
Your team's current state and available actions are:
GLOBAL STATE
Now your job is to provide the next best action for each agent. You must provide single action for each agent. These actions must be exactly ONE of the agent's available actions, including the 'do nothing' action. Do not propose multiple actions per agent.
Specify your action plan in the following format with agent names in all caps:
<rcasoning>(any reasoning or calculationsx//rassnnng>>
<AGENT> 'MY NEXT ACTION*/AGENTT>
For example:
AGENT A> 'acttoon</AGENN A>
AGENT B> 'action B>
Make you include enough details in cach action such explicit target coordinate locations.
30

---

Published in Transactions on Machine Learning Research (12/2025)
A.13 Prompts of HMAS-2: Feedback
Prompt for HMAS-2: Feedback
You are AGENT X, an AGENT-TYPE Agent in cooperative multi-agent robotic task.
Your team's task is:
CURRENT TASK
Your team's provious state action pairs at each step are:
STEP HISTORY
Your team's current state and available actions are:
GLOBAL STATE
The initial action plan from the central planner is:
INITIAL PLAN
Now your job to provide feedback to the action plan specifically regarding your agent. the plan is satisfactory, the feedback should only be 'ACCEPT'
Remember, you AGENT X AGENT-TYPE Agent, located at POSITION.
<reasoning> (any reasoning or caleulationss</rrssonini
<foodback> 'Foddback's < boaiockk'/foddaatei /feodbbcck>
31

---

Published in Transactions on Machine Learning Research (12/2025)
A.14 Baseline Pseudo Codes
A.14.1 CAMON Algorithm
Algorithm 1: CAMON Implementation
Data: Agent set A, environment env, time limit T max, target score MaxScore, task task Leader A[0]:
state env.reset();
for + to Tao do
GlobalData t f}:
/* Parsing Observations and Generating Perceptions
foreach a E A do
Og - (ettsserratiion,sstttt))
Pa LLM__ Luu__eeeeratePercpptioo,o (oa);
agentData, - (Pa: action,, Ha);
append agentData, to GlobalData:
/* Generating Plan
10
foreach E A
11
if action, * None then L continue;
12
13
if a= Leader then
14
plan + LLM MeMMGeeraaaePPlatttaaa (task, A, GlobalData, Ma);
15
else
16
proposal +- LLM PropPoooooPPPaattaaaa (task, Pa: Ha. Ma);
plan LLM Lew.LLereeeeesssse..eee. (prroopppaaa (proposal, task, Ma. GlobalData):
17
18
Leader
/+ Assigning tasks to agents forcach task, plan do
19
20
L action, + L.M.TTranslaaeekciiotttta..
E ():
/* Executing Actions foreach E A do
22
23
Ca + Exeeuueeaciion(cction.
24
if action is complete then
25
append action, to Ha; action, = None;
26
27
E[a]+ cai (state, score) + env.step(E))
28
29
if score 2 MamScore then
30
break;
In our CAMON Implementation, a leader agent initially chosen as shown in line 1. Then, for each timestep, each agent parses their observations(o.) from the environment states, and then generates text perception (pa) given those observations through the PERCEPTION module (A.2). This perception (p.) is combined with the agent's current action (action,) and its action history (Ho) to create agentData,, which is then appended to the ClobalData.
Then, each agent checks their current action (action.)). If it is still active (meaning it is multi-step and not. complete), the agent will do nothing, as seen in line 12.
If not, and the agent is the leader agent, will independently generate a new team plan through LLM_ GMMGeeerratt,,n, A.5 in line 14. If the agent is not the leader agent, instead it will propose
32

---

Published in Transactions on Machine Learning Research (12/2025)
plan through LLM LLM__roppee_ ProposePlan, (A.6), in line 16. This plan will then be reviewed by the leader agent through LLM www.eeeeelaa.uuueec
A.7 in line 17. Then the agent will become the new leader agent.
Next, the finalized plan will be parsed into tasks for each agent and translated into a standardized action format through LLM Tra.ssaatttttciinn, (A.4) in line 20. This action (action,) will be each agent's 's active action.
Now since all agents have an active action, each agent will generate an executable vector (ea) through our EXECUTION Module in line 23. If the current action (action.) is complete single-step, it will then be set to None and appended to the agent's 's action history.
Lastly, all executable tensors will be combined and executed within the Wildfire Enviromment in line 28. If the max score has been reached, the algorithm ends.
33

---

Published in Transactions on Machine Learning Research (12/2025)
A.14.2 COELA Algorithm
Algorithm 2: COELA Implementation
Data: Agent set A, environment env, time limit T maxx, largel MaxScore, task task state env.reset));
M (+ (}
for t 1+-10 to Tmar do
/* Parsing Observations and Generating Perceptions
foreach do
Og f (sttDserrattion,sstttt))
Pa huan_eeeerateeerrcppiio....
/* Proposing Message and Choosing Action
foreach A do
if action, * None then
9
continue;
10
ereposedMesssage LLM Propossseesssgee (task, Pa. Ha, M);
11
chosenAction t- LLM, ChooseAction, (task, proposedMessage, Pa: Ha. M);
12
if Alosen.Action = proposedMessaage then
13
append proposedMessagg to M
14
action, + NoAction
15
else action, + L.M.Transsaateecciiiccccoocaclcl
16
17
E + 0: /* Executing Actions forcach a do
18
19
ea < -Exeauteecciion/ccttonm)))
20
if action is complete then
21
append action, to Hai
22
action, None;
23
E(a] ta; Cai
(state, score) + env.step(E);
if score 2 MamScore then
26
break;
In our COELA Implementation, message history (M) is initially cmpty in line 2. Then, for each timestep, each agent parses their observations(o.) from the environment states, and then generates text perception (Pa) given those observations through the PERCEPTION module (A22).
Then, each agent checks their current action (action.). If it is still active (meaning it is multi-step and not complete), the agent will do nothing, as seen in line 9.
However, if not, cach agent proposes message send through LLM ProposeMessage, lask, the agent's perception (p.), the agent's 's action history (H.), and the leam's chat. history (M).
A.8 given the team's
Then, given the proposed message, through LLM ChooseAction, or to send the proposed message.
A.9 the agent chooses to execute an action
If the agent chooses to send the proposed message, then that message is appended to the message history (M) and No Action chosen in lines 13-14.
However, if the agent chooses different action, that action is translated into a standardized action format through LLM Transsateettiion, (A44 in line 16. This action (action,) will be each agent's active action.
34

---

Published in Transactions on Machine Learning Research (12/2025)
Now since all agents have active action (including No Action), each agent will generate an executable vector (ea) through our EXECUTION Module in line 19. Tf the current action (actiona) is complete or single-step, it will then be set to None and appended to the agent's action history.
Lastly, all executable tensors will be combined and executed within the Wildfire Environment in line 24. If the max score has been reached, the algorithm ends.
35

---

Published in Transactions on Machine Learning Research (12/2025)
A.14.3 Embodied Algorithm
Algorithm 3: Embodied Implementation
Data: Agent set A, environment, env, time limit, max, task task, communication rounds C state env.reset(); for t +--1 to mar do
/* Parsing Observations and Generating Perceptions
foreach a E E do
4
Og f GetObservvtiinn(u, state);
Pa LLM Get.eertteerrroppoooon.... 0g);
/* Communication Rounds
for c to C' do
forcach A do
8
messages, +- LLM__ [eneerrtteeeeesggeetta... Pa: Has Ma); foreach (recipient, msg) messages, do
9
10
append mag to Mai append mag to Mrecipienti
11
/* Generating Actions foreach E do
12
13
generatedAction LLM, Generateactioon Pa; Ha, Ma);
14
action, + LLM Tnw. oenareaeeeelttcooeeeeeerereellll
15
E + 0:
/* Executing Actions
16
foreach a E do
17
6a < +=xxeuteecctioo/(ctttoo)))
if action is complete then
19
append action, to Haj
20
action, None;
F(a] Gaj
Cai
22
(state, score) + env.step(actions)
23
if score 2 MamScore then
24
L break;
In our Embodied Implementation, start each timestep with all agents parsing their observations (o,) from the enviromment state in line 4, and then generating text perception (pa) given those observations through the PERCEPTION module (A.2) in line 5.
Then, for e communication rounds, each agent generates messages for any recipient agent through LLM_ Geeeretteessagees... (A.10) in line 8, given the team's task, the agent's perception, action history ((Ha), and message history (Ma). These messages are then parsed and added to the corresponding agents' message histories ((Mreesptenz)) in line 11.
Then after all communication rounds, each agent generates their next action with LLM_ GenerateMessages, A.11 in line 13, given the team's 's task, the agent's perception, action history (Ha), and message history (Ma). This generated action is then translated into a standardized action format through LLM Translateaction, A.4 in line 14. This action (action.) will be each agent's 's active action.
Now since all agents have an active action (including No Action), each agent will generate an executable vector (ea) through our EXECUTION Module in line 17. the current action (action,) complete singlo-step, it will then be set to None and appended to the agent's action history.
Lastly, all executable tensors will be combined and executed within the Wildfire Environment in line 22. If the max score has been reached, the algorithm ends.
36

---

Published in Transactions on Machine Learning Research (12/2025)
A.14.4 HMAS-2 Algorithm
Algorithm 4: HMAS Implementation
Data: Agent A, environment env, time limit T maxx task task
Leader A(0];
GlobalState +- (::
StepHistory +- ( state t env env.rese()):
for 1+1 to mar do
/* Parsing Observations and Generating Perceptions
foreach E do
Ou + GetObservation(a, state);
Pa LLM Let.eerrtttPorrpptooooo.a. 0z);
append to (lobalState(d]:
/* Iterating Plans
10
valid + False
11
while valid False do
12
valid +- True
13
plan < horn__GGeerattePaaaaaddrrrro (task, GlobalState, StepHistory, Review);
/* Reviewing Plan
14
Review = ()
15
foreach E A do
16
feedback, +- LLM LLM_LLM__W___ prrieeelaan,ppaaaa GlobalState, Step History); if feedback, is REJECT then
17
18
append feedback, to Review; valid + False;
19
20
foreach task, plan do
21
L action, +- LLM LLLM__TTaaaeet_ Traaaassslaaeeeccio((taa..))
/* Executing Actions actions + ();
22
23
foreach E A do
24 Ca <- ==xecuteecctiom(actioon
25
if action is complete then
26
append action, Hai
27
action, None;
28
F(a] <- Pa;
29
append (GlobalState, actions) to StepHistory;
30
(state, score) + www.step(actions))
if score 2 MamScore then
32
break;
In our HMAS-2 Implementation, leader agent initially is chosen shown in line 1. Then, for each timestep, each agent parses their obscrvations(oa) from the environment states, and then generates text perception (Pa) given those observations through the PERCEPTION module A.2 This perception(p.) is then appended to the GlobalState.
Then, the leader agent generates a plan for the team through LLM Www_GGnerrtePPaaaaadea team's task, the Global State, the Step History, and reviews from other agents in line 13. This will provide the leader agent with all agents' current perceptions and past, perceptions and actions.
A.12 given the
37

---

Published in Transactions on Machine Learning Research (12/2025)
Then each agent will review the plan and provide feedback through LLM ReMieeennn,,n, the leedback is to reject the plan, the feedback will be appended to Review and valid will be set to False. This will cause lines 12-19 to be repeated, reiterating the plan.
AA.13 in line 16. If
When the plan does not have any rejection, the finalized plan will be parsed into tasks for each agent and translated into standardized action format through LLM TranslateAction, (A.4) line 20. This action (action.) will be each agent's active action.
Now, since all agents have active action, each agent will generate an executable vector (ea) through our EXECUTION Module in line 23. If the current action (action,) is complete or single-step, it will then be set to None and appended to the agent's 's action history.
Lastly, all executable tensors will be combined and executed within the Wildfire Environment in line 28. If the max score has been reached, the algorithm ends.
38

---

Published in Transactions on Machine Learning Research (12/2025)
A.15 VLM Perception Module Ablation Study
We conducted an evaluation to assess the accuracy of our perception module versus the GPT4o VLM on 100 sample observations. Given the same game frames, the perception module received the associated minimap ASCII representation while the VLM received the image directly. Both were prompted by the same perception prompt (see Appendix A.2), with slight adjustments where the VLM was given a color key instead of an ASCTT key.
An unbiased JudgeLLM (GPT4o) judged both perception summaries against the ground truth data with respect to accuracy in locating features such as fires and civilians. The prompt for the Judge LLM is follows:
JudgeLLM Evaluation Prompt
Ground Truth Data (extracted from the environment): {ground truth json
Perception Text to Evaluate: "(perception_tens " Evaluate this perception text and provide:
1. Fire Detection Score (0-10): How accurately did it identify fires (ignited, on fire, extinguishing cells)?
Civilian Detection Score (0-10): How accurately did it identify civilians?
Respond using this format:
<fire _fire_score>Nuutee NUMBE>NUMBBRRRHH > score>
Jooe:-NUMBER<<dhvllaal > score> <explanation>>
Brief explanation of your scoring < </explanation>
VLM vs Parception Module Comparison
Accuracy Scores
Average Time per Sample
3,004
Arns tey
Dovelee Total
Deletee
Whr/Tee Comparison (Overall Accuracy)
10.0%
4714
20
16.1%
V. Krs
Figure 9: Behavior Competency Scores (BCS) by algorithm.
These figures demonstrate that the perception module is more accurate in our setting and does not significantly lag in terms of speed or cost.
39

---

Published in Transactions on Machine Learning Research (12/2025)
A.16 Computational Performance
Table 6: Resource usage with 20 agents across different map sizes.
Map Length (cells)
CPU Usage (%)
RAM Usage
250 (62,500)
27.2
206 MB
400 (160,000)
30.1
262 MB
600 (360,000)
25.5
353 MB
800 (640,000)
26.8
482 MB
1000 (1,000,000)
36.4
604 MB
Table 7: Resource usage with different agent counts on 100-size map.
Agent Count
CPU Usage (%)
RAM Usage
20 Agents
25.4
182 MB
100 Agents
27.5
266 MB
500 Agents
20.0
1034 MB
1000 Agents
21.6
1990 MB
2000 Agents
25.3
3835 MB
40

---

Published in Transactions on Machine Learning Research (12/2025)
A.17
Emergent behaviors observed in dialogues
Table 8: Examples each CREW-Wildfre Behavioral Goals
Behavioral Goal
Explanations
Examples of dialogue snippets that reflect such behaviors
Task Designation (TD)
Agents explicitly split tree- cutting targets so no effort overlaps.
We should divide the target locations...
AGENT + (11, 16) to cut trees.
AGENT - (13, 16) to cut trees.
AGENT - (11, 3) to cut trees.
Agent Capitalization (AC)
Firefighter delegates tree-line task to Bulldozer with superior cutting.
AGENT 1: "Since you have exceptional
tree-cutting abilities, could you focus on the line from (26, 6) to (26, 10)? AGENT 2 and will support you as needed."
AGENT 3: "Confirmed, will focus on cutting trees from (26, 6) (26, 10). Cur- rently moving towards the target area."
Spatial Reasoning (SR)
Uses relative offsets to locate and frame the fire zone spa- tially.
"AGENT 3 at (130, 200). Fire roughly 12 11 S. Requesting confirmation assis- tance."
Observation Sharing (OS)
Broadcasts civilian position and hazard stalus for rescue action.
"AGENT 5: Four civilians at (230, 124). No fires. Request Firefighters to escort to safe zone."
Realtime Coordination (RC)
Synchronizes boarding with he- licopter arrival.
AGENT 1: "All agents, prepare for transport by AGENT 7. We will move towards the target location [92.0, 11.0)"
AGENT 5: "Ready for transport.
AGENT 7, please initiate pick-up and transport operation planned"
AGENT 7: "Initiating pick-up and transport operation"
AGENT 7: "Confirming all Firelighter Agents are on board. Departing now"
Plan Adaptation (PA)
Detects new fire spread; re- quests aerial support, altering strategy.
"AGENT 12: Fire spreading north (238,
198). Need acrial backup from Helicopter 13 ASAP."
Objective Prioritization (OP)
Balances search mission with stand-by evacuation planning.
"AGENT now searching for civilians ahead of fire at (520, 565). Helicopter 10, stay ready to assist evacuation."
41

---

Published in Transactions on Machine Learning Research (12/2025)
A.17.1
Example Calculation of BCS for CAMON (algorithm) and Realtime Communication (behavioral goal)
Level
Raw score 8a,8
Target Tp
Baseline B
NSa.t
Transport FF (S)
6.00
6
0
1.000
Transport FF (L)
10.00
12
0
0.833
Rescue S+R R T
0.00
10
0
0.000
Suppress L+T+3
-729.67
0
-929.67
0.281
Full Environment
-5571.67
0
5722266
0.038
1.000 + 0.833 + 0.000 + 0.281 + 0.038
BCSCAMON,RC
0.430.
5
A.18 Behavior Competency Scores (BCS) Radar Chart
#ehavioral Competency Score (BCS)
CAMON
AC
COELA
Embodied
HMAS_2 2
1.0
0.8
0.6
0.4
0.2
05
TD
RC
PA
Figure 10: Behavior Competency Scores (BCS) by algorithm.
42

---

Published in Transactions on Machine Learning Research (12/2025)
A.19 Random Seeds and Other Hyperparameters
For all the LLMs, we set the lemperalure to 0 to achieve deterministic results and single completion per decision step. All experiments used GPT-4o (model ID: gpt-4o-2024-08-00 with context window of 128,000 tokens.
Baseline-specific hyperparameters were set as follows:
Embodied Implementation:
Communication Rounds per Timestop:
Message Lifespan: timesteps
COELA:
Max Messages: 30 messages per chat history
Note: COELA uses maximum message count while Embodied uses message lifespan because COELA maintains a single continuous chat while Embodied uses independent chats that are synchronized.
The generation seeds for the tasks are listed in Tab.
Table 9: Seeds Used Across All Levels
Level
Seeds Used
Cut Trees: Sparse (Small)
375, 483, 43, 6370, 9964, 2097, 25808, 83248, 48320, 94510 212, 981, 1530, 5382, 9405
Cut Trees: Sparse (Targe)
Cut Trees: Lines (Small)
9259, 4881, 8456, 59497, 66768, 78914
Cut Trees: Lines (Large)
820, 5406, 6503
Scout Fire (Small)
4651, 6841, 7593, 1012, 8528, 29751, 36346, 42589, 43846, 62563
Scout Fire (Large)
5324, 3603, 8592, 43126, 70576
Transport Firefighters (Small)
283, 2461, 2478, 7622, 7647
Transport Firefighters (Large)
741, 7305, 9528, 8079, 6232
Rescue Civilians: Known Location (Small)
9502, 3972, 6545, 5884, 8491, 7723, 30743, 51358
Rescue Civilians: Known Location (Large)
7979, 1539, 2269, 7152, 5226
Rescue Civilians: Search and Rescue
966, 7377, 7285
Rescue Civilians: Search + Rescue + Transport
8208, 150, 2577
Suppress Fire: Extinguish
2994, 4936, 4847
Suppress Fire: Contain
733, 7765, 8049
Suppress Fire: Locate and Suppress
5280, 2142, 2628
Suppress Fire: Locate + Transport + Suppress
6309, 3821, 6117
Full Environment
6434, 9424, 9500
43


Reviews:
[{"review_id": "rcyzyUJTp8", "reviewer": "Reviewer_vdyu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Authors introduce an open-source benchmark built atop the CREW simulation platform, designed to evaluate scalability, robustness, and coordination capabilities in complex, dynamic environments allowing for the procedural generation of response scenarios (Use case chosen is wildfire fighting). \nAmong its features  are terrain map generation and support for different types of agents (4 introduced).\nThe paper claim is to have built a benchmark specifically designed to evaluate Agentic multi-agent LLM systems under conditions of real-world scale and complexity. The main benchmark features are the support for: \n      - Heterogeneous agents (4 types introduced, 2000+ handled),\n      - Complex and dynamic terrain (1 million+ cell maps),\n      - LLM-compatible multi-agent framework, \n      - Perception and Execution modules that enable language-based agents to interpret sensory information and generate executable actions,\n      - Realistic objectives under: \n              o Partial observability,\n              o Stochastic environments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "In the evaluation use case (Wildfire fighting simulation) fire dynamics are represented through cellular terrain decomposition enhanced with slope, wind, vegetation type, moisture etc.\nThe challenge in such evaluations is to increase number and diversity of agents, complexity of environments that they interact in, with complex perception capabilities that could be transformed int textual communication elements.\nAn interesting start could be in repurposing of the current use case to fundamentally similar situations (health care management of contagious diseases for example) allowing to minimize context and environment setup time and cost and the reuse of trained models.\n\nThe limitations section is well laid and enumerates the various simplifications needed at this stage to evaluate the proposed benchmark but also traces paths to future improvements.", "audience_yn": "Yes", "audience_explanation": "This wok constitutes a good instantiation effort of the CREW platform introduced in a separate paper previously published in this venue.", "requested_changes": "Nevertheless, a few aspects need clarification:\n- In “3.2 Environment Design: Wildfire Simulation”:\n      o Q1: How does θslope scale this influence based on terrain steepness?\n      o Q2: While the wind vector w⃗ represents the wind speed and direction, what does the  ∆⃗x  direction vector magnitude represent\n      o “Moisture ratio attenuates the likelihood of ignition in wetter regions”:\n              Q3: Does this make it binary? \n              Q4: Any baring on propagation speed? \n- In “3.3 Agent Design”:\n      o “Observations support third-person agent image data”:\n              Q5: Care to clarify?\n- In “3.3.1 Perception and Execution Modules”\n      o “Given that our core environment requires precise, discrete and continuous control, we developed”:\n              Q6: Can you please detail how these 3 aspects (precision, discretization and continuous control) are required or present them selves in this benchmark?\n- In “3.4 Pillars of CREW-Wildfire Design”:\n      o “Task complexity” seems a misnomer:\n              Q7:  Shouldn’t the benchmarking suite pillar be adaptation to task complexity in the simulation scenario?\n- In “4 CREW-Wildfire Benchmarking Suite”:\n      o “Where an algorithm is conceptually challenged”:\n              Q8:  Care to clarify?\n- In “Procedurally Generated Task Levels”:\n      o “We define 12 procedurally generated levels (Fig. 7)”:\n              Q9: Is this the right reference or Table 2?\n              Q10: Are these 12 or 16 levels? (Same for Table 3)\n- In “3.4 Pillars of CREW-Wildfire Design”: \n      o “For heterogeneous agents, role heterogeneity demands coordination through complementary capabilities”:\n              Q11: As the various defined agents have a set of shared capabilities (‘Move forward etc.), How are those leveraged? \n- In “A.2 Perception Module”: \n      o “AGENT X, and your current location is POSITION, and thus your minimap view will be the range X : [x0 − x1], Y : [y0 − y1]”:\n              Q12: Aren’t minimap end coordinates [x0 – xi], Y : [y0 – yi], i instead of 1 where i depends on Agent’s minimap defined observation range?\n- In “5.3 Behavioral Analysis”: \n      o “Step 1: Level Normalization. The range of raw scores across different task levels is different due to different scoring functions. “:\n              Q13: Normalization is generally applied for comparison reasons of variables with different ranges/scales. When using different scoring functions, does this “normalization” (or mapping) sill yield interpretable results ? \n- In “3.4 Pillars of CREW-Wildfire Design”: \n      o “tackling the inherent challenges that a large and complex cooperative task, such as wildfires creates”. \n              Clarification: Isn’t the task rather “wildfire fighting”?", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2mjDztaJEE", "reviewer": "Reviewer_qfL5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces CREW-wildfire, a simulator for wildfire responses, as a multi-agent agentic (in the LLM-sense) environment. This environment requires detecting fires, searching for survivors, rescue, and managing fires. The system is made to be compatible with LLMs, by translating the environment details into language for use by language models to make decisions which can be translated into actions in the environment. This environment itself demands capabilities that we would want in our agents. These include large maps, heterogeneous agents, partial observability, stochastic dynamics, and long-horizon planning objectives, as stated in the paper. There are four kinds of heterogeneous  agents: firefighters, bulldozers, drones, and helicopters. It supposedly offers twelve levels (though table 2 and 9 appear to show 16 levels), that represent differents kinds of tasks in different forms. The paper tests several frameworks on these 12 (or 16?) tasks. It benchmarks their scores on all the tasks/levels and additionally produces a behavior-competency score to provide insight on the behavioral properties of different algorithms.", "claims_and_evidence_yn": "No", "claims_explanation": "I will begin with claims that are indeed validated.\n\n> A new open-source benchmark for evaluating LLM-based multi-agent Agentic systems in procedurally generated, physically grounded, and high-stakes disaster response environments.\n\nThe paper indeed does this.\n\n>CREW-Wildfire bridges these gaps by offering a fully open-source, embodied, and highly scalable environment centered on evaluating and benchmarking LLM-based multi-agent Agentic AI frameworks. It supports\nheterogeneous agents, complex and dynamic terrain, and realistic objectives under partial observability and\nstochastic conditions. By combining these features with support for low-level control and high-level language\nreasoning, CREW-Wildfire provides a unique testbed for evaluating large-scale collaboration, perception,\nand planning in Agentic multi-agent systems.\n\nRoughly speaking, I do think this paper mostly achieves this.\n\n> We implement and evaluate several state-of-the-art LLM-based multi-agent Agentic AI frameworks, uncovering significant performance gaps that highlight the unsolved challenges in large-scale coordination, communication, spatial reasoning, and long-horizon\nplanning under uncertainty\n\nThis paper introduces an environment that is argued demonstrates limitations in existing methods.\nOne weakness of this paper, in terms of a lack of convincing evidence, is a clear signal that the environment indeed demands these skills.\nNotably, this paper lacks any evidence or demonstration that this task can be handled appropriately. For example, as the system is set up, it seems that minimal effort would be required to produce a human baseline for this task, if support is made for the low-level interface. Particularly since many of the tasks do not require several agents. At the moment, we have to take it somewhat on faith that agents with certain competencies would indeed perform well in this environment.\n\n\nOne thing that is not clear to me, are the mechanisms of communication. One of the stated contributions has to do with communication. I realize some of the baselines have leaders that can communicate to other agents. What is not clear from the text of the paper, however, is how the agents communicate. Figure 4 lists out the abilities of the different types of agents, none of which involve actual communication. It does beg the question then, how the different entities are meant to communicate in a manner that is semantically consistent with wildfire responses. For example, are the agents allowed to communicate with each other on the algorithm-side (as opposed to communicating through the environment) as per the pseudocode? Is this meant to simulate radio communication?\n\n> To address this, CREW-Wildfire Benchmarking Suite provides infrastructure for systematic evaluation, testing, and\nbehavioral analysis\n\nThe paper lists what the behavioral goals are, but in the paper (including table 8), there is no enumeration of which behavior goals correspond to which tasks/levels. Moreover, the behavior goal designations are entirely qualitative in nature. Unfortunately, I have no means to evaluate the validity or correctness of the (unlisted) behavior goal labels with respect to the tasks.\n\nThe behavioral competency score is also hard to validate, and is unconvincing as evidence. It seems that each level or task has potentially multiple behavioral goals assigned to it. However, when computing the BCS, there is no clear way to decouple the degree to which each task requires competency in each behavioral goal. This can lead to a large amount of correlation that makes it extremely difficult to know, or even believe, whether the BCS is capturing the behavior meant to be measured.\n\nAs a side note, it is not clear the maximum score of 0 is indeed accurate for open-ended tasks. Is it possible to achieve a score of 0 on the open-ended tasks with negative penalties?\n\n\nOther general issues:\n- Only 3 seeds are run for everything, which definitely hurts the convincingness of the results.\n\n\nOverall, I think this could be an effective environment for the development and evaluation of multi-agent agentic systems. Much of my concern has to do with my ability to verify the efficacy given what I have in the paper. A large part of this paper is that it offers an environment meant to test capabilities not offered by other environments, which indeed it seems to do. However, where this paper seems to fall short in terms of convincingness is its ability to clearly demonstrate those capabilities (which are largely qualitative with no strong validated measurements or proxies) are captured by the environment as well as demonstrating the inefficacy of baselines on said capabilities.", "audience_yn": "Yes", "audience_explanation": "This paper introduces the CREW Wildfire environment, which in itself certainly does seem to offer something different and useful to a community interested in multi-agent agentic collaboration. The problem setting itself is of interest, and indeed, by my assessment, can be used to investigate research questions.", "requested_changes": "Critical Adjustments:\n- It appears like there are 16 levels in tables 2 and 9. Clarifying whether there are 12 or 16 levels is important, as the entire paper's empirical results hinge on this.\n\n\nImportant adjustments:\n- The pseudocode seems to suggest that the baselines do not have hyperparameters. Clarifying whether this is the case is helpful for understanding. \n- The costs of running an environment are naturally of critical importance for the community and readers to understand whether the environment should be used. Figure 7 does not make it clear how many API calls come from the baseline methods versus actually making the API calls for running the environment itself. Clarification here is key.\n- I am looking at the pseudocode for HMAS, Embodied Implementation, COELA, and CAMON. It does not appear that they have any learning. If there is no learning, I think it is important clarify in the text.\n\n\nHelpful adjustments:\n- Should use \\citep (instead of cite or citet). For example, the entire related work section has citations in text that is grammatically incorrect. These should be parenthetical or numbered citations (whatever is required by the TMLR format)\n- Fig. 7 is referenced, seemingly out of place, in section 4. Resolving this would benefit the clarity.\n- \"A.18 Random Seeds and Other Hyperparameters\" (no hyperparameters are mentioned?)\n- A.17: \"compentency\" should be \"competency\"\n- In the execution model prompt on page 20, \"PRIMATIVES\" should be \"primitives\"", "broader_impact_concerns": "While this is open-ended with the use of language models, this work is firmly a research project. I do not foresee any issues with broader impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "v8GTmUmK4X", "reviewer": "Reviewer_6WkG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces CREW-Wildfire, a benchmark environment for evaluating LLM-based multi-agent systems in complex, large-scale wildfire response scenarios. The authors argue that existing benchmarks lack the scale, complexity, and realism needed to properly test multi-agent coordination capabilities. Built on the CREW human-AI teaming platform, CREW-Wildfire features procedurally generated environments with up to 2000+ agents, heterogeneous roles (drones, helicopters, bulldozers, firefighters), partial observability, and stochastic dynamics. The paper evaluates four LLM-based agent frameworks (AutoGen, CAMON, LangGraph, CrewAI) across 12 benchmark levels measuring 7 behavioral competencies. Results show current frameworks struggle significantly with spatial reasoning, real-time coordination, and plan adaptation.\n\n**Strengths**\n\n- The benchmark addresses a limitation in current multi-agent evaluation with most existing work focusing on small-scale, fully observable, or simplified environments. The wildfire domain serves as a prototypical complex coordination task. In particular, supporting 2000+ agents substantially exceeds existing benchmarks.\n- The 7 behavioral competencies (Task Designation, Agent Capitalization, Spatial Reasoning, etc.) provide interesting metrics beyond just task success and the Behavior Competency Score (BCS) methodology allows diagnosing specific failure modes.\n- The results clearly demonstrate current LLM frameworks fail dramatically on complex coordination tasks which is valuable negative evidence for the community.\n\n**Weaknesses**\n\n- Perception module ablations: It would have been good to provide ablations for the specific implementation of the perception module. For example, using a strong VLM to map the image data to text instead of encoding it into ASCII first. \n- BCS methodology: The target and baseline definitions seem somewhat arbitrary. The authors do not provide experiments to check how sensitive the BCS scores to their particular choices.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper’s claims are well grounded in the experimental evidence provided by the authors.", "audience_yn": "Yes", "audience_explanation": "This paper tackles an important problem and introduces a technically impressive benchmark that could be valuable for the community.", "requested_changes": "The following changes could strengthen the paper:\n\n- Analyze the agent communication and perform a more detailed trajectory analysis showing patterns beyond individual examples. While the authors show examples of failure cases, it would be interesting to see statistics and a more high-level analysis as well. E.g., how often does a certain failure case happen?\n- Provide a VLM baseline. The authors use GPT4o for their experiments already. This model also has has vision capabilities. It could be interesting to feed the image to the model directly instead of converting it to ASCII first.\n- The paper focuses on task success but doesn't measure communication efficiency, redundancy, or other coordination-specific metrics beyond the behavioral goals.", "broader_impact_concerns": "No concerns", "additional_comments": "- What are the numbers in Table 3 showing?\n- What happens if you give agents perfect communication or full observability? How much does partial observability matter?\n- How does performance decrease with the number of agents?\n- What makes \"Large\" harder than \"Small\". Is it just map size or other factors? How was difficulty validated?\n- The \"Do Nothing\" baseline is useful but extremely weak. Would it be possible to try a scripted heuristic baseline? That could be more informative.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)