Title: LLM-PySC2: Starcraft II learning environment for Large Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Model

Decision-Making

Multi-Agent

StarCraft II Environment
Abstract: The tremendous potential has been demonstrated by large language models (LLMs) in intelligent decision-making problems, with unprecedented capabilities shown across diverse applications ranging from gaming AI systems to complex strategic planning frameworks. However, the StarCraft II platform, which has been widely adopted for validating decision-making algorithms in the past decade, has not yet provided substantial support for this emerging domain. To address issues that LLMs cannot interface with the hundreds of actions of the pysc2 backend and the lack of native support for multi-agent (MA) collaboration, we propose the LLM-PySC2 environment. This is the first environment that offers LLMs the complete pysc2 action space with sufficient multi-modal information and game Wiki knowledge. With an asynchronous query architecture, the environment efficiently interacts with LLMs that maintain a constant latency regardless of the scale of the agents' population. In the experiments, we evaluated LLMs' decision-making performance in both the macro-decision and micro-operation scenarios, with traditional StarCraft II Multi-Agent Challenge (SMAC) tasks and a series of new proposed. Results indicate that LLMs possess the potential to achieve victories in complex scenarios but cannot constantly generate correct decisions, especially in the recovered pysc2 action space and MA settings. Without task-relevant instructions, the pre-trained models suffer from issues such as hallucinations and inefficient collaboration. 
Our findings suggest that StarCraft II still challenges in the era of large models, revealing that there is a lot to do to develop an advanced LLM decision-making system, and the proposed LLM-PySC2 environment will support future development of LLM-based decision-making solutions.

LLM-PySC2: Starcraft II learning environment for Large Language Models

Abstract

The tremendous potential has been demonstrated by large language models (LLMs) in intelligent decision-making problems, with unprecedented capabilities shown across diverse applications ranging from gaming AI systems to complex strategic planning frameworks. However, the StarCraft II platform, which has been widely adopted for validating decision-making algorithms in the past decade, has not yet provided substantial support for this emerging domain. To address issues that LLMs cannot interface with the hundreds of actions of the pysc2 backend and the lack of native support for multi-agent (MA) collaboration, we propose the LLM-PySC2 environment. This is the first environment that offers LLMs the complete pysc2 action space with sufficient multi-modal information and game Wiki knowledge. With an asynchronous query architecture, the environment efficiently interacts with LLMs that maintain a constant latency regardless of the scale of the agents' population. In the experiments, we evaluated LLMs' decision-making performance in both the macro-decision and micro-operation scenarios, with traditional StarCraft II Multi-Agent Challenge (SMAC) tasks and a series of new proposed. Results indicate that LLMs possess the potential to achieve victories in complex scenarios but cannot constantly generate correct decisions, especially in the recovered pysc2 action space and MA settings. Without task-relevant instructions, the pre-trained models suffer from issues such as hallucinations and inefficient collaboration. Our findings suggest that StarCraft II still challenges in the era of large models, revealing that there is a lot to do to develop an advanced LLM decision-making system, and the proposed LLM-PySC2 environment will support future development of LLMbased decision-making solutions.

1 Introduction

The remarkable progress of LLMs has not only enhanced their reasoning capabilities but also positioned them as multitask strategists, even without post-training on specialized domains. Unlike reinforcement learning (RL) based agents, LLMs exhibit advantages in better context understanding, knowledge utilization, and human-AI interactions, acting in a wider range of zero-shot scenarios like gaming (1)-(12) , robot manipulation/navigation (13)-(17) , financial and trading (18)-(20).

However, there is still a lot to do to release the potential of LLM decision systems. Current works are mostly limited to prompt engineering (6)(7) , reflection (4)(8)(9)(22) , LLM workflow (3)(18)(21) to dismantle tasks into smaller tasks. These works enable LLMs to act better in diverse scenarios, but the knowledge-learning problem for a specific domain remains unsolved.

Player1

Player2

Build-in Bot

Human

LLM server1

LLM Services

Analysis:...

Prompt: ...

Images

GLYPH(cmap:d835)

GLYPH(cmap:d835)

Wrapper

Agent3

Multi-Agent System

Communication Actions:

<MessageTo(Agentxx,

'''

...

'''

)>

<MessageTo(Channel-1,

'''

...

'''

)>

Prompt:{system prompt}

{example prompt}

{observed info}

Actions:

Team Stalker-1:

<AttackUnit(0x1000a2001)>

Communication Info:

From Channel-1: \n\t

Agent1: ...

From Agent xx:

...

Commu In: ...

Actions:...

Commu A:...

Analysis:

In game time ..., we meet

group of enemies roaches ...

Agent4

Agent1

Aye!

To Agent2:

I need help

Listen: all

go to [x,y]

Agent for

Stalkers

Superior

Commander

Agent for

Colosuss

Agent for

Disruptors

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

Wrapper

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

LLM server2

GPT-3 backend

LLM server3

GPT-3 backend

LLM server4

GPT-3 backend

(Commander)

Agent2

To Agent3:

copy that

Camera Move

Script Agent

RL Agent

RL model

text obs

profile

LLM client4

wiki data

last action

commu-in

commu-out

query

response

GLYPH(cmap:d835)

GLYPH(cmap:d835)

image obs

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)4

茶

人

M

M

Currently, most LLM decision-making solutions are developed in relatively simple environments, resulting in ignorance of LLM's shortcomings. For example, MineDojo(23) is relatively comprehensible for LLMs and exhibits a high tolerance for errors, while some other works simplify the policy space of the environment(5)(6). Earlier works, such as StanfordTown(24), do not even concern decision-making ability but is more focused on LLMs' behaviors.

The StarCraft II environment, well known for its complexity, has been widely used as a validation platform for decision algorithms in the past decade, supported multi-agent research such as VDN(25), Qmix(26), MAPPO(27), the milestone algorithm Alpha-Star(28) and DI-Star(29). It is precisely the extremely high complexity that makes it the most authoritative verification platform for decisionmaking algorithms. However, since the vector interfaces are not compatible with LLMs, the StarCraft II environment does not support complete interactions with LLMs in the past few years.

Existing LLM Starcraft II environments, such as Swarm Brain(5), TextStarCraft II(TSC2) (6), have the problem of severely limiting the action space. They cut off most unit control operations and reduce continuous action space to discrete. Although the over-simplified environments have attracted attention for LLM decision research in the past years, they hindered further research due to the lack of complexity. Other works like (30)(31) do not support complete games.

At the same time, the support of current platforms for multi-agent systems is insufficient. Currently, most LLM multi-agent systems(12)(18) expose only a single agent to interact with the environment, while others act as modules for data processing or aggregating. Other works focus on conducting social simulations, emphasizing the accuracy of simulation (32)-(34) rather than promoting multiagent collaboration.

To provide support for LLM decision-making, we developed LLM-PySC2 , an environment derived from the StarCraft II Learning Environment (SC2LE)(35). This environment expanded the action space to complete pysc2 action space, allowing agents to perform fine-grained operations and unit skills. We also provide agents with comprehensive observations, including images and Wiki Knowledge(36).

It is worth noting that this is a platform with native multi-agent framework. We enable all kinds of multi-agent cooperation such as centralized decision-making and distributed decision-making. To

avoid an increase in waiting time as the number of agents grows, we build an asynchronous query architecture to maintain the latency of multi-agent queries.

In experiments, eight new scenarios were proposed. Unlike the SMAC(37) tasks, these tasks require more on task understanding and usage of unit skills. Mainstream LLMs are evaluated in both the complete StarCraft II games and mini scenarios. Results indicate that pre-trained LLMs have possess zero-shot decision-making ability but lack the ability to make consistently effective decisions. Without task-specific training, pre-trained LLMs cannot always find the key elements for victories. They fail to identify the important aspect of the situation, making mistakes in analysis and even dealing damage to allies sometimes.

Our contributions can be concluded as follows:

(1) We propose the first LLM StarCraft II framework with a complete pysc2 action space and provide a structured Wiki knowledge database of all units' information.

(2) We provide native support for multi-agent collaboration in our platform, paired with an asynchronous architecture that ensures a stable latency regardless of the population of LLM agents.

(3) We propose several new evaluation scenarios for LLM decision-making and evaluate LLMs' performance in both the macro-decision scenarios and the scenarios for micro-operations.

Problems of the LLM decision system are also discussed in the final sections of our paper. Results indicate that current LLMs cannot effectively handle complex StarCraft II scenarios due to serious hallucinations and lack of domain knowledge. How to increase the ability of LLMs in complex decision-making problems, at an acceptable cost, still poses a challenge in the era of large models and remains an unsolved problem.

2 LLM-PySC2 environment

2.1 Framework

The LLM-PySC2 environment is built on the player level of SC2LE. As shown in Figure 1, two players fight against each other and play the role of interacting with the pysc2 backend. They directly control the camera, select units, collect observations, and execute actions.

To precisely control the whole system, a multi-agent framework is designed. Agents of the system collaborate through natural language communication. At each step time t , agent i with profile p i get the observations o i t from the environment, queries remote LLM for analysis ana i t and strategy stg i t , communication messages m i t and actions a i t :

Then the player sends the joint action to the environment and transmits messages to assigned agents:

Pseudo code of the interaction and query process can be seen in Appendix A.

2.2 Actions

Actions are the most important part of a decision-making problem. In LLM-PySC2, textual actions play the role of the interface for large models and the environment. These actions are defined as:

where args refer to screen or minimap coordinates, unit tag, or their combination. Compared to discrete text actions, these actions avoid clipping the policy space and neglecting StarCraft II complexity. Details of the actions are provided in Appendix B.

In SC2LE, there are about 500 original for controlling Protoss, Terran, and Zerg. Most of them require additional parameters such as a screen or minimap position. These actions further constitute a huge policy space, making it one of the most complex environments for decision-making problems.

As shown in Fig 2, there are more than 100 text actions for Protoss agents in the LLM-PySC2 environment, which can be classified as unit control, unit skills, building, researching, training, etc. Different from other environments, these actions increase the theoretical performance of optimal policy but also raise the challenge of generating correct actions.

<Stop()>

<Hold_Position()>

<Attack_Unit(tag)>

<Move_Screen(screen)>

<Move_Minimap(minimap)>...

Unit Trainning Actions (17):

<Train_Adept()>

<Train_Zealot()>

<Train_Stalker()>

<Train_Phoenix()>

<Train_Colossus()> ......

Unit Warpping Actions (6):

<Warp_Zealot_Near(tag)>

<Warp_Stalker_Near(tag)> ......

Research Actions (17):

<Research_WarpGate()>

<Research_PsiStorm()>

<Research_ProtossAirArmor()>

<Research_ProtossAirWeapons()> ......

Build Actions (Easy Mode1, 15):

<Build_Nexus_Near(tag)>

<Build_Assimilator_Near(tag)>

<Build_Pylon_Near(tag)>

<Build_Forge_Near(tag)>

<Build_Gateway_Near(tag)>......

<Build_Nexus_Screen(screen)>

<Build_Assimilator_Screen(screen)>

<Build_Pylon_Screen(screen)>

<Build_Gateway_Screen(screen)> ......

Single Unit Contol, basic actions (3):

<Select_Unit_Attack_Unit(tag, tag)>

<Select_Unit_Move_Screen(tag, screen)>

<Select_Unit_Move_Minimap(tag, screen)> ......

Resource Block Actions (2):

<Lock_Nexus_Near(tag)>

<Lock_Assimilator_Near(tag)>

Transporter Actions (2):

<Load_Unit(tag)>

<Unload_Screen(screen)>

Unit Abilities (17):

<Ability_GuardianShield()>

<Ability_Blink_Screen(screen)>

<Ability_PsiStorm_Screen(screen)>

Single Unit Control, Abilities (17):

<Select_Unit_Ability_GuardianShield(tag)>

<Select_Unit_Ability_Blink_Screen(tag, screen)>

<Select_Unit_Ability_PsiStorm_Screen(tag, screen)> ......

<Morph_Archon()>

<Morph_ObserverMode()>

Protoss Action Space

<Ability_StasisTrap_Screen(screen)>

<Ability_ForceField_Screen(screen)>

<Ability_NexusMassRecall_Near(tag))> ......

<Build_Nexus()>

<Build_Assimilator()>

<Build_Pylon()>

<Build_Forge()>

<Build_Gateway()> ......

Scan Actions (4):

<Worker_Scan()>

<Zealot_Scan(t)> ......

Easy Control Actions (3):

<All_Units_Attack()>

<All_Units_Defend()> ......

Unit Warpping Actions (Easy Mode, 6):

<Warp_Zealot_Near(tag)>

<Warp_Stalker_Near(tag)> ......

Recognize Process of an Action

Bridge Object =

{'name': 'Select_Unit_Blink_Screen', 'arg': [

'tag'

,

'screen'

],

'func': [(2, pysc2.lib.actions.select_point, ('select',

'screen_tag'

)), (180, pysc2.lib.actions.Effect_Blink_screen, ('now',

'screen'

))]}

Recogonized Action:

<Select_Unit_Move_Screen(

0x1000c0001

,

(16,32)

)>

PySC2 functions:

pysc2.lib.actions.select_point('select',

(45, 66)

)

pysc2.lib.actions.Effect_Blink_screen('now',

(16,32)

)

0x1000c0001

Find the bridge object

Find the position of

Generate pysc2 functions

2.3 Observation

Observation provides fundamental support for decision-making. Given the distinct requirements of different agents, we developed an interface that offers each agent the observations specifically suited to their tasks. Additionally, with multi-modal observations that convey rich semantic and visual information, we released the potential for a deeper understanding of the situation, solving the problem that the previous environment had only observation of unit quantity information.

LLM-PySC2 Observation

System Prompt

Example Input

Example Output

GLYPH(cmap:d835)

GLYPH(cmap:d835)

Warpper

Environment

Task Description

Global info

Team unit info

Game Knowledge

Valid Actions

......

Text Observation

Team Stalker-1 Info:

Controlled Units: ......

Nearby Ally Units: ......

Nearby Enemy Units: ......

Game Time: 00:50

Minerals: 200, Gas: 50

Valid Actions:

Team Adept-1:

<Stop()>

<Attack_Unit(tag)>

<Move_Screen(screen)>

......

Tasks: Team Adept-1' task: Kill as much as enemy workers as possible.

Relevant Knowledge:  Protoss.Adept: ......,   Zerg.Queen: ......,   ......

RGB Feature

RGB Minimap

RGB Screen

Feature map

Minimap

Image Observation

Screen Image

Image Observation

Text Observation

Feature map

Last step Actions:

Team Adept-1: ......

Communication info: ......,   Valid targets: ......,   Communicat Actions: ......

Unit Count:  our unit: ......,    our buildings: ......,   spotted enemy: ......

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

2.3.1 Text Observation.

As illustrated in Fig. 3, an observation wrapper is implemented to process the relevant text observations for each agent. This wrapper includes a set of functions tailored for handling different types of text information. Once all parts of the text observation are generated, they will be aggregated into OpenAI query messages, which include system prompts, example inputs and outputs, as well as a series of images. These messages are then sent to the LLM server for querying responses.

Considering possible user requirements, we expose all the observation interfaces in the open-source code repository. It is possible to customize the wrapper to generate other kinds of text observations. More detailed examples of text observation can be seen in Appendix C.

2.3.2 Image Observation.

In StarCraft II scenarios, image observation provide information such as terrain and relative position. It is almost inevitable to use images to describe such higher-dimensional information.

In LLM-PySC2, we provide four kinds of image observation: RGB-Screen, RGB-Minimap, RGBFeature, and Original-Feature-Maps. Image observation wrapping functions collect the image from the pysc2 backend, adding auxiliary lines and annotations to facilitate the coordinate recognition by LLMs. The image will be encoded into a base 64 string and will be added to the message to query the LLMs for analysis, actions, and communication behaviors.

2.4 Multi-Agent System

Disassembling complex problems into small tasks in a multi-agent system has become a basic solution. Different large models interact through natural language, coordinating their behaviors and managing the massive StarCraft II system together.

(1) Macro Decisions: Combat

(2) Macro Decisions: Develop

(3) Micro Operations: Precise Combat

(4) Micro Operations: Precise Build

<xxx_Scan()>

<All_Units_Attack()>

Agent

CombatGroup1

Agent

CombatGroup2

Agent

CombatGroup5

Agent

Commander

GLYPH(cmap:d835)

GLYPH(cmap:d835)

3

Agent

Developer

unit training buildings

researching buildings

<Reasearch_xxx()>

<Train_xxx()>

Communication

Agent

Builder

GLYPH(cmap:d835)

GLYPH(cmap:d835)

2

Execute

Execute

GLYPH(cmap:d835)

GLYPH(cmap:d835)

1

Execute

workers

Communication

Communication

(1)

(2)

(2) Enable Easy Build Mode

Team Probe-1: ...

Team Probe-2: ...

<Build_xx()>

G

In LLM-PySC2, agents collaborate by communicating with each other. They can discuss in a channel or directly send messages to another agent. As shown in Fig. 1, at each step, received messages will be added to observation, and the agent can respond to others by generating Communication actions shaped as < MessageTo ( TargetName, ′′′ content ′′′ ) > . When the agent receives messages, the received messages will be displayed in their origin form. An Agent should analyze both the observed situation and the requests/information from others, and finally generate actions and reply to their teammates.

As shown in Fig. 4, we define four kinds of agents responsible for (1) macro-decisions for combat deployment, (2) macro-decisions for economic development, (3) micro-operations for combat, and (4) micro-operations for building. Agents for macro-decisions organize other agents to work together, while agents for micro-operations execute specific actions. Note that, agents of LLM-PySC2 query in independent threads, ensuring a constant waiting time when the number of agents increases.

The multi-agent system supports both centralized and decentralized decisions in the environment. Two 'Easy Modes' are also provided for simplifying some aspects that researchers are not very concerned about, among which 'Easy Build' disables the agent Builder and helps researchers concentrate more on multi-agent collaboration in the combat, while 'Easy Control' disables the agents CombatGroups and helps researchers concentrate more on planning and multi-modal information processing.

3 Experiments

In this section, we introduce two series of experiments: (1) Experiments for macro-decisions, i.e. complete StarCraft II game ; (2) Experiments for micro-operations, including classic SMAC scenarios and eight new tasks that require units to use their skills and achieve assigned goal. To distinguish micro-operation scenarios from the traditional SMAC environment, we refer to these two groups of experiments the LLM-SMAC task group and the LLM-PySC2 task group .

Combined with the complete StarCraft II games, these experiment scenarios constitute one of the most comprehensive experiment groups in LLM decision-making and support research on enhancing LLMs' abilities in reasoning, planning, learning, and multi-agent cooperation.

We use the Kill/Death (KD) ratio and Winning Rate (WR) to evaluate the performance of the LLMs:

where V killed _ units and V dead _ units refers to killed units' value and dead units' value of the LLM's camp. The higher the value of KD and WR, the better the performance of LLMs.

3.1 Experiments for Macro-Decisions

3.1.1 Experiment Settings

Complete games demand real decision-making abilities, such as analyzing situations, planning for tactic strategy, deceiving the opponent, and engaging with the enemy at the right time. To evaluate the performance of LLM macro-decisions, we tested the three modes in the Simple64 map: (1) easy control + easy build (ECEB); (2) standard control + easy build (SCEB); and (3) easy control + standard build (ECSB). Considering the poor performance of LLM in complex settings, we do not further test the last mode (4) standard control + standard build (SCSB).

For games with easy control settings, we enable the agent Commander to directly control all units to attack, defend, retreat, and scan for information. For the standard control settings, we enable agents named CombatGroup-i to precisely control different kinds of unit to move, attack, and use skills.

For games with easy build settings, the agent Developer can build buildings by generating actions < Build _ BuildingName () > . For games that enable standard build, the agent Developer can only train units and upgrade technologies, and has to communicate with the agent Builder to build in specific coordinates [ x, y ] by generating actions < Build _ BuildingName _ Screen ([ x, y ]) > .

In these experiments, we give each agent a client for querying GPT-4o-mini. For macro-decision agents, we provide relevant text information and minimap images. For standard unit control agents, we provide text observation and both the screen image and minimap images. Examples of observations, responses for different agents, and detailed experimental settings can be seen in Appendices C and D.

(a) expand a new base

(b) build buildings

(c) train/warp units

(d) attack

(e) defend

(f) retreat

回口

品品

口

00

0口0口

1953

3.1.2 Experiment Results

In the macro-operation tasks (complete StarCraft II games), we conducted 30 repeated experiments from level-1 (very easy) to level-7 (very hard/elite). As shown in Table 1, two agents in the ECEB mode control the whole system via discrete actions and perform nearly the same as in TSC2 (6). At level-5, LLMs can only win about 30% of the games and nearly lose all games at level-6 and above.

In SCEB and ECSB modes, LLMs perform worse due to the recovered complexity from the complete action space and the higher demand for collaboration. In SCEB mode, the Easy Build part develops the economy and military strength the same as in ECEB mode, but agents for micro-operations frequently make mistakes in command, resulting in a 0% winning rate from level-3 onwards. Notably, in ECSB mode, while the Builder agent controls workers for standard construction operations, it maintains certain competitiveness from level-1 to level-3, achieving a 60% winning rate at level-3. However, as the difficulty increases further, issues such as inappropriate building placement and suboptimal defense layouts become evident, causing the winning rate to drop sharply to 17% at level-4 and resulting in complete inability to win from level-6 onwards.

To better assess the statistical significance of the winning rates and account for the uncertainty associated with the limited sample size, we calculated Wilson Score Confidence Intervals. This interval provides a reliable confidence interval estimate for a binomial proportion, particularly suitable for small sample sizes or proportions near 0 or 1 . The formula for the Wilson interval is as follows:

where ˆ p is the observed winning rate, n is the number of trials ( 30 in this case), and z is the 1 -α/ 2 quantile of the standard normal distribution, i.e., z = Φ -1 (1 -α/ 2) . For a 90% confidence interval ( α = 0 . 10 ), z ≈ 1 . 645 . The upper and lower limits of the interval, denoted as WP upper and WP lower , are reported in the table in red and green, respectively.

3.2 Experiments for Micro-Operations

3.2.1 Experiment Settings

SMAC is a well-known benchmark for multi-agent reinforcement learning (MARL) approaches. We provide compatible support for SMAC tasks. Note that, unlike the SMAC tasks that n units are controlled by n agents, the LLM-SMAC units are controlled in groups. It is not recommended to compare the LLM-based method with the MARL-based method due to different control frequencies.

In the LLM-PySC2 task group, eight new experimental scenarios were constructed. These tasks introduce unit skills into the experiments. Unlike SMAC, which focuses only on incoming combat, LLM-PySC2 requires an understanding of task description, planning attack routes, and utilizing skills to achieve the goal. Tasks 1 to 4 are designed as single-agent tasks, while tasks 5 to 8 are designed as multi-agent tasks. Agent settings are the same as for the standard control mode of the complete StarCraft II game. More detailed settings can be seen in Appendix D.

(d) task4: Blink-Stalker combat

(e) task5: large scale combat (type1)

(f) task6: large scale combat (type2)

品

ooo.

OCP

'0000

3.2.2 Experiment Results

In the micro-operation tasks, we conducted 20 repeated experiments for each LLM (except GPT3.5-turbo which evaluates 50 games). As shown in Table. 2, all the tested LLMs act poorly in LLM-SMAC scenarios, similar to works such as (30). LLMs make obvious mistakes that do not move their long-range combat units, even when attacked by melee units in 3s_vs_3z. In 2s3z, the agent for Stalkers sometimes escapes from the battlefield, resulting in the quicker death of ally Zealots.

In the LLM-PySC2 task group, we evaluate the performance of 9 models. Results in Table. 3 demonstrate that LLM suffers from hallucinations and the lack of knowledge (More details are provided in the Discuss section and Appendix E).

Two findings derive from these results: (1) Reasoning models such as GPT-o1-mini cannot significantly improve the decision-making ability in an environment never seen before; (2) Scaling law does not work well in decision-making problems that Llama3.1-405b does not significantly outperform Llama3.1-70b (but enough parameters is crucial for basic decision-making ability). These problems are possibly due to a lack of relevant knowledge and instructions in the pre-training stage.

To avoid the situation that all tasks achieve 100% winning rates several years after its proposal, we set three difficulty levels for the LLM-PySC2 task group. As the level grows, it will be more difficult for the LLMs to reach the goal due to additional enemy units or upgrades. We evaluate the performance of GPT-3.5-turbo in these tasks, as shown in Table. 4, and serve it as a baseline for future research.

3.3 Latency and Token Cost.

Since StarCraft II is a real-time environment, we also report the runtime latency and token consumption to assess the practicality of LLM-based decision-making. Table 5 summarizes the average observation size, response size, and runtime latency by agent type in complete-game settings using GPT-3.5. The environment-side latency (i.e., observation processing and action parsing) remains

below 0.1 seconds per step, which is negligible compared to the model-side inference cost. As shown in the table, LLM response latency (5.2-5.6 seconds per query) is the dominant overhead, demonstrating that the PySC2 execution pipeline is efficient and that LLM inference is the true computational bottleneck.

4 Discussion

In this section, we discuss three challenges in LLM decision making. These challenges significantly reduce performance, severely hindering the application of the LLM-based decision-making system.

Lack of domain knowledge . Correct and sufficient knowledge is the prerequisite for correct decisions. However, there is no guarantee that all knowledge across all fields are introduced in the pre-training phase As a result, LLMs may not realize that 49 additional supplies are far beyond the demand for StarCraft II games, or know that shields recharge in the 2s_vs_1sc scenario.

Hallucinations and mistakes . Hallucinations and mistakes have an inevitable impact on the decisionmaking process. LLM suffers from (1) input-conflicting hallucinations that generate invalid actions; (2) fact-conflicting hallucinations that mistake ally units for enemy units; and (3) context-conflicting hallucinations that mistake screen coordinates for minimap coordinates.

Inefficient collaboration . Effective information exchange is critical for multi-agent collaboration. However, LLMs generate communication messages with a lot of non-essential and incorrect information. At the same time, they tend to unconditionally trust their teammates and ignore possible errors in the incoming information, which severely damages the performance in StarCraft II games.

These problems hinder the further application of LLM-based intelligent decision-making systems, waiting for further research and solutions. Examples of these problems are provided in Appendix E.

5 Conclusion

In this paper, we introduce a new environment for LLM decision-making, the first environment that accommodates the complete continuous PySC2 actions, and the first LLM StarCraft II environment with a multi-agent framework and communication system. In experiments, we evaluated the performance of mainstream LLMs in complete StarCraft II games and both the LLM-SMAC and LLM-PySC2 task groups, among which the LLM-PySC2 task group is a brand-new experimental scenario that we designed for large models. Results show that LLMs can make decisions and generate valid actions but cannot make effective decisions consistently. Still, the quality of the decision is relatively low and there are several problems such as hallucinations, poor utilization of knowledge of the game, and lack of understanding of the world. Results indicate that learning in the deployment environment is necessary for LLM-based decision-making solutions. We hope the LLM-PySC2 environment can promote research on LLM learning methods, helping LLM-based decision-making methods better adapt to task scenarios.

References

Tan, W., Ding, Z., Zhang, W., Li, B., Zhou, B., Yue, J., Xia, H., Jiang, J., Zheng, L., Xu, X., Bi, Y., Gu, P., Wang, X., Karlsson, B. F., An, B., et al. , Lu, Z. Towards General Computer Control: A Multimodal Agent for Red Dead Redemption II as a Case Study. In ICLR 2024 Workshop on Large Language Model (LLM) Agents , January 2024.

Xu, Y., Wang, S., Li, P., Luo, F., Wang, X., Liu, W., et al. , Liu, Y. Exploring Large Language Models for Communication Games: An Empirical Study on Werewolf. arXiv preprint arXiv:2309.04658 , 2023.

Qian, C., Liu, W., Liu, H., Chen, N., Dang, Y., Li, J., Yang, C., Chen, W., Su, Y ., Cong, X., Xu, J., et al. , Sun, M. ChatDev: Communicative Agents for Software Development. arXiv preprint arXiv:2307.07924 , 2023.

Hua, W., Liu, O., Li, L., Amayuelas, A., Chen, J., Jiang, L., Jin, M., Fan, L., Sun, F., Wang, W., et al. , Zhang, Y. Game-Theoretic LLM: Agent Workflow for Negotiation Games. arXiv preprint arXiv:2411.05990 , 2024.

Shao, X., Jiang, W., et al. ,Liu, M. SwarmBrain: Embodied Agent for Real-Time Strategy Game StarCraft II via Large Language Models. In arXiv preprint arXiv:2401.17749 , 2024. URL: https://arxiv.org/abs/2401.17749 .

Ma, W., Mi, Q., Zeng, Y., Yan, X., Wu, Y., Lin, R., et al. , Wang, J. Large Language Models Play StarCraft II: Benchmarks and a Chain of Summarization Approach. In Advances in Neural Information Processing Systems , volume 37, pages 133386-133442, 2024.

Z. Li, C. Lu, X. Xu, R. Qi, Y. Ni, L. Jiang, et al. , X. Guo. Hierarchical Expert Prompt for Large-Language-Model: An Approach Defeat Elite AI in TextStarCraft II for the First Time. arXiv preprint arXiv:2502.11122 , 2025.

Xu, X., Li, Z., Lu, C., Qi, R., Ni, Y ., Jiang, L., Liu, X., Zhang, X., Fang, Y ., Huang, et al. , Li, Z. Reflection of Episodes: Learning to Play Game from Expert and Self Experiences arXiv preprint arXiv:2502.13388, 2025.

Zhu, X., Chen, Y., Tian, H., Tao, C., Su, W., Yang, C., Huang, G., Li, B., Lu, L., Wang, X., Qiao, Y., et al. , Dai, J . Ghost in the minecraft: Generally capable agents for open-world environments via large language models with text-based knowledge and memory arXiv preprint arXiv:2305.17144, 2023.

Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., et al. , Anandkumar, A. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023.

Feng, Y., Wang, Y., et al. , Lu, Z. Llama rider: Spurring large language models to explore the open world. arXiv preprint arXiv:2310.08922, 2023.

Y. Li, S. Liu, T. Zheng, M. Song. Parallelized Planning-Acting for Efficient LLM-based Multi-Agent Systems. arXiv preprint arXiv:2503.03505 , 2025.

Liu, H., Zhu, Y., Kato, K., Tsukahara, A., Kondo, I., et al. , Hasegawa, Y. Enhancing the LLM-Based Robot Manipulation Through Human-Robot Collaboration. IEEE Robotics and Automation Letters , 2024.

D. Shah, B. Osi´ nski, S. Levine. LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action. In Conference on Robot Learning , pages 492-504, March 2023.

P. Doma, A. Arab, X. Xiao. LLM-Enhanced Path Planning: Safe and Efficient Autonomous Navigation with Instructional Inputs. arXiv preprint arXiv:2412.02655 , 2024.

Jin, Y., Li, D., Shi, J., Hao, P., Sun, F., et al. , Fang, B. RobotGPT: Robot Manipulation Learning From ChatGPT. IEEE Robotics and Automation Letters , vol. 9, no. 3, pp. 2543-2550, 2024.

Dorbala, V. S., Mullen, et al. ,Manocha, D. Can an Embodied Agent Find Your 'Cat-Shaped Mug'? LLM-Based Zero-Shot Object Navigation. IEEE Robotics and Automation Letters , vol. 9, no. 5, pp. 4083-4090, 2023.

Xiao, Y., Sun, E., et al. , Wang, W. TradingAgents: Multi-Agents LLM Financial Trading Framework. arXiv preprint arXiv:2412.20138 , 2024.

Ouyang, K., Liu, Y., Li, S., Bao, R., et al. , Sun, X. Modal-adaptive Knowledge-enhanced Graphbased Financial Prediction from Monetary Policy Conference Calls with LLM. In Proceedings of the Joint Workshop of the 7th Financial Technology and Natural Language Processing, the 5th Knowledge Discovery from Unstructured Data in Financial Services, and the 4th Workshop on Economics and Natural Language Processing , pages 59-69. Association for Computational Linguistics, Torino, Italia, 2024.

Y. Li, S. Wang, H. Ding, H. Chen. Large Language Models in Finance: A Survey. In Proceedings of the Fourth ACM International Conference on AI in Finance , pages 374-382, November 2023.

Z. Zeng, W. Watson, N. Cho, S. Rahimi, S. Reynolds, et al. , M. Veloso. FlowMind: Automatic Workflow Generation with LLMs. In Proceedings of the Fourth ACM International Conference on AI in Finance , pages 73-81, November 2023.

J. Xu, W. Du, X. Liu, X. Li. LLM4Workflow: An LLM-Based Automated Workflow Model Generation Tool. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering , pages 2394-2398, October 2024.

Fan, L., Wang, G., Jiang, Y., Mandlekar, A., Yang, Y., Zhu, H., Tang, A., Huang, D., et al. , Anandkumar, A. MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge. Advances in Neural Information Processing Systems , vol. 35, pages 18343-18362, 2022.

Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., et al. , Bernstein, M. Generative Agents: Interactive Simulacra of Human Behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology , pages 1-22, October 2023.

Sunehag, P., Lever, G., Gruslys, A., Czarnecki, W., Zambaldi, V., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, et al. , Graepel, T. Value-Decomposition Networks for Cooperative Multi-Agent Learning. In arXiv preprint arXiv:1706.05296 , 2017. URL: https://arxiv. org/abs/1706.05296 .

Rashid, T., Samvelyan, M., De Witt, C. S., Farquhar, G., Foerster, et al. J.,Whiteson, S. Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning. In Journal of Machine Learning Research , volume 21, number 178, pages 1-51, 2020.

Yu, C., Velu, A., Vinitsky, E., Gao, J., Wang, Y., Bayen, A., et al. ,Wu, Y . The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games. In Advances in Neural Information Processing Systems , volume 35, pages 24611-24624, 2022.

Vinyals, O., Babuschkin, I., Czarnecki, W., Mathieu, M., Dudzik, A., Chung, J., Choi, D., Powell, R., Ewalds, T., Georgiev, P., Oh, J., Horgan, D., Kroiss, M., Danihelka, I., Huang, A., Sifre, L., Cai, T., Agapiou, J., Jaderberg, M., Vezhnevets, A., Leblond, R., Pohlen, et al. , C., Silver, D. Grandmaster Level in StarCraft II Using Multi-Agent Reinforcement Learning. Nature , vol. 575, no. 7782, pages 350-354, 2019.

DI-star Contributors. DI-star: An Open-source Reinforcement Learning Framework for StarCraft II. 2021.

Ma, W., Fu, Y., Zhang, Z., Li, G., et al. Ghanem, B. VLMs Play StarCraft II: A Benchmark and Multimodal Decision Method. arXiv e-prints , arXiv:2503, 2025.

Y Deng, Y Yu, W Ma, Z Wang, W Zhu, J Zhao, Y Zhang SMAC-Hard: Enabling Mixed Opponent Strategy Script and Self-play on SMAC arXiv preprint arXiv:2412.17707, 2024.

Tang, J., Gao, H., Pan, X., Wang, L., Tan, H., Gao, D., Chen, Y., Chen, X., Lin, Y., Li, Y ., Ding, B., Zhou, J., Wang, J., et al. ,Wen, J. GenSim: A General Social Simulation Platform with Large Language Model Based Agents. arXiv preprint arXiv:2410.04360 , 2024.

Hua, W., Fan, L., Li, L., Mei, K., Ji, J., Ge, Y., Hemphill, L., Zhang, Y. War and Peace (WarAgent): Large Language Model-Based Multi-Agent Simulation of World Wars. arXiv preprint arXiv:2311.17227 , 2023.

Gürcan,Ö. LLM-Augmented Agent-Based Modelling for Social Simulations: Challenges and Opportunities. In HHAI 2024: Hybrid Human AI Systems for the Social Good , pages 134-144, 2024.

Vinyals, O., Ewalds, T., Bartunov, S., Georgiev, P., Vezhnevets, A., Yeo, M., Makhzani, A., Küttler, H., Agapiou, J., Schrittwieser, J., Quan, J., Gaffney, S., Petersen, S., Simonyan, K., Schaul, T., van Hasselt, H., Silver, D., Lillicrap, T., Calderone, K., Keet, P., Brunasso, A., Lawrence, D., Ekermo, A., Repp, J., et al. Tsing, R. StarCraft II: A New Challenge for Reinforcement Learning. arXiv preprint arXiv:1708.04782 , 2017.

Blizzard Entertainment. StarCraft II. StarCraft Wiki , [Online]. Available: https:// starcraft.fandom.com/wiki/StarCraft_II .

Samvelyan, M., Rashid, T., de Witt, C., Farquhar, G., Nardelli, N., Rudner, T., Hung, C., Torr, P., Foerster, J., et al. Whiteson, S. The StarCraft Multi-Agent Challenge. In arXiv preprint arXiv:1902.04043 , 2019. URL: https://arxiv.org/abs/1902.04043 .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims presented in the abstract and introduction align precisely with the contributions and scope of the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discussed both the problems of LLM decision-making in the Discuss section and listed some limitations of our environment in supplementary materials Appendix G.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Our code is available in Anonymous GitHub (link: https://anonymous. 4open.science/r/LLM-PySC2-Anonymous-0E0D ), and the experiment results mentioned in the paper can be reproduced by source code.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Our code is available in Anonymous GitHub (link: https://anonymous. 4open.science/r/LLM-PySC2-Anonymous-0E0D ), with source code of the environment, quick start script, documents, and other material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The experiment setup and results are mentioned in the Experiment section. More detailed settings and examples of the model's input and output are also listed in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The two indices (Winning Rate and Kill/Death ratio) involved in our experiment usually do not use error bars (1). The winning rate itself is a statistical value, and the raw data of winning and losing are distributed in 1, 0 (i.e. winning/losing). (2) For a single game, the KD ratio has the possibility of dividing by zero (i.e. no unit lost). Directly calculating the ratio of each game before calculating the mean and variance may lead to errors. Therefore, the KD ratio is calculated by dividing the total loss of enemy units of all experiments by the total loss of ours, to avoid the error of dividing by zero.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We introduced the hardware settings for the environments in Appendix D.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We claim the Code of Ethics in Appendix G.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We claim the Broader Impacts in Appendix G.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our works developed the former RL environment into an LLM decisionmaking environment, all the experiments dealt within the StarCraft II game. We do not release a new model, dataset or image generators that may lead to misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We provide citations and source links for existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We introduced LLM-PySC2, an environment with complete pysc2 action space. All the assets (such as game maps) are already released in Anonymous GitHub (link: https://anonymous.4open.science/r/LLM-PySC2-Anonymous-0E0D ).

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [Yes]

Justification: This paper introduces an LLM decision-making environment and evaluates the performance of some LLMs in it.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Appendix A. Pseudo Code

A.1 LLM-PySC2 Rollout Process

Algorithm 1 LLM-PySC2 Rollout Process

A.2 Query Process for an Agent

Algorithm 2

Appendix B. Action Space

Most units share the actions above, so we listed here to avoid mention repetitive unit control actions in subsequent appendices. Here, F refers to pysc 2 .lib.actions.F UNCTION , the same as in the following.

In standard building mode, a worker will be chosen as building-worker at the beginning of the game or the time the worker dead. The 'Builder' agent will control the worker to build buildings using the actions mentioned above.

In easy-build mode-1, the agent 'Builder' does not need to provide precision position, but a tag of nearby buildings. The LLM-PySC2 program will autonomously find a position near the unit with given tag and build new buildings there.

In easy-build mode-2, the agent 'Builder' does not need to provide any additional information of where to build the building. the program will automatically find a position for above actions. Experiments of ECEB mode and SCEB mode use these actions as the Developer's building actions.

Researching actions are actually complex actions of combination pysc2 functions, they require first selecting a research building and then starting the research. The program will autonomously find idle building for these functions, select the building and execute the pysc2 functions for technology upgrades.

Unit training actions share the same pre-process of researching actions. To avoid stocking a lot of resources in the training queue, the program only trains units in idle buildings. To avoid spending too many tokens on finding suitable buildings, the program autonomously searches idle buildings for these actions.

For protoss, some of the unit training actions will change to unit warpping actions after WarpGate technology upgrades. They need to choose a tag for power field provider (such as Pylon) to warp unit there. In easy-warp mode, the program will autonomously find valid position for unit warping actions.

We provide easy control actions, a series of actions similar to TextStarCraft-II unit control actions. For researchers who focus on studying LLM-based planning (develop the economy) or VLMbased decision making(precisely build buildings), simplifying unit control actions can provide great convenience.

In LLM-PySC2 environment, agent controls its unit in a group, using above actions. The program will select the units belong to the agent, and then execute LLM-generated actions.

In some senerios, precisely controlling single unit is key to the victory, especially in SMAC tasks and early stage of the game (against high level opponent). We provide single unit control actions for this senarios, and LLM can use the actions whenever they need to improve performance of micro-operations.

Appendix C. Query message, Prompt, Examples of Observations and Responses

C1. Query message and Prompt

Decision Rules of the System Prompt

BASIC_COMMAND_RULES = """

Analyse following aspect in your decision process:

(Combat Deployment) Analyse the situation, always make deployment for attack/defend/retreat at each step. As default choice, ask your units to defend your area. If you have enough units, rise attack to defeat your enemy.

(Scan Deployment) If you are prepared for combat, make scan deployment to find out enemy's strengths. Note that unit for scan will be killed by enemy units.

(Final Combat) If game time > 12:00 and you have enough units (DO NOT have to be 200 supply), raise attack to defeat the enemy."""

BASIC_DEVELOP_RULES = """

Analyse following aspect in your decision process:

(Supply) If run out of supply (less than 10) and no building for supply is under construction, build Pylon/OverLord/SupplyDepot (depend on your race).

(Economy Building: Base) If you have enough minerals (more than 400), build Nexus/Hatchery/CommandCenter (depend on your race).

(Economy Building: Gas) If you run out of gas (much less than minerals), build Assimilator/Extractor/Refinery (depend on your race).

(Building: Unit Training) If you have too less unit training buildings, or there are abundant resources but all the unit training buildings are working, build unit training buildings.

(Building: Research) If you have too less research buildings, or there are abundant resources but all the research buildings are working, build research buildings.

(Unit Training/Warping) If you have enough idle unit training buildings but few combat units, or have a lot resource, train/warp units as much as possible.

(Tech Upgrading) If you have idle research buildings and enough resource, or have a lot resource, update your technology.

8. (Early Stage Expand) If you do not have 'the second base building (such as Nexus)', the

'CyberneticCore', the 'TwilightCouncil' and 'first two Gateway', try to build them as quick as

possible.

(Middle Stage Develop) During the middle stage of the game, try to build buildings for training high value units, and train units as much as possible (especially high value units) to increase strength.

(Final Stage Develop) During the final stage of the game, train or warp more units to fight with enemy, do not build building if we have enough buildings."""

BASIC_BUILD_RULES = """

Analyse following aspect in your decision process:

(Minimap Position) According to image 'rgb_minimap', where is/are our base/bases and where should we go? give minimap position. (Our base units and buildings are green points/squares in the minimap)

(Build) You can build on any position of the screen (unless it is blocked by other buildings) direct use the action <Build_XXX_Screen([x, y])>. You can build more than one buildings at a step by generate many action <Build_XXX_Screen([x, y])>.

(Move) You should move to a plain location near the base building, and build buildings there. Don't be far away from the base building, keep base building in your sight(screen), unless you are building a new one.

(Actions Sequence) First <Build_xxx_Screen(screen)> or <Build_xxx_Near(tag)> then <Move_xxx(xxx)> (note that, do not move and build at the same screen position)"""

BASIC_COMBAT_RULES = \"""

Concentrating firepower is always necessary, attack different unit at the same time will definitely reduce killing speed and leading to terrible result. Always concentrating all teams' fire at one unit that (1)with highest DPS(most valuable) (2)most vulnerable (3)closest."""

C2. Examples of Textual Observations

C3. Examples of Image Observation

12

15

18

21

12

15

18

21

12

15

18

21

2

15

18

21

12

15

18

21

15

18

21

2

15

18

18

21

中

12518

21

15

18

21

12

15

18

21

12

15

18

21

15

18

21

6

15

18

3

6

9

12

15

18

21

12

15

18

21

12

15

18

21

15

18

21

1

3

12

15

18

21

3

12

15

18

21

15

18

21

3

15

18

21

15

18

21

15

18

21

81

1624

32

40

56

24

32

40

48

56

40

48

56

8

16

24

32

40

48

56

16

24

32

40

48

56

8

16

24

32

40

48

56

40

48

56

8

16

24

32

40

48

56

16

24

40

48

56

16

24

32

40

48

56

12

15

18

21

D

12

15

18

15

18

12

15

18

21

12

15

18

21

2

15

18

21

12

15

18

21

12

15

18

3

9

12

15

18

21

12

15

18

21

2

1518

21

3

9

12

15

18

12

15

18

21

2

15

18

15

18

21

3

6

9

12

15

18

21

15

18

3

6

12

15

18

21

9

12

15

18

8

16

24

40

48

56

24

32

40

48

56

8

16

24

32

40

48

56

32

40

48

56

16

24

40

48

56

16

24

32

40

48

56

8

16

24

32

40

48

56

16

24

32

40

48

56

8

16

24

40

48

56

8

16

24

32

40

48

56

16

24

32

40

48

56

16

24

32

40

48

56

32

40

48

56

8

16

16

24

32

40

8

16

24

32

40

48

56

16

24

32

40

48

56

8

16

24

32

40

48

56

8

16

24

32

40

48

56

C4. Examples of LLM Responses

Example of  Response (Agent Commander, Easy Control Mode) Example of  Response (Agent Commander, Easy Control Mode)

Analysis: Analysis:

Our current resources (340 minerals and 502 vespene) are enough to reinforce with additional units if necessary. The Warp Gates are idle, so more reinforcements can be warped in. Additionally, the Shield Battery and Pylon construction are progressing, which will help maintain our position. However, the limited supply (26 supply left) means we must carefully consider which units to produce. necessary. The Warp Gates are idle, so more reinforcements can be warped in. Additionally, the Shield Battery and Pylon construction are progressing, which will help maintain our position. However, the limited supply (26 supply left) means we must carefully consider which units to produce.

The enemy is currently defending with a small group of units, including a Stalker and a Zealot. The enemy base has minimal defense, with only one Warp Gate and Pylon in the area. Our Zealots have already begun the assault on enemy workers, but reinforcements are still on the way. We need to reinforce our offensive, manage resources, and ensure the continued supply of troops. The enemy is currently defending with a small group of units, including a Stalker and a Zealot. The enemy base has minimal defense, with only one Warp Gate and Pylon in the area. Our Zealots have already begun the assault on enemy workers, but reinforcements are still on the way. We need to reinforce our offensive, manage resources, and ensure the continued supply of troops. Our current resources (340 minerals and 502 vespene) are enough to reinforce with additional units if

Actions:

Communications: Communications:

<MessageTo(Developer, '''Request reinforcements: warp in more Zealots to sustain the attack on enemy workers. Consider warping in additional Stalkers as well for support against any enemy Stalker reinforcements. The Shield Battery and Pylon construction must be prioritized for sustainability. Deploy more Probes for resource gathering if necessary to maintain unit production. Proceed with caution to avoid detection by enemy Queens in the vicinity. We need to overwhelm the enemy quickly.''')> <MessageTo(Developer, '''Request reinforcements: warp in more Zealots to sustain the attack on enemy workers. Consider warping in additional Stalkers as well for support against any enemy Stalker reinforcements. The Shield Battery and Pylon construction must be prioritized for sustainability. Deploy more Probes for resource gathering if necessary to maintain unit production. Proceed with caution to avoid detection by enemy Queens in the vicinity. We need to overwhelm the enemy quickly.''')>

Example of  Response (Agent Developer, Easy Build Mode) Example of  Response (Agent Developer, Easy Build Mode)

Analysis:

Analysis: We have a strong mineral and gas economy with 2820 minerals and 282 vespene gas. Our supply is currently at 58 out of 102, leaving us with 44 supply available. We have 2 Gateways that are both currently working, which limits our unit production. The Commander has instructed us to initiate the Warp Gate technology upgrade and train additional Zealots, as well as construct an extra Gateway to enhance our unit production capabilities. Given our current resources and the need to bolster our forces, we should prioritize the Warp Gate research, train as many Zealots as possible, and build an additional Gateway. We have a strong mineral and gas economy with 2820 minerals and 282 vespene gas. Our supply is currently at 58 out of 102, leaving us with 44 supply available. We have 2 Gateways that are both currently working, which limits our unit production. The Commander has instructed us to initiate the Warp Gate technology upgrade and train additional Zealots, as well as construct an extra Gateway to enhance our unit production capabilities. Given our current resources and the need to bolster our forces, we should prioritize the Warp Gate research, train as many Zealots as possible, and build an additional Gateway.

Actions:

<Build_Gateway()>

Communications:

your directives.''')> Communications:

Communications: <MessageTo(Commander, '''Initiating Warp Gate technology upgrade, training additional Zealots, and constructing an extra Gateway to enhance unit production capabilities. All actions are aligned with <MessageTo(Commander, '''Initiating Warp Gate technology upgrade, training additional Zealots, and constructing an extra Gateway to enhance unit production capabilities. All actions are aligned with your directives.''')>

Communications: <MessageTo(Commander, '''Engaging SCVs as instructed. Targeting the damaged SCV first. Will report any changes in enemy movements.''')> <MessageTo(Commander, '''Engaging SCVs as instructed. Targeting the damaged SCV first. Will report any changes in enemy movements.''')>

Example of  Response (Agent Commander, Standard Control Mode)

Analysis:

The current situation shows that we have a total of 5 Zealots engaged in combat against enemy SCVs. The enemy has a significant number of SCVs (9) and is operating a Barracks and an Orbital Command, indicating they are likely preparing for a larger military presence. Our resources are sufficient to train additional units, and we are currently researching Warp Gate technology. Given that the game time is 6:12, we should continue to disrupt the enemy's economy while preparing for a larger assault.

Actions:

<All_Units_Attack()>

Communications: Analysis:

<MessageTo(CombatGroup0, '''Continue targeting SCVs to disrupt their economy. Reinforcements are on the way, and we will prepare for a larger assault once we have sufficient forces. Maintain pressure!''')> The current situation shows that we have a total of 5 Zealots engaged in combat against enemy SCVs. The enemy has a significant number of SCVs (9) and is operating a Barracks and an Orbital Command, indicating they are likely preparing for a larger military presence. Our resources are sufficient to train additional units, and we are currently researching Warp Gate technology. Given that the game time is

<MessageTo(Developer, '''Initiate training of additional Zealots and continue Warp Gate research. We need to strengthen our forces for the upcoming attack. Current resources are sufficient for multiple unit trainings.''')> 6:12, we should continue to disrupt the enemy's economy while preparing for a larger assault. Actions:

Example of  Response (Agent Commander, Standard Control Mode)

Communications:

<MessageTo(CombatGroup0, '''Continue targeting SCVs to disrupt their economy. Reinforcements

Example of  Response (Agent CombatGroup0, Standard Control Mode)

are on the way, and we will prepare for a larger assault once we have sufficient forces. Maintain pressure!''')>

Analysis:

<MessageTo(Developer, '''Initiate training of additional Zealots and continue Warp Gate research.

The current situation shows that Team Zealot-1 has already engaged the enemy and is in close

We need to strengthen our forces for the upcoming attack. Current resources are sufficient for multiple proximity to several enemy units, including SCVs and a BarracksTechLab. The Commander has

unit trainings.''')>

instructed to focus on eliminating SCVs quickly, which are vulnerable and have low health. The Zealots are in a good position to attack the SCVs, especially since one of them is already damaged. The

previous action of selecting a unit to attack another unit is still in progress, and the movement towards

[14, 12] is also relevant for positioning.

Actions: Team Zealot-1: Example of  Response (Agent CombatGroup0, Standard Control Mode)

<Select_Unit_Attack_Unit(0x1020c0001, 0x101680001)>  # Targeting the SCV with 45 health

<Attack_Unit(0x101680001)>  # Attack the SCV <Move_Screen([14, 12])>  # Move to a better position if needed Communications: <MessageTo(Commander, '''Engaging SCVs as instructed. Targeting the damaged SCV first. Will report any changes in enemy movements.''')> The current situation shows that Team Zealot-1 has already engaged the enemy and is in close proximity to several enemy units, including SCVs and a BarracksTechLab. The Commander has instructed to focus on eliminating SCVs quickly, which are vulnerable and have low health. The Zealots are in a good position to attack the SCVs, especially since one of them is already damaged. The previous action of selecting a unit to attack another unit is still in progress, and the movement towards [14, 12] is also relevant for positioning.

Analysis:

Actions:

Team Zealot-1:

<Select_Unit_Attack_Unit(0x1020c0001, 0x101680001)>  # Targeting the SCV with 45 health

<Attack_Unit(0x101680001)>  # Attack the SCV

<Move_Screen([14, 12])>  # Move to a better position if needed

Communications:

<MessageTo(Commander, '''Engaging SCVs as instructed. Targeting the damaged SCV first. Will report any changes in enemy movements.''')>

Example of  Response (Agent Developer, Standard Build Mode)

Analysis:

We currently have 49 supply used and 21 supply left, which means we can train more units. We have 3850 minerals available, allowing us to build additional structures and train more Zealots. The Builder is already constructing a Pylon, which will help with our supply issues. However, we need to prioritize building a Cybernetics Core and additional Gateways to enhance our unit production capabilities. The Commander has recommended sending our Zealots to attack the enemy SCV, which will disrupt their operations. We should also continue training Zealots to prepare for a larger attack. Example of  Response (Agent Developer, Easy Build Mode)

Actions: Analysis:

<Train_Zealot()>

<Train_Zealot()>

<Train_Zealot()>

<Train_Zealot()>

<Train_Zealot()>

<Build_CyberneticsCore_Screen(screen)>  # Note:This action will not be executed by Developer

<Build_Gateway_Screen(screen)>               # Note:This action will not be executed by Developer

Communications: <Train_Zealot()>

<MessageTo(Builder, '''Please prioritize the construction of the Cybernetics Core and additional Gateways to maximize our unit production capabilities. We will send our Zealots to attack the enemy SCV to disrupt their operations. Continue building as planned.''')> <Train_Zealot()> <Train_Zealot()> <Train_Zealot()>

<MessageTo(Commander, '''We are training additional Zealots and constructing the Cybernetics Core and Gateways as per your recommendation. We will prepare for a larger attack soon.''')> <Train_Zealot()> <Build_CyberneticsCore_Screen(screen)>   # Note: This action will not be executed by Developer

We currently have 49 supply used and 21 supply left, which means we can train more units. We have

3850 minerals available, allowing us to build additional structures and train more Zealots. The Builder

is already constructing a Pylon, which will help with our supply issues. However, we need to prioritize

building a Cybernetics Core and additional Gateways to enhance our unit production capabilities. The

Commander has recommended sending our Zealots to attack the enemy SCV, which will disrupt their

operations. We should also continue training Zealots to prepare for a larger attack.

Actions:

<Build_Gateway_Screen(screen)>                # Note: This action will not be executed by Developer

<MessageTo(Builder, '''Please prioritize the construction of the Cybernetics Core and additional

Gateways to maximize our unit production capabilities. We will send our Zealots to attack the enemy

Example of  Response (Agent Builder, Standard Build Mode)

SCV to disrupt their operations. Continue building as planned.''')>

<MessageTo(Commander, '''We are training additional Zealots and constructing the Cybernetics

Analysis:

Our current base is located at the minimap position [15, 27]. We have a Nexus at [16, 5] and several

Pylons nearby. The last actions attempted to build a Gateway and a Pylon, but the Gateway construction

Core and Gateways as per your recommendation. We will prepare for a larger attack soon.''')>

failed. We need to prioritize building a Cybernetics Core and an additional Gateway as per the

Developer's instructions. We have enough minerals (3850) and supply (21 left) to build these structures.

The Pylon at [12, 14] is currently under construction (89% complete), so we should wait for it to finish

before building more structures in that area. Example of  Response (Agent Builder, Standard Build Mode)

Actions: Analysis:

Team Builder-Worker-1: <Build_CyberneticsCore_Screen([12, 15])> <Build_Gateway_Screen([12, 16])> <Move_Screen([12, 13])> Communications: Our current base is located at the minimap position [15, 27]. We have a Nexus at [16, 5] and several Pylons nearby. The last actions attempted to build a Gateway and a Pylon, but the Gateway construction failed. We need to prioritize building a Cybernetics Core and an additional Gateway as per the Developer's instructions. We have enough minerals (3850) and supply (21 left) to build these structures. The Pylon at [12, 14] is currently under construction (89% complete), so we should wait for it to finish before building more structures in that area.

<MessageTo(Developer, '''Initiating construction of Cybernetics Core and additional Gateway. Will

keep you updated on progress.''')>

Actions:

Team Builder-Worker-1:

<Build_CyberneticsCore_Screen([12, 15])>

<Build_Gateway_Screen([12, 16])>

<Move_Screen([12, 13])>

Communications:

<MessageTo(Developer, '''Initiating construction of Cybernetics Core and additional Gateway. Will keep you updated on progress.''')>

C5. Examples of Received Communication Messages

Received Communication of agent Commander (first 20 steps, partial)

{"0":""}

{"1":"Communication information:\n\tFrom Developer:\n\t\tInitiating Chrono Boost on Nexus to enhance mineral production. Current resources are low, and I will focus on building supply structures as soon as resources allow.\n\n"}

{"2":"Communication information:\n\tFrom Developer:\n\t\tInitiating construction of a Pylon to increase supply capacity. Once completed, I will prioritize building a Gateway for unit production as per your instructions.\n\n"}

{"3":"Communication information:\n\tFrom Developer:\n\t\tPylon construction is underway to increase supply capacity. I will initiate Chrono Boost on the Nexus to enhance mineral production as per your instructions. We will prepare to construct a Gateway as soon as resources allow. Confirming the plan.\n\n"}

{"4":"Communication information:\n\tFrom Developer:\n\t\tPylon construction is nearly complete. I will prioritize the construction of a Gateway immediately after. We will enhance our military presence to prepare for potential enemy engagements.\n\n"}

{"5":"Communication information:\n\tFrom Developer:\n\t\tInitiating Chrono Boost on the Nexus to enhance mineral production. Building an additional Pylon to increase supply capacity. Gateway construction is ongoing. Will keep you updated on resource availability.\n\n"}

{"6":"Communication information:\n\tFrom Developer:\n\t\tWe are prioritizing the completion of the Gateway and will build an additional Pylon to increase supply capacity. This will enable us to train military units as soon as the Gateway is operational. Current resources are being managed effectively.\n\n"}

{"7":"Communication information:\n\tFrom Developer:\n\t\tThe Gateway is currently at 64% completion. I will use Chrono Boost on the Nexus to enhance mineral production. We are also building an additional Pylon to increase our supply capacity. I will keep you updated on resource availability.\n\n"}

{"8":"Communication information:\n\tFrom Developer:\n\t\tWe are prioritizing the construction of an additional Pylon and the Gateway to ensure we can start training military units promptly. The Gateway is currently 87% complete. We will keep you updated on our progress.\n\n"}

{"9":"Communication information:\n\tFrom Developer:\n\t\tThe Gateway is operational, and I am initiating the construction of a Nexus to expand our economy. I will also train five Zealots to bolster our forces. Chrono Boost will be activated on the Nexus as soon as it is completed to enhance mineral production. We are preparing for unit production swiftly.\n\n"}

{"10":"Communication information:\n\tFrom Developer:\n\t\tWe are currently low on resources and cannot train units or build additional structures at this time. I will construct a Pylon to increase our supply capacity as soon as we gather enough minerals. We will prioritize the completion of the Gateway and initiate unit training once resources allow. Thank you for your guidance.\n\n"} ......

{"20":"Communication information:\n\tFrom Developer:\n\t\tWe are training additional Zealots to bolster our forces while the new Nexus and Pylon are under construction. We will be ready for potential enemy engagements shortly.\n\tFrom CombatGroup0:\n\t\tTeam Zealot-1 is maintaining defensive positions and ready to engage if any enemy units are spotted. We are moving slightly forward to enhance our defensive stance. Awaiting further instructions.\n\n"}

Received Communication of agent Commander (steps 25 to 105, partial)

{"25":"Communication information:\n\tFrom Developer:\n\t\tWe are training additional Zealots and constructing another Gateway to enhance our unit production capabilities. The process is being expedited as we prepare for potential engagements. Current resources are sufficient for this operation.\n\tFrom CombatGroup0:\n\t\tTeam Zealot-1 is engaging an enemy Probe and will maintain defensive positions as instructed. We will report any further threats detected.\n\n"}

{"35":"Communication information:\n\tFrom Developer:\n\t\tWe are training additional Zealots and building a Pylon to support our supply needs. The Cybernetics Core is nearing completion, and we will be ready for an offensive shortly.\n\tFrom CombatGroup0:\n\t\tContinuing to maintain pressure on the enemy. No enemy units sighted yet. Moving towards the designated coordinates for regrouping. Will report any sightings immediately.\n\n"}

{"45":"Communication information:\n\tFrom Developer:\n\t\tWe are training additional Zealots and building another Assimilator to enhance gas production. This will help us maintain momentum for future upgrades and unit production. We will keep you updated on our progress.\n\tFrom CombatGroup0:\n\t\tWe are continuing to engage the Zergling and holding our position as instructed. Reinforcements are welcomed. We will remain vigilant for any incoming threats.\n\n"}

{"55":"Communication information:\n\tFrom Developer:\n\t\tWe are initiating Warp Gate research and training additional Zealots as per your request. Expect reinforcements shortly for the upcoming offensive.\n\tFrom CombatGroup0:\n\t\tWe are maintaining pressure on the enemy and preparing for a coordinated attack. Current position is [22, 37]. Reinforcements are expected soon. Will continue to engage.\n\n"}

{"65":"Communication information:\n\tFrom Developer:\n\t\tWe are constructing a Pylon to increase our supply capacity and training additional Zealots as requested. We will maintain pressure on the enemy. Expect reinforcements shortly.\n\tFrom CombatGroup0:\n\t\tWe are engaging the enemy unit and will continue to press the attack. Moving to a better position for a coordinated assault.\n\n"}

{"75":"Communication information:\n\tFrom Developer:\n\t\tWe are building a Pylon to increase supply capacity and warping additional Zealots as requested. Reinforcements will be ready shortly. Maintain offensive momentum!\n\tFrom CombatGroup0:\n\t\tWe are engaging the ChangelingZealot and will prepare for the coordinated attack. Awaiting additional Zealots for support.\n\n"}

{"85":"Communication information:\n\tFrom Developer:\n\t\tWe are initiating the warp-in of additional Zealots to reinforce our frontline. A Pylon is being constructed to increase our supply capacity. Expect reinforcements shortly.\n\tFrom CombatGroup0:\n\t\tWe are currently engaging enemy Changeling Zealots. Holding position and preparing for follow-up attack as instructed. Will remain vigilant against potential Mutalisks and Zerglings.\n\n"}

{"95":"Communication information:\n\tFrom Developer:\n\t\tBuilding a Pylon to increase supply and will initiate Charge upgrade for Zealots immediately after. We will warp in additional Zealots as soon as supply allows.\n\tFrom CombatGroup0:\n\t\tEngaging the Mutalisks and focusing fire on the Changeling Zealot as instructed. Reinforcements are welcomed!\n\n"}

{"105":"Communication information:\n\tFrom Developer:\n\t\tWe are constructing an additional Pylon to increase our supply capacity and have initiated research on Warp Gate technology to expedite unit training. Reinforcements will be ready shortly.\n\tFrom CombatGroup0:\n\t\tWe are holding our position and focusing fire on the Mutalisks as instructed. Reinforcements are awaited. We will stay strong!\n\n"}

......

Received Communication of agent Commander (steps 105 to 125, partial)

{"115":"Communication information:\n\tFrom Developer:\n\t\tWe are constructing an additional Pylon to increase our supply capacity and will warp in Zealots immediately to counter the Mutalisks. Expect reinforcements shortly.\n\tFrom CombatGroup0:\n\t\tWe are focusing fire on the Mutalisks as instructed and holding our position. Reinforcements are awaited.\n\n"}

{"125":"Communication information:\n\tFrom Developer:\n\t\tWe are constructing an additional Pylon to increase our supply capacity for reinforcements. Over.\n\tFrom CombatGroup0:\n\t\tWe are engaging the Mutalisks and securing the area as instructed. Current status:attacking the closest Mutalisk and moving to a safer position. Over.\n\n"}

{"135":"Communication information:\n\tFrom Developer:\n\t\tWe are constructing additional Pylons to increase our supply capacity. This will allow us to train more units and strengthen our forces. Over.\n\tFrom CombatGroup0:\n\t\tWe are engaging the enemy Changeling Zealots and pushing forward. The Nexus is under attack, but we will maintain pressure on the enemy. Over.\n\n"}

{"145":"Communication information:\n\tFrom Developer:\n\t\tWe are warping in additional Zealots as requested to strengthen our frontline. The new Pylon is under construction and will be completed shortly. We are also researching Warp Gate technology to enhance our capabilities. Over.\n\tFrom CombatGroup0:\n\t\tWe are engaging the enemy Hydralisks and Roach as instructed. Continuing to apply pressure. Over.\n\n"}

{"155":"Communication information:\n\tFrom Developer:\n\t\tWe are warping in additional Zealots and Adepts to strengthen our forces. An Assimilator is being constructed to enhance gas income, and Warp Gate technology is being researched to improve unit production efficiency. Over.\n\tFrom CombatGroup0:\n\t\tWe are engaging the Mutalisks as instructed and will provide cover for the advancing Zealots. Over.\n\n"}

{"165":"Communication information:\n\tFrom Developer:\n\t\tReinforcements are being warped in. Four additional Zealots will be ready shortly to strengthen our forces. Over.\n\tFrom CombatGroup0:\n\t\tWe are engaging the Mutalisks as instructed and will focus fire to eliminate them quickly. Over.\n\tFrom CombatGroup7:\n\t\tWe are engaging the enemy Changeling Zealot and will scout for additional units. Maintaining pressure on their production. Over.\n\n"}

{"175":"Communication information:\n\tFrom Developer:\n\t\tPylon construction initiated to increase supply. Warping multiple Stalkers to counter air threats as per your directive. Over.\n\tFrom CombatGroup0:\n\t\tWe are focusing fire on the Mutalisks as instructed and moving to a safer position. Reinforcements are welcomed. Over.\n\tFrom CombatGroup1:\n\t\tWe are engaging the enemy forces while ensuring the safety of our units. Moving to a safer position near the Pylon. Requesting support if needed.\n\tFrom CombatGroup7:\n\t\tWe are currently engaging the Mutalisk and Zergling. Once these threats are neutralized, we will proceed to disrupt the enemy's economic structures as instructed. Over.\n\n"}

{"185":"Communication information:\n\tFrom Developer:\n\t\tWe are warping additional Zealots and Stalkers to reinforce our forces. A Pylon is also being constructed to increase supply capacity. Reinforcements will be ready shortly. Over.\n\tFrom CombatGroup0:\n\t\tWe are continuing to focus fire on the Mutalisks and Zerglings as instructed. Moving to a strategic position for better coverage. Over.\n\tFrom CombatGroup1:\n\t\tWe are engaging the Mutalisks as instructed. Focusing fire on the one with low health. Over.\n\tFrom CombatGroup7:\n\t\tWe are currently engaging the Mutalisk and Zergling. Once these threats are neutralized, we will proceed to disrupt the enemy's economic structures as instructed. Over.\n\n"}

.......

C6. Examples of Communication Actions

Communication Messages from agent Commander (steps 0 to 200, partial)

{"0": "Communications:\n    <MessageTo(Developer, '''We need to prioritize building a Gateway for unit production. Currently, we have enough Probes for resource gathering, but we need military units to prepare for potential enemy encounters. Please initiate construction of a Gateway as soon as possible.''')>"}

{"10": "Communications:\n    <MessageTo(Developer, '''We need to prioritize gathering minerals to increase our supply capacity and complete the Gateway. Please proceed with constructing a Pylon as soon as we have enough resources.''' )>"}

{"20": "Communications:\n    <MessageTo(Developer, '''Please prioritize the training of additional Zealots to strengthen our frontline. We need to be prepared for any potential enemy engagements as we expand our economy.''' )>\n    <MessageTo(CombatGroup0, '''Maintain your defensive positions and continue to scout for enemy movements. Be ready to engage if any threats are detected.''' )>"}

......

{"90": "Communications:\n    <MessageTo(Developer, '''Please prioritize building a Pylon to increase our supply capacity. We need to warp in additional Zealots to strengthen our forces against the enemy's aerial units.''' )>\n    <MessageTo(CombatGroup0, '''Coordinate with the Zealots to prepare for a potential assault from the Mutalisks. Ensure we have cover for our ground units and be ready to reposition as necessary.''' )>"}

{"100": "Communications:\n    <MessageTo(Developer, '''Please initiate the construction of additional Pylons to increase our supply capacity for more Zealots. We need to prepare for a potential offensive against the enemy.''' )>\n    <MessageTo(CombatGroup0, '''Continue focusing fire on the Mutalisks and maintain defensive positions. We will reinforce with additional Zealots shortly.''' )>"}

......

{"170": "Communications:\n    <MessageTo(CombatGroup0, '''Continue focusing fire on the Mutalisks and regroup for a coordinated attack. We need to eliminate them quickly to reduce pressure on our forces. Over.''')>\n    <MessageTo(CombatGroup7, '''Prepare to disrupt enemy economic structures as soon as the Mutalisks are dealt with. Your infiltration will be crucial for our victory. Over.''')>\n <MessageTo(Developer, '''Reinforcements are needed as soon as supply allows. Focus on building additional Zealots to maintain pressure on the enemy. Over.''')>"}

{"180": "Communications:\n    <MessageTo(CombatGroup0, '''Continue focusing fire on the Mutalisks. Ensure they are neutralized to protect our ground forces. Reinforcements are on the way. Over.''')>\n <MessageTo(CombatGroup1, '''Maintain pressure on the Overseer. Once it is neutralized, regroup with the Zealots to assist in taking down the remaining air threats. Over.''')>\n    <MessageTo(Developer, '''We need to prioritize warping in additional Stalkers to counter the Mutalisks and build a Nexus for economic expansion. Over.''')>"}

{"190": "Communications:\n    <MessageTo(CombatGroup0, '''Continue focusing fire on the Mutalisks and engage any Zerglings that approach. Maintain optimal positioning for maximum effectiveness. Over.''')>\n    <MessageTo(CombatGroup1, '''Keep engaging the Mutalisks and use Blink to reposition as necessary. Ensure the Zealots are covered. Over.''')>\n    <MessageTo(Developer, '''Pylon construction is critical to address our supply issues. Please expedite the warping of additional Stalkers to counter the air threats. Over.''')>"}

Communication Messages from agent Developer (Easy Build mode, steps 0 to 200, partial)

{"0": "Communications:\n    <MessageTo(Builder, '''We need to gather more minerals to build a Pylon and expand our supply. Please focus on mining to increase our resources.''' )>\n <MessageTo(Commander, '''Current status: We are low on resources and unable to build any military or research buildings. Requesting support to gather more minerals.''' )>"}

{"10": "Communications:\n    <MessageTo(Builder, '''Confirming the construction plan for the Gateway. Please ensure it is built immediately after the Pylon is completed. We need to ramp up our military unit production as soon as possible.''' )>\n    <MessageTo(Commander, '''We are currently waiting for resources to complete the Gateway. Probes are gathering minerals to expedite the process. Will keep you updated on the construction status.''' )>"}

{"20": "Communications:\n    <MessageTo(Builder, '''Please proceed with building the Pylon to increase our supply capacity. We need to ensure we can train more units as soon as possible.''' )>\n <MessageTo(Commander, '''We are prioritizing the training of additional Zealots and will build a Pylon to increase supply. We will also focus on gathering Vespene gas for future upgrades.''' )>"}

{"30": "Communications:\n    <MessageTo(Builder, '''Please proceed to build the Cybernetics Core immediately after the Pylon is completed. We need to enhance our technology for unit upgrades. Keep me updated on the progress.''')>\n    <MessageTo(Commander, '''We are training additional Zealots and will build the Cybernetics Core as soon as the Pylon is complete. Defensive posture is maintained with current units. Will keep you updated on our progress.''')>"}

......

{"120": "Communications:\n    <MessageTo(Builder, '''Please proceed to build an Assimilator immediately after the Pylon is completed. We need to ensure a steady gas supply for future unit production and upgrades.''')>\n    <MessageTo(Commander, '''We are prioritizing the warp-in of additional Zealots to maintain a defensive posture as instructed. Research on ground weapons and armor is also underway.''')>"}

......

{"180": "Communications:\n    <MessageTo(Builder, '''Continue building the additional Gateways at [15, 12] and [15, 13]. We need to increase our unit production capacity quickly.''')>\n <MessageTo(Commander, '''We are warping in additional Zealots and Adepts to bolster our forces. Researching ground weapons and armor for enhanced combat effectiveness.''')>"}

{"190": "Communications:\n    <MessageTo(Builder, '''Please continue building additional Pylons and an Assimilator to increase our supply capacity and gas income. We need to ensure we can support our unit production. Also, confirm the status of the Pylon construction.''' )>\n    <MessageTo(Commander, '''The previous attack has completed successfully. We are currently reinforcing our units and upgrading our technology. We will continue the assault once we have sufficient forces and upgrades.''' )>"}

{"200": "Communications:\n    <MessageTo(Builder, '''Please prioritize building additional Pylons to increase our supply capacity. We need to ensure we can support our upcoming assault with enough reinforcements. Confirm once completed.''')>\n    <MessageTo(Commander, '''We are preparing reinforcements with Stalkers and Sentries. Upgrades for ground weapons and armor are also in progress. We will maintain pressure on the enemy's Hatchery and Overseers as planned.''')>"}

Appendix D. Experimental Settings with Multi-Agent Settings

Appendix E. Examples of the problems in LLM decision-making

E.1 Hallucination Examples in Complete StarCraft II Games (standard contorl mode)

our zealots

enemy sentry

715

3130

74

1392

/115

59/94

Z15

2牌

6.43/1352

12

技(A)健括

地面自动市

Example-1, misjudge the situation of unit strength (before query) . 2 Zealots engage with a lot of enemy units.

enemy units

(long-range attack)

our units

(melee attack)

435

240*

452

29/39

33/47

所有化家

所MAg

cChrgre

A

[11.12

cul Lco

1966code

o.

21092254

28

125 / 14.50

(A然后

地国自动

Example-2, poor understanding of combat mechanism (before query) . 3 Zealots engage with 5 long-range-attack units.

enemy air unit

our zealots

(melee attack，ground only)

2427

4840

744

1430

67/102

78/102

：S1p(11:75)

1.110

Z18181142）

Str- p19(184）

11.52 / 14.50

铁药

联单位

Example-3, try to attack unattackable units (before query) . CombatGroup0 find an enemy Raven in the road to enemy base.

all zealots lost

3120

640

78

1316

57/94

85/15

所有玩家

E: Zealot-1 Step63 (06:64:20)

Tain_Zealot()

所有玩家

所有现家

eTrain_Stakes

含: Zealot-1 Step53 (06:64:46)

：Zealot-15p0(D6:4:42)

Zealot- Step53 (06:44.51)

5.46 /1352

M12.12]

所有玩家

:Zealet-1.5hep63 (06.46.73)

探机

跑

Example-1 misjudge the situation of unit strength (after executing actions) . LLM tries to kill the sentry but is killed by enemy units.

3 <Move_Screen(

[11, 12]

)>

1 <Attack_Unit(

0x101dc0001

)>

2 <Attack_Unit(

0x102280001

)>

33/47

33/39

[所有以家]

10rde0001s

MLLMPyt2:Zealo-1 Step27 (00:26:25)

Zeslot-1 Pep27 (00:26:74)

Z127(002679）

1,Pep27 (D0:26.07)

327 / 1450

按（A)健后

地国自动项市

Example-2, poor understanding of combat mechanism (after executing actions) . LLM tries to kill two units then retreat but finally fails.

cannot attack air units,

but try to attack

<Attack_Unit(

0x103880002

)>

2442

4875

760

1438

67/102

78/102

11

LLMPyc2-Zed51

ssndm

nSkBp119（11:5415）

11.54 / 14.5)

有家M

Alck

（A然后

地国自动欢选

Example-3, try to attack unattackable units (after executing actions) . LLM tries to kill the Raven using ground units melee attack.

Response of CombatGroup0 in Example-1

Analysis:

The current situation shows that Team Zealot-1 is engaged with enemy units, specifically two enemy Zealots and a Sentry with low health. The Commander has instructed to maintain pressure on the enemy Probe and prepare for engagement with the Stalkers. The enemy Sentry is vulnerable with only 26 health remaining, making it a prime target. Additionally, the enemy Probe has 40 health and is also a valid target. Concentrating fire on the Sentry first will likely eliminate it quickly, allowing for a more effective engagement with the Stalkers afterward.

Actions: Team Zealot-1: <Select_Unit_Attack_Unit(0x102d00001, 0x102d00001)>  # Attack the Sentry <Select_Unit_Attack_Unit(0x101580001, 0x101580001)>  # Attack the Probe <Move_Screen([12, 12])>  # Move to a better position for engagement ）

E.2 Hallucination Examples in Complete StarCraft II Games (develop and build)

game time 5:23

start building

Cybernetics Core

61/86

6202 /62s

[所有玩家]

Move_Sereen[17,10]

nt（0x102480001

Zeot-1S45(05:18:30)

传送

Example-4, build important building too late . Cybernetics Core usually built before 3min, without this building, most unit will be locked.

too many zealots

which can not attack air units

hard to defeat heavy units

game time 7:26

A2/78

eTrain,Staked

历玩家

<Train_Zealons

有玩家

有玩

<Trar_Stakero

有玩家

726/2029

Train_Zewlotja

Example-5, train too many low-level units (before query) . Developer has already trained a lot of zealot at game time 7:26.

a lot buildings built

near the first Nexus

15:00/2127

Example-6, build too many buildings around base building 1 (standard build mode, screen of base building 1) . Builder only build around the first base and they do not build enough unit training buildings due to frequently calling for build at invalid position.

first

Cybernetic Core

game time 5:43

9261

0

67/94

2601

1111

29/47

1:ep48(05:38:48)

ot-1Sp4805

Zealot-1 Step48 (05:38:56)

542 / 2029

Move_Screen[17,10]

[所有玩家]

ot-1 Step48 (05:38.62)

Step48 (05:39.11)

控制芯

Example-4, build important building too late. Construction of Cybernetics Core finishes after game time 5:40.

<Train_Zealot()>

31C

222

1207

开自购

斯有动购

所有

127 /225

传送

Example-5,  train too many low-level units (after executing actions) . LLM call for Zealot training even after Stalker/Adept/Sentry available.

nearly no building built

around the second Nexus

12430

1690

108/148

onoe

4677

60/70

15:00/21:27

Nexus

Example-6, build too many buildings around base building 1 (standard build mode, screen of base building 2) . There is enough area for build buildings around the second base. But the Builder never ask the probe go to the second base to build buildings.

Response of Developer in Example-5

Analysis:

We currently have 13 supply left, which is quite low. We need to build a Pylon to increase our supply capacity. Additionally, we have a significant amount of minerals (3230) and some gas (218), allowing us to train more units. The Commander has requested the expedited training of additional Zealots, which aligns with our current strategy to maintain pressure on the enemy...

Actions:

Communications:

<MessageTo(Commander, '''We are building a Pylon to increase supply and training additional Zealots to maintain pressure on the enemy. ''')>"

Hallucination Analysis of the Response in Example-5

Agent Commander do not realise that we already have too mant Zealots, and continues sending messages to Developer to train more Zealots.

Agent Developer completely followed the commander's intention but lost his own thinking. Even the Developer knows that actions like <Train_Stalker()> and <Train_Adept()> is available, it do not send the information to Commander and ask for pemission for training other units.

E.3 Hallucination Examples in micro-operation scenarios

Adept

shadows

Enemy

units

Example-7, skill use (correct strategy) .

Adepts should use the shadow to avoid damages.

enemy workers

(can noly

attack air units)

skill GravitonBeam

(raise a unit into air)

Example-8, skill use (before query) . Last step Pheonixes use its skill GravitonBeam raise a worker to the air and finished it successfully.

3 Disruptors

(powerful area

attack units)

0:10

screen [32, 32]

(code v0.1, screen 128x128)

minimap

[32, 32]

A

Enemy

units

(direct move

to enemy base)

ManAgnLLuPyic2

<Attack_Unit(0x1012c0001)>

enemy anti-air

units

two abandoned

the teammates

one Disruptor

killed by enemy

skills available

(they do not use it)

Example-9, collaboration (after executing actions) . Disruptors retreat before using skills to kill any of the enemy unit.

<Ability_PsiStorm_Screen([32, 32])>

HighTemplar

(area attack unit)

Enemy

units

Response of CombatGroup0 in Example-8

**Analysis:**

The task is clear: we need to kill as many enemy Drones as possible until all units are dead.

**Team Phoenix-1** has three units, two of which are low on health, and one with significant health remaining. We also need to pay attention to the proximity of the enemy Drones to maximize damage.

The Drones are spread across the map, but the most immediate threat is the Drone at [9, 8] which is directly in range (distance 0) from the team. The other nearby Drones are also very close, at distances of 3 to 7.

Two Phoenix units are under attack, and one is being healed. The situation suggests that focusing on eliminating nearby Drones quickly is the best course of action.

**Strategy:**

Focus fire on the Drone at [9, 8], as it is closest and will reduce the enemy's presence on the map significantly.

After eliminating the Drone at [9, 8], shift attention to the Drone at [9, 12] and [8, 13] (next closest units).

Prioritize attacking Drones over other enemy units (like the Hatchery or Queens) to complete the task efficiently.

**Actions:**

Team Phoenix-1:

<Attack_Unit(0x101380001)>  # Target the Drone at [8, 13] to ensure damage continues

The actions focus on eliminating the nearest Drones and continuing the task of killing as many Drones as possible.

Hallucination Analysis of the Response in Example-8

(1) As shown in the observation text, no Drone has the buff 'Buffs.GravitonBeam',which means all the enemy workers are on the ground, and our Pheonix can not attack ground units.

(2) As shown in Fig. E6 (3), the agent used to release the skill

<Select_Phoenix_Ability_GravitonBeam_Unit(tag, tag)>

to successfully killed a worker, but it do not realise this skill is the key to make enemy units attackable.

Appendix F. Examples pf Structured StarCraft II Wiki Knowledge

Appendix G. Other supplementary materials

Limitations.

Although the LLM-PySC2 environment provides a lot more features than existing environments, there are still some limitations. For example: (1) We found problems in LLM-based decision-making but have not yet found a solution that is good enough. How to learn domain-specific knowledge in the environment effectively remains a problem. (2) Due to the limited decision-making ability of the LLMs, many professional operations (such as multi-line attacks, invisible units harassment, airdrop, changing homes, etc.) have not appeared in our experiments. As a result, it is quite challenging for us to test these advanced tactics, leading to potential bugs when dealing with advanced strategies. (3) Considering that Protoss is enough for decision-making method verification, we suspend the work of Terran and Zerg. Full support for these two races will be provided in the future.

Impact Statements.

This work presents a new environment for evaluating LLM decision-making performance and developing learning algorithms. With complete StarCraft II action space, our environment provides far more complexity than other decision-making platforms. With the asynchronous query architecture, the environment efficiently interacts with LLMs that maintain a constant latency regardless of the scale of the agents' population. Advancement in the environment reduces the obstacles of large model decision research and prepares conditions for post-training LLMs in sequential decision problems. The phenomena observed in the experiments will attract more attention to the LLM hallucinations and lack of knowledge in specific domains.

Broader Impact.

Our work opens up new opportunities for applying LLM in gaming and other multi-agent collaboration applications. Given that a lot of work on LLM learning methods is still supervised learning or human feedback reinforcement learning on data sets, our environment may promote the research of LLM in-environment learning. Potential negative impacts include misuse of the environment to develop autonomous decision-making systems without enough safety control and human control interfaces, which may generate uncontrolled, unexpected, or dangerous behavior in the application. We suggest strengthening the research on LLM security while promoting the research on large-model autonomous decision making.

Code Of Ethics.

This research fully adheres to the NeurIPS Code of Ethics. Violence behaviors of LLMs, such as attacking and harassing are confined entirely to the Real-Time Strategy game StarCraft II, just like previous similar works like SC2LE and SMAC. Potential risks of misuse and our suggestions are discussed in Section Broader Impact . This work does not involve or address any topics related to discrimination, races mentioned in the paper are the camps in the StarCraft II game.

Peer reviews:
comments: SUMMARY:
This paper transforms StarCraft II into an environment manageable by multimodal large language models. The author set up various in-game tasks and evaluated different types of large language models within this environment.

STRENGTHS AND WEAKNESSES:
Strengths
- Game AI itself is an interesting topic, and the multi-modal approach aligns with the current development direction of large models. The StarCraft II environment is also quite challenging, making it suitable for evaluation.
- The designed method is detailed and reasonable, and experiments have proven that large models can exhibit a certain level of capability.

Weaknesses
- Lack of intuitive description of the final capabilities. For example, what human-level equivalent was achieved? How do previously proposed agents perform on these tasks?
- Ablation studies for the method design are lacking. For example: The proposal that LLMs should return reasons for decisions—does this improve decision-making ability? By how much?
- The paper suggests in the experimental section that reasoning models similar to GPT-o1 perform poorly in unseen scenarios. This seems to contradict other similar works. For example, Gemini recently used its multi-modal reasoning capabilities to complete the game Pokémon Blue. Perhaps this is because the models used in this paper are not new enough; considering GPT-o3/o4-mini or Gemini-2.5-Pro might be worthwhile.
- The other models used are generally not new enough; further supplementation could be considered.

QUESTIONS:
See weaknesses.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe research introduces LLM-PySC2, a novel StarCraft II learning environment designed for LLMs, addressing the current lack of support for LLMs in complex real-time strategy games. Main features of this environment include complete PySC2 action space, multi-modal information, game knowledge from a Wiki and an asynchronous query architecture to manage interactions with multiple LLM agents. Experiments show that while LLMs can make decisions, their performance can be inconsistent, exhibiting issues like hallucinations and inefficient collaboration due to a lack of domain-specific knowledge and instruction. The authors hope LLM-PySC2 will foster advancements in LLM-based decision-making.\n\nSTRENGTHS AND WEAKNESSES:\n__Strengths__\n\nCompared to the existing benchmark that evaluates LLM policy on SC2, the environment introduced in this paper provides support for complete action space.\nAlso provide support for multi-modal observations such as RGB screen and RGB feature which would be very valuable for benchmarking multi-modal LLMs.\n\n__Weaknesses__\n\nIt’s hard to assess the novelty/impact of this paper due to the limited evaluation presented in this paper. For example, the author mentioned the main contribution of this work as the support for complete action spaces. While this makes the environment more challenging, it’s difficult to understand immediately what kind of failure modes this immediately exposes that future work can research on.\n\n__Limited evaluation__\n\nThe evaluations in this paper seem restricted to comparison between different LLM backbones. This offers limited insights on how LLM-backed agents compare with existing state-of-the-art RL agents (e.g. AlphaStar etc). Without comparison with more “traditional” RL agents, it is difficult to understand the strengths and weaknesses of LLM agents which I think is crucial for future work.\n\nNevertheless, the authors did mention that “It is not recommended to compare the LLM-based method with the MARL-based method due to different control frequencies.” I am not really sure about what the authors mean. Why can’t we run the LLM and MARL -based agents at their respective control frequency and evaluate the results?\n\nI would like to see how the LLM-based agents perform compared to existing MARL approaches in order for me to raise my score.\n\n__Nit__\n* Rather than putting the illustration for all hallucination examples together. It would be more useful to present the illustration followed by each failure analysis separately. That would make the readers’ life easier when trying to understand the examples.\nFor the tables on evaluation performance. Do you run the evaluation with multiple seeds? If not I would consider doing it since SC2 are highly complex environments where the variance may be huge. If so, it would be great if you can include the confidence intervals.\n\nQUESTIONS:\n__Question on asynchronous query architecture__\n\nCan the authors explain whether the asynchronous query architecture refers solely to calling the LLM APIs concurrently either by multi-threading or asyncio? If so, I would consider this a basic implementation necessity rather than a major feature contribution.\n\n__Questions on wiki knowledge__\n\nThe author mentions that LLM-PySC2 offers game wiki knowledge. I don’t find much details on this other than the information offered as part of the prompt. Can the authors clarify whether this is what they mean by wiki knowledge?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe main contribution of this paper is to propose the LLM-PySC2 environment, which is the first environment to provide a complete PySC2 action space for LLMs, and supports multi-agent collaboration and communication.The article introduces the environment framework, action space, observation space and multi-agent system in detail, and evaluates the performance of LLMs in macro decision-making and micro operation scenarios through experiments.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. This paper proposed the first LLM decision environment supporting a complete PySC2 action space, providing a new research platform for the application of LLMs in complex strategy games.\n\n2. Through experiments in the complete StarCraft II game and the newly designed LLM-PySC2 task group, the potential and limitations of LLMs in complex decision problems, particularly in the areas of hallucinations and insufficient knowledge are revealed.\n\n**Weaknesses**\n\n1. While comparisons have been made with some existing methods, systematic comparisons are lacking for other potentially applicable baseline methods, such as different LLM architectures or traditional reinforcement learning methods.\n\n2. Although the article mentioned the universality of the environment, it did not specifically explain or demonstrate how to support the operations of both the human and Zerg races. The focus of the article is to demonstrate the functions and potential of the LLM-PySC2 environment, mainly illustrating the issue through the case of the Protoss.\n\n3. Although the article provides the framework of the environment and experimental results, the detailed description of how to specifically implement key parts such as action recognition and observation transformation is not in-depth enough. For example, how to map text actions to specific PySC2 function calls? How do regular expressions precisely match various possible action formats, and how to handle situations with format errors or incompleteness.\n\n4. This paper introduces the efficiency performance of the action recognition module in handling large amounts of text and complex actions, as well as how to deal with action texts generated by LLMS that do not meet expectations, such as through what mechanisms to correct or ignore incorrect actions.\n\n5. The paper does not conduct ablation experiments on each component of the LLM-PySC2 environment (such as multimodal observation, asynchronous query architecture, different action space Settings, etc.) to evaluate the contribution of each component to the overall system performance.\n\n6. The main contribution of the paper is to propose an environment rather than a method, which seems more suitable for Benchmark Track rather than Main Track.\n\nQUESTIONS:\n1. A key issue needs to be clarified: the efficiency performance of the action recognition module in handling large amounts of text and complex actions, as well as how to deal with action texts generated by LLMS that do not meet expectations, such as through what mechanisms to correct or ignore incorrect actions.\n\n2. To enhance the observation and conversion efficiency, whether optimization measures have been taken, such as how to reduce data transmission volume and lower processing latency, as well as the impact of these optimizations on the overall system performance.\n\n3. It is necessary to explain how the action recognition and observation conversion module ADAPTS to different game versions, maps and task types, as well as how to conveniently add or modify the types and contents of actions and observations to support new research requirements.\n\n4. The paper does not have specific experimental verifications for independent modules such as action recognition and observation conversion. Although these modules are key components of the environment, no specialized tests have been conducted on their performance, accuracy, and efficiency. Instead, the focus has been placed on the LLMs's performance in the overall decision-making task.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
This work introduces LLM-PySC2--the first StarCraft II environment that offers large language models the full pysc2 action space. The environment also provides multimodal observations and supports native multi‑agent communication through an asynchronous query architecture. Although LLMs demonstrate the potential to win in complicated contexts, it is still hard for them to constantly make correct actions. These could due to the common issues of LLMs such as hallucinations and inefficient collaboration. Most importantly, LLM-PySC2 provides a platform that encourages further research and development of LLM-based StarCraft II agents.

STRENGTHS AND WEAKNESSES:
**Strengths**
1. LLM-PySC2 is the first StarCraft II framework that expose a complete pysc2 action space to LLMs, along with a structured Wiki knowledge database.

2. The environmental setup is clear and reasonable.

3. This work considers representative experimental scenarios including macro-decisions experiments, micro-operations experiments, and evaluates over a board range of LLMs.

4. LLM-PySC2 provides a platform that promotes the development of LLM-based decision-making approaches, especially on StarCraft II.


**Weaknesses**
1. Only the Protoss is considered. Terran and Zerg are not included. 
2. This work does not test the LLMs that are fine-tuned using RL techiniques like PPO, GRPO.
3. Latency and token cost are not reported. 
4. Hallucination analysis is a bit qualitative, with no quantitative evidences.
5. Minor issues: Spacebar is not properly used in Table 2.

QUESTIONS:
1. Please consider adding Terran and Zerg. 
2. Please provide quantitative analyses for hallucination
3. Please evaluate more open-source LLMs e.g., Qwen, Mistral, etc.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-03