StarCraft II Arena: Evaluating LLMs in Strategic Planning, Real-Time Decision Making, and Adaptability

Authors: Wenjie Tang, Yuan Zhou, Erqiang Xu, Keyan Cheng, Minne Li, Zhiyuan Wang
Primary area: datasets and benchmarks
Keywords: benchmark evaluation, large language model, LLM-based agent, strategic reasoning, real-time decision-making.

Abstract:
StarCraft II plays an important role in developing AI agents for real-time strategic reasoning due to its complex nature. However, people usually draw conclusions of how competent their agents are according to the level of the built-in agents in StarCraft II which they can win in terms of the final success rate. Little intermediate quantitative information is considered while human-in-the-loop analysis is time inefficient, which results in inadequate reflection of the true strategic reasoning ability. In this work, we propose StarCraft II Arena, a well-designed benchmark for evaluating the strategic planning, real-time decision-making, and adaptability capabilities of large language models (LLMs) agents. We introduce using fine-grained capability metrics, allowing for targeted capture and analysis of specific capability, and further propose a detailed decision trace to enhance the understanding of LLM behavior. We demonstrate the utility of such a benchmark by evaluating several state-of-the-art LLMs in various setups. Our results reveal distinct performances in long-term strategy development, real-time decision-making, and adapting to environmental changes. Such results show that the StarCraft II Arena offers a deeper insight into the decision-making process of LLMs and has the potential to become a challenging and comprehensive benchmark for strategic reasoning.

Decision: reject
Reviewer scores: 3, 3, 3, 3

Reviews and discussion:
<thread>
creation date:
1730450732593

thread_type:
official_review

<thread_object>
creator: reviewer_hoan

<content>
summary:
summary: The paper presents StarCraft II Arena as a benchmark for evaluating LLMs in strategic planning and decision-making capabilities. While it introduces fine-grained metrics and decision tracking mechanisms, the work needs substantial clarification regarding its novel contributions, metric selection justification, and experimental methodology.

strengths:
1. Well-structured evaluation framework with proposed fine-grained metrics
2. Comprehensive testing across multiple LLM models
3. Detailed decision tracking system for behavior analysis
4. Clear visualization of experimental results
5. Systematic approach to evaluating different aspects of LLM capabilities

weaknesses:
1. Metric Selection and Justification:
- APM/EPM metrics appear borrowed from traditional StarCraft II evaluation without clear justification of their relevance to LLM agents
- No discussion of how these metrics specifically reflect LLM decision-making capabilities
- Missing analysis of whether traditional StarCraft II performance metrics are appropriate for language models
2. Experimental Design Limitations:
- Build-in AI difficulty level not specified
- Race and map selection criteria not documented
- Limited experimental scope (10 games per model)
- Absence of LLM vs LLM experiments
- No evaluation against human players
- Limited testing of open-source models, reducing reproducibility
3. Theoretical Foundation and Novelty:
- Limited discussion of why LLMs are suitable for StarCraft II
- Insufficient comparison between LLM agents and traditional RL agents
- Evaluation metrics show significant overlap with existing work
- Need clearer articulation of novel contributions
- Better contextualization within existing literature required

questions:
1. Evaluation Metrics:
- How do APM and EPM meaningfully evaluate LLM agent performance when their decision-making process is fundamentally different from traditional agents?
- What is the relationship between these metrics and LLM reasoning capabilities?
- Have you considered developing metrics specifically designed for language model evaluation?
2. Comparative Analysis:
- Could you clarify the key methodological differences between your work and "Large Language Models Play StarCraft II"?
- What novel insights does your evaluation framework provide that weren't captured in previous work?
- How does your fine-grained capability analysis advance the field beyond existing benchmarks?
3. Experimental Design:
- Why weren't LLM vs LLM experiments included in the evaluation?
- What prevented the inclusion of human player comparisons?
- Could you explain the decision to limit testing of open-source models?
- How would the results differ with head-to-head language model competitions?
4. Implementation Details:
- How are LLM actions translated into game commands?
- What are the time constraints for model inference?
- What specific prompt engineering techniques were used?
- What is the role of temperature and other sampling parameters?
5. Theoretical Framework:
- How does the decision tracking system account for the unique characteristics of language model reasoning?
- What theoretical justification supports the choice of strategic planning metrics?
- How do you ensure the evaluation framework captures the full range of LLM capabilities?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730648387779

thread_type:
official_review

<thread_object>
creator: reviewer_ywgt

<content>
summary:
This paper introduces a benchmark called Starcraft II Arena, which aims to evaluate the decision-making, planning, and adaptability of large language models (LLMs) within a strategic gaming environment. While traditional benchmarks for Starcraft II assess agents based on a single overall metric—win rates against built-in opponents—the authors argue that a more detailed analysis of LLM performance is necessary. The main contribution of this paper is a methodology designed for an in-depth evaluation of the capabilities of LLM agents.

strengths:
The key idea of this paper, introducing a benchmark for a more in-depth multi-agent evaluation of LLMs, is significant and useful.

weaknesses:
Overall, the paper lacks clarity and depth in describing both the technical implementation and practical contributions.

### **Major comments**

1. Unclear contribution: The paper does not effectively justify why this benchmark must exist as a standalone contribution rather than an addition to existing Starcraft II resources. The contribution seems limited to a collection of scripts and metrics, which could likely be integrated into the existing environment without creating a separate benchmark.

2. Lack of implementation details: Key technical aspects of the implementation are insufficiently described, making it hard to understand the benchmark's novelty and how it's technically realized. Several things are not clear, such as:
   - Integration: How are LLM agents integrated with StarCraft II? How can users use the benchmark? Does the benchmark use a custom API or an interface for this?
   - Decision Tracking: How is decision-making tracked and analyzed? While Table 3 provides a decision trajectory, details of how this is analyzed and used are missing.
   - Computational Requirements: What hardware/software is necessary to run this benchmark effectively? This information is critical for usability but is absent.
   - Opponents: Are the LLMs evaluated with built-in agents or newly introduced opponents? The fact that agents are evaluated against built-in agents in Starcraft II is mentioned as a limitation, but it is unclear whether the authors change this in their benchmark.

3. Incomplete metric information: The metrics lack context. For instance, while Appendix A.1 outlines the metrics, there are no defined ranges, leaving the reader unsure of how to interpret scores. For example, how should a Real-Time Decision score of 21.12 versus 37.51 in Table 4 be interpreted? Similarly, terms such as “effective” actions in EPM or “collected vespene” are not unexplained, reducing the metrics’ interpretability (how do we know that these are the right metrics to assess decision-making and planning?).
 
4. Missing benchmark discussion and limitations: A discussion about future development and limitations of the benchmark is missing, which limits the reader's understanding of the benchmark's intended scope and future extensions.

5. Figure 2 indicates a large variance. Why are there no error bars in the tables?
6. It's important to have the prompt included in the appendix or supplement. Was it possibly in a supplement that I cannot access?

### **Minor comments** (These did not affect my score)
- Abstract: Lines 016-019 are a bit difficult to understand; consider rephrasing
- Figure 2: It’s unclear what this Figure is meant to convey, and the Figure lacks labeled y-axes.
- In Section 4.3, line 367 states "Definitions and methods for these metrics will be further detailed in the figure 4.3." This seems to refer to a table, possibly Table 3, rather than a figure. 
- In Table 3, "OBSERVERtgreater" should probably be "OBSERVER."
- Lines 323 + 350 state that screenshots illustrating decision traces will be provided in the appendix, but these are not included
- I don't understand what is meant when the authors state that civilization and the other games are not "strategic and tactical" in Table 1. Additionally, Werewolf is clearly an imperfect information game. The authors should reconsider this table because I believe many of the entries are inaccurate.
- Why is the score in Table 4 unnormalized? It's an incomprehensible number as it stands.

questions:
See above
</content>
</thread_object>

</thread>
<thread>
creation date:
1730706154221

thread_type:
official_review

<thread_object>
creator: reviewer_nweb

<content>
summary:
The paper presents StarCraft II Arena as a benchmark for evaluating LLMs in strategic planning and decision-making capabilities. While it introduces fine-grained metrics and decision tracking mechanisms, the paper needs substantial clarification on its novel contributions and experimental methodology.

strengths:
1. Well-structured evaluation framework with fine-grained metrics
2. Comprehensive testing across multiple LLM models
3. Interesting decision tracking system for behavior analysis

weaknesses:
1. Insufficient experimental details:
- Build-in AI difficulty level not specified
- Race and map selection criteria not documented
- Limited experimental scope (10 games per model)
2. Theoretical foundation needs strengthening:
- Limited discussion of why LLMs are suitable for StarCraft II
- Insufficient comparison between LLM agents and traditional RL agents
- Unclear theoretical justification for chosen metrics
3. Literature positioning:
- Need more thorough comparison with existing StarCraft II benchmarks
- Better contextualization of contributions needed
- Clearer differentiation from existing approaches required

questions:
1. Can you elaborate on the key differences between this work and existing StarCraft II benchmarks, particularly "Large Language Models Play StarCraft II"? What are the novel contributions that advance the field?
2. The paper would benefit from a deeper discussion of why LLMs are suitable for StarCraft II:
- What unique capabilities do LLMs bring compared to traditional RL approaches?
- How does the decision-making process differ between LLM agents and RL agents?
- What insights have you gained about LLM capabilities through this work?
3. Experimental details requiring clarification:
- What was the difficulty level of the built-in AI?
- How were races and maps selected?
- Why was 10 games chosen as the sample size? Why not do more experiments and test open-soruce model
- What statistical analyses support the conclusions?
4. Theoretical foundation:
- How do the proposed metrics relate to fundamental LLM capabilities?
- What theoretical guarantees or limitations exist for the evaluation framework?
- How generalizable is this benchmark to other strategic decision-making tasks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730923981075

thread_type:
official_review

<thread_object>
creator: reviewer_pea8

<content>
summary:
The paper presents StarCraft II as a benchmark for evaluating reasoning capabilities of LLMs. The paper also presents a set of metrics for evaluating these models, which are based on the domain itself (e.g., amount of resources the play collects). Finally, the paper presents results of several LLMs on this benchmark.

strengths:
The idea of having a challenging benchmark for reasoning with LLMs is interesting and appealing. I also think that computer games can be a good benchmark for this type of evaluation. The choice of a real-time strategy game is particularly good, given the response latency of these systems.

weaknesses:
My main concern with the paper is that it is only half-baked in the sense that the presentation should be improved substantially before being accepted for publication.

My concerns with presentation range from the low level to the high level.

**Low-level concerns**

- The paper has a dangling sentence in line 241. It seems that the explanation of Table 2 was inadvertently commented out from the paper.
- It is not clear what Equation 1 is trying to convey. What is its role in the paper? What is the parameter $\tau$ that is passed as a parameter but not used in the equation?
- The paragraph starting in line 137 discusses the need for communication between the agents, but this is not discussed in the paper. I can see SCII having this need if agents play in a team, but this doesn't seem to be the setting evaluated in the paper.
- In line 106, the paper states that early RL algorithms learn through trial and error. Isn’t this the case for current RL algorithms too?
- Some of the tables are not referred to in the text. For example, I am not sure when I should read Table 3, and I don’t think the paper discusses it at all.

**High-level concerns**

One of the key points I was looking for in the paper was the input-output system for the benchmark. Is the LLM receiving images or text as input? How is the input defined? How much time do they have to reason? What happens when they timeout? The text mentions that smaller models perform better than larger models in the micromanagement part of the game. Is this simply because they are able to respond quicker? These are some of the key questions the authors would need to address in the paper. This is what the readers will be after when understanding whether SCII is a suitable benchmark for LLM evaluation.

Overall, the paper is only half-baked, as the authors need to fix all these presentation issues and adjust the discussion of their results and benchmark. There is clearly a system running, as the paper also includes tables of results. However, it is not possible to understand how this system works as the description is missing.

questions:
Please see the questions in the "high-level concerns" section above.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes StarCraft II Arena as a benchmark for evaluating LLMs' strategic planning and decision-making capabilities, introducing metrics and decision tracking mechanisms. While presenting an interesting direction, the paper suffers from insufficient technical details, unclear differentiation from existing benchmarks, and inadequate experimental methodology.