Kati8/ICLR_2023_2025 huggingface dataset:
authors: Wenjie Tang
authors: Yuan Zhou
authors: Erqiang Xu
authors: Keyan Cheng
authors: Minne Li
authors: Zhiyuan Wang
reviews: 
Comments: summary: The paper presents StarCraft II Arena as a benchmark for evaluating LLMs in strategic planning and decision-making capabilities. While it introduces fine-grained metrics and decision tracking mechanisms, the work needs substantial clarification regarding its novel contributions, metric selection justification, and experimental methodology.
Strengths: 1. Well-structured evaluation framework with proposed fine-grained metrics
2. Comprehensive testing across multiple LLM models
3. Detailed decision tracking system for behavior analysis
4. Clear visualization of experimental results
5. Systematic approach to evaluating different aspects of LLM capabilities
Weaknesses: 1. Metric Selection and Justification:
- APM/EPM metrics appear borrowed from traditional StarCraft II evaluation without clear justification of their relevance to LLM agents
- No discussion of how these metrics specifically reflect LLM decision-making capabilities
- Missing analysis of whether traditional StarCraft II performance metrics are appropriate for language models
2. Experimental Design Limitations:
- Build-in AI difficulty level not specified
- Race and map selection criteria not documented
- Limited experimental scope (10 games per model)
- Absence of LLM vs LLM experiments
- No evaluation against human players
- Limited testing of open-source models, reducing reproducibility
3. Theoretical Foundation and Novelty:
- Limited discussion of why LLMs are suitable for StarCraft II
- Insufficient comparison between LLM agents and traditional RL agents
- Evaluation metrics show significant overlap with existing work
- Need clearer articulation of novel contributions
- Better contextualization within existing literature required
Recommendation: 3
Reviewer confidence: 5
Soundness/correctness: 1
Clarity: 2
reviews: 
Comments: The paper presents StarCraft II Arena as a benchmark for evaluating LLMs in strategic planning and decision-making capabilities. While it introduces fine-grained metrics and decision tracking mechanisms, the paper needs substantial clarification on its novel contributions and experimental methodology.
Strengths: 1. Well-structured evaluation framework with fine-grained metrics
2. Comprehensive testing across multiple LLM models
3. Interesting decision tracking system for behavior analysis
Weaknesses: 1. Insufficient experimental details:
- Build-in AI difficulty level not specified
- Race and map selection criteria not documented
- Limited experimental scope (10 games per model)
2. Theoretical foundation needs strengthening:
- Limited discussion of why LLMs are suitable for StarCraft II
- Insufficient comparison between LLM agents and traditional RL agents
- Unclear theoretical justification for chosen metrics
3. Literature positioning:
- Need more thorough comparison with existing StarCraft II benchmarks
- Better contextualization of contributions needed
- Clearer differentiation from existing approaches required
Recommendation: 3
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 2
reviews: 
Comments: The paper presents StarCraft II as a benchmark for evaluating reasoning capabilities of LLMs. The paper also presents a set of metrics for evaluating these models, which are based on the domain itself (e.g., amount of resources the play collects). Finally, the paper presents results of several LLMs on this benchmark.
Strengths: The idea of having a challenging benchmark for reasoning with LLMs is interesting and appealing. I also think that computer games can be a good benchmark for this type of evaluation. The choice of a real-time strategy game is particularly good, given the response latency of these systems.
Weaknesses: My main concern with the paper is that it is only half-baked in the sense that the presentation should be improved substantially before being accepted for publication.

My concerns with presentation range from the low level to the high level.

**Low-level concerns**

- The paper has a dangling sentence in line 241. It seems that the explanation of Table 2 was inadvertently commented out from the paper.
- It is not clear what Equation 1 is trying to convey. What is its role in the paper? What is the parameter $\tau$ that is passed as a parameter but not used in the equation?
- The paragraph starting in line 137 discusses the need for communication between the agents, but this is not discussed in the paper. I can see SCII having this need if agents play in a team, but this doesn't seem to be the setting evaluated in the paper.
- In line 106, the paper states that early RL algorithms learn through trial and error. Isn’t this the case for current RL algorithms too?
- Some of the tables are not referred to in the text. For example, I am not sure when I should read Table 3, and I don’t think the paper discusses it at all.

**High-level concerns**

One of the key points I was looking for in the paper was the input-output system for the benchmark. Is the LLM receiving images or text as input? How is the input defined? How much time do they have to reason? What happens when they timeout? The text mentions that smaller models perform better than larger models in the micromanagement part of the game. Is this simply because they are able to respond quicker? These are some of the key questions the authors would need to address in the paper. This is what the readers will be after when understanding whether SCII is a suitable benchmark for LLM evaluation.

Overall, the paper is only half-baked, as the authors need to fix all these presentation issues and adjust the discussion of their results and benchmark. There is clearly a system running, as the paper also includes tables of results. However, it is not possible to understand how this system works as the description is missing.
Recommendation: 3
Reviewer confidence: 5
Soundness/correctness: 2
Clarity: 1