Title: Large Language Models Play StarCraft II:Benchmarks and A Chain of Summarization Approach
Benchmark: TextStarCraft II
Summary: TextStarCraft II turns the full StarCraft II video‑game into a purely text interface so that large language‑model (LLM) agents can play the game through natural‑language commands.
The paper also introduces “Chain of Summarisation” (CoS), a cache‑like, multi‑frame prompting procedure that lets an LLM compress streams of observations and issue batches of actions fast enough to keep up with real‑time play. ​It also provided a benchmark, which enables the evaluation of LLMs’ real-time strategic decision-making and long-term planning abilities through natural language. 
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Real-time strategic decision-making, long-term planning, and natural language understanding, reasoning, flexibility
Phenomenon defined by authors: Yes
Phenomenon definition: "Real-time strategy decision-making and long-term planning are critical AI challenges, necessitating rapid, tactical decisions and strategic adaptability over time." (page 1)
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: The task involves controlling a StarCraft II agent using natural language to issue macro-level strategic commands. Micro-level commands are managed by rule-based scripts.
Task item definition: A single item is one complete 1:1 match played on a competitive ladder map at a chosen AI difficulty.	
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: While the game is a real-time strategy game, the task is presented as a text-based interaction, which is a representation of the real task rather than a complete or partial real task. The objective of defeating an opponent in SC2 is a representative goal of the actual game.
Dataset size detail: The paper mentions fine-tuning open-source models using the "entire dataset of GPT3.5-turbo-16k interaction logs with TextStarCraft II."  However, the specific sizes of the train and validation sets are not explicitly reported for this interaction log dataset. There are results presented for fine-tuning on different subsets of win data, with the largest subset being "FULL DATASET (ALL GAMES)" used for fine-tuning, but its size isn't quantified with a number.
Dataset metadata available: Yes
Dataset metadata detail: difficulty level, opponent style, map name
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code), Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall), Correlation (Matthew's correlation, Pearson's r), Win Rate, Population Block Ratio (PBR), Resource Utilisation Ratio (RUR), Average Population Utilization (APU), Technology Rate (TR)
Metric aggregation: Simple Mean
Metric statistics: Simple mean and standard deviation
Metric subscores: Yes
Metric subscores detail: Authors publish PBR, RUR, APU, TR alongside win‑rate.
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: The benchmark is itself realistic
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: They show that most LLMs can beat the built-in AI (LV5), GPT-4 ranks highest in SC2 knowledge per Grandmaster experts, and fine-tuned models play on par with Gold-level humans. They also highlight the LLM agent’s interpretable and adaptable strategy compared to AlphaStar.
Taxonomy: Reasoning / Planning
New taxonomy: General Capability / Planning