CivRealm: A Learning and Reasoning Odyssey in Civilization for Decision-Making Agents
Authors: Siyuan Qi
Shuo Chen
Yexin Li
Xiangyu Kong
Junqi Wang
Bangcheng Yang
Pring Wong
Yifan Zhong
Xiaoyuan Zhang
Zhaowei Zhang
Nian Liu
Yaodong Yang
Song-Chun Zhu
Venue: ICLR 2024 spotlight (2024)
Keywords: Interactive Environments
Benchmark
Reinforcement Learning
Language Agent
Multi-agent
Primary area: datasets and benchmarks
TL;DR: We introduce an interactive environment benchmark grounded in the Civilization game for reinforcement learning (RL) and language agents.

Abstract: The generalization of decision-making agents encompasses two fundamental elements: learning from past experiences and reasoning in novel contexts. However, the predominant emphasis in most interactive environments is on learning, often at the expense of complexity in reasoning. In this paper, we introduce CivRealm, an environment inspired by the Civilization game. Civilization’s profound alignment with human society requires sophisticated learning and prior knowledge, while its ever-changing space and action space demand robust reasoning for generalization. Particularly, CivRealm sets up an imperfect-information general-sum game with a changing number of players; it presents a plethora of complex features, challenging the agent to deal with open-ended stochastic environments that require diplomacy and negotiation skills. Within CivRealm, we provide interfaces for two typical agent types: tensor-based agents that focus on learning, and language-based agents that emphasize reasoning. To catalyze further research, we present initial results for both paradigms. The canonical RL-based agents exhibit reasonable performance in mini-games, whereas both RL- and LLM-based agents struggle to make substantial progress in the full game. Overall, CivRealm stands as a unique learning and reasoning challenge for decision-making agents. The code is available at https://github.com/bigai-ai/civrealm.

Published as a conference paper at ICLR 2024
CivRealm: A LEARNING AND REASONING ODYSSEY IN Civilization FOR -ECKSIGN-MAKING AGENTS
Styuan Qi
Shuo Chen
Yexin Li""
Xiangyu Kong" Junqi Wang"
Bangeheng Yang'
Pring Wong
Yifan Zhong 12
Xiaoyuan Zhang
Zhaowei Zhang 2
Nian Liu
Yaodong Yang*!
Song-Chum Zhu
National Key Laboratory of General Artificial Intelligence, BIGAI Peeingg University BUPT
ABSTRACT
The generalization of decision agents encompasses two fundamental ele- ments: learning from past experiences and reasoning in novel contexts. However, the predominant emphasis in most interactive environments is on learning, often at the expense of complexity in reasoning. In this paper. introduce CivRealm, an environment inspired by the Civilization game. Civilization's ' profound align- ment with human society requires sophisticated learning and prior knowledge, while its ever-changing space and action space demand robust reasoning for gen- eralization. Particularly. CivRealm sets up an imperfect-information general-sum game with changing number of players; it presents a plethora of complex fea- tures, challenging the agent to deal with open-ended stochastic environments that require diplomacy and negotiation skills. Within CivRealm, we provide inter- faces for two typical agent types: tensor-based agents that focus on learning, and language- based agents that emphasize reasoning. To catalyze further research, we present initial results for both paradigms. The canonical RL-bbaed agents ex- hibit reasonable performance in mini-gan mini-games, whereas both RL- and LLM- based agents struggle to make substantial progress in the full game. Overall, CivRealm stands unique learning and reasoning challenge for decision-making agents. The code is available at https: / / gighub........ com/bic bainnbiiaiaii -a -a / //ii/// vveaam.
INTRODUCTION
Human intelligence behaves very differently from contemporary AI. From the earliest use of stone tools, our ancestors spent 18, 88,000,000 years progressing to the control of fire [21]. If we were to learn and explore in a manner similar to modern AI agents, it would take even longer to transition from the industrial age to the information age. This due to the exponential growth of our toolkit and action space A, where finding meaningful development trajectory in the vast realm of possibilities would seem nearly impossible. Yet, human society not only thrives but also at an ever-accelcrating pace. In fact, it merely took 247 years from the invention of the steam engine to the birth of the digital computer [19; 30]. This is truly remarkable from a decision-malee learn from past experience and reason in novel contexts that defies the probabilistic expectations.
perspective: it is our ability to
In this paper, we present CivRealm, an interactive environment designed to push the boundaries of macisinn-makin agents' learning and reasoning. CivRealm is inspired by the iconic game Civi- lization, where each agent acts as a player to guide a civilization, mirroring the course of human history. Within this game, players strategic planning, diplomacy, and warfare. The game rules profoundly align with the mechanics governing human society, and decision-making in the game span from long-term strategic vision (e.g., development prioritization) to fine-graing tactical maneuvers (e.g., unit control) (Figure l).
confronted with decisions including resource management,
As multi- aulti-agent interactive environment, CivRealm distinguishes itself through unique features that demand robust reasoning capabilities to adapt to its wer-changing conditions (Table 1). It estab- lishes an imperfect-information gameplay. Within this setting, agents must adeptly navigate open-ended and stochastic environ- ments with rapidly expanding state and action spaces, driven by technological advancements and societal progress. Additionally, effective interaction and communication between agents necessitate diplomatic and negotiation skills, akin to challenges encountered in real-world decision- -ecciiin-mmaing....
general-sum game where the number of players fluctuates during
"Equal contribution. Corresponding author.

---

Published as a conference paper at ICLR 2024
ROME
GREECE
CARTHAGE
EGYPT
(a) Game interface.
(b) Examples of long- tere-temm strategies.
The gameplay of Civilization [73] requires deep reasoning, involving long-term strategic planning and fine -raiiid tactical controls. The figure depicts a hypothetical situation that resembles a historical scenario, with the Romans securing Sicily against Carthage while nurturing friendly diplomatic relationship with a declining Egypt. This decision made in a highly complex con- text: players need to consider various aspects of long-term developmental strategies like technology, military, and diplomacy in the given geographical and diplomatic context. They also engage fine- grained control actions, e.g., border exploration, vessel building, and road construction.
Figure I:
As interactive environments become increasingly complex [6; 76; 70; 2; 8; 15; 42; 43; 37; 44; 18), it is important to note that many lack the same degree of reasoning sophistication and dynamic complexity as CivRealm. One closely related environment is StarCraft [79]. However, being a real- time strategy game, it places a greater emphasis on quick tactical reactions, with a typical game concluding within 30 minutes for human players. In contrast, Civilization is turnbbased strategy game that simulates the development human society, where a single game can span from several hours even days to complete. This extended duration reflects the need for profound reasoning in this expanding decision space. On the other hand, statistics [83] show that human players learn to play the game reasonably well within 23 hours, which typically amounts to about 3-4 full games. This presents the learning challenges for decision-making agents to match human speed.
In addition to the full game, CivRealm offers mini-games across three core aspects: development, battle, and diplomacy. The development mini-games challenge players to nurture their civilization's 's growth, including aspects like population, production, and economy. In battle mini-games, players need to master the control of multiple units against opposing forces. The diplomacy mini-games demand players to employ diplomatic actions (e.g., trading), to foster their civilization's prosperity.
To stimulate future research, we provide both tensor and language APIs, accommodating two typical genres of decision -masing agents: reinforcement learning (RL) and large language models (LLM) agents. We have developed canonical RL method as well as two language model-based approaches. The first approach BaseLang mirrors AutoGPT [72], while the second, named Mastaba, is a hier- archical amalgamation of individual BaseLang models. We present the preliminary results of initial endeavors. The RL. agent demonstrates reasonable performance in mini-games. However, for both RL- and LLMdbased agents, substantial progress in the full game remains challenge.
In summary, this paper makes three major contributions. First, we introduce an interactive envi- ronment for a Civilization-baae game that poses new challenges in both learning and reasoning. enriched with mini-g games and support for two distinct types of APIs. Second, we outline the design of bascline methods and present a language model-based approach. Third, we share the preliminary results of our explorations, laying the groundwork for further research in this landscape.
2 RELATED WORK
Complex Interactive Environments. Interactive decision-making environments have exhibited increasing complexity with two key dimensions. Task complexity has grown in both physics-based and game environments, transitioning from simpler forms like MuJuCo [76] and ALE [6] to more realistic [1; 54; 59; 66; 44] or more intricate open-ended settings (e.g., MineCraft [37; 18] and so on [2; 39; 38; 64]) that feature partially observable worlds and tasks with real-world resemblances.
However, unlike CivRealm, their state and action spaces typically remain static during gameplay.
Simultaneously, the complexity has expanded from single-agent scenarios to multi-agent challenges. Multi-agent environments introduce additional difficulties due non-unique learning objectives,
2

---

Published as a conference paper at ICLR 2024
Table 1: Comparison with existing environments. CivRealm features the following characteristics for learning and reasoning, Imperfect info: The full state is partially observable. Stochastic: The dynamics of the environment
-enemmterminitti.. Multi-goal: There are multiple victory condi- tions in the game. Dynamic space: The state and action space of single player change dynamically game. Multi-agent: There are multiple interacting players in the game, General-sum: It is mixed motive game, where cooperation and competition coexist. Changing players: The number of players can increase or decrease during a single game. Comm.: players can explicitly communi- cate in the game. Tensor & Lang.: The environment provides both tensor and language APIs.
Environment
Imperfect info Stochastic Multi goal Dynamic Multi-pe Mulli-agent Gen Changing players Comm.
Tensor & Lang.
MineDojo [18]
MPE [50]
x
Hanabi [5]
8
1
x
/
/
X
x
Hold cm [10]
/
X
Diplomacy [51]
Melting pot [13]
/
X
Google Football [2]
x
/
X
X
/
X
X
X
X
Stratego [53]
v
/
X
SMAC [5S]
Dota 2 [8]
X
StarCraft II [79]
v
x
X
CivReabe
non-stationarity, and diverse information structures [90; 84]. Examples such as MPE [50], Melting Pot [43], Neural MMO [70], Stratego [53], Diplomacy [51; 17], and StarCraft [79; 80), each offer distinct perspectives on the complexities of multi-agent decision-making. In comparison, CivRealm stands out for its comprchensive features for generalization, characterized by ever-changing in in-game conditions. For more detailed comparison, please refer to Table 1. In summary, CivRealm offers platform for assessing decision-making agents' reasoning abilities at a broader scale.
Reinforcement Learning Agents with Reasoning. Reinforcement learning (RL) has witnessed significant growth with a focus on model-free methods [47; 61; 62; 22; 80). As a culmination, AlphaStar [80] achieved mastery in StarCraft II. However, it still has generalization issues, relying heavily on human demonstrations and prolonged training. To overcome these challenges, knowledge representation and reasoning are crucial as shown early work on Civilization [12; 11]. To integrate RL and reasoning, typical ways include
methods that select actions based on potential outcomes [13; 68; 69; 60], model-based RL that learns dynamic model of the environment [49; 36; 86; 40; 23; 24; 26; 67; 28], and hierarchical RL that employ temporal and state abstractions [35; 78; 63; 16; 25; 3]. Despite these advances, solving highly complex tasks remains difficult for RL.
agents. CivRealm comprehensively benchmarks these agents' generalization ability.
Agents with Learning. In the past two years, there has been sig- nificant progress in the development of LLM-based agents for task planning and reasoning in (multi- agent) interactive environments [14; 55; 87; 46; 89; 52; 56; 34; 29; 45; 27]. These agents have evolved from not incorporating learning [82; 93; 91] to becoming more adaptive with memory [71], self-reflection mechanisms [85; 65; 48], and learning capabilities [81; 92; 88; 65]. Nonetheless, re- search has revealed certain shortcomings in the reasoning capabilities of LLMs [7; 77; 9; 32]. Given the extensive human knowledge LLMs have acquired, CivRealm an ideal environment to as- sess their ability to learn from interactions and apply knowledge for reasoning.
Large-L Lage---anngag-mddee
ENVIRONMENT
In this section, we begin by introducing the open-ended characteristics of CivRealm, followed by its engineering features. Subsequently, we describe both the full game and the mini-games supported by CivRealm, along with a discussion of their research values for building decision-making agents.
There are various characteristics this environment that make open-ended (Table I): Imperfect information, where players only have access to information discovered by their own units and cities. Stochastic dynamics with random events and crises that can disrupt plans. Multiple victory paths possible (e.g.. conquer, science, or highest score). requiring a balance between economic expansion, military development, diplomacy, culture, and technology. A dynamic game space with continuous changes in state and action space for single agent. Multi-agent interactions with built-
3

---

Published as a conference paper at ICLR 2024
The Brozz A ge
The G Gunpowder Age
The Industrial Age
The Space A Ag
Technology
Building
nanary
City wail
plant
Space infin
Unit
Settler
Transport
Shodth plane
Diplomacy
Unknown
Pesce
War
Alliance
Figure 2: Civilization evolves as the game unfolds, and the potential state and action space explode. This figure focuses on 4 of the ages, wherein technological advancements unlock greater number of buildings and units. Throughout the course of the game, the state can grow from 10t5 to 10550, and the action space can expand from 10 to 10 166 (S D). This figure only shows some example elements; the full game includes 87 types of technologies, 68 types of buildings, 52 types of units, 6 government types, and 5 diplomatic states, all subject to the rule sets used and are customizable.
in Al players other models, providing the potential for self-play. General-sum game that allows alliance formation during gameplay, which changes the game structure and makes the victories of different players non-exclusive. Changes in the number of players during game due revolts or conquers, leading to significant alterations in the joint state and action space. Communication between players through diplomatic actions and natural language chat, allowing agents to their natural language capabilities. In summary, CivRealm presents unique challenges and complexities, making it an open-ended testbed for decision-makinn making agents. Please see A. for more details.
framework. CivRealm empowers each agent to act as a player in the open-source turn-based strategy game Freeciv [74]. CivRealm employs a server-proxy-cienn framework and implements proxy APIs so that a server hosts the game and the proxy establishes the connection between agents (i.e., clients) and the server. The proxy distributes the game states received from the server to each agent and submits the actions returned by agents to the server. By this design, agents with various architectures can seamlessly engage in Freeciv by interpreting the observations provided by the proxy and generating actions that adhere to CivRealm's 's specifications. baweed-agent-frinnenee
agent-architecture-aggie
Freeciv is a turn-based game that operates without the need for real- time reactions. This affords players ample time for thoughtful deliberation. This pace aligns well with the operation of LLM agents, which typically demand substantial time for inference.
Evaluation platform for generalization ability. CivRealm offers multiple convenient methods to create novel scenarios, such as generating random maps with diverse landscapes and varying player and unit numbers, or modifying the rule sets that define the fundamental game rules. These elements result in new configurations, demanding agents to reason the underlying game mechanics rather than relying solely on memorized experiences and public knowledge. Therefore, CivRealm serves as an effective platform for assessing the generalization capabilities of decision decision-making agents. Support for a variety of tasks. CivRealm offers wide range of learning and reasoning tasks. These tasks include not only the comprehensive full game of Freeciv, but also smaller-scale mini-games designed using Lua scripts. In $ 3.2, we will provide detailed deseriptions of these tasks.
3.1 FULL GAME DESCRIPTION
In CivRealm, players take the role of civilization leaders with the objective of guiding their civi- lization from its humble beginnings greatness, where full games can last from several hours to several days. Civilizations evolve through eras, with an explosion in the number of controllable objects as the game progresses, resulting in vast state spaces and joint actions (Figure 2). Decisions in the game have multifaceted impacts, encompassing both long-term strategic consequences and short-term tactical outcomes. This complexity necessitates thought process that carefully weighs the implications of these decisions since greedy moves can easily be non-optimal in the long term.
Observations. Instead of directly processing pixel data of the game interface, extract repre- sentative discrete information from graphics observed during human gameplay. These observations encompass data related the map, units, cities, government, technology, and diplomacy. The map
4

---

Published as a conference paper at ICLR 2024
werk
Tide
Trelen (12 (12
Development: TransBuildCity
Development: BuildIfai
Development: City TelTTileWoldde
gho you
you give me deanced Flight?
very good
And how about Electronies!
Yos, would make
Battle: Ladera
Battle: NavalBattleModera
Diplomacy: TradeTechs
Figure 3: Examples of different types of the designed mini-games.
information includes whether a particular tile is observable, where a "tile" refers to a square space the grid-hased map. The map provides details like the terrain type, owner of the tile, resource output, additional infrastructure, and units present on the tile. The unit information provides insights into unit's health, location, owner, attack/defense strength, remaining movement points (indicating the actions the unit can take in a turn), maintenance costs, etc. The city information covers details such city's location, owner, size, population, shield value, resource production, and more. The government information indicates the current government type of the civilization, the tax rate, etc. The technology information displays the technologies that have been researched and the technology currently being researched. The diplomucy information comprises data regarding diplomatic rela- tionships with other players. For a comprehensive list of these observations, please refer to A.1.1.
Actions. We have implemented rich set of action classes that encompass the five primary facets of gameplay: unit, city, government, technology, and diplomacy. The unit actions are responsible for controlling player's units. They can be categorized into three main types: engineering actions, which handle tasks like city construction, planting, mining, and more; movement actions, including and military actions, such as attacking, fortifying,
moving, transportation, embarking, and bribing, etc. The city actions pertain to the development and management of a city. They include unit production, building construction, city worker assignment, and more. The government actions allow players to change their government type to gain corresponding political benefits, adjust tax rates to balance economic expansion and citizen happiness, etc. The technology actions enable players to set immediate or long-term goals for their technology research. The diplomacy actions empower players initiate negotiations, such as trading technologies, negotiating ceasefires, forming alliances, etc. For an exhaustive list of the implemented actions, please refer to A.1.2.
Evaluation Metrics. CivRealm offers evaluation metrics to assess playing performance across var- ious dimensions, including population, constructed cities, researched technologies, produced units, explored land, etc. An aggregated score provided for overall evaluation. Please refer to A.1.3.
3.2 MINI-GAME BENCHMARKS
CivRealm requires players to balance multiple factors, including economic expansion, military de- velopment, diplomatic influence, cultural achievements, and technological research, when making decisions. To provide comprehensive benchmark and support for various learning paradigms (e.g., curriculum learning), built collection of mini-games that focus specific aspects of the full game. These mini-games are generated using the same rule sets as the full game, but with a smaller map size and fewer players. The mini-games master specific skills to achieve victory. The key features of the benchmark are as follows.
designed to be challenging. requiring agents to
Unlimited task generation. The benchmark offers automatic mini-gan game generation tool, which introduces randomness across various aspects, including the map landscape, initial position of units and cities, tile resources, etc. Consequently, this provides access inexhaustible pool of mini-
5

---

Published as a conference paper at ICLR 2024
1256
125
-243
Iracceeriie on meseeee Joface celeve
850
377
3.00
1.25
1.59
135
2.60
322
pewrgoyer Rerrnce O1 Strarce-ninteerr
(a) BuildCity statistics.
(b) LandBattleModern statistics.
(c) TradeTechs statistics.
Figure 4: The generated mini-games diverse and balanced. (a) Terrain and resource distribution and corresponding joint food/production/trade metrics. (b) Unit numbers and strength: inter-player
ratio's distribution. (c) Technology number and value: inter-player difference's distribution.
games, allowing the assessment of agents' generalization abilities in smaller-scale tasks.
High customizability. The benchmark utilizes Lua scripts to specify the reward structure of mini- games conveniently. This modular design enables users customize the victory condition of mini- games and train algorithms with different paradigms (e.g., curriculum learning and meta learning), Compatibility with full game. Mini-games adhere the same input and output format as the full game. Agents trained in mini- gamie can be seamlessly migrated to the full game, and vice versa.
In CivRealm, we design 10 types of mini-games and use the generation tool produce 10,000 instances for each type. They can be categorized into three groups: development, battle, and diplo- macy (as exemplified in Figure 3). The development mini-games aim nurturing cities, focusing on factors like population growth, production efficiency, and economic prosperity. These elements are closely linked to civilization's is overall strength and advancement. The battle mini-games revolve around tactical warfare between opposing groups of units. Skilled commanders aim defeat their enemies while minimizing their own losses. The diplomacy mini mini-games represent a unique feature of civilization games. Mastering diplomatic skills is pivotal, as maintaining positive relationships with other players can significantly influence the course of gameplay. Please refer to A.2 for specific game definitions and the criteria use determine victory in these games.
Task variation statistics. The benchmark manipulates crucial configurations in each mini-game to increase the diversity of mini-game instances ($ A.2.2). In development mini-games, introduce variations in terrain types and extra infrastructures on tiles, both of which impact resource outputs (food/prdduction/rrade))) In battle mini-games, we alter the number of units and their attack/defense strengths. In diplomacy mini-g mini-games, we adjust the number and type of technologies held by each player. Figure 4 illustrates the benchmark effectively generates a diverse set of Task difficulty level. Owing to the varied configurations of mini-game instances, we classify their difficulty into three levels: easy, normal, and hard ($ A.2.1), based on whether the player controlled by our agent holds advantage or disadvantage compared to the opposing player.
METHODS
We design three approaches as baseline methods, including: 1) canonical tensor-based RL method inspired by AlphaStar [80], 2) an LLM method BaseLang that works similarly as AutoGPT, and 3) Mastaba, hicrarchical amalgamation of individual BascLang models. In this section, introduce these models as well as the challenges encountered. For more discussion, please see $ E.
4.1 TENSOR- BASED REINFORCEMENT LEARNING
Challenges in CivRealm for based RL methods include the following. Complex dynamics: CivRealm's complex game mechanics mimic human society, posing difficulties for tensor-batt based learning. involves diplomacy, economics, technology, and military strategy, making it hard for both model-free and model-based approaches. Overwhelming information: agents face extensive information from units, cities, players, etc, while lacking semantic understanding. Multi-Level ac- tions: CivRealm's 's action space is complex and hierarchical, requiring sequential decision-man -maiing and neural network adaptation. Sparse, delayed rewards: Rewards are sparse, delayed, and asyn- chronous due the turn-based and society-simulating nature CivRealm. Diverse victory paths:

---

Published as a conference paper at ICLR 2024
ang
Mastaba
World Obs
Advio
dovioo
Ec ASA., Sellas sum pruce earrsiie
laot ofhec cor'l
Knowlrdge
L1Mss
Zoots-ost Obs
Thrught
Maanar Find
Le cutella Sollan Aboor bul
Losk up
Reasoring
Heewsning
en v.dis daleee
Zoot-in 0be
Uraus Pank Manu Tita
Command
Conmmrand
(Ganda
Maxail
gual diai welk
Action
NT jegeb
Action
Thaughts
Figure 5: Architecture of LLM-based approaches. Left: BaseLang. Right: Mastaba.
CivRealm offers multiple paths to victory, complicating RI. training that relies reward signals. Network Design We implement a hierarchical approach inspired by AlphaStar [80] to handle di- verse inputs effectively. Our network has three main parts (Figure 11). Representation: we use MLP, Transformer, and CNN models to extract features based on input type (vector, sequence, or image-based). These features globally connected through a transformer, and an RNN combines current-state features with a memory state for historical context. Action Selection: we leverage the learned representations
make decisions. The actor module selects the primary action category (e.g., unit, city, government, or turn termination), followed by pointer network choosing the spe- cific action ID. Value Estimation: we include value prediction head to enable actor-critic learning, sharing the representation for training efficiency. We train the network using Proximal Policy Opti- mization [62] and parallelize tensor environments with Ray [4] (see $ E.1.3 for training details).
4.2 BASELANG: BASELINE LANGUAGE-BASED AGENT
Necessity and challenges The need for developing an LL.M-based agent in CivRealm arises from two factors. 1) LLMs are capable of task generation and problem-solving, augmented by their exten- sive human knowledge base. Their capabilities and knowledge are promising for solving decision- making problems. 2) LLM is not yet sufficiently sophisticated for complex reasoning [77; 7]. In CivRealm, LLM is advantageous due to its ability to use natural language, prioritize strategic game- play, and handle diplomatic interactions. However, constructing such an agent faces several chal- lenges, including managing multiple in-game roles, handling sparse and complex observations, ad- dressing the long-term impact of actions, and improving the agent's maiisinnnmaiinggg over time. Design We design the bascline language-based agent composed of three components (Figure 5): observation, reasoning, and commands. For observation, 5x5 tile-based observation is employed, centered on each unit's location, optimizing tactical information provision while accommodating strategic depth. The reasoning module mimics AutoGPT [72] and outputs in three stages: thought, reasoning, and command. Commands empower the agent with the choice between "manual and history search" and "final decision" commands, enabling data retrieval from a vector database or selecting available actions execute based environmental and historical context. Finally, indi- vidual LLMs are assigned to each unit, with their context histories, to facilitate detailed planning.
4.3 MASTABA ENHANCING BASELANG BY A HIERARCHICAL STRUCTURE
To facilitate cooperation between independent entities, Mastaba introduces a hierarchical structure, organizing LLM workers, observations, and decision-making into pyramid-like structure. LLM Workers. Within Mastaba, LLM workers are structured as two layers. At the pinnacle is the "advisor". tasked with overseeing all other LLM workers. The advisor monitors the holistic nationwide perspective, including unit counts, city metrics, and enemy/player information. At the operational level, Mastaba maintains LLM workers that resemble BaseLang's structure.
Observation. Mastaba adopts pyramid-like map view, condensing data from a 15 15 tile region into 9 blocks, each spanning 5 x tiles. This design enables entities to grasp information within broader range while managing prompt loads effectively, thereby elevating map awareness.
Decision-making Mastaba's decision decision-making making workflow follows its agent structure. The advisor
Ancient Egyptian tomb before pyramids. The first pyramid in Egypt considered stack of Mastabas.

---

Published as a conference paper at ICLR 2024
1.0
0.8
Success Rate
0.8
Success Rale
normal
normal
hard
hard
0.6
0.3
0.6
Success
0.4
0.2
0.4
easy
0.2
normal
hard
0.2
0.1
10K
20K
30K
40K
10K
20K
30K
40K
10K
20K
30K
40K
Step
Step
Step
(a) City TindWrrder
(b) SettlerBuildCity
(c) LandBatteDefendCCit
Figure 6: Performance the RL method over training on three mini-tasks; the diverse mini-tasks exhibit varying degrees of stochasticity and complexity that lead different success rates. In the "SentlerBuildCit"" task, the RL method exhibited shortcut learning [20] in hard level.
Researched Techs
100
30
10
75
Scores
20
Units
25
0
0.0M
0.5M
1.0M
1.5M
2.0M
0.0M
0.5M
1.0M
1.5M
2.0M
Step
Step
Step
(a) Game scores
(b) technologies researched
(c) units built
the full game for the RL method. CivRealm provides different metrics to gauge the performance of the model from different aspects to provide further insights; these metrics also reflect the multi-goal nature of the environment ($ A.1.3).
Figure 7: Scores over the training
initiates each turn with nationwide assessment, encompassing cities, units, and potential threats. generates suggestions during each turn and communicates them to other LLM workers, who in- dependently select actions for their entities. Additionally, workers possess the capability query vector database for knowledge, enabling informed decisions based on manual or stored experiences.
5
EXPERIMENTS
5.1 TENSOR -BASED REINFORCEMENT LEARNING
To evaluate the performance of our tensor baseline across various mini-tasks and full games, we kept the same architecture and training syper-pprameters and trained our models from scratch. Minitask For each minitask of given difficulty level, we trained tensor baseline models for over 40,000 steps with random seeds. Figure shows 3 representative tasks with the success rates. While a small number of development minitasks, such as 'CityTileWndeer' could be solved with high success rate (~ 90%), most minitasks still pose hierarchy of increasingly difffcuult challenges to our tensor baseline. These results highlight the strengths and limitations of current tensor-based RL models in handling different types of tasks in CivRealm. Specifically, RL models exhibit better learning capabilities for tasks offering immediate rewards and requiring relatively short-term plan- ning. However, when confronted with tasks with sparse and delayed rewards, requiring long-term strategic planning, these models encounter significant difficulties in identifying viable solutions.
Fullgame For full games, we trained tensor baseline models for over million steps. The overall performance is measured by the average score at the end of each game as shown in Figure 7. The score is multi-facet metric calculated from weighted sum of important game stats including pop- ulation, technology researched, units built/killed, etc. Our RL model exhibits progressive learning, primarily focusing on unit production to enhance its score. However, is important to note that this approach is myopic in nature. A more strategic and long-term approach involves prioritizing the establishment additional cities to foster exponential growth in both technology and economics, although this strategy results in an initial decrease in population (and subsequently the score) in the short term. These observations underscore the existing limitations RL methodologies. In this context, CivRealm emerges ideal platform for future research endeavors.
8

---

Published as a conference paper at ICLR 2024
TS
T 30
T 100
T200
T200
5
T 20
113
120
299
Figure 8: Evolution of the civilization. Top: BaseLang: Bottom: Mastaba. From left right, this visual illustrates the progression of the empire through various phases: inception, establishment of a new colony, peak expansion of the realm, encountering invasion, and approaching collapse.
Mastabe
Mastaba BaseLang
Mestaba
BaseLang
15
Baselang
musary UnIES
population
8
1.0
4
2
0
100
200
300
400
1000
200
300
400
100
200
300
400
turn
turn
turn
Figure 9: Stats on metrics of LL.M-based agents against game turns for 50 individual games each.
5.2 LANGUAGE-BASEI AGENTS: BASELANG AND MASTABA
We compared Mastaba with BaseLang 10 maps with distinet geography. Each method plays full games on each map. Figure 8 and Figure 9 show the qualitative and quantitative results, respectively. Note that one of the games lasts longer than the others, resulting in no variance in the later stage.
As depicted in Figure 9, Mastaba's 's development significantly outpaces that of BaseLang. Mastaba evolved with a larger population, greater number of military units, and higher score. Figure shows Mastaba achieves a more extensive explored and settled area by turn 113, surpassing Base- Lang's progress by turn 210. In some games, BaseLang manages survive longer than Mastaba. This is consequence of Freeciv's 's built-in mechanics, which trigger pirate invasion when player's score exceeds a certain threshold. Consequently, Mastaba encounters invasions at an earlier stage.
Our experiments offer valuable insights into trol large number of objects (e.g., units, cities) based on their local observations due to the context limit of LLM. BaseLang's sluggish expansion pace underscores the absence of a global perspective, whereas Mastaba's pyramid-like structure empowers the advisor to oversee the overall progress, establish comprehensive context, and thus assist each local object in making well-informed de- cisions. The performance contrast between Mastaba and BaseLang highlights the necessity of a hierarchical decision architecture for tackling the complex scenarios presented by CivRealm.
methods. Firstly, it is challenging to con-
Secondly, these results expose the weakness in LLM's grounding ability. Despite exposure to a vast amount of human knowledge, LLM-based agents still exhibit inefficiencies in their reasoning within CivRealm, which mirrors human society. Even with global context, Mastaba manages to achieve only moderate economic expansion performance while failing defend against pirate invasions. This signifies LLM's 's difficulty in discerning the most critical issues within the current context and its struggles to effectively balance various aspects of gameplay. Consequently, CivRealm serves as an ideal platform for evaluating and advancing the reasoning capabilities of LL-M-based agents.
CONCLUSION
We present CivRealm, distinctive challenge for doeision-making agents, placing simultaneous em- phasis on learning and reasoning abilities, which are evaluated across various mini-games and full games using diverse metrics. Experiments in CivRealm unveil limitations of contemporary methods, with tensor methods often yielding myopic strategies, and LLM-based agents still struggling with intricate reasoning tasks. To surmount these challenges, future research directions could involve in- tegrating the strengths of RL and LLM, creating decision-mail making agents that combine adaptability and sophisticated reasoning. CivRealm serves as an ideal testing ground for these approaches.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENT
The work of CivRealm is built upon several open-source projects, including Freeciv [74], Freeciv- web [75], FCIV-NET [73], and Freeciv-bot [57].
This work was supported by the National Natural Science Foundation of China. This work was also supported by Wuhan East Lake High-Tech Development Zone, National Comprehensive Ex- perimental Base for Governance of Intelligent Society.
REPRODUCIBILITY STATEMENT
To help readers understand the environment details, we discuss the tensor-based API $ B, language-biis based inl C, and method details in E. For LLM experiments, used GPT3.5-turbo provided by Azure's OpenAl API.
ETHICS STATEMENT
CivRealm can potentially contribute to the development of more socially aware Al agents, which could have applications in areas like policy-making economic simulations, conflict resolution, and education. Moreover, the learning from such simulations could offer insights into the dynamics of human society, historical event outcomes, and potential future societal trajectories. Overall, the societal impacts of CivRealm and similar social simulation platforms can profound, aiding development of Al that is more aligned with human values and societal goals.
The development of CivRealm is committed to fostering cultural and social inclusivity. It is de- veloped upon the open-source game Freeciv [74], with conscious effort made to incorporate wide array civilizations. Should any instances of marginalization or misrepresentation arise, we encourage and facilitate contributions to the source code for rectification.
REFERENCES
[1] Josh Abramson, Arun Ahuja. lain Barr, Arthur Brussee, Federico Carnevale, Mary Cassin, Rachita Chhaparia, Stephen Clark, Bogdan Damoe, Andrew Dudzik, al. Imitating interactive intelligence. arXiv preprint arXiv:2012.05672 2020. 2
[2] Joshua Albrecht, Abraham Fetterman, Bryden Fogelman, Ellie Kitanidis, Bartosz Wróblewski, Nicole Seo, Michael Rosenthal, Maksis Knutins, Zack Polizzi, James Simon, et al. Avalon: A benchmark for rl generalization using procedurally generated worlds. Advances in Neural Information Processing Systems, 35:12813-12822 2022. 2
[3] Marein Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experi- ence replay. Advances neural information processing systems, 30, 2017. 3
[4] Anyscale, Inc. Ray: Productionizing and scaling python ml workloads simply, 2023. URL
https: // /www. ray.aa.... ri/. 7,
[5] Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch, Mare Lanctot, H Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, et al. The hanabi challenge: A new frontier for ai research. Artificial Intelligence, 280: 103216, 2020.
[6] Mare Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
[7] Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: Lims trained on "a is b" fail to learn "b is a". arXiv preprint arXiv:2309. 12288, 2023. 3,7
10

---

Published as a conference paper at ICLR 2024
[8] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme. Chris Hesse, et al. Dota with large scale deep reinforcement learning. arXiv preprint arXiv: 2019. 2,
[9] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07255: 2021.
[10] Michael Bowling, Neil Burch, Michael Johanson, and Oskari Tammelin. Heads-up limit hold'em poker is solved. Science, 34766218;:445149,, 2015. 3
[11] Satchuthanan R Branavan, David Silver, and Regina Barzilay. Non-linear monte-carlo search in civilization ii. In AAAI Press//nternational Joint Conferences on Artificial Intelligence, 2011.
[12] SRK Branavan, David Silver, and Regina Barzilay. Learning to win by reading manuals in monte-carlo framework. Journal of Artificial Intelligence Research, 43:661-704 2012.
[13] Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter Cowl- ing, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and Al in games, 4(1):1-43, 2012.
[14] Micah Carroll, Rohin Shah, Mark Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-ai coordination. Advances in neural information processing systems, 32, 2019. 3
[15] Benjamin Ellis, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Mahajan, Jakob N Foerster, and Shimon Whiteson. Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning. arXiv preprint arXiv:2212.074489 2022. 2
[16] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint
2018. 3
[17] Meta Fundamental AI Research Diplomacy Team (FAIR)t, Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, et al. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science, 37866624) 2022. 3
[18] Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended em- bodied agents with internet-scale knowledge. In Advances in Neural Information Processing Systems, 2022. 3
[19] Eugene S Ferguson. The origins of the steam engine. Scientific American, 210(1)998-107, 1964.
[20] Robert Geirhos, Honn-Herrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665-673, 2020.
[21] John AJ Gowlett. The discovery of fire by humans: long and convoluted process. Philosoph- ical Transactions of the Royal Society B: Biological Sciences, 371(1696):0155164,, 2016.
[22] Tuomas Haarnoja, Aurick Zhou, Picter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pp. 1861-1870. PMLR, 2018. 3
[23] Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv: /9/..//222200 /2.0//00, 2019. 3
[24] Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02199 2020.

---

Published as a conference paper at ICLR 2024
[25] Danijar Hafner, Kuang- Huei Lee, lan Fischer, and Pieter Abbeel. Deep hierarchical planning from pixels. Advances Neural Information Processing Systems, 35:26091-26104, 2022.
[26] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse do- mains through world models. arXiv preprint arXiv:2301.0410. 2023.
[27] Cheng Han, James Chenhao Liang, Qifan Wang, Majid Rabbani, Sohail Dianat, Raghuveer Rao, Ying Nian Wu, and Dongfang Liu. Image translation as diffusion visual programmers. International Conference on Learning Representations, 2024.
[28] Nicklas Hansen, Xiaolong Wang, and Hao Su. Temporal difference learning for model predic- tive control. arXiv preprini arXiv:2203.0 04955, 2022.
[29] Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong. Zhen Wang, Daisy Zhe Wang, and Zhit- ing Hu. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305. 14992, 2023.
[30] Douglas Rayner Hartree. The eniac, electronic computing machine. Nature, 158(4015): 500-506, 1946.
[31] Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, et al. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308 2035022000.0052, 2023." 3
[32] Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey, arXiv preprint arXiv:2212.104000 2022. 3
[33] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, 2022.
[34] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. Conference on Robot Learning. pp- 1769-1782. PMLR, 2023. 3
[35] Matthias Hutsebaut Hutsebaut-Buysse, Kevin Mets, and Steven Latré. Hierarchical reinforcement learn- ing: A survey and open research challenges. Machine Learning and Knowledge Extraction, 4 (1):172-221, 2022.
[36] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in neural information processing systems, 32, 2019. 3
[37] Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell, The malmo platform for artificial intelligence experimentation. In Ijcai, pp. 4246-4247, 2016. 2
[38] Arthur Juliani, Ahmed Khalifa, Fincent-Pierre Berges, Jonathan Harper, Ervin Teng, Hunter Henry, Adam Crespi, Julian Togelius, and Danny Lange. Obstacle tower: generalization challenge in vision, control, and planning. In Proceedings of the 28th International Joint Conference Artificial Intelligence, pp.
2019. 2
[39] Arthur Juliani, Vincent-Pierre Berges, Ervin Teng, Andrew Cohen, Jonathan Harper, Chris Elion, Chris Goy, Yuan Gao, Hunter Henry, Marwan Mattar, and Danny Lange. Unity: A general platform for intelligent agents. arXiv preprint 18Xv::109.00227,, 2020.
[40] Rahul Kidambi, Aravind Rajeswaran, Pranceth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. Advances in neural information processing sys- 2020.
tems,
[41] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference Learning Representations, 12 2014. 27
12

---

Published as a conference paper at ICLR 2024
[42] Karol Kurach, Anton Raichuk, Piotr Stanczyk, Michal Zajac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, et al. Google research football: A novel reinforcement learning environment. In Proceedings of the AAAl conference on artificial intelligence, volume 34, pp. 4501 -510, 2020. 2,3
[43] Joel Z Leibo, Edgar A Duefiez-Guzmal Alexander Vezhnevets, John P Agapiou, Peter Sune- hag, Raphael Koster, Jayd Matyas, Charlic Beattie, Igor Mordatch, and Thore Graepel. Scal- able evaluation of multi-agent reinforcement learning with melting pot. In International ference on machine learning, pp. 6187-6199. PMLR, 2021. 2,3
[44] Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martin-Martín, Chen Wang, Gabrael Levine, Michael Lingelbach, Jiankai Sun, et al. Behavior- 1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation. In Conference on Robot Learning, pp. 80-93. PMLR, 2023. 2
[45] Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for" mind" exploration of large language model society. In Thirty-seventh Conference Neural Information Processing Systems, 2023.
[46] Wenhao Li, Dan Qiao, Baoxiang Wang, Xiangfeng Wang, Bo Jin, and Hongyuan Zha. Se- mantically aligned task decomposition in multi-agent reinforcement learning. arXiv preprint arXiv:2305. 10865, 2023.
[47] Timothy P Lillicrap. Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint
02997, 2015.
[48] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refine- ment with self-feedback. arXiv preprint arXiv:2303. 17651, 2023. 3
[49] Thomas M Moerland, Joost Broekens, Aske Plaat, Catholijn M Jonker, et al. Model-based reinforcement learning: A survey. Foundations and Trends in Machine Learning, 16(1):
1-118, 2023. 3
[50] Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi- agent populations. In Proceedings of the AAAI conference on artificial intelligence, 2018.
[51] Philip Paquette, Yuchen Lu, Seton Steven Bocco, Max Smith, Satya O-G, Jonathan K Kum- merfeld, Joelle Pineau, Satinder Singh, and Aaron C Courville. No-press diplomacy: Modeling multi-agent gameplay. Advances in Neural Information Processing Systems, 32, 2019.
[52] Joon Sung Park, Joseph C O'Brien, Carrie J Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv preprint 2rXiv:2304.03444 2023.
[53] Julien Perolat, Bart De Vylder, Daniel Hennes, Eugene Tarassov, Florian Strub, Vincent de Boer, Paul Muller, Jerome T Connor, Neil Burch, Thomas Anthony, et al. Mastering the game of stratego with model-free multiagent reinforcement learning. Science, 378 996. 2022. 3
[54] Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In Proceedingy of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 8494-8502, 2018. 2
[55] Xavier Puig, Tiammin Shu, Shuang Li, Zilin Wang. Kuan-Hong Liao, Joshua Tenenbaum, Sanja Fidler, and Antonio Torralba. Watch-and-help: A challenge for social perception and human-ai collaboration. arXiv preprint arXiv:2010.09899. 2020.
[56] Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. Communicative agents for software development. arXiv preprint a7X71:2307.07924, 2023. 3
13

---

Published as a conference paper at ICLR 2024
[57] Christian Rockstroh. Freeciv-bot, 2023. URL https:://gttaab.ccc/ccttttc
Freeciv-bot. 10
[58] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon White- son. The starcraft multi-agent challenge. arXiv preprint Seti::9992.0400000 2019. 3
[59] Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF international conference on computer vision, pp.
2019.
[60] Julian Schritwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Si- mon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588(7839):004-699, 202.
3
[61] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015.
[62] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint (rXiv:1707.06347 2017. 3, 7,
[63] Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics- aware unsupervised discovery of skills. arXiv preprint vrXivv1907.0166 2019. 3
[64] Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits: An open-domain platform for web-based agents. In International Conference on Machine Learning, pp. 3135-313 3144. PMLR, 2017. 2
[65] Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: autonomous agent with dy- namic memory and self-reflection. arviv preprint mrXiv:2303.11366 2023.
[66] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mot- taghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 10740-10749, 2020. 2
[67] Harshit Sikchi, Wenxuan Zhou, and David Held. Learning off-policy with online planning. In Conference on Robot Learning, pp. 1622-1633, PMLR, 2022.
[68] David Silver, Aja Huang. Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanc- tot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529 (7587):484 489, 2016.
[69] David Silver, Julian Schritwieeser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-35,, 2017.
[70] Joseph Suarez, Yilun Du, Phillip Isola, and Igor Mordatch. Neural mmo: A massively multiagent game environment for training and evaluating intelligent agents. arXiv preprint arXiv:1903.00784 2019. 2,3
[71] Theodore Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L Griffiths. Cognitive chitectures for language agents. arXiv preprint arXiv:2309.02427, 2023. 3
[72] The AutoGPT team. Autogpt: the heart of the open-source agent ecosystem. 2023. 2, 7,
[73] The FCIV-NET project. Feiv-net: the version of the freeciv strategy game, 2023. URL https: //gl thu.. com/ feiv feieeete................m 2, 10
14

---

Published as a conference paper at ICLR 2024
[74] The Freeciv project. Freeciv.org/ 4,
Freeciv: An empire-building strategy game, 2023. URL http://
[75] The Freeciv-web project. Freeciv-web: An open-source tur-based strategy game, 2023. URL https: //github.. com/ www.eeeewweeeeeeeeeeee.e..eeeeee 10
[76] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: physies engine for model-based control. In 2012 TEEF/RSJ International Conference on Intelligent Robots and Systems, pp- 0026-5033. IEEE, 2012. doi: B.LL00/IROS.2002.63660000
[77] Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Large language models still can't plan (a benchmark for Ilms on planning and reasoning about change). In NeurlPS 2022 Foundation Models for Decision Making Workshop, 2022. 3,7
[78] Vivek Veeriah, Tom Zahavy, Matteo Hessel, Zhongwen Xu, Junhyuk Oh, lurii Kemaev, Hado P van Hasselt, David Silver, and Satinder Singh. Discovery of options via meta-learned subgoals. Advances in Neural Information Processing Systems, 34:29861-29877 2021. 3
[79] Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schritwieser,, et al. Starcraft it: A new challenge for reinforcement learning. arXiv preprint 1rXiv:1708.064 04778,,
2017. 2.3
[80] Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et Grand- master level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350- 354, 2019. 3, 6, 7.
[81] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305. 16291, 2023.
[82] Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint 2rXivv2332.011560 2023.
[83] The HongLong MongLoonTTBeen ToBeat Website. Sid meiers civilization vi. com/gamm/33
2023. 2
[84] Yaodong Yang and Jun Wang. An overview of multi-agent reinforcement learning from game theoretical perspective. arXiv preprint arXiv:201 1.00583, 2020.
[85] Shunyu Yao, Jeffrey Zhao. Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2022.
[86] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances Neural Information Processing Systems,
14142, 2020. 3
[87] Ceyao Zhang, Kaijie Yang, Siyi Hu, Zihao Wang, Guanghe Li, Yihang Sun, Cheng Zhang, Zhaowei Zhang, Anji Liu, Song-Chun Zhu, al. ProAgent: Building proactive cooperative Al with large language models. arXiv preprint arXiv:2308, 11339, 2023.
[88] Danyang Zhang, Lu Chen, Situo Zhang, Hongshen Xu, Zihan Zhao and Kai Yu. Large language model is semi- parameertt reinforcement learning agent. arXiv preprint arXiv:2306.079.1 07929, 2023.
[89] Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B Tenenbaum, Tianmin Shu, and Chuang Gan. Building cooperative embodied agents modularly with large language models. arXiv preprint 200700230 2023.
15

---

Published as a conference paper at ICLR 2024
[90] Kaiqing Zhang, Zhuoran Yang, and Tamer Başar. Multi-agent reinforcement learning: A lective overview of theories and algorithms. Handbook of reinforcement learning and control, pp. 321-384, 2021. 3
[91] Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren's 's song in the ai ocean: A survey on hallucination large language models. arXiv preprint MXXi:2309,01219, 2023. 3
[92] Andrew Zhao, Daniel Huang. Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. Expel: Llm agents are experiential learners. arXiv preprint 201444 2023. 3
[93] Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyu Yang, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, et al. Ghost the minecraft: Generally capable agents for open-world enviroments via large language models with text-based knowledge and memory. arXiv preprini arXiv:2305.77144 2023.
16

---

Published as a conference paper at ICLR 2024
Appendices
Content
Environment
18
A.1 More on Full Game and CivRealm Features
18
A.2 Mini-game Details
22
Tensor Environment
22
B.1 Observation
22
B.2 Action
23
C Language Environment
23
C.1 Observation
C.2 Action
25
Estimation State Action Space Sizes
25
Method Details
26
E.1
Tensor-based Reinforcement Learning
26
E,2
BaseLang: Baseline Language-basee Agent
27
E,3 Mastaba: Enhancing BaseLang by Hicrarchical Structure
29
More Experiment Results
29
17

---

Published as a conference paper at ICLR 2024
A ENVIRONMENT
A.1 MORE ON FULL GAME AND CivRealm FEATURES
In the full game, each player acts as civilization leader. The agents can be customized to be controlled by our agents or by the built-in AI algorithms. The objective of players is to guide their civilization from its humble beginnings to the greatest civilization and one full game can last from several hours to several days'. As depicted in Figure 2, civilizations evolve through eras from the Bronze Age to the Space Age and the number of controllable objects (units, cities, diplomatic relations, etc.) explodes as the game progresses. In addition, each decision made typically carries a multi-faceted impact, encompassing both long-term strategic consequences and short-term tactical outcomes. It is worth noting that a favorable tactical outcome may not necessarily translate into a positive strategic outcome. For instance, the immediate construction of city at the beginning of the game can yield greater resources in the early stages (a tactical advantage). In contrast, settling in a resource-rich area after thorough exploration may result in substantial resource accumulation over the long haul (a strategic advantage). This complexity underscores the necessity for a learning and reasoning process that judiciously weighs the implications of long-term and short-term decisions.
Besides the long rrcision-making horizon, multi-faceted decision impacts, and huge state-action spaces, our environment exhibits additional characteristics (Table 1) that elevate its complexity:
Imperfect info. Players typically only gain the information discovered by their own units and cities, resulting in partially observable states. Players may also obtain others' vision by diplomatic actions.
Stochastic. The dynamics of the environment is stochastic. Moreover, there exist random events and crises that can disrupt plans, forcing players to adapt on the fly and make tough decisions.
Multi-goal. There are multiple victory paths, i.e., (1) military: conquering all other civilizations; (2) science: being the first civilization that launches a spacecraft destined for Alpha Centauri; and (3) time: obtaining the highest score, computed based on criteria such civilization size, wealth, cultural accomplishments, and scientific advancements, before reaching a predetermined number of turns, in case the first two conditions are not met. These paths necessitate delicate balance between economic expansion, military development, diplomatic influence, cultural achievements, and technological research, which poses high requirement for learning and reasoning.
Dynamic space. As the game unfolds, players continuously produce new units, construct additional cities, engage in battles, and conquer other players. Consequently, the state and action space of player undergoes dynamic changes throughout the gameplay. Designing an effective decision model for the agent to adapt to this evolving space presents significant challenge.
Multi-agent. Multiple players can interact with one another, including hand-crafted Al players provided by Freeciv the server side. CivRealm allows multiple agents connect the same game simultancously, facilitating self-play training.
General-sum. Players are free to form alliances or wage war against others, rendering the full game a general-sum game that necessitates considerations of both cooperative and competitive strategies.
Changing players. The number of players can fluctuate during game due factors like revolts or civilization conquests, introducing new players controlled by built-in AI or removing existing ones. Such changes often result in significant alterations to the state-action space.
Comm. Players can communicate explicitly using two types of communication: diplomatic actions (e.g., adding clause) and natural language chat through a chat box. This feature enriches player interactions and enables LLM agents to fully leverage their natural language processing capabilities.
A.1.1 OBSERVATIONS
In Freeciv, players primarily interact with map composed of tiles where units and cities are situated. Additionally, Freeciv offers three tabs for conducting operations related to technology, diplomacy, and government. Consequently, the observations we extract from the Freeciv graphics encompass information pertaining map tiles, units, cities, technology, diplomacy, and government.
Accordingg to httph///wwngggggoommomm///////sa
18

---

Published as a conference paper at ICLR 2024
CivRealm provides set of feature layers for the observations of maps. We list these layers in Table 2, where M x N denotes the map size. Specifically, status, type of terrain, owner of files, unit owner, and city owner are layers with categorical values; infrastructures and output are layers with binary values; unit distribution layers are with scalar values. The upper bound of the unit number on tile and the player number is 255, covering all possible scenarios in gameplay. The status being O means a tile has never been explored, being 1 means a tile has been explored before while being covered by war fog currently, and being 2 means a tile is within the vision range of the objects controlled by our agent. The rype of terrain and owner of tiles denote the terrain type and owner of each tile, respectively. The infrastructure and output represent whether each tile has the corresponding infrastructure and resource output, respectively. The unit distribution means how many units of each specific type are on each tile. The unil owner and city owner record the owner of the units and city on each tile respectively.
Observations of unit are listed Table 3. For each unit in the game, it belongs to other players, its features of common unit field can be known, while features of both common unit field and my unit field can be known if it belongs to our agent. For features of my unit field, its default value is set as -1 for units of others. The upper bounds of some features, which can be theoretically infinite, are set to very large values to cover all possible scenarios in gameplay. These settings also apply to the subsequent features related diplomacy (Table 4), government (Table 5), city (Table 6), and technology (Table 7). Please refer to these tables for details.
A.1.2 ACTIONS
We list the details of the actions supported by CivRealm in Table 11. These actions fall into five groups: unit, city, diplomacy, government, and technology. The unit actions demand precise control from players, enabling them to explore the map, execute tactical maneuvers, and construct infras- tructure on the map. Additionally, there are miscellaneous unit actions we do not mention in the main text, including unit upgrading. trade route establishment, etc. The city actions empower play- ers to produce units within their citics and enhance urban development to boost production outputs. The diplomacy actions facilitate the establishment of diplomatic relationships and the exchange of information, such as vision and technology. The government actions grant players the ability to instigate revolutions change their government type and adjust tax, science, and luxury weights to balance various aspects of their civilizations. The technology actions enable players to define the direction of their technological development.
It is worth noting that certain actions are parameter-free, meaning their targets are self-evident (e.g., increasing the tax rate). Conversely, other actions are parameterized. requiring additional parame- ters to specify their targets (e.g., the embarking action necessitates specifying which boat the unit embarks on). Due to the dynamic change (i.e., the number of units, cities, and players) in Freeciv, the parameter space of parameterized actions can also vary, which requires additional consideration when designing greision-makinn agents.
The specific design of the parameter space for unit actions also deserves detailed explanation. It's essential to note that the game map can be of arbitrary size, and certain unit actions require specifying target on the map. Consequently, if we were to allow unit actions to specify arbitrary targets anywhere on the map, the action space would become excessively vast. To address this challenge, we restrict units to specifying targets within range of nine tiles. This range includes
Table 2: Observations of Map.
Fields
Attributes
Value domains
Descriptions
Status
0,2)
Type of terrain
00, 13
Size: M
Basic map
Owner of tiles
(0, 255]
Infrastructures
34 layers of size M N 6 layers of size N output types
0 or 1
Output
Unit owner
Units and city on each tile
Size: M x
City owner
(0, 255)
Unit distribution
52 layers of size M x for 52 unit types
19

---

Published as a conference paper at ICLR 2024
Table 3: Observations of Unit.
Fields
Attributes
Value domains
Descriptions
[.
X-coordinate
[]
Y-coordinate
Owner
[5,]5
Player the unit belongs to
HP
00 65535]
Health point of the unit
Produce cost
Cost needed to produce this type of unit
Veteran
Whether the unit
or 1
transport
Whether unit can transport other units
Unit type
One of 52 unit types
[0, 51]
Common unit field
Obsoleted by
The unit type this unit can upgrade to
Attack strength
Affect the attack success rate
Defense strength
[0, 63535)
Firepower
The damage of a successful attack
Unit ID)
Moves left
Actions the unit can take in this turn
My unit field
Home city
City supports this unit
[0,32767]
Upkeep shield
Upkeep gold
Resources needed to support this
Upkeep food
Table 4: Observations of Diplomacy.
General
Attribules
Values
Descriptions
Player ID
[o, 255]
Team
Name
text
Is alive
Or
Common player field
Score
65336)
Turns alive
How many turns the player has lived for
Nation
[0, 559)
Embassy
Describe there embossies between players
text
Love
Describe players' attitudes others
Mood
Peaceful Combat
My player field
calegorical vector ol my diplomacy states with other players; anmistice, war, ceasefire, etc.
Diplomacy
[0.6]
Table 5: Observations of Government.
General
Attributes
Values
Descriptions
Government
[]
Common government fields
Government name
government
[0,6]
Goal revolution
Gold
[0, 65535]
treasury
Revolution finishes
current revolution complete
My government fields
Science
Tax
[0, 1001
Goverment investment for each aspect.
Sum to 100.
Luxury
the eight neighboring tiles (north, northwest, west, southwest, south, southeast, east, and northeast), along with the tile on which the unit currently located. For targets beyond this predefined range, units must first move to neighboring position and then designate those targets. This approach ensures that we maintain a manageable unit action space.
A.1.3 EVALUATION METRICS
CivRealm adopts game scores across 16 dimensions provided by the game engine to assess playing performance, including population, economics, production, cities, researched technologies, military units, wonders, research speed, land area, settled area, gold, units built, units killed, units lost, and
20

---

Published as a conference paper at ICLR 2024
Table 6: Observations of City.
General
Attributes
Value domains
Descriptions
City
text
X
[0, .]]
X-Coordinate
Common city field
Y
[0,N]
Y-Coordinate
Owner
Player this city belongs
[0,5
Size
City ID
Food stock
Shield stock
Granary size
Buy
Cost buy the undergoing production finish current production
Turns complete
Luxury
Science
[0,32767]
Food
Gold
Resource outputs in each turn
Shield
Trade
Bulbs
City
City corruption
City pollution
My city field
Growth in
text
turns for city population to grow
State
[0. 2]
City state: disorder, peace, etc. Unit or building Unit or building type being produced
Production kind
00,
Production value
[0, 67
People angry
People unhappy
00, 127]
Number of people each mood
People content
People happy
Surplus food
Surplus gold
[- 32768, 32767]
Surplus shield
Surplus trade
Can build unit
Binary vectors corresponding units buildings
Can baild building
0 or
Having Buildings
Last completion
32767)
Turn No. when city
completed the last production
Table 7: Observations of Technology.
General
Attributes
Values
Descriptions
Reseurch name
text
Common technology fields
Researching
[7]
the technology being researched
Tech of each type
0
each technology been researched
Bulbs researched
Accumulated technology bulbs
Tech upkeep
Cost keep current technologics
[0. 65535
Science cost
My technology fields
Researching cost
Tech goal
[0,87]
Techs researched
Last researched technology
units used, well aggregated score for the overall performance evaluation. This aggregated score aggregates multiple factors, i.c., population, researched technologies, wonders, units built and killed, culture, and if spaceship criteria is achieved by the player.
21

---

Published as a conference paper at ICLR 2024
Table 8: Definition of mini- gami-.
Category
Name
Introduction
SettlerBuildCity
Move suitable for building a city.
WorkerBuildinfra
Command workers to build infrastructures for improving cities.
Development
CityTile
Arrange work tiles to speed up producing a world wonder. Transport settlers by ships to another continent and build cities.
TransBuildCity
LandRattee|Ancieent
Defeat enemy units on land tiles (units from various ages).
Medieval, Industry. Modemn]
Battle
LandBatteAltackCCiy
Conquer enemy city.
LandBatteDefendCity
LundBarleDefendCiiy against enemy invasion for certain number turn.
NavalBattle
Defeat enemy fleet ocean (with Middle Times frigates). Defeat enemy flect on the ocean (with several classes of modern ships).
NavelBatleModde
Diplomacy
14
Techs
Techs
Trade technologies with another civilization.
A.2 MINI-GAME DETAILS
We provide the concrete definition of each mini- game in Table 8. The mini games belong to three classes -evelopment, battle, and diplomacy, covering a diversity of typical sub-tasks of the Civiliza- fion game. The development class of mini-games is closely related to city tile's combined resource outputs (food, production, and trade), while the battle class of mini-games focuses more on unit movement/attack operations. The player should be aware of the unit attack/defensc/HIP points to make full use of them. As for the last class of mini-game: diplomacy, it is about the art of trade and negotiation to achieve maximal interests for the player's civilization.
mini-games
A.2.1 VICTORY CONDITIONS, SCORE/REWARD SETTINGS AND DIFFICULTY LEVELS
Based on each mini-game's characteristies, we design set of evaluation principles for them. For instance, assess development mini-games by verifying whether the combined resource outputs (food, production, and trade) from city tiles surpass a predetermined threshold. The success of battle mini-games is determined by the successful annihilation of enemy units the conquest/defense of designated cities. We determine the achievement of diplomacy mini-games based on whether the agent negotiates favorable agreements with other players, such exchanging valuable technologies. For the details of reward settings and the computation of victory conditions, please refer to Table 9.
We additionally partition the mini-games into different difficulty levels (easy, normal, and hard) to better evaluate the models' capability as well as enable curriculum design future work. Specific criteria for the difficulty level divisions can be found in Table 9.
A.2.2 MINI-GAME RANDOMIZATION
To ensure sufficiently diverse and balanced mini-game instances, we conduct set of randomization processing during mini-game generation. Table 10 gives exact perspectives on which randomization operations are performed for each type of mini-game.
B TENSOR ENVIRONMENT
Fix Space Size Deep reinforcement learning algorithms generally demand constant size of the ob- servation space and the action space, but the numbers of units and cities vary through the gameplay of CivRealm, introducing inherent conflicts. To solve this problem, tensor environment embeds size-varying observations into a large constant-size observation space by truncating excessive enti- ties and masking out con-existent ones.
B. OBSERVATION
The tensor environment provides API for tensor models. Observation of tensor environment is big tensor including all observations of map, unit, city, diplomacy, government, and technology in $ A.1.I.
22

---

Published as a conference paper at ICLR 2024
Table 9: Score, victory, reward, and difficulty of mini-games.
1D
Scone Setting
Victory Canditton
Stepuise Rewand
Tifiiully
product trade
2ttp-q!
Danis
bytie
S.255 land,
Eany
Eety P)
nurmoll 25
Ti
1
the mumber nanss complete the
No
hart,
Ngii number city built on another land
N2
normal,
Treer
15
599
((Nautt) - Manu Mauite
((Num))
of player
A [Naatgonnn) - Nini Nitay Mami 4[Natyoool] aou Wayy NN
Mata Masy
Nett 0
(Naagaant) ((Nonoomn)) ((Nom))
hard, nurmanl easy.
Maua// 21111 Mosil Na S Namnd aum 0.000 Smn man
Mang
12
8 ((Naunt) Nonr N A(Nanat N mom
Ne Wanw//g > 0.5 & No Nan (N S
((Num))
14
Nauhh. 2-
AL-kkill level
Table 10: Randomization setting of mini-games.
ID
Terrain
Resource
Unit
City location
Tech degrec
type
location
type
location
type
location
number
1
/
3
/
/
5-9
/
V
10
/
11
12
13
14
B.2 ACTION
There are five types of actors in the tensor environment, i.e., unit, city, diplomacy, government, and technology. Each type of actor has its corresponding action space and the overall action space is the joint action space all actor types. For an actor, each action in its action space is denoted by an action class name associated with parameters. For example, targel tile is denoted by the action key: goto_num where goto is the action class name and num is a parameter describing the direction, thus action Go to the north tile is denoted as goto I where the parameter corresponds to the north direction.
action in action class Go
chosen action returned the tensor environment a triplet including the actor type, actor ID, and the action key. For example, assuming action Go to the north tile is chosen for a unit whose ID is 121, then the action triplet returned to the tensor environment (unit, 121, goto_I).
C LANGUAGE ENVIRONMENT
C.1 OBSERVATION
Language environment provides API for language models. Observation of language environment is a dictionary containing a world observation, and the observation of each actor.
To understand the global situation and circumstances, world observation consists of multiple tences summarizing the game board. includes statistics on our units, cities, enemy units, enemy cities, and if are in time of peace war, etc., for example, "We have 10 units: 3 Warriors, Workers, Settlers, Diplomacy and I Explorer: We can see 4 enemy units. We have 5 cities of a total size of 14. We can see I enemy city and other cities. We are under attack.".
Observation of each actor consists of the actor's 's name, a list of its available actions, zoomed-out, and zoomed-in observation dictionary corresponding to its macro- and micro-observations. The
23

---

Published as a conference paper at ICLR 2024
Table 11: The actions supported by CivRealm
Component
Action description
Parameter
Go
tile
the target tile
Enter the farget file for random events
a target mooring an ocean Disemburk from larget beat mocring in an ocean tile
the target bost unit
Unlond all units carried by the transporter unit Bourd a boat mocring in the city of the current tile
Dehoard a boat mooring, in the city of the current tile
Fortffy in the current tile
Attack unit larget tile
the target tile
players to join
target unit
Conquer a city helongs to other players
Sabotage a city belongs to other players
the target city
Steal technology from a city helongs to other players
Mine in the current tile
Irrigate in the carrent tile
Unit
Build road in the current tile
Build railroad the curceut tile
Build city in the current tile
Ruild airhase in the current tile
Build fortress in the current tile Transform the terrain of the current tile
Pillage an infrastructure in the current tile
Cultivate the forest the current tile plain
Upgrade
Dishand
itself save cost
Keep the current activity in this turn
Set the unit's home city as the city in the current tile
Join the city in the current tile (increase city population)
Sell goods in the target city's 's marketplace
Investigate tanget city belongs other players
the target city
Establish embassy target city belongs other players
Establish route
unit's home city turget city
Chaose a working tile for city
the target tile
Do not week on a tile
Buy building unit
City
Change the type of specialist
type of the target specialist
Sell building
the lurget building
Construct building
Produce unit
the target unit
Start a negotiation
End negotiation
Accept treaty
target player ID
Cancel meaty
Cancel
Diplomacy
clause
target player ID target clause type target player giver ID target technology target player + giver ID bow much gold
trading tech clanse
Add a trading gold clause
Add trading city clause
targot player ID + giver ID target city ID target player ID parameters of the target clause
Remove clause
Revolution
the target Govennment
Government
Set nites luxury + science tax Set a current research goal
luxury science tax
Technology research
the target technology
Set a future research goal
zoomed-in observation dictionary corresponds to a mini-map, centered on the location of the actor, with a customized length and width. The zoomed-out observation dictionary is similar but corre- sponds to a larger sub-map, with a larger length and width, at the expense of granularity. Hence, the zoomed-in observation describes the detailed surroundings of the actor, while the zoomed-out ob- servation represents its general perception of the distant surroundings. Instead using coordinates that not natural language friendly,
tile dir num_nummmmmmm dir dir_num num block dir num dir num
24

---

Published as a conference paper at ICLR 2024
to describe location related to the actor. For example, tile north eest 1 means the tile whose coordinates are (1, related to the actor's location. The tile hilessouth__ 1_ west 2 means the tile whose coordinates are (-2, -1) related to the actor's 's location.
Please refer to Figure 10 as an example of an actor's observation dictionary. Zoomed-in observation corresponds to 5 x mini-map centered on the actor's location, thus it contains 25 tiles. Zoomed- out observation corresponds to a 15 x 15 sub-map. In this example, we define that a block made up of x tiles, thus the above 5 5 mini-map is the block where the actor is currently located, and the remaining tiles in zoomed-out observation make up the 8 surrounding blocks. Based on these settings, we get the observation of the actor. Each tile or block has a list containing its status, terrain, and the infrastructures on it. Besides, the information on units and cities, as well as their owners is also on the list. Unit and city owners are also associated with diplomacy tags, for example, units on current_ file belong to myself player; 3 cities in block_north__ / belong to an Alliance player.
C.2 ACTION
The action returned to the language environment is a triplet including the type of the actor, actor ID, and the name of the chosen action in available_actions.. For example, in Figure 10, the type of the actor is unit, its actor ID is 121, assuming that action Go to the north tile has been chosen, then the final action return to language environment is (unit, 121, move North).
D ESTIMATION OF STATE ACTION SPACE SIZES
The following estimations focus on a temporal section of the game. However, viewing the game as a long-term planning task, the analysis may change lot, but the result would also be of great complexity. If we consider that the building actions are temporal, our decision is not single-turn Markovian, the state space would be multiple times larger on the exponent.
On Turn 5. Under ruleset Ci Civ2Civ3, each nation started with 2 Settlers, 2 Workers, and an Explorer. On turn 5, each Settler has acted at most times, so could appear 121 possible places as itself, and could build a city at 81 positions. Each city originally had no less than 5 building options, and 24 working options. Each Worker, at the same time, could occur at similar positions, and when some work is done, the range it occurs will be smaller. Explorers could move 3 tiles per turn. So state space for 2 Settlers (or the cities they turned into) is about ((11 + 81 * 5 24)²
1521- 5 * 24 121) x 5 x 10" large (square of single Settlers status, counted in cases: a city is not built/ / city is built, with city options, then excluding all conflicting cases where cities
too close to each other. Halving means two Settlers are identical in practice). Workers has states about ((121 + 81 3)2 121 81 * 3) 66066. Explorer contributes 312 961 states. So total state space of size about x 10
With all units in their original form (no cities built), explorers can move in 9 directions and stay (9 moves in total), each Worker has 9 moves and two working options on each tile on average. Settlers have 9 moves and a build city. So in this case, the action space is of size 19800.
With 100 units and 50 cities. simple observation is: on map of 80 50 where half of the tiles are lands (2000), each unit has 1000 free places to stay (the other half are deep in enemies' territory). Thus units may contribute 1000000 100 10000 different states. For the 50 city locations, suppose each may find 10 options on average when they are built, then 1050 additional complexity becomes factor. The cities" current improvement brings lot more complexity. Given each city 20 possible improvements to build, the number of states for each city may go up to 220 xe 10 providing 10000 states in total. Therefore, we may conclude that the state space is at least 10650 in size.
About the space of actions. 100 units bring at least 10000 actions from moving and sentry / defense / building. As for cities, with relatively high research levels, each city could produce at least 20 different units/improvements, thus the action space for cities is 2050 x 10 In total, we would estimate the action space no smaller than 10 even without considering the actions on tech, resea rch, tax rate, di diplomacy, etc.
25

---

Published as a conference paper at ICLR 2024
E METHOD DETAILS
E.1 T'ENSOR-BASED REINFORCEMENT LEARNING
E.1.1 CHALLENGES
CivRealm presents several challenges for contemporary reinforcement learning methods, which can be summarized as follows:
Complicated environment dynamics. CivRealm has an intricate and multifaceted game mecha- nism that closely parallels the complexities of human society, making it an exceptionally challenging domain for both model-free and model-basee tensor- lensor-based learning methods. The game's 's mechan- ics encompass a wide array of elements, including diplomacy, economics, technology. and military strategy, with decisions having far-reaching consequences akin to historical societal developments. This intricate interplay of factors mirrors the multifaceted dynamics real-world human societies, rendering challenging for tensor-based agents effectively capture and navigate. Whether adopt- ing model-free or model-based approaches, these agents struggle to interpret the nuanced interac- tions, complex dependencies, and evolving conditions within the game, underscoring the nature of this environment and the need for innovative strategies to tackle its dynamics.
Overwhelming observation information. One of the primary challenges faced by tensor-based RL agents is the assimilation of vast amounts of information in manner that lacks semantic un- derstanding of the tensor values. the context of CivRealm, the observation space is exceptionally rich and expansive, providing intricate details about units, cities, players, governments, and more. However, not all of this information holds immediate relevance for effective decision-making during each turn. The dynamic, ever-changing nature of the information, coupled with its variable length, presents a formidable obstacle for the agent to discern the significance of individual components and to establish connections among different pieces of information. Moreover, the inherent dynamism of the game, involving fluctuating player numbers and evolving game conditions, further compounds the complexity of this challenge.
Dynamic multi-Level action space. CivRealm's action space is complex and hierarchical, making it impractical to model actions uniformly. Instead, actions need to be decomposed into multiple levels, with decisions made sequentially for each component. This introduces design challenges for neural networks and the selection of optimal actions, as the agent must navigate through this multi-level action hierarchy.
Sparse, delayed, and asynchronous rewards. In the context of CivRealm, rewards are sparse, delayed (e.g.. it can take many turns to finish a building), and often asynchronous. The agent receives a reward only when specific events or tasks are accomplished, making it challenging to provide timely feedback for learning. Many crucial actions, such exploration, do not yield immediate rewards, and even actions that can lead to rewards are not individually rewarded. Instead, rewards typically accumulate over a series of actions in multiple turns, making it difficult for the agent to attribute its actions to future rewards. Additionally, rewards are received only at the end of a turn, which may mislead the agent into favoring turn-ending actions over actions that contribute to long- term objectives.
Diverse diininn//erminationn conditions. Achieving victory in CivRealm is multifaceted, with var- ious development paths, including military, technological, and temporal strategies. The evaluation of the agent's 's state varies depending the chosen strategy, which complicates RL training. Tradi- tional RL methods heavily reliant on reward signals may struggle to adapt to these diverse pathways success, where cooperation and competition coexist.
In summary, CivRealm poses significant obstacles for RI agents, necessitating innovative ap- proaches to tackle the issues of sparse rewards, overwhelming information, complex actions, and diverse victory conditions. Addressing these challenges will be essential for creating intelligent agents capable of mastering the game of CivRealm.
1.1 FORMULATION OF GAME AS MDP
We consider a discrete-time iafinite-horizon Markov decision process (MDP) defined by a tuple (S, A, po, P, R., 7), where S C ded is the space of states, A dR the space of actions, po (8o)
26

---

It is worth noting that, in the context of turn- bune- game CivRealm, we consider timestep in the action level instead of the turn level. That is, every action Published as a conference paper at ICLR 2024
the distribution over initial states So. P(st+1|st, te)the transition function, R. the reward function, and y E (0. 1] a discount factor. I ( ss; f) is the policy parameterized by 0 E I The objective of an RL algorithm is to train policy that maximizes the expected sum of discounted rewards: J(0) = Er--((::)) E ('r(s; (ir,,a) az)
taken within turn will lead to change in timestep, and turn-done is also considered to be action.
E. NETWORK DESIGN
To effectively handle multi-source and variable-length inputs, we draw inspiration from AlphaS- tar [80] and implement serialized hierarchical feature extraction and action selection approach. This method involves generating layered actions and predicting value function outputs, and our neu- ral network architecture comprises three main components: representation learning, action selection, and value estimation.
Representation. At the representation level, we adopt hierarchical structure. In the lower layer, we extract controller features using various models like MLP, Transformer, and CNN, depending on whether the input single vector, sequence, or image-based. These extracted features are then fed into transformer to facilitate attention across different entities, creating globally meaningful representations. Additionally, we utilize an RNN to combine the current-state features with the memory state, enabling conditional policy decisions based on the state history.
Action selection. At the action selection level, we leverage the learned representations to make decisions. In the actor selection module, we determine the primary action category to be executed, including options like unit, city, government, or termination. Subsequently, we employ a pointer network to select the specific action ID to be executed, followed by the determination of the exact action to be performed.
Value estimation. To enable the of actor-eritic algorithm, we incorporate a value prediction head after the representation learning phase. This shared representation part of the network benefits both the actor and critic, enhancing training efficiency.
Training. We use the Proximal Policy Optimization (PPO) [62] algorithm to train the agent. To mitigate the on-policy sample complexity of PPO, we harness Ray [4] for parallelizing tensor en- vironments, optimizing training speed and efficiency. We configured the actor update for 5 epochs, employing a clipped value loss with clip parameter of 0.2, and using one mini-batch per epoch. The coefficients assigned to the entropy term and value loss were 0.01 and 0.001, respectively. The length of each episode was set at 125 steps, and we collected training data across parallel environ- ments. The learning rate for the Adam[41] optimizer was established at 0.0005, with an optimizer epsilon of 0.00001. These parameter settings were carefully chosen to maintain a balance between the effectiveness of learning and the stability of the algorithm.
E.2 BASELANG: BASELINE LANGUAGE-BASER AGENT
LLM's emergent capabilities grant it a vast foundation of human knowledge, empowering it in vari- areas such as task generation, open-world long-term planning, and solving complex problems.
In CivRealm, LLM has three distinct advantages. Firstly, our environment prioritizes long-term planning and strategic gameplay over low-level control, allowing the agent to interact with the en- vironment in natural language rather than precise control actions. Secondly, Freeciv is a turn-based game, not requiring real-time interaction, providing ample time for L.M engage in long-term planning. Thirdly, diplomatic operations are challenging for tensor- algorithms as agreements are usually achieved through conversations, but natural language allows for direct diplomatic inter- actions. Additionally, LLM's knowledge encompasses a wide range of human civilization evolution knowledge, including history, warfare, politics, technology, etc., making it more akin to human players in terms of prior knowledge.
E.2.1 CHALLENGES
Due to the high complexity of the Freeciv game itself, there are many challenges in the process of constructing a language-base agent, including:
27

---

Published as a conference paper at ICLR 2024
Multiple role-playing. Different from agents for Minecraft, the agents for CivRealm must play different roles at a time: controller of different units with different locations and abilities, Mayer of different cities, leader of the nation deciding research directions and diplomatic strategies. A language agent must play all these different but related identities simultaneously in a turn.
Sparse and complex observation. In the early stages of the game, there are extensive unseen areas in the player's ' field of view. This makes it necessary for early-stage agents to first learn to efficiently explore the environment using powerful prior knowledge, posing a significant challenge in constructing these agents. Additionally, the vast map size (4000-tile maps for our full game setup) and complex information on each tile overwhelm the context window of all language models. Thus, LLM agents have to prioritize reading the map properly and efficiently.
Long-term effect of actions. Freeciv is game that requires a strong focus on long-term strategy and planning, where early moves can have an impact on decisions hundreds of turns later. However, the context length limitation of LLMs restricts their analysis to short time-frames, and figuring out valuable action information within sparse reward temporal trajectory data is significant challenge.
How does the agent improve itself. Training or fine-tuning LLM is highly challenging task, and the lengthy time span of CivRealm significantly increases the difficulty of collecting high-quality data. Therefore, we can only choose to use an in-context approach, continuously enhancing I.I.'s decision-making ability based experiential information gathered during interactions with the en- vironment. However, extracting core information from such a complex game remains a challenge, making it diffieult to let the agent improve itself effectively.
E.2.2 STRUCTURE
We adopted an AutoGPT-like approach build baseline language- based agent. AutoGPT [72] is a memory-equipped automated LLM agent capable of making autonomous decisions in context. The following sections will delve into observation, reasoning, commands, and structure separately. ion.
Observation. Due to the various challenges mentioned above, we need to provide our language- based agent with a more effective natural language-basee observation. While, at the strategic level, players require very detailed map information for effective long-term planning, for each specific unit, providing only the most basic information needs can fulfill its tactical requirements. Specifically, we designate an area of 5x5 tiles, totaling 25 tiles, centered on the location of each unit, as the observation provided to the agent.
Reasoning. For the reasoning module, we largely follow the design of AutoGPT, employing three modules: thought, reasoning, and command, to analyze information. The thought module deter- mines what actions the agent should take in the current state. The reasoning module analyzes the current information to understand what additional actions are needed to achieve the current thought and why. The command module summarizes the information from the previous two parts and presents specific steps for each operation, which in our case is a command to execute. The integration of these three parts forms the reasoning module of the language-based agent.
Commands. In our design, we allow language-based agents to choose between two commands in the planning stage: "manual and history search" and "final decision." " For the former, we first store a document related to Freeciv in a vector database. The agent can then use this command to query the database in a way that retrieves relevant information from the document based on semantic similarity. As for the latter, the agent selects actions for the currently controlled unit based on the current environmental information and the historical context information.
We represent this observation information using JSON, where the keys represent the relative posi- tions of each tile with respect to the current unit's tile. For example, "north_ 1_east_2 _" indicates a position one tile north and two tiles east from the current unit's 's tile. The value information for each tile corresponds to the resources present on that tile and distinguishes between friendly and enemy agents. We consolidate the information from these 25 tiles the observation for the unit currently controlled by the agent. For unexplored areas, we will not provide tile information.
Structure. After constructing language- based agent, created such agent for each unit, each having its own separate context history for planning and decision- making. In the initial design, used one agent operate all players, but due the inherent context limitations of the language
28

---

Published as a conference paper at ICLR 2024
model, it not possible to perform very long-term planning for unit's behavior. Therefore, believed it was necessary to build a separate agent for each unit.
When the interaction information exceeds the maximum context limit, use the Conversation- Summmay BuffrMeemrry module of Langchain to summarize the historical information. In addition to this, we established unit list within the system, taking turns allowing language-based agent to provide the corresponding action decisions, thereby achieving planning and control for each unit.
E.3 MASTABA: ENHANCING BASELANG BY A HIERARCHICAL STRUCTURE
BaseLang faced challenges due to the independent, isolated behavior of its entities. Mastaba builds upon BaseLang and succeeds in managing a nation in the game. Each entity operated independently with limited communication through observations, hindering efficient cooperation. To mitigate these issues, BaseLang constrained each entity's 's view to 5 x 5 tile area, limiting long-range planning- sigmificant drawback when communication between entities is restricted. Mastaba attempts to re- solve these problems by introducing primary pyramid structure to organize observations, agents, and decision decision-making, where different layers observation will be fed into different levels of LLM units for decision-making
E.3.1 DESIGN OVERVIEW
Hawk-Eye mapview. Inspired by the concept of hawk-eye vision, we have designed multi-layer observation. Recognizing that entities do not necessarily require detailed information beyond a local 5 x 5 tile area, condense data from 15 x 15 tile region into 9 blocks, each measuring 5 x 5 tiles. The central block represents the immediate local area, while the remaining eight blocks correspond adjacent directions. This arrangement forms 3 x grid, illustrated in Fig. 5. This pyramid-like structure strikes a balance between map complexity and the richness of observations.
The role of the advisor. In agent organization, each entity is associated with an independent LLM instance for communication. Additionally, we introduce a crucial entity called the "advisor" to oversee all other instances. The advisor responsible for monitoring fundamental game information concerning the entire nation. This includes data such as the total number of units (both military and militara,,,,, the number of cities owned by the entity, and the quantities of units and cities belonging to both enemies and other players. Ultimately, the advisor receives nation status report, indicating whether the entity is currently being invaded, conducting invasions, maintaining a state of relative peace, or experiencing a lack of communication. The advisor's 's primary role is to generate suggestions during each turn and disseminate them to all other agent prompts. This advisor entity plays pivotal role at the apex of hierarchical structure within the agent framework.
Specialized worker units. The structure of our worker units closely follows the BaseLang frame- work. In addition to the general worker prompt from BaseLang, we introduce three distinet types of specialists: Settlers, Workers, and Explorers. The key distinction lies in the examples provided within their instruction prompts, which are tailored to their respective unit types. For the general LLM worker, we continue to employ the instruction prompt from Basel.ang for all other entities. Furthermore, enhance the output accuracy by requiring each worker to reiterate the available actions in their "thoughts".
Decision workflow. Mastaba follows a decision-making workflow structured like a pyramid. Dur- ing each turn, the advisor initiates the decision-making process with an overarching assessment of the nation, taking into account cities, units, and potential threats from enemies. This decision is then communicated to the prompts of all other worker units. Subsequently, each worker unit indepen- dently selects an action for the entity it controls. Workers also have the opportunity to query a vector database to obtain knowledge from manual or stored experiences. Following a query, a worker must make a decision regarding the action to be taken by the entity under its control.
F MORE EXPERIMENT RESULTS
We provide more results of our benser-based RL method for mini-games in this section.
29

---

Published as a conference paper at ICLR 2024
infol "tn_tin_nni info [[ [''nntt]] [121]
"nmmee 'arrroo 121',
avvailable_actions" Kekepp activity', 'set Homecity', "upgrade', "cancel order",
mmove North', 'nove East', 'nove NorthWeett
observations':
"nininap': :
terrent_tile': Desert', '1 Irrigotion', '1 Road", '1 Reilrvad",
'2 Phatanx', '4 Warrion', '1 Chariot',
tumil owners are: myself player_0', cities nyselr player_e'l, "tile_north__ ['1 Hills', '1 Mine', '1 Road'],
"tite_sovth_1": Ocean'l,
'tile_east_1': I'1 Desert', '1 Irrigation', '1 Road ),
'tile_wost_1': ['1 Plains', '1 Irrigation', '1 Road ' "tite_nortt___eastt_1" Mountains'],
"tite_north_3__wst__11 ['1 Desert', '1 Irrigation', '1 Road"l,
"tite_sout__leest_..1 ['1 Plains", '1 Irrigation', '1 Road', '1 River", Workers', 'unit owners are: ny self player_0'), 'tile_sotth_1_wett_1''' ['1 Ocean'),
'tile_north__2" ['1 Hills'],
'tile_north_2_eatt__1, ['1 Forest', '1 Road"],
"tite_north___west__." ['1 Hills'),
"tile_nortt_2_eastt_2" ['1 Forest', '1 Road
'tite_nortt_22_wett_21" ['1 Grasstand', '1 Road
'tile_south_2'" 0cean'],
'tile_south_2_east__1 ['1 Ocean'),
"tile_south_2_wwst... ['1 Ocean'],
"tile_soutt_22eastt_." ['1 Desert', '1 Road', '1 Railroad']
'tile_south_2_wwstt.2" ['1 tile Lnexp"
'tile_east_2': ['1 Hills'],
'tile_nortt_1_osst_22:: ['1 Mountains ].
'tile_soutt__eeast__2: ['1 Desert', '1 Irrigation', '1 Road', River"]
'tilo_wost_2': ['1 Ocean', Whales'],
'tiile_north___west_22' ['1 Forest', '1 Road"),
"tile_south_1_west 2 Ocean']
upper_mapp {
surrent_block':
['1 tile unexplored',
'7 Ocean', Desert', Forest',
'1 Grassland', '4 Hills', '2 Mountains', '2 Plains',
Irrigation', Mine', '12 Road', Reilroad', River',
'1 Wheat', '1 Whales', '1 Workers', '2 Phalanx', 14 Warrior', Chariot', 'unit owners are: myself player 0',
cities myself player_8'), ['11 Farest', '4 Grasstand', Road',
'block_north_1.
cities Alliance player_2'1,
"lock_south_11" ('13 tiles unexplored', '7 Ocean' '1 Forest', '1 Grassland ]
'block_eest_11 ['16 Desert', '3 Forest', '1 Irrigation', '4 Mine'
'block_west_1'; ('2 tiles unexplored', '7 Ocean', Swamp', '1 Mine!),
"btack_nortt__eeestt.1 ['15 Forest', '3 Workers'],
"block_north...weet. ['6 Forest', Grassland"],
'block_south_..east..rr [33 Ocean', '2 Desert', '8 Forest', '3 Hills',
'2 Jungle', '1 Mountains', '2 Plains', '1 Mine"], block_south___wwet___p ('5 tiles unexplored', Ocean', Grassland')
Figure 10: language environment observation example. The length and width of the mini-map and upper-map, as well as the size of the block, can be customarily set.
30

---

Published as a conference paper at ICLR 2024
Value
Action Selection
Critic MIP
Actor Type MLP
Major Categery
Actor ID Pointer Net
Action Type MIP
of Actions
Representation
GRU
Clobal Reprosenitiinn
Global Transformer
MLP
M.P
Transformer
Transformer
Transformer
Transformer
Transformer
ResNet
Single Vectur
Sequeace
Emage
Rules
Player
Other players
Other Cities
Oterr Units
My Cittles
My Units
Map
Figure 1I: Network architecture of the tensor-based reinforcement learning agent.
0.44
0:39
10-00
9
Figure 12: Battle ancient era.
0115
gexx
Figure 13: Battle attack city.
Figure 14: Battle defend city.
304
300
304
304
300
305
304
300
1.0%
2.0
30C
Remandk
Nalle
Figure 15: Battle industry era.
0.50
63
Figure 16: Battle info cra.
Figure 17: Battle medieval.
31

---

Published as a conference paper at ICLR 2024
100
Figure 18: Battle modern era.
0.51
0.22
27M
250
Figure 19: Battle naval.
5:1E
find
700
20%
200
500
105
Figure 20: Battle naval modern.
rart
500g
5450
Figure 21: Development build city.
Baca
Figure 22: Development build infra.
330
CX
500
240
D6 Rerands
Sille Bon
Stp sward
Figure 23: Development citytile wonder.
BESE
CC1
-042
Figure 24: Development transport.
0.00
Figure 25: Diplomacy trade tech.
32


Reviews:
[{"review_id": "JelbU9YL6v", "reviewer": "Reviewer_3D5Q", "summary": "This is a dataset and benchmark paper that introduces the CivRealm environment. It supports a number of game features based on the Civilization game, including an agent-agnostic framework, a friendly interface, and a tensor-based agent interface, supporting a variety of custom tasks with custom goal definitions, etc. This is a very exhaustive benchmark for a long-horizon strategy-based game. The paper also discusses the performance of three methods (one tensor-based and one LLM-based).", "questions": "1. How robust is the server-client architecture? Did you do any experiments to test it?\n2. Is there support for designing custom games? I see Lua script being mentioned to specify the reward structure, new goals, etc. Is there also a support for new maps?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This dataset and associated framework is very useful and will support a lot of research for multi-agent settings involving various kinds of agents. \n2. The whole framework is very novel.", "weaknesses": "I'm not sure how robust the client-server architecture used in the framework is. There are no comments on the robustness of this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hpFxQl449O", "reviewer": "Reviewer_4Csd", "summary": "The paper introduces a novel testbed environment inspired by the Civilization game, called *CivRealm*, which is a multi-agent multi-goal long-horizon challenging environment. The authors also implement RL and LLM baseline agents and show that they struggle to make substantial progress in the full game.", "questions": "1. Why the success rate of the RL agent in some mini-games (e.g., SettlerBuildCity) is higher for the hard setting than for the easier settings?\n2. Can the authors comment on the reason for instability and the sudden drop in performance in Figure 9?\n3. It would be great if the authors could provide videos of the experiments or visualization ways in the future code repo.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The authors present a novel and challenging testbed for agent studies, which is a big contribution.\n2. The experiments on the proposed agents also set up baselines that follow-up works can improve upon.\n3. The environment also has mini-games that can benefit the (multi-agent) RL community.\n4. The paper is clearly written and easy to follow.", "weaknesses": "1. Some details are not clear enough. As this is a multi-agent environment, what are the baseline RL and LLM agents play against? I didn't seem to find such details in the description.\n2. The proposed baselines are not working in the (full game) environment. Maybe it is better to also set different difficulty levels (e.g. by different task horizons) for the full game so that later research can more easily be evaluated on the benchmark.\n3.  Is Figure 2 the real situation that the LLM agents discover or the expected situation? If it is what the LLM agents discover, why does Tit still behave poorly?\n4. The text in Figure 4 is too small to view.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "csYGUgxyvN", "reviewer": "Reviewer_XJ4N", "summary": "The primary contribution of this paper is the introduction of the CivRealm environment, which is built on Civilization VI, a complex and popular video game. This environment offers a platform to study long-term planning, multi-agent interactions, and intricate decision-making etc.", "questions": "This environment is based on an open-source game http://freeciv.org/. From the development aspects, what is the main modifications/effort has been done upon the open-source game?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper doesn't just introduce an environment but also provides benchmark tasks and baseline.\n- The API provides a comprehensive log of game state and actions.\n- It allows AI agents to interact at both high (strategic) and low (tactical) levels.\n- It focus on long-term planning which is crucial to agents", "weaknesses": "As a work in the area of datasets and benchmarks, I think a lot of the details are in the code, and I don't have good confirmation of those details just by virtue of the text in the paper. This environment is based on an open-source game http://freeciv.org/. I don't know how different is the open-source game and the environment introduced by the authors. It would be excellent if authors could give reviewers an early release of code to review.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)