Emergent collective intelligence from massive-agent cooperation and competition
Authors: Hanmo Chen
Stone Tao
Jiaxin Chen
Weihan Shen
Xihui Li
Sikai Cheng
Xiaolong Zhu
Xiu Li
Venue: Submitted to ICLR 2023 (2023)
Keywords: Reinforcement Learning
Multi-agent System
Emergent Behavior

Abstract: Inspired by organisms evolving through cooperation and competition between different populations on Earth, we study the emergence of artificial collective intelligence through massive-agent reinforcement learning. To this end, We propose a new massive-agent reinforcement learning environment, Lux, where dynamic and massive agents in two teams scramble for limited resources and fight off the darkness. In Lux, we build our agents through the standard reinforcement learning algorithm in curriculum learning phases and leverage centralized control via a pixel-to-pixel policy network. As agents co-evolve through self-play, we observe several stages of intelligence, from the acquisition of atomic skills to the development of group strategies. Since these learned group strategies arise from individual decisions without an explicit coordination mechanism, we claim that artificial collective intelligence emerges from massive-agent cooperation and competition. We further analyze the emergence of various learned strategies through metrics and ablation studies, aiming to provide insights for reinforcement learning implementations in massive-agent environments.

Under review as a conference paper at ICLR 2023
INTELLIGENCE FROM MASSIVE-AGENT COOPERATION AND COMPETITION
EMERGENT COLLECTIVE
Anonymous authors Paper under double-blind review
ABSTRACT
Inspired by organisms evolving through cooperation and competition between dif- ferent populations Earth, study the emergence of artificial collective intel- ligence through massive-agent reinforcement learning. To this end, We propose new massive-agent reinforcement learning environment, Lux, where dynamic and massive agents in two teams scramble for limited resources and fight off the darkness. In Lux, we build our agents through the standard reinforcement learn- ing algorithm in curriculum learning phases and leverage centralized control via a pixel-to-pixe -ixel policy network. As agents co-evolve through self- self-play, we observe several stages of intelligence, from the acquisition of atomic skills to the develop- ment of group strategies. Since these learned group strategies arise from individ- ual decisions without explicit coordination mechanism, claim that artificial collective intelligence emerges from massive-agent cooperation and competition. We further analyze the emergence of various learned strategies through metrics and ablation studies, aiming to provide insights for reinforcement learning imple- mentations in massive-agent environments.
INTRODUCTION
Complex group and social behaviors widely exist in humans and animals on Earth. In vast ecosys- tem, the simultancous cooperation and competition between populations and the changing environ- ment serve natural driving force for the co-evolution of massive numbers of organisms (Wolpert) Tumer T999: Dawkins & Krebs 1979). This large-scale co-evolution between populations has enabled group strategies for tasks individuals cannot accomplish (Ha & Tang 2022). Inspired by this self-organizing mechanism in nature, i.c., collective intelligence emerges from massive-agent coop- eration and competition, propose to simulate the emergence of collective intelligence through training reinforcement learning agents a massive-agent environment. We hope this can become stepping stone toward massive-agent reinforcement learning research and inspiring method for complex massive-agent problems.
Recent progress in multi-agent reinforcement learning (MARL) demonstrates its potential com- plete complex tasks through multi-agent cooperation, such as playing StarCraft2 (Vinyals et al. 2019) and DOTA2 (Berner et al. 2019). However, the number of agents is still limited to dozens in those scenarios, far away from natural populations. To support large-scale multi-agent cooperation and competition, we reintroduce the massive-agent setting into multi-agent reinforcement learning. To this end, we propose Lux, cooperative and competitive environment where hundreds of agents in two populations scramble for limited resources and fight off the darkness. We believe Lux is suitable testbench for experimenting with collective intelligence because it provides an open envi- ronment for hundreds of agents to cooperate, compete and evolve.
From the algorithmic perspective, the massive-agent setting poses great difficulties to reinforcement learning algorithms since the credit assignment problem becomes increasingly challenging. Some research (Lowe al. 2017) focuses on the credit assignment problem between multi-agents, how- ever, it lacks the scalability to massive-agent scenarios. To overcome that. we present a centralized control solution for Lux using pixel-to-pin pixel modeling architecture (Han et al. 2019) coupled with Proximal Policy Optimization (PPO) (Schulman et al. (017)) algorithm. Using that solution, we avoid the problem of credit assignment, with up a 90% win rate versus the state-of-the-ar policy

---

Under review as a conference paper at ICLR 2023
Isaiah al. 2021) proposed by the Toad Brigade team (TB) which won first place in the Lux Al competition on Kaggle
Through self-play and curriculum learning phases, observe several stages of the massive-agent co-evolution, from atomic skills such as moving and building to group strategies such as efficient territory occupation and long-term resource management. Note that group strategies arise from indi- vidual decisions without any explicit coordination mechanism or hierarchy, demonstrating how col- lective intelligence arises with co-evolution. Through quantitative analyses, further evidence shows that collective intelligence can emerge from massive-agent cooperation and competition, leading to behaviors beyond our expectations. For example, agents learn stand in diagonal row and move as a whole to segment off parts of the map as shown in Figure || Without any prior knowledge, this efficient strategy emerges from spontaneous exploration. Furthermore, we perform detailed ablation study to illustrate some implementation techniques which may be helpful in massive-agent reinforcement learning.
(a) Blue is our policy and Yellow is TB.
(b) Yellow is our policy and Blue is TB.
Figure I: Two episodes between our policy and TB where our Workers stand in diagonal row. Our agents discover it as efficient way expand the territory and limit the enemy's movement.
Our main contributions are 1) we reintroduce massive-agent reinforcement learning as scenario for studying collective intelligence and propose new environment, Lux, as starting point. 2) provide evidence that collective intelligence emerges from co-evolution through massive agents' cooperation and competition in Lux. 3) we discuss the implementation details our solution, which may provide valuable insights into massive-agent reinforcement learning.
RELATED WORK
Multi-Agent Environments. Many environments such Multi-agent Particle Environment (MPE) (Lowe et al. 2017) and Google Research Football (Kurach et al. 2020) are proposed to study multi- agent cooperation and competition. For multi multi-agent cooperation, StarCraft Multi-Agent Challenge (SMAC) (Samvelyan et al. 2019) provides a common testbench. However, SMAC focuses on de- centralized micromanagement scenarios with only approximately 30 agents play. massive- agent environments, Neural MMO (Suarez et al. 2021) is an open-ended Massively Multi-player Online (MMO) game environment with up 1024 agents. MAgent (Zheng et al., 2018) grid world environment that supports up to million agents. We propose Lux, massive-agent rein- forcement learning environment, which can support thousands of agents simultaneously acting at step. Unlike previous massive-agent aggent environments, Lux incorporates game dynamics that are similar to |attlecode| (2022) and MiniRTS (Tian et al.) (017). Moreover, Lux scales up the number of agents with frequent spawns and deaths, which opens up the potential for complex strategies in such large-scale and highly dynamic scenario.
Saal-Time-Strannn
(RTS)
Credit Assignment in MARL. Credit assignment between agents (Chang et al. 2003) is crucial challenge in multi-agen, cooperation. Several value-based multi-agent algorithms (Sunehag et al. 2017 Rashid al. 2018 Iqbal & Shal (018)) decompose global value into individual values using a linear model or neural network, which can be viewed as an implicit way of credit assignment. Another way of doing this computing an agent-specific advantage function. For example, Foerster et al. (2017) uses counterfactual regret to measure contributions to the team. In complex games, Berner al. (209)) and Ye et al. (2020) use hand-crafted team-based rewards for each agent as an explicit method of credit assignment. Compared to the implicit value decomposition method, this
httpp:://// www. www.wwww........ow..w.owwwwwww.. kaggle. hoom/c/lux-aai -1222/
2

---

Under review as a conference paper at ICLR 2023
explicit reward-shaping method requires prior domain knowledge and lacks generalization ability. However, both of them are limited to small population scenarios and are hard to scale to massive and dynamic agents. Han al. (2019) handles this problem using grid- wise centralized policy instead of decentralized policy. It uses a convolutional neural network to map from pixel-wise observations to actions each pixel, which avoids the credit assignment problem while achieving efficient multi- agent collaboration. Following this, we adapt this pixel-to-pel to-pieel architecture to the Lux environment with the PPO algorithm (Schulman al. 2017) and curriculum learning phases.
Collective Intelligence and Emergence Behaviors.
Collective Intelligence, including self- 1999; Woolley et al., 2010). has a long organization and emergent behaviors (Wolpert & Tumer
history connected with biological and economic studies. Research on emergent behavior usually emphasizes that group strategies emerge from multi-agent co-evolution in designed environment rather than hand-crafted collaboration mechanisms. [Baker et al. (2019) uses reinforcement learn- ing agents and autocurricula (Leibo et al. 2019) in the Hide-and environment, leading to the emergence of tool use. Yang al. (2018) proposes using million-aggent reinforcement learning to study how the agents' grouping behaviors will change with the environmental resources. Zheng ett al. (2021) uses two-level, deep RL framework to train agents and social planner in nomic environment, where optimal taxation policy emerges as the result of co-adaptation. [Johanson] al. (2022) studies the emergence of bartering behavior in microeconomics -aaeiccc---eeddd environment with producers and consumers. Dynamics in those environments usually induce agents' behaviors within human comprehension, thus limiting the possible emergent strategies. Since RTS games pro- vide perfect Petri dish for collective intelligence, our study absorbs RTS game dynamics into the environment where simple rules may induce complex group strategies.
LUX
Like the Earth, a suitable environment for collective intelligence to evolve must support massive agents' competition and cooperation. For that purpose, we propose an open-sourced environment Lux, where hundreds of agents in two teams compete for resources and build cities as illustrated in Figure
Resources
Uranium
CaCoal
Tree
Agents
Workers
CityTiles
Figure 2: A snapshot of Lux with hundreds of agents in teams. Workers can collect resources and build CityTiles. At night, CityTiles and Workers need fuel to stay alive and will be consumed by darkness fuel runs out. The team that owns more cities wins the end.
Setups. The map is a 2D square grid of size 12 to 32, scattered with different resources. episode consists of 360 turns, split into 9 Day/Night cycles of 30 days and 10 nights. There are two basic units named Worker and CityTile. Each team starts with one Worker and one CityTile. Workers can collect resources and build CiryTiles and CityTiles can also build new Workers. Workers collect adjacent resources automatically and convert resources to fuel when standing upon a friendly CiryTile. At night, CityTiles and Workers consume fuel to stay alive and will be consumed by darkness if the fuel runs out. At the end of the game, the team with more CiryTiles wins. More details about the environment in Appendix A.

---

Under review as a conference paper at ICLR 2023
Observation and Action Space. For observation, each team has perfect information about the game state, including the global map, its own, and the opponent's information. For actions, each team needs to make decisions for every Worker and CityTile. A Worker can move in 4 cardinal directions and build a CityTile when it has enough resources. Workers however cannot move onto tile with an enemy CityTile Worker. A CityTile can build Worker research to increase the team's 's research points. Sufficient research points unlock the ability for the team's Workers to mine high-level resources that convert more fuel like coal and uranium.
MARL in Lux. For MARL research, Lux raises a challenging situation for multi-agent modeling and the credit assignment problem. Distinguished from other environments, the number of agents in Lux is massive and dynamic. For 32 x 32 map. the number of agents in team can rise to 1000. Moreover, Workers and CūyTiles are built and lost all the time, bringing difficulty for multi- agent modeling of dynamic agents. A carefully-desinee in small-scale problems; however, with massive and dynamic agents, it becomes impractical due to the combinatorial complexity. Furthermore, the win-or-lose sparse reward throws another challenge on the hard exploration.
credit assignment scheme may be useful
RTS in Lux. At first sight, Lux seems like pocket-sized RTS game like StarCraft2. Agents in Lux need to balance economic decisions and individual control, which requires high-level coordination between hundreds of agents. However, the major difference between Lux and RTS games is the way of controlling units. In RTS games, the low-level unit actions are executed by fixed rules, which allows human players Al to focus on -trrttsstaaeeiees and economic decisions. In Lux, atomic actions such as moving and building are all controlled by the learned policy, resulting in an action space of approximately 10 magnitudes beyond StarCraft2 (Vinyals al. 2019). Thus, successful policy needs to learn atomic skills and group strategies together, which is significant in the emergence of collective intelligence.
4 METHODOL.OGY
Overall, our policy is trained using the standard algorithm PPO (Schulman et al. 2017) with Gen-
eralized Advantage Estimation (GAH) (Schulman et al.) 2016). For massive-agent agent coordination, we use pixel-to-pixee architecture the centralized policy (Han et al. 2019 Isaiah et al., [02T], tak- ing both observations and actions as images and using the ResNet (He al. (0166) structure as the backbone. To address the sparse reward problem, we design three phases with different rewards as a progressive curriculum. For clarity, refer to the "agent" each unit on the map and refer to the "policy" as the centralized policy network that controls every agent.
4.1 PIXEL-TO-PIXEL ARCHITECTURE
We model the massive-agent control problem using centralized policy with a pixel-to-pine archi- tecture (Han et al. 2019). The policy network takes images as input observations and outputs actions over each pixel in the form of an action map. More implementation details are in Appendix A.2 A.2
Policy Network Architecture. The architecture of our policy network is pictured in Figure The input image (C x H x W' consists of C' channels containing information about itself, the opponent, and the global state. We use a ResNet-style convolutional network as the backbone. For actions, we use a convolutional layer with kernel size and output channels the action dimensions. Moreover, we use a flattened layer and a fulle-connected layer for the value estimation as the critic. A valid action mask is used eliminate unnecessary exploration.
Why Centralized Policy. In Lux, our policy needs to control hundreds of agents each time step. While the decentralized policy in MARL is computationally efficient and easy to scale, it needs a carefully-desineel credit assignment mechanism. In Lux, however, as agents are massive and dy- namic, the credit assignment problem becomes increasingly challenging. To avoid that, we adopt a centralized policy controlling every agent over the map. This pixel-to-pixel architecture with a con- volutional network leverages the advantage of centralized and decentralized methods. Convolutional layers work as a parameter-sharing mechanism across agents, similar to shared policy networks in decentralized methods. This parameter-share sharing mechanism improves learning efficiency via data reuse. Furthermore, the deep stacked structure provides large receptive field for global information
4

---

Under review as a conference paper at ICLR 2023
Input C di W
Representation x 1f W
Action Dimension
Output
Residual Blocks
Conv
Softmax
Worker
Workers
Actions
CityTiles
Softmax
City Tile
Global-Time
CityTile
Figure 3: Policy network architecture. C is the input channels and H, W denote the map height and width. E is the feature map channel through the backbone. Output channels AWoker/CaayTie the corresponding action dimensions.
extraction and multi-agent communication, which naturally avoids the trouble of credit assignment (Han et al. 2019).
4.2 CURRICULUM TRAINING PHASES
The objective of agents in Lux is to own more CityTiles than the opponent, but the final result only provides a sparse reward (1 for win, -1 for lose), resulting in the hard exploration problem (Badia al. 2020). Reward shaping is a common method to handle this problem in reinforcement learning- et al.
However, hand-crafted rewards can easily direct agents into specific behavioral patterns with limited strategies. Hence, we design three phases with different rewards as progressive curriculum. First, we use dense reward to guide the policy towards basic skills. Then we gradually reduce the learning signals and utilize the sparse reward to encourage the policy to explore more diversified strategies.
Phase 1: Dense Rewards for Basic Skills. At first, basic skills. Specifically, four kinds of behaviors are given rewards, namely, the increase of Workers and CityTiles, Research Points and fuel. More details in Appendix A.2
hand-crafted dense rewards to encourage
Phase 2: Sparse Reward with Scaled Signals. In Phase 2, a reward is given only when an episode ends. However, our policy still needs guidance through long- term reasoning. We modify the reward with slight signal about the win condition, i.c. = /Nse Nople where Neaty and N denote the number of our own and the enemy's 's CityTiles, encouraging to own more CityTiles for the win.
Phase 3: Win-or-Lose Sparse Reward. The win-or-lose sparse reward (1 for win and -1 for lose) applied in the final phase. After human-des deennnd guidance in the first two phases, the win-or-lose sparse reward encourages our policy explore more advanced strategies.
5 EMERGENT COLLECTIVE INTELLIGENCE
Through massive-agent cooperation and competition, we have observed three stages of our agents' evolution. Training from scratch, agents quickly acquire atomic skills such as collecting resources and building cities. After around 5 million episodes, an elementary level of coordination appears on the regional scale with dozens of agents. As training proceeds, the coordination expands from re- gional to global scope, which includes long-term economic decisions and precise control of hundreds of agents. Those global strategies naturally arise from individual decisions due to massive-agen in- teraction and co-evolution without any explicit coordination mechanism, signifying the emergence of collective intelligence.
5.1 ATOMIC SKILLS
The first step of our agents is to get a grasp of atomic skills. Guided by dense rewards, Workers learn to move toward resources to collect fuel, and build and fuel the CityTiles, as shown in Figure
5

---

Under review as a conference paper at ICLR 2023
4a However, this stage agents more likely work alone and unable make group decisions. For example, Workers tend to build more CityTiles than they can support, leading to sudden loss of large cities as illustrated in Figure 4b as they run out of fuel.
(a) Workers collect resources and build CityTiles
(b) CityTiles run of fuel and collapse.
Figure 4: Illustration of atomic skills in self-play episode. Agents acquire atomic skills such as collecting resources and building CityTiles. However, due to lack group coordination, CityTiles often burn out fuel and collapse.
5.2 REGIONAL COORDINATION
As training proceeds, regional coordination appears, which involves dozens of agents in a local area. For example, agents learn carefully choose locations before building CityTile and develop self- organizing patterns for occupying resources efficiently. We describe a few examples of regional strategies:
Construction Planning, As CityTiles built next to each other can share fuel and reduce cost at night, agents gradually learn that the locations of CityTiles are important in city survival and fuel saving. We find that agents discover several patterns of construction planning as visualized in Figure 5 1) build CityTiles near the resources for quicker access to fuel sources. 2) build CityTiles long row to form cities that act like the Great Wall to prevent enemies' aggression. 3) build CiryTiles in blocks to reduce fuel costs at night.
(a) CityTiles built near resources.
(b) CityTiles built in
(c) CityTiles built in blocks.
Figure 5: Three emergent patterns of construction planning. a) build near resources for quick access fuel. b) build in row as the Great Wall for defense. c) build in blocks save fuel.
We use the city survival ratio (the final number of CiryTiles divided by the most number of CityTiles one episode) to measure how these building patterns work quantitatively. As shown in Figure ba the regional-scale construction planning effectively helps CityTiles fight off the darkness.
Territory Division. We have also observed a self-organizing structure where several Workers stand in diagonal row shown in Figure [ Those Workers simultaneously move forward and backward as a whole to keep the formation, and when any of them die, a new Worker nearby will fill in. In this shape, they can effectively guard and expand the team's territory and limit the enemy's movement. We measure a statistic called Five-Diagonal (how many times five or more Workers stand in diagonal row in game) to investigate how often this strategy is utilized. Results in Figure (b illustrate that the frequency our agents use this strategy generally increases with training in the long term, indicating it is acquired strategy rather than circumstance.

---

Under review as a conference paper at ICLR 2023
Prond
Paac
(a) City survival ratio: final number divided by most number Cityliles in one episode.
(b) Five-diagonal: how many times five more Workers stand diagonal row one episode.
Figure 6: Quantitative analysis of region coordination using city survival ratio and five- diagonal. Both metrics are evaluated using self-play. a) The city survival ratio increases as training continues, indicating that the regional construction planning effectively helps CityTiles live long and prosper. b) The frequency of the five-diagonal shape increases during the course of training, which demonstrates the gradual acquisition of this strategy.
5.3 GLOBAL STRATEGIES
As in micro-manageman management scenarios of SMAC (Samvelyan et al. 2019). regional coordination of dozens of agents often found multi-agent cooperation. However, our agents go far beyond that, achieving much larger-scale coordination between hundreds of agents. We provide interpretation and analysis of several global strategies as follows:
Sustainable Development. A key component in Lux is the balance of city development and re- source consumption. the early stages, the rapidly growing cities often face severe fuel shortages. Gradually, our policy learns to develop cities at sustainable speed in tune with fuel production depending on the resouree distribution and the opponent's behavior. Another phenomenon we have observed is the retention of trees. As trees the only renewable resource in Lux, forest protection significant in securing long-term fuel supplies. Our agents intentionally preserve trees from exces- sive deforestation and build CiryTiles near the woods in defense of the enemy's aggression. Another metric, total wood collect (the total collected woods divided by originally spawned woods) used to measure how this forest protection strategy influences our fuel supplies. Results in Figure 7a show how these protection strategies significantly improve the utilization efficiency of wood, resulting in our agent collecting more than 500% of the original wood on the map at times.
40.300
13.000
(a) Total wood collect: total wood collected di- vided by originally spawn woods.
(b) Total fuel: total fuel storage one episode using our latest model a fixed map.
Figure 7: Quantitative analysis of global strategies using total wood collect and total fuel. The metrics are evaluated using self-play. a) The utilization efficiency of wood increases as our policy grows the sense of forest protection. In one episode, our fuel storage accumulates until turn 240. After that, it tries to build more CityTiles for the win.
All In For The Win. Another surprising strategy is that when the episode is about to end, our policy will rapidly harvest all the protected trees and try to build as many CityTiles as possible for the win as shown in Figure 7b Furthermore, we observe that sometimes cities retain very little fuel at the end an episode, evidence that almost all the resources have been fully utilized, as any resources left at the end would be waste. Efficiently using all remaining resources before the end is very challenging because it needs the overall calculation CiryTiles, in addition to precise control of every agent. We think this strategy perfectly demonstrates the emergence collective intelligence through the combination of long-term economic decisions and maseive-agent mobilization.
total fuel consumption by all Workers and

---

Under review as a conference paper at ICLR 2023
EXPERIMENTS
In this section, we perform ablation studies to reflect on our policy implementation and general reinforcement learning algorithms under massive-agent settings: 1) we investigate the necessity of curriculum learning phases by training with different procedures. Results demonstrate that our cur- riculum design can help tackle the hard exploration problem caused by sparse rewards in the early stages of training and encourage the emergence of complex strategies beyond human design. 2) further demonstrate the generalization ability of our model across different map sizes. When eval- uating on maps of size 32, the policy trained size 12 still retains some basic strategies. After fine-tuning phase of only 1.8 million episodes, the transferred policy achieves 90% win rate against TB on maps of size 32. The results indicate our model can learn generalizable representations suit- able for the environment through learned spatial structures via convolutional layers. 3) we compare our centralized policy against a standard decentralized solution with arrfully-designed team-based rewards. The centralized policy achieves a 98% win rate. See implementation details in Appendix A.2 The decentralized policy implementation and experiment are in Appendix A..3.
6.1 DESIGN OF CURRICULUM LEARNING PHASES
We perform experiments investigate the necessity of our curriculum learning phases. Five differ- ent procedures are applied: a) Only Phase I; b) Phase I and 2 without Phase 3; c) Phase and 3, without Phase 2; d) Phase 1, 2, and 3 (the original procedure); e) Only Phase 3.
Only
106%
50%
48%
366
26%
10%
2
0 28 25 Episodes (n 10*)
1.5
15
29
18
Figure 8: The win rate curves from different training phases. All win rates evaluated against TB on maps size 12. Training with direct sparse rewards results 0% win rate, while training only with Phase dense rewards converges at around 50%. Compared to Phase 1+2, Phase 1+3 improves slower and results in a lower win rate of around 70%. Phase 1+2 achieves an 85% win- rate, and Phase 1+2+3 further boosts above 90%.
Phase utilizes dense rewards which are fundamental for helping the centralized policy acquire atomic skills for individual agents. As shown in Figure g our policy can hardly learn any basic skills when directly training with sparse rewards, resulting in a win rate of around zero due the hard exploration problem.
Phase utilizes a scaled sparse reward which plays two roles in the whole learning procedure, accelerating learning and improving performance. First, continuous learning with dense rewards converges at a 50% win rate, but the win rate rapidly rises to 70% with Phase 2. On the other hand, without Phase 2, switching from Phase 1 to Phase 3 is more challenging with lower performance even after training for longer period. This shows that the scaled sparse reward can work as a proper transition between dense rewards and win-or-loss sparse reward (applied in Phase 3) it explicitly tells the policy that owning more cities is the key to winning.
Phase 3 utilizes sparse win-loss reward which further boosts the final performance to above 90%. As the Phase training converges to a win rate of 85% without Phase 3, the win-loss sparse reward pushes our policy to go further and explore, resulting in overall 90% win rate.
8

---

Under review as a conference paper at ICLR 2023
6.2 GENERALIZATION OF REPRESENTATIONS FOR REINFORCEMENT LEARNING
We provide clear evidence that the learned representations from the convolutional neural network and reinforcement learning algorithms can be generalized different map sizes. First, we directly transfer the policy net trained on maps of size 12 size 32, As shown in Figure 2 basic skills are retained on larger maps such as Workers collecting and fueling cities, even showcasing some structured city construction planning surround and protect wood resources. Secondly, after an ad- ditional aue-tuning phase of around 1 million episodes. the policy quickly adapts to larger maps and achieves overall 90% win rate against TB, while training from scratch uses 1.6 million episodes for a 20% win rate as shown in Figure 10
.Larrin
Loennig
105%
65%
85%
75%
66%
55%
45%
856
15%
56
*pposees (o
Figure 9: Illustration of the general- ization ability in a self-play episode. When the policy trained on maps of size 12 is directly transferred to 32, some strategies are retained such as Workers building and fueling cities with plans.
The win-rate curves on X 32 maps of training from scratch and transfer. Both are evalu- ated against TB. After fine-tuning phase of million steps, the transferred policy achieves 90% win rate, while training from scratch uses 1.6 million episodes for 20% win rate.
Figure 10:
The results demonstrate the generalization ability of our model, which provides insights into speed- ing up policy training on large maps: we can pre-train policy small maps and then transfer it to large maps with a fine-tuning phase. This procedure significantly reduces the training time because smaller maps faster for environment simulation and network update. For example, the Lux environment simulation on CPU is 2.5x slower on maps of size 32 than size 12, and the policy network update on GPU is 5x slower maps of size 32. As training on maps of size 12 both time-saving and computationally efficient, our "Pre-train and Fine-tune" scheme achieves a higher win rate with fewer training hours.
DISCUSSION AND FUTURE WORK
We have demonstrated that collective intelligence can emerge from massive-agent cooperation and competition. As proof of concept, we propose Lux, an environment hosting hundreds of agents and incorporating RTS game dynamics. Through standard reinforcement learning algorithms and pixel- to-pixel centralized modeling, we observe several stages of agents' strategy evolution. Our agents exhibit ambitious group strategies based on accurate individual control of massive agents without explicit coordination mechanisms, signifying the emergence of collective intelligence.
We hope our work with Lux will be stepping stone toward artificial collective intelligence. In Lux, we observe the number of agents can reach up to 2000 in a single timestep. but this still pales in comparison to the millions or even billions of organisms cooperating and competing in nature. The Lux environment can be easily extended to host more agents as the experiments in Appendix A.4 but simulation and inference become extremely slow reaching the million-agent level. Going forward, the environment design and engineering as well as the training algorithm need lot of modifications to adapt to such scale. We also acknowledge that the RTS game dynamics in Lux may not directly coincide with real-world problems. However, with Lux blueprint, economic rules and dynamics like Zheng et al. (202I) can be incorporated, which may provide some reference for economic decisions and policies in the real world.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Zhaohan Daniel Guo, Bilal Piot, Steven Kapturowski. Olivier Tieleman, Martín Arjovsky, Alexander Pritzel, Andrew Bolt, and Charles Blundell. Never give up: Learning directed exploration strategies. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
Bowen Baker, Ingmar Kanitscheider, Todor M. Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch, Emergent tool use from multi-agent autocurricula. CoRR, abs/1909, htts/199.005525. 007528, 2019. URL BuRLhttp://axxiii org/ /wws/999.... 07528
Battlecode. Battlecode, 2022. URL. https:://bbtte lecode.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung. Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Christopher Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota with large scale deep reinforcement learning. CoRR, abbs/191..0660. 2019. URL http: arxiv. org/ abs /1912 06680
Yu-Han Chang, Tracey Ho, and Leslie Pack Kaelbling. All learning is local: Multi-agent learning in global reward games. neural information processing systems, 2003.
Richard Dawkins and John Richard Krebs. Arms races between and within species. Proceedings of the Royal Society of London. Series B. Biological Sciences, 205(1 161):489-511 -119,,5111111111 1979.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. national conference artificial intelligence, 2017.
David Ha and Yujin Tang. Collective intelligence for deep learning: A survey of recent develop- 2022.
ments. Collective Intelligence,
Lei Han, Peng Sun, Yali Du, Jiechao Xiong, Qing Wang, Xinghai Sun, Han Liu, and Tong Zhang. Grid-wise control for multi-agent reinforcement learning in video game AI. In Ka- malika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Con- ference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2576-2585. PMLR, 09-15 Jun 2019. URL htt.http... http://pooceee..../ee/ / ml wtt.en.........s////...... T.press/997
han.9a.hnr 9tmll
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Shariq Iqbal and Fei Sha. Actor-attention-crii critic for reinfoorccmme learning. international conference on machine learning, 2018.
Pressman Isaiah, Kirwin Liam, and Sturrock Robert. Kaggle Lux AI 2021, 12 2021. URL https:
//gi com Isa ahP ressman/ rtseaa///a//////aaalaa.ae.
AT ___2_]
Michael Bradley Johanson, Edward Hughes, Finbarr Timbers, and Joel Leibo. Emergent bartering behaviour in multi-agent reinforcement learning. CoRR, abs/ 06660, 2022. doi: 10.48550/ 2205. 06760
arXiv.2205.06760 htRlssttt::/// http://dooo..og///// worg///l...... 4855448555//t/
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 1rXiv:1412.6980 2014.
Karol Kurach, Anton Raichuk, Piotr Stanczyk, Michal Zajac, Olivier Bachem, Lasse Espeholt, Car- los Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, and Sylvain Gelly, Google research football: A novel reinforcement learning environment. In The Thirty- AAAI Con- ference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Arti- ficial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAA! 2020, New York, NY, USA, February 7-12, 2020, pp. 4501-4510. AAAI Press, 2020.
10

---

Under review as a conference paper at ICLR 2023
Joel Z Leibo, Edward Hughes, Mare Lanctot, and Thore Graepel. Autocurricula and the emergence of innovation from social interaction: A manifesto for multi-agent intelligence research. arXiv preprint arXh::1903.00744 2019.
Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed cooperative-competttiy environments. Advances in neural informa- tion processing systems, 30, 2017.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder de Witt, Gregory Farquhar, Jakob Foer- ster, and Shimon Whiteson, Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. arXiv: Learning, 2018.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philiph H. S. Torr, Jakob Foerster, and Shimon Whiteson. The StarCraft Multi-Agent Challenge. CoRR, abs/1902.04043. 2019.
John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1 abs/1707.06347, 2017. URL http::/erxiv..rrr wwwpxiiiirrg/a.....
7707.0630
Joseph Suarez, Yilun Du, Clare Zhu, Igor Mordatch, and Phillip Isola. The neural mmo platform for massively multiagent research. In J. Vanschoren and S. Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021. URL
wttpp////ccpppppee/222/2 roundl. rouuuu
file/4445e.88111552222ee22222222 Paper
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Leibo, Karl Tuyls, et al. calopsdecomooosttoon networks for cooperative multi-agent learning. arXiv preprint arXiv: arXiv:1706.05296 2017.
Yuandong Tian, Qucheng Gong, Wenling Shang, Yuxin Wu, and C. Lawrence Zitnick. Elf: An extensive, lightweight and flexible research platform for real-time strategy games. Advances in Neurul Information Processing Systems (NIPS), 2017.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czamecki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning, Nature, 575(7782)3350335 2019.
David H Wolpert and Kagan Tumer. An introduction to collective intelligence. arXiv preprint cs/9908014, 1999.
Anita Williams Woolley, Christopher F Chabris, Alex Pentland, Nada Hashmi, and Thomas W Mal- one. Evidence for a collective intelligence factor in the performance of human groups. science, 33006004):666-688. 2010.
Yaodong Yang, Lantao Yu, Yiwei Bai, Ying Wen, Weinan Zhang, and Jun Wang. A study of ai pop- ulation dynamics with million-agent reinforcement learning. In Proceedings of the 17th Interna- tional Conference on Autonomous Agents and MultiAgen Systems, AAMAS '18, pp. 2133-2135, Richland, SC, 2018. International Foundation for Autonomous Agents and Multiagent Systems.
Deheng Ye, Guibin Chen, Wen Zhang, Sheng Chen, Bo Yuan, Bo Liu, Jia Chen, Zhao Liu, Fuhao Qiu, Hongsheng Yu, al. Towards playing full moba games with deep reinforcement learning. Advances in Neural Information Processing Systems, 33:621-632, 2020.
Lianmin Zheng, Jiacheng Yang, Han Cai, Ming Zhou, Weinan Zhang, Jun Wang, and Yong Yu. Magent: A many-agent reinforcement learning platform for artificial collective intelligence. In Thirty-Second AAAI Conference Artificial Intelligence, 2018

---

Under review as a conference paper at ICLR 2023
Stephan Zheng, Alexander Trott, Sunil Srinivasa, David C. Parkes, and Richard Socher. The Al economist: Optimal economic policy design via two-level deep reinforcement learning. CoRR, abs/2108.02755,, 2021. URL https: http/aiivvvv... abs//bb///2/ 02755
A APPENDIX
A.1 DETAILED RULES OF LUX
For ease of understanding, the environment rules including unit types and action spaces are simpli- fied in the main text. In this part, we provide a detailed description of the environment design and rules. The version of Lux we use compatible with the version on Kaggle Lux S1 competition The rules can also be found at https: www. lux- tux-ai.. org/ /specscs 2021 and the following text is a reformatted and slightly modified version of the original rules.
Background.
The night is dark and full of terrors. Two teams must fight off the darkness, collect resources, and advance through the ages. Daytime finds a desperate rush to gather and build the resources to carry you through the impending night. Plan and expand carefully- any city that fails to produce enough light will be consumed by darkness.
Environment. In the Lux AI Challenge Season 1, two competing teams control team of Units and CityTiles that collect resources to fuel their Cities, with the main objective to own as many CityTiles possible at the end of the turn-based game. Both teams have complete information about the entire game state and use that information to optimize resource collection, compete for scarce resources against the opponent, and build cities gain points. Each competitor must program their policy in their language of choice. Each turn, your agent gets 3 seconds to submit their actions, excess time saved across turns. In each game, you are given pool of 60 seconds that is tapped into each time you go over turn's 's 3-second limit. Upon using up all 60 seconds and going over the 3-second limit, your agent freezes and can no longer submit additional actions.
The Map. The world of Lux is represented as 2D grid. Coordinates increase east (right) and south (down). The map always a square and can be 12, 16, 24, or 32 tiles long. The (0, 0) coordinate at the top left.
[0,0]
(0.y)
(x,0)
[x-y]
Figure 11: The specification of the map in Lux.
The map has various features including Resources (Wood, Coal, Uranium), Units (Workers, Carts), CiryTiles, and Roads. In order to prevent maps from favoring one player over another, it is guaranteed that maps are always symmetric by vertical or horizontal reflection. Each player will start with a single CityTile and single Worker that CityTile.
Resources. There are kinds of resources: Wood, Coal, and Uranium (in order of increasing fuel efficiency). These resources collected by Workers, then dropped off once Worker moves top of a CityTile to then be converted into fuel for the city. Some resources require research points before they are possible
collect. Wood in particular can regrow. Each turn, every wood tile's wood amount increases by 2.5% of its current wood amount rounded up. Wood tiles that have been depleted will not regrow. Only wood tiles with less than 500 wood will regrow.
https: wwww://wwwww//..////////////. kaggle. cwm/ htom/c/lun -
12

---

Under review as a conference paper at ICLR 2023
Table 1: The specifications of resource collection and convert.
Resource Type
Research Points Pre-requisite
Fuel Value per Unit
Units Collected per Turn
Wood
0
1
20
Coal
50
10
5
Uranium
200
40
2
Collection Mechanics. At the end of each turn, Workers automatically receive resources from all adjacent (North, East, South, West, or Center) resource tiles they can collect resources from according to the current symmetric formula:
Iterating over uranium, coal, then wood resources:
Each unit makes resource collection requests collect even number of resources from each adjacent tile of the current iterated resource such that the collected amount takes the unit's cargo above capacity. E.g. Worker with 60 wood adjacent to 3 wood tiles asks for 14 from each, receives 40 wood, and wastes 2.
All tiles the current iterated resource then try to fulfill requests. If they can't, they make sure all unfulfilled requests get an equal amount, and the leftover is wasted. E.g. if Workers are mining a tile of 25 wood, but one of them is only asking for 5 while the others are asking for 20 wood each, then first all Workers get wood each, leaving 5 wood left over for 3 more Workers with space left. This can be evenly distributed by giving wood each to the last 3 Workers, leaving wood left that is then wasted.
Workers cannot mine while CityTiles. Instead, if there is least Worker a CityTile, that CityTile will automatically collect adjacent resources at the same rate as a Worker each turn and directly convert it all to fuel. The collection mechanic for CiryTile the same as Worker and you can treat a CityTile as an individual Worker collecting resources.
Actions. Units and CityTiles can perform actions each turn given certain conditions. In general, all actions are simultaneously applied and are validated against the state of the game at the start of a turn. The next few sections describe the Units and CiryTiles in detail.
City Tiles. A CiryTile is building that takes up one tile of space. Adjacent CityTiles collectively form a City. Each CityTile can perform single action provided the CityTile has a Cooldown 1.
Actions:
Build Worker Build Worker unit on top of this CityTile (cannot build a Worker if the current number of owned Workers carts equals the number of owned CityTiles).
Build Cart Build Carts unit on top of this CityTile (cannot build cart if the current number owned Workers + carts equals the number owned CityTiles).
Research- Increase your team's Research Points by
Units. There are two unit types, Workers, and Carts. Every unit can perform a single action once they have Cooldown < 1. All units can choose the move action and move in any of the 5 direc- tions, North, East, South, West, or Center. Moreover, all units can carry raw resources gained from automatic mining or resource transfer. Workers are capped at 100 units of resources and Carts are capped 2000 units of resources. Whenever unit moves on top of friendly CityTile, the City that CityTile forms converts all carried resources into fuel.
There can be at most one unit on tiles without CityTile. Moreover, units cannot move on top of the opposing team's 's CityTiles. However, units can stack on top of each other on friendly CityTile. If two units attempt to move to the same tile that not a CityTile, this considered a collision, and the move action is canceled.
Workers. Actions:
Move Move the unit in one of 5 directions, North, East, South, West, or Center.
Pillage Reduce the Road level of the tile the unit is by 0.5.
13

---

Under review as a conference paper at ICLR 2023
Table 2: The specifications of cooldown.
Unit Type
Base Cooldown
CityTile Worker
10
2
Cart
3
Table The specifications of file burn, Redi denotes the number of adjacent friendly CityTiles.
Unit Type
Fuel Burn in City
Fuel Burn Outside City
CityTile
23 - X Radj
N/A
Worker
0
10
Cart
0
4
Transfer Send any amount of a single resource-type from a unit's cargo to another (start- of-turn) adjacent Unit, up to the latter's cargo capacity. Excess is returned to the original unit.
Build CityTle- Build CityTile right under this Worker, provided the Worker has 100 total resources of any type their cargo (full cargo), and the tile is empty. the building is successful, all carried resources consumed, and CätyTile is built with 0 starting resources.
Carts. Actions:
Move
Move the unit in of 5 directions, North, East, South, West, Center.
Transfer Send any amount of a single resource-type from a unit's cargo to another (start- of-turn) adjacent Unit, up the latter's cargo capacity. Excess is returned the original unit.
Cooldown. CityTiles, Workers, and Carts all have cooldown mechanic after each action. Units and CiryTiles can only act when they have Cooldown 1. At the end of each turn, after Road has been built and pillaged, each unit's Cooldown decreases by 1 and decreases by the level of the Road the unit is on the end of turn. CityTiles are not affected by road levels, and cooldown always decreases by 1. The minimum Cooldown is 0. After action is performed, the unit's Cooldown will increase by a Base Cooldown, as specified in Table 2
Roads. As Carts travel across the map, they start to create roads that allow all Units to move faster. At the end each turn, Cart will upgrade the road level of the tile ends on by 0.75. The higher the road level, the faster Units can move and perform actions. All tiles start with road level of 0 and are capped at 6. Moreover, CityTiles automatically have a max road level of 6. Workers can also destroy roads via the pillage action which reduces road levels by 0.5 each time. If City is consumed by darkness, the road level of all tiles in the City's 's CiryTiles will go back to 0.
Day/Night Cycle The Day/Night cycle consists of a 40-turn cycle, the first 30 turns being day turns, the last 10 being night turns. There are 360 turns in a match, forming 9 cycles. During the night, Units and Cities need to produce light survive. Each turn of the night, each Unit and CityTile will consume an amount of fuel, see Table 3 for rates. Units in particular will use their carried resources to produce light whereas CityTiles will use their fuel produce light. Workers and Carts will only need to consume resources if they are not on CityTile. When outside the City, Workers and Carts must consume whole units of resources to satisfy their night needs, e.g. if Worker carries 1 wood and 5 uranium them, they will consume a full wood for 1 fuel, then full unit of uranium to fulfill the last fuel requirements, wasting 37 fuel. Units will always consume the least efficient resources first.
Lastly, at night, Units gain 2x more Base Cooldown. Should any Unit during the night run out of fuel. they will be removed from the game and disappear into the night forever. Should City run out of fuel, however, the entire City with all the CiryTiles it owns will fall into darkness and be removed from the game.
14

---

Under review as a conference paper at ICLR 2023
Game Resolution order. To help avoid confusion smaller details of how each turn is resolved, we provide the game resolution order here and how actions are applied. Actions in the game are first all validated against the current game state to see if they are valid. Then the actions, along with game events, are resolved the following order and simultaneously within each step:
1. CityTile actions along with increased cooldown.
2. Unit actions along with increased cooldown.
3. Roads are created.
4. Resource collection.
5. Resource drops on CityTiles.
6. If night time, make Units consume resources and CityTiles consume fuel.
7. Regrow wood tiles that are not depleted to O.
8. Cooldowns are handled/computed for each unit and CityTile.
The only exception to the validation criteria is that units may move smoothly between spaces, mean- ing two units are adjacent, they can swap places in one turn. Otherwise, actions such as one unit building a CityTile, then another unit moving on top of the new CityTile, are not allowed as the current state does not have this newly built city and units cannot move on top of other units outside of CityTiles.
Win Conditions. After 360 turns the winner whichever team has the most CityTiles on the map. If that is tie, then whichever team has the most units owned on the board wins. If still tie, the game is marked as a tic. A game may end early if team no longer has any more Units or CityTiles. Then the other team wins.
A.2 ADDITIONAL IMPLEMENTATION DETAILS
Detailed information on our policy implementation is illustrated in this section, including feature engineering, network design, and reinforcement learning algorithm implementation.
PPO implementation. Standard PPO loss (Schulman et al., 2017) is used as the policy loss to optimize the policy net and to estimate the advantage, we apply GAE (Schulman et al.) 2016) with the trajectory length as 32. For Phase training with dense rewards, we set the GAE parameter 0.99. For Phase 2 and 3 training with sparse rewards, we set = , = 1. We apply PPO2 with a clip operation and set the clipping ratio e 0.1. Mcan squared loss is used to optimize the critic's value head and the weight of the value loss is set 0.5. The entropy loss is also computed to encourage exploration with a coefficient of 0.1. We use Adam (Kingma & Ba 2014) as the optimizer with the learning rate le 4. For computational resources, we use an NVIDIA-V 100 GPU and 600 CPU cores.
A = 0.95, the discount factor
Input. Input features fed into policy network consists of two parts: 1) the vector containing global information such as timestep and total fuel. Features in the global information vector and their corresponding data specifications are listed in Table 4 2) the image input containing the map information and the locations of our own and enemy's Workers and CityTiles are listed in Table
Dense Reward Design. The detailed dense reward design listed in Table 6 Four types of rewards are given for specific behaviors of CiryTiles and Workers. The total reward is the sum of four sub- rewards.
Data Preprocessing, The whole data preprocessing procedure is illustrated in Figure 12 The global information vector is split into two parts, the one-hot features of dimension and the other features of dimension 18. The one-hot features are fed into linear layer with an output dimension of 9 for embedding. For unifying input into image shapes, the embedding vector of length 9 and the other global information vector of length 18 are expanded as the image sizes in separate channels, where every pixel in each channel is of the same value. After that, these expanded image features x9x H W and 18 H W. H and W are the map height and width) along with the original image features ( 37 x H x W) are passed through separate convolutional neural networks with a kernel size of 1. Then three parts of input images concatenated together in the channel dimension. Through
15

---

Under review as a conference paper at ICLR 2023
Table 4: Input features Part I: global information vector.
Feature Description
Type
Range
Normalization Coefficient
Current Cycle
One-Hot
[0,8]
N/A
Current Turn In This Cycle If At Night
One-Hot
[0,39]
N/A
One-Hot
[0.1]
N/A
Own CityTile Numbers
Int
[0,1024]
100
Enemy CityTile Numbers
Int
[0.1024]
100
Own Unit Numbers
Int
[0,1024]
100
Enemy Unit Numbers
Int [0,1024]
100
Own Research Points
Int [0,200]
200
Enemy Research Points
Int
[0,200]
200
Own Total Fuel
Int [0,10%]
2300
Enemy Total Fuel
Int
[0,10%]
2300
Own Average Fuel Per CityTile
Float
[0,10
230
Enemy Average Fuel Per CityTile
Float
[0,10*]
230
Own Total Fuel Cost
Int
[0,10*]
230
Enemy Total Fuel Cost
Int
[0,10*]
230
Own Average Fuel Cost Per CityTile
Float
[0.23]
23
Enemy Average Fuel Cost Per CityTile
Float
[0,23]
23
If Own Team Can Collect Coal
Bool
[0.1]
N/A
If Enemy Team Can Collect Coal
Bool
[0,1]
N/A
If Own Team Can Collect Uranium
Bool
[0.1]
N/A
If Enemy Team Can Collect Uranium
Bool
[0.1]
N/A
Global information vector
Map-feature
(09)
(37
One-hot (51)
Others (1.6)
Embedding
Expand
Expand
1x1 Conw2D (w)
1x1 Conv2D (38 MW)
Concatenate
1x1 Conv2D WQ
Concatenate
Conv2D (128 #
Output
Figure 12: The Data Preprocessing Procedure. Global information is split into two parts, one-hot and others. One-hot vector is first embedded through fully-comnected layer and then expands the map size. After passing the 1 x Conv2D one-hot and other features are concatenated in channels, going through another 1 Conv2D, The global features and map features are concatenated with another Conv2D with input channels as 64 and output channels as 128.
another convolutional layer with a kernel size 1, input channels as 64, and output channels as 128, the tensors the ResNet backbone input is in shape 128 H x W.
16

---

Under review as a conference paper at ICLR 2023
Table 5: Input features Part 2: image features.
Feature Description
Type
Normalization Coefficient
If No Worker
Bool
N/A
If Own Worker
Bool
N/A
If Enemy Worker
Bool
N/A
If No Cart
Bool
N/A
If Own Cart
Bool
N/A
If Enemy Cart
Bool
N/A
If No CiryTile
Bool
N/A
If Own CityTile
Bool
N/A
If Enemy CityTile
Bool
N/A
Road Level
Float
6
Worker Cooldown
Float
10
If Worker Can Act
Bool
N/A
Cart Cooldown If Cart Can Act
Float
10
Bool
N/A
CityTile Cooldown If CityTile Can Act If It is Resource Wood Amount If Wood Can Regrow
Float
10
Bool
N/A
Bool
N/A
Int
100
Bool
N/A
Coal Amount
Int
100
Uranium Amount
Int
100
Worker Wood Carry Amount
Int
100
Worker Coal Carry Amount
Int
100
Worker Uranium Carry Amount If Worker Reaches Carry Limit
Int
100
Bool
N/A
Cart Wood Carry Amount
Int
100
Cart Coal Carry Amount
Int
100
Cart Uranium Carry Amount
Int
100
CityTile Fuel Cost
Int
100
CityTile Average Fuel
Float
230
If CiryTile Can Survive Tonight
Bool
N/A
Fuel CityTile Needed to Survive
Int
230
If Worker is at CityTile
Bool
N/A
If Cart is at CiyyTile
Bool
N/A
X Relative Distance to Center
Float
N/A
Y Relative Distance Center
Float
N/A
Table 6: Design details of Phase dense rewards.
Units
Behaviors
Weights
Research Points Increases
0.01
CityTTiles
Units Built
0.5
Fuel Increases
0.0001
Workers
CityTiles Built
1
ResNet Backbone. The ResNet backbone consists of 8 Residual blocks. Each Residual block comprises two convolutional layers and a Squeezz-and-ExciiaioofffS33 layer. The detailed structure of the Residual Block is shown in Figure 13
Output. After the ResNet backbone, we use multiple heads for the output actions and value. The leameed representation from the ResNet backbone is first passed through Spectral Normalization layer. For the action head, we use three separate heads for the Workers, Carts and CityTiles. Each head convolutional layer with kernel size and output channels as the corresponding action
17

---

Under review as a conference paper at ICLR 2023
Residual Block
SE Block
Input (128-H1-P)
Input
(128
5x5 Conv2D
AvgPool2D
(126)
LReLU
5-5 Conv2D
ReLU
(8)
LReLUU
FC
(1289)
Sigmoid
Block
Expand
LReLU
ReLU
Output (328 11
Dutput 21 WW
Figure 13: Residual Block Design and Sguueze-and-Exxitatie Layer. Each Residual block con- sists convolutional layers (kernel size 5, padding 2, stride 1) and LeakyRel.U as the activation function. snneerre-and-Exxiiatiionn layer consists of a 2D Average Pooling, and two fully-connected layers.
dimensions (19 for Worker, 17 for Cart and 4 CityTile.. For the critic's head, we use an Average Pooling to transform the representation of size 128 x H x W to vector of length 128. Then we use a fully-connective layer to get a single value for the critic's estimation.
Representation
(L2S 20-00
Spectral Norm
(120-27-03
1-1 Com2D
1-1 Conv2D
1xt Conv2D
AvgPool2D
(1--1119)
(17-0-03
(4-11-19
(121)
Worker Actions
Cart Actions
CityTll Actions
Value
Figure 14: Output actions and value. First through spectral normalization layer, three action heads, and value head are appended. 1 x Conv2D used for output actions of Worker, Cart, and CityTile. AvgPool2D and fully-connected layer are used for value estimation.
Valid Action Mask. Valid action mask is common technique in reinforcement learning elim- inate unnecessary explorations and accelerate the learning process. We calculate the valid action mask based on the following rules:
Workers: All actions are invalid when cooldown > 1. When cooldown < 1, moving to enemy CiryTiles or tiles with other Workers on it is invalid; moving to friendly CityTile is always valid; and building a Citylile is valid only when its resource achieves 100. Doing nothing is always valid.
When cooldown 1, building Worker is valid when the number of workers are less than the number of CityTiles; research is valid when the team's research point < 200. Doing nothing always valid.
CityTiles: All actions are invalid when cooldown 1.
A.3 DECENTRALIZED POLICY IMPLEMENTATION AND ABLATION STUDIES
In this section, detailed information on our decentralized policy implementation is described, in- cluding input features, network design, and rule-based agents.
18

---

Under review as a conference paper at ICLR 2023
Table Decentralized policy input: global information.
Feature Description
Type
Range
Number of Agents Observed
Int
320
Global CityTile Number
One-hot
[0,320]
Global Unit Number
One-hot
[0,320]
Current Cycle
One-hot
[0,9]
Current Turn this cycle
One-hot
[0,39]
If At Night
Bool
N/A
Own Research Point If Own Team Can Collect Coal
Int
[0,200]
Bool
N/A
If Own Team Can Collect Uranium
Bool
N/A
Table 8: Decentralized policy input: self information.
Feature Description
Type
Range
Location X
Int
[0,31]
Location Y
Int
[0,31]
Location X
One-Hot
[0,31]
Location Y
One-Hot
[0,31]
Type
Bool
N/A
If At City
Bool
N/A
Alive
Bool
[N/A
Cooldown
One-Hot
[0,9]
If At Night
Bool
N/A
Wood Carry Amount
Int
[0,100]
Coal Carry Amount
Int
[0,100]
Uranium Carry Amount
Int
[0,100]
Wood Carry Amount
One-Hot
[0,100]
Coal Carry Amount
One-Hot
[0,100]
Uranium Carry Amount
One-Hot
[0,100]
Fuel
Int
[0,4000]
Fuel
One-Hot Hot [0,4000]
Input. The input of our decentralized policy can be divided into four parts: global information (listed in Table 7), self information (listed in Table 8) other agents (team and enemy) information (listed in Table 9 and map information (listed in Table 10).
Reward Shaping. Each agent receives three types of rewards: 1) Its reward encourage cer- tain behaviors such as survival, building cities, collecting resources, and fueling cities. 2) CityTile reward. Though the CityTile a rule-based reward. this reward is used guide the Workers' behav- ior to support the CityTiles. 3) Team reward. It consists of final win reward and average reward of the team encourage cooperation among agents.
Network Design. The input is split into seven parts, i.e., global features, self features, friend Worker features, friend CityTile features. enemy Worker features, enemy CiryTile features, and image fea- tures. For the former six vector features, use six different two-layer fully-connected networks for feature extraction. And for those features involving multiple units, we perform max pooling along units. For the image features, tions. Then those representations are concatenated together and passed through two fully-connected layers for the actions and values.
three convolutional layers and flatten the learned representa-
Rule-based Agents. For simplicity, we only use Worker as reinforcement learning agents and CiryTiles rule-based agents. The decision rules of CiryTiles simple and intuitive: 1) Build a Worker Tily can. 2) If it cannot build a Worker and the team's 's research point < 200, research. 3) Do nothing otherwise.
19

---

Under review as a conference paper at ICLR 2023
Table 9: Decentralized policy input: other agents' information.
Other Agents
Feature Description
Type
Range
Friend
Bool
N/A
Location X
Int
[0,31
Location Y
Int
[0,31]
Distance
Int
[0,62]
Own Worker x 160
At City
Bool
N/A
Cooldown
One-Hot
[0,3]
Wood Carry Amount
Int
[0,100]
Coal Carry Amount
Int
[0,100]
Uranium Carry Amount
Int
[0,100]
Is Friend
Bool
N/A
Location X
Int
[0,31]
Location Y
Int
[0,31]
Distance
Int
[0,62]
Own CityTile x 160
Cooldown
One-Hot
[0,9]
Average Fuel Per CityTile
Float
[0,2300]
Fuel Cost Per Night
Int
[0,23]
Can Survive Tonight
Bool
N/A
Fuel Needed to Survive Tonight
Int
[0,230]
Is Friend
Bool
N/A
Location X
Int
[0,31]
Location Y
Int
[0,31]
Distance
Int
[0,62]
Enemy Worker x 160
At City
Bool
N/A
Cooldown
One-Hot
[0,3]
Wood Carry Amount
Int
[0,100]
Coal Carry Amount
Int
[0,100]
Uranium Carry Amount
Int
[0,100]
Is Friend
Bool
N/A
Location X
Int
[0,31]
Location Y
Int
[0,31]
Distance
Int
[0,62]
Enemy CityTile 160
Cooldown
One-Hot
[0.9]
Average Fuel Per CityTile
Float
[0,2300]
Fuel Cost Per Night
Int
[0,23]
Can Survive Tonight
Bool
N/A
Fuel Needed to Survive Tonight
Int
[0,230]
Compared with Decentralized Control. To illustrate our pixel-to-ixel centralized control solu- tion, we perform comparative experiment with the decentralized control solution. Competing with the decentralized control solution, the centralized control solution achieves 98% win rate comput- ing by 100 runs. As shown in Figure [6 the decentralized policy can acquire basic skills such as collecting and building CityTiles. Encouraged by the ream-based reward, decentralized agents even acquired a basic level of regional cooperation. However, since the cooperation is induced by pre-engineered rewards, it can only be applied to special scenarios and cannot be extended to other complex situations. For example, advantage initially, but due their cooperation lacking adaptivity, our agents gradually build cities surrounding them and limit their development to gain the advantage. As a result, at Turn 120, the centralized policy has taken control of every resource the map. Moreover, more group strategies emerged from the evolution of the centralized policy, for instance. being aggressive in sending some Workers to occupy and protect the key resources from its opponent.
a local map with woods, the decentralized agents are at an
20

---

Under review as a conference paper at ICLR 2023
Global feature
Self feature
Unit feature
Citytile feature
Map feature
(698)
(603)
(t 160 Dning)
2 160 (itie)
(1711x115)
FC ReLU
FC ReLU
ReLU
4x4 Conv2D
AAARAA
(120)
(126)
(128 - 168 thitss
(120 100 (tler)
(22 x4
ReLU
FC ReLU
FC ReLU
ReLU
4x4 Comv2D
ReLU
MaxPool
MaxPool
3x3 Conv2D
(F1)
(61)
ReLU
(66+6+
Flatten FC
FC ReLU
(5-2)
ReLUU
(178)
Actions
Value
Figure 15: Decentralized policy net architecture. For the global and self features, we use two fully-connected layers for extraction. For unit features and CityTile features, layers and then apply MaxPooling along the units. Three convolutional layers with a flattened and linear layer are used for map features. Then the outputs are concatenated together, using two fully- connected layers for the output actions and value.
linear
Turn 100
Turn 115
Decentralized
Centralized
Turn 130
Advantage
Figure 16: One episode between the decentralized and centralized policy. Yellow is the decen- tralized policy, and Blue is the centralized policy. In local battle, Blue is at an advantage at first, but with better coordination, Yellow turns things around within just 30 turns.
21

---

Under review as a conference paper at ICLR 2023
Table 10: Decentralized policy input: map information.
Map Information
Feature Description
Type
Range
Wood Here
Bool
N/A
Wood Reserves
Int
[0,1000]
Coal Here
Bool
N/A
Resource Map
Coal Reserves
Int
[0.1000]
Uranium Here
Bool
N/A
Uranium Reserves
Int
[0,1000]
Is Friend Worker
Bool
N/A
Is Enemy Worker
Bool
N/A
Worker Cooldown
Int
[0,3]
Worker Map
Worker Wood Carry Amount
Int
[0,100]
Worker Coal Carry Amount
Int
[0,100]
Worker Uranium Carry Amount
Int
[0,100]
Is Friend CityTile
Bool
NIA
Is Enemy CityTile
Bool
N/A
CityTile Cooldown
Int
[0,9]
Average Fuel Per CityTile
Float
[0,2300]
City Map
Fuel Cost Per Night
Int
[0,23]
If Can Survive Tonight
Bool
N/A
Fuel Needed Survive Tonight
Int
[0,230]
Road Level
Bool
N/A
Table 11: Decentralized policy reward design.
Reward Type
Feature Description
Weights
Worker Death Penalty
-1
Worker Survive One Night Turn
0.05
Worker Survive Ten Night Turns
0.5
Worker Reward
Worker Build a CityTile
per CityTile
Built City Fuel Saving
0.05 x (23- Fuel Cost)
Worker Fuel Increase
0.005 per fuel
Worker Fuel Donation
0.01 per fuel
CityTile Death penalty
CityTile Survive One Night Turn
0.05
CityTile Survive Ten Night Turns
0.5
CityTile Reward
CityTile Research Point Increase
0.02 per point
Research Point reaches
Research Point reaches 200
4
CityTile Build a Worker
Final Win Reward
100
Team Reward
Team Average Reward
0.1 x average of friend reward
A.4 MORE GENERALIZATION STUDIES
In section 6.2
demonstrate the generalization of model by transferring the policy trained on maps of size 12 to size 32. More studies are conducted to further investigate the generalization ability of our proposed model larger maps. Results show that even transferred to larger maps, model still retains a surprising ability of massive-agent coordination.
We use the model trained 32 32 maps the base model and evaluate it different map sizes without fine-tuning. On maps of sizes 48 and 64, our policy shows the fantastic mastery of massive-agent coordination shown in Figure 17
22

---

Under review as a conference paper at ICLR 2023
Figure 17: Policy transfer on maps size 64. There are 1069 CityTiles and 1059 units for the orange team, and 779 Tiles and 768 units for the blue team. In larger maps, policy demon- strates the generalization ability of coordination between thousands agents.
Figure 18: Policy transfer on maps of size 128. The large map and the cooldown mechanism limit the ability build large cities fulfilling the map. However, policy still exhibits skills and strategies they acquire on 32 x 32 maps. This large-scale setting eventually causes the web viewer unresponsive.
23

---

Under review as a conference paper at ICLR 2023
Furthermore, make bold attempt on the 128 X 128 maps. However, due to the cooldown mechanism, agents can hardly travel across the map within 360 turns, which makes it impossible to build large cities fulfilling the map like they do in 32 32 maps. Moreover, in larger map, the environment simulation is much slower, which takes about 30 minutes for one episode. It indicates that although Lux has the scalability for millions of agents, the game core and the dynamics need lot of modification to adapt to larger scales. Nevertheless, we find our agents still exhibit skills and strategies they acquire on 32 32 maps in Figure 18 which demonstrates the potential of our method at mallion-ageent scale.
24


Reviews:
[{"review_id": "xXqhXQdw4I", "reviewer": "Reviewer_ndgW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a massively multi-agent (thousands of agents) learning environment, Lux, in which teams compete in a custom Real-Time Strategy (RTS) grid world with limited resources. There are two types of agents on each team, Workers and CityTiles, with distinct actions available. The main contributions of the paper, as indicated by the authors, are (1) the introduction of the Lux learning environment; (2) evidence that there are emergent collective behaviors including local strategies such as regional coordination and global strategies such as sustainable development; (3) the implementation details of a particular algorithm using curriculum training that appears to have a very high success rate against the winner of a competition in developing agents for this environment.", "summary_of_the_review": "This paper needs to clarify what the contributions are (environment vs. curriculum training). If it is the environment, then this should be either submitted to a different venue or anonymized properly before submission. Also, there are a number of questions about what it would take to train agents in this environment, presumably already described in the competition rules referenced in the Appendix. If it is the curriculum training approach, then there needs to be further study of the effect that this has relative to \"the base algorithm\" it is used with.", "strength_and_weaknesses": "Strengths:\n- This paper demonstrates multi-agent reinforcement learning in a complex domain with limited resources.\n- The paper is written clearly and the appendix has many details aiding reproducibility.\n- The Lux environment is very compelling and could definitely be a source of important new results in multi-agent reinforcement learning, especially in the context of limited resources.\n\n\nWeaknesses:\n- One of the biggest weaknesses is the nature of the first contribution (introducing the environment itself). This paper mentions introducing this environment, but then cites the Doerschuk-Tiberi and Tao (2021) GitHub page and associated competition. If this paper is coming from the authors of the repo, then this is clearly breaking anonymity. Otherwise, it is not clear why the authors are stating this as a contribution.\n- The paper does not adequately explain why the algorithm that is used here has an \"up to 90% win rate\" against the Isaiah et al. (2021) policy; it appears that this is entirely due to the curriculum training, as mentioned in the caption to Figure 10; otherwise the win rate is only 20%. If that is the case, is the contribution about the effectiveness of curriculum training? If so, how would curriculum training benefit another algorithm in this environment, such as the Isaiah et al. (2021) policy?\n- While using a central policy for all of the agents in the solution makes sense from a practical perspective, it primarily makes the action space enormous but removes the individual independence of agents. Would it be possible to train individual agents in this framework, and what would this look like?\n- The particular methodology described in Section 4 for a particular neural architecture does not make it clear how difficult it would be to train different neural architectures or to use different algorithms with this framework.\n- The accompanying video is not sufficiently self-contained; some text in the appendix explaining what the video demonstrates would be useful.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UvEvnY14xpQ", "reviewer": "Reviewer_evMn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors present Lux, an environment for multi-agent reinforcement learning at scale. The environment is evaluated through simulations using a curriculum learning solution. The main contribution of this work resides in the environment itself, which allows for a simulation of multi-agent interactions at scale. However, the dynamics of the game itself does not appear particularly insightful. The implementation of the game itself is also rather standard.", "summary_of_the_review": "\nThe authors present an interesting environment, which is made available to the research community as open source software. However, the environment itself is not particularly innovative compared to the state of the art. The evaluation of the platform considering a curriculum algorithm solution is rather standard. The paper is not well structured and impossible to understand without reading the Appendix. Key sections are not in the main paper. \n\nThe reviewer also would like to stress the fact that the paper does not appear to be fully anonymized. There is a direct citation of the GitHub page of the project with the names of the authors of the manuscript.\n\n", "strength_and_weaknesses": "General Note\n\nIt appears that the paper is not anonymized since the authors say \"To this end, we propose Lux (Doerschuk-Tiberi & Tao, 2021), a cooperative and competitive environment where hundreds of agents in two populations scramble for limited resources and fight off the darkness\". It appears that the authors of the work are linked to Doerschuk-Tiberi & Tao.\n\nStrengths\n\n- The release of a platform for the research community is commendable.\n- The emerging patterns are interesting, but somehow expected since they are the direct consequence of the reward structure of the game.\n\nWeaknesses\n\n- The paper is not self-contained. Key parts of the paper are in the Appendix. It is not possible to understand the dynamics of the game without checking the Appendix. Several key experimental results are in the Appendix.\n- It is difficult to see a clear scientific contribution in this work. The reviewer understands that the authors are making available a new environment for the community, but \n- Most of the description of the technical details of the implementation of Lux and its evaluation are in the Appendix. It is worth noting that the some of the notation itself cannot be found in the main paper.\n- The actual contribution of this work is unclear, since it mixes the presentation of the platform and the evaluation with a specific curriculum learning algorithm.\n- The curriculum learning solution used by the authors is rather standard. The results are not particularly insightful.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "P2shM3giQ6m", "reviewer": "Reviewer_Qnaj", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper seems to be deanonymized. \"To this end, we propose Lux (Doerschuk-Tiberi & Tao, 2021)\" which links to this github repo: \"https://github.com/Lux-AI-Challenge/Lux-Design-2021\". If this is not the case, I will provide a full review. ", "summary_of_the_review": "TBD", "strength_and_weaknesses": "TBD", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "-3eKRfJ1Of", "reviewer": "Reviewer_Sp1k", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new benchmark to train a large number of agents in an environment with limited available resources. It also explores the connection of learning atomic / low-level skills with that of learning social strategies such as cooperation, coordination, and competition.", "summary_of_the_review": "The paper proposes a new massive-agent RL framework that could act as a benchmark for but testing agents' ability to cooperate, coordinate or learn different kinds of social behaviors.", "strength_and_weaknesses": "The proposed environment captures the essential elements of learning diverse social behaviors required to sustain in a society with limited resources.\n\nOne concern I have with the work is the lack of comparison to prior frameworks. I'm skeptical as to what additional features Lux provides that are not already there in previous massive-agent frameworks like MAgent. The comparisons done in the paper do not highlight the main advantage of using Lux over previous frameworks.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject