GoBigger: A Scalable Platform for Cooperative-Competitive Multi-Agent Interactive Simulation
Authors: Ming Zhang
Shenghan Zhang
Zhenjie Yang
Lekai Chen
Jinliang Zheng
Chao Yang
Chuming Li
Hang Zhou
Yazhe Niu
Yu Liu
Venue: ICLR 2023 poster (2023)
Keywords: Reinforcement Learning
Environment
Cooperation
Competition
Scalable

Abstract: The emergence of various multi-agent environments has motivated powerful algorithms to explore agents' cooperation or competition. Even though this has greatly promoted the development of multi-agent reinforcement learning  (MARL), it is still not enough to support further exploration on the behavior of swarm intelligence between multiple teams, and cooperation between multiple agents due to their limited scalability. To alleviate this, we introduce GoBigger, a scalable platform for cooperative-competition multi-agent interactive simulation. GoBigger is an enhanced environment for the Agar-like game, enabling the simulation of multiple scales of agent intra-team cooperation and inter-team competition. Compared with existing multi-agent simulation environments, our platform supports multi-team games with more than two teams simultaneously, which dramatically expands the diversity of agent cooperation and competition, and can more effectively simulate the swarm intelligent agent behavior. Besides, in GoBigger, the cooperation between the agents in a team can lead to much higher performance. We offer a diverse set of challenging scenarios, built-in bots, and visualization tools for best practices in benchmarking. We evaluate several state-of-the-art algorithms on GoBigger and demonstrate the potential of the environment. We believe this platform can inspire various emerging research directions in MARL, swarm intelligence, and large-scale agent interactive learning. Both GoBigger and its related benchmark are open-sourced. More information could be found at https://github.com/opendilab/GoBigger.

Published as a conference paper at ICLR 2023
GOBIGGER: A SCALABLE PLATFORM FOR COOPERATIVE-COMPETTIY MULTI-AGENT REINNORCEMENT LEARNING
Ming Zhang" Shenghan Zhang", Zhenjie Yang", Lekai Chen', Jinliang Zheng', Chao Yang", Chuming Li', Hang Zhou', Yazhe Niu?, Yu Liu "SenseTime Research, 'Shanghai AI Laboratory, Shanghai, China izhangning, zhangshenghan, zhouhang2 esensetime com
jtw.eeenngaaaaannnnnnneneeeeeee yangzhenjie, chenlekai, whengjinliang
(1 (lichuming, niuyazhe, lel.liiunnn liuyu] @ @pjlab. org.en
ABSTRACT
The emergence of various multi-agent environments has motivated powerful al- gorithms to explore agents' cooperation or competition. Even though this has greatly promoted the development of multi-agent reinforcement learning (MARL), it is still not enough to support further exploration on the behavior of swarm in- telligence between multiple teams, and cooperation between multiple agents due to their limited scalability. To alleviate this, we introduce GoBigger, scalable platform for pepperative-oopptttti is an enhanced environment for the Agar-like game, enabling the simulation multiple scales of agent intra-team cooperation and inter-team competition. Com- pared with existing multi-agent simulation environments, our platform supports multi-team games with more than two teams simultaneously, which dramatically expands the diversity of agent cooperation and competition, and can more effec- tively simulate the swarm intelligent agent behavior. Besides, in GoBigger, the cooperation between the agents in team can lead to much higher performance. We offer diverse set of challenging scenarios, built-in bots, and visualization tools for best practices in benchmarking. We evaluate several state-of-the-art algorithms on GoBigger and demonstrate the potential of the environment. We believe this platform can inspire various emerging research directions in MARL, swarm intelligence, and large-scale agent interactive learning. Both GoBigger and its related benchmark open-sourced. More information could be found at
multi-agent interactive simulation. GoBigger
INTRODUCTION
The swarm behavior of multi- multi-agent systems (MAS) widely exists in nature and human society. In MAS, individual agent pursues their goal and interacts with each other in local areas, following the rules of cooperation or competition, and then the intelligent behavior of the agent group forms the complex collective behaviors. The phenomena of collective behaviors can be found in the flocking birds [Bhattacharya & Viesek| 22100), molecular motors (Chowdbury) 2006), human crowds (Helbing et al. 2000). and traffic systems (Kanagaraj & Treiber (2018). To understand and simulate such phenomena, some rule-based models (Castellano et al. (209) can simulate the swarm behavior in an unconstrained environment with random movement. However, in a complex interactive environment such as intra-cellular molecular motor transport, where the interaction of agents is time-varying and updatable, it is challenging to recover the underlying collective behaviors by manually designing the controllers or rules.
Interactive simulation of multi-agent systems can provide significant convenience for multi-agent learning algorithms, Some existing multi-agent simulation environments mainly focus on the coop-
DDenotes equal contributions.
1 Correspnnning author

---

Published as a conference paper at ICLR 2023
Create Environments
Mup Size
Research Workflow in GoBigger
Evaluation
Sinple Mulli Agents))
Agsinal
Obervaaton Transfon
Policy Optimization
Local Bots
Public
Player Nunder 41.
Sell Flay Leugue Training
Lenderfiouml
Differenn policies leagie
Cooperation
Basic Units
Actions of Balls
Coopettiion
Fred
Split
Saure
TiCce
Eject
Figure I: New users can follow the given research workflow, while advanced users can customize the configuration of the environment and define new tasks based on GoBigger. The lower left part shows the basic units (balls) and the related actions. The lower right part shows the training in the league with many games going on at the same time, from where cooperative and competitive behaviors of agents can be observed.
eration (Xuan al., 2001 between agents, such as Di-DeeHandd (Chen et al. 2022). Multi-A Agent MuJoCo (Peng et al. (021)) PressurePlate Mclnroe (2022). RWARE (Papoudakis et al. (021])
and Flatland (Mohanty al. 2020). These environments offer simulation designs for cooperation between agents but do not care about multi-agent competition. With the rapid development of the
field of multi-agent reinforcement learning, more and more multi-agent simulation environments focus both cooperation and competition, including LBF (Papoudakis et al.] (2020). MALMO
(Johnson et al., 2016), DM Lab (Beattie et al., 2016), DM Lab2D (Beattie et 2020), Derk's Norén (2020). PommerMan (Resnick et al. 2018). StarCraft Multi-Agent Challenge (SMAC)
(Samvelyan et al.
2019). Google Research Football (Kurach et al.] 2019). Multi-Agent Particle
Environment (MPE) (Mordatch & Abbeel 2017), Hide and Seek (Baker al.] 2019). and Neural
2021). Agents in these environments are divided into different teams to achieve intra-team cooperation and inter-team competition. However, most of them consist of up to two teams, named 2 x N mode. That means they can not handle situations where multiple teams cooperate or compete with each other, which necessary for the research on the swarm behavior of multi-agent systems. Besides, the performance gap caused by different levels of cooperation is not significant in most environments including SMAC, Google Research Football, and NeuralMMO. For the most popular multi-agent environment SMAC, recently SMACv2 (Ellis 2022) shows that agents can get high performance after dropping teammates' observations. The author uses only the identity and observation of agents in the training phase under QMIX (Rashid et al. 2018) and MAPPO Yu et al., 202T) to achieve the same performance as that using the global state additionally. And the replay videos of these environments with well-trained agents could not show the complex cooperation. For those environments that emphasize cooperation through game mechanics, the observations of them too simple that different multi-agent algorithms can easily reach high performance. More details about the various multi-agent simulation environments are shown in Table [
MMO Suarez et al.
In this paper, propose scalable platform, GoBigger, aiming delve into cooperative-competiti multi-agent reinforcement learning for swarm intelligence between multiple teams. Different from the previous multi-agent simulation environment. GoBigger is a scalable environment that enables the simulation of various teams and agents in each team. In other words. in the M x N game mode of GoBigger, M means the number of teams in the environment, and N means the number of players in each team. This new game mode dramatically expands the way of agent cooperation and competition and can more effectively simulate the swarm intelligent agent behavior. In addition, GoBigger makes agents with intra-team cooperation and inter-team competition achieve higher performance according to the restriction of game mechanics and rules, which is approved in Section 6.1 We offer a diverse set of challenge scenarios in GoBigger for best practices in benchmarking. In most of the given scenarios, each player in team is controlled by an independent agent that has to act based on only its local observation or all teammates' observations. Meanwhile, GoBigger is
complex
2

---

Published as a conference paper at ICLR 2023
Table 1: Comparison of GoBigger with other related multi-agent simulation environments. Agents Size denotes agent scale in the interactive environment, where M and N mean the number of competitive teams and the agent number each team, respectively, in the M N settings. Actions denotes whether the action space is continuous, discrete, or hybrid. Obs denotes whether to support the partial observation. + Coop and Comp indicates the importance cooperation and competition. means there is no cooperation competition in the environment.
Agents Size
Actions
Obs
Coop
Comp
SMAC (Samvelyan al.
2019)
N
Discrete
Partial
GRF (Kurach al.
2019)
N
Discrete
Partial
+
+++
Bi-DexHands (Chen et al.
2022)
N
Discrete
Full
+++
MPE (Mordatch & Abbeel
2017
2 x N
Hybrid
Full
++
++
Hide-and-Seek (Baker et al.
2019)
2 N
Discrete
Partial
MAMujocc Peng al.
2021)
N
Continuous
Partial
NeuralMMO (Suarez al.
2021)
X N
Discrete
Partial
+
++
GoBigger (Ours)
M X N
Hybrid Partial
+++
++
environment with well-designed action space and observation space, giving partial view for all agents in the game. A reproducible benchmark including several state-of-the -fae--ffthh-rrttt algorithms under different scenarios accessible for users quickly get started with GoBigger and demonstrate the potential of the environment. GoBigger also features game systems configurable for users to research on individual aspects of intelligence and on combinations thereof, with given rule-based built-in bots and visualization tools to make it easier for users to evaluate their agents. The overview of the research workflow for users shown Figure]]
2 RELATED WORK
Most multi-agent reinforcement learning environments mainly designed for cooperation competition between no more than two teams of agents. SMAC (Samvelyan et al. 2019) consists of carefully designed micro scenarios and necessitates learning one or more micromanagement techniques to defeat the enemy. But the SOTA algorithms can achieve high scores even without cooperation between agents. Google Research Football (Kurach et al.] 2019) is a novel open- source reinforcement learning environment provides different modes for both single-agent and multi-agent settings, while there is baseline indicating multi-agent cooperation and competition in the environment. Furthermore, both SMAC and Google Research Football can not handle situations where multiple (more than two) teams in a game, making them unable to be used for the research on swarm behaviors. Agar (staghuntrpg) implemented with python is also available and is evaluated on recent works (Tang et al. 2021). However, focuses on cooperation and competition between two players but not multiple teams, and its scenarios not complex. Neural MMO (Suarez al. 2021) a massively multi-agent environment for Al research, which support spans to 1024 agents and minute- hours-long time horizons. It also allows M N game mode in its scenarios, and agents need to cooperate with teammates to fight against other teams in the map. According to its given baselines, we find that agents in Neural MMO focus more on competition with other agents but less on cooperation up to now. And open-sourced agents in the challenge are more likely to play on their own instead of cooperating with teammates.
3 DESIGN PRINCIPLES AND SYSTEM OVERVIEW
GoBigger is inspired by the popular online multiplayer game Agar (SA). In regular full game of Agar with OFA(One For All) mode, players cooperate with their teammates and fight against other teams for the final champion. On this basis, many modifications have been made to allow more
3

---

Published as a conference paper at ICLR 2023
cooperation and competition in GoBigger. Each player, which is represented by one more balls (dubbed clone ball), increases its size by colliding and merging with other balls within a bounded rectangular area in limited time. The larger the size of clone balls, the higher the player's 's score. At the beginning of the game, a single controllable clone ball (one player) is randomly spawned on the map. In addition to the clone balls, there three uncontrollable neutral balls in the game map (dubbed as food ball, spore ball, and thorn ball), as shown in Figure
3.1 BASIC UNITS
Food balls are neutral resources and they are always fixed on the map. Clone balls can eat food balls, and then absorb their size. Food balls will be randomly and continuously generated until the number of available food balls reaches the maximum.
Food Ball:
Spore Ball: Spore balls ejected by clone balls. They will stay on the map and can be eaten by any other clone balls.
Thorn Ball:
Thorn balls another kind of neutral resource in the game. Different from food balls, they have a larger size and less quantity on the map. In addition, they can eat the spore balls ejected from clone balls and absorb their size. When a clone ball eats a thorn ball, it will evenly split into several small balls. It means player can push a thorn ball to an opponent' 's clone ball, force it to split, and eat its small balls.
A player consists of several clone balls and they are the only controllable balls. In addition, a clone ball can eat other smaller balls including food balls, thorn balls, spore balls, and clone balls (from teammates opponents) by covering their center. Clone balls of an agent can be merged after a certain time. Clone balls will decay in size to ensure they will not grow too large. all clone balls of player are eaten, this player will respawn on the map randomly in the next frame. player can control clone balls by the following three actions in total:
Clone Ball:
Move: Moving can help a clone ball eat other balls or escape from danger. The bigger the clone ball, the slower it moves.
Eject: Ejecting a spore ball can help a clone ball drop some size and move faster (Figure D
When a clone ball ejects, a new spore ball will appear on the clone ball's moving direction at high speed but quickly slow down and stop.
Split: Splitting helps clone ball evenly split into two clone balls (Figure I.. Besides, player can have at most 16 clone balls. To move faster, player can split and turn the clone halls into smaller ones and get higher speed.
3.2 GAME SYSTEMS
The base game representation is limited map comprising the units declared in Section Note that the map is not a grid map, which means that the continuous state is a complex challenge for agents. At the same time, GoBigger makes the operating efficiency of the game acceptable for users. The whole game can be split into the following systems: resources, cooperation, and competition.
Resources:
Resource system designed for navigation and multi-task reasoning. In GoBigger, food balls and thorn balls can help the agents to grow larger. Eating food balls is safe but slow while cating thorn balls is dangerous but quick. There are food balls and thorn balls randomly distributed on the map. The agent needs to decide which ball to eat according to its current size and the state of other agents around it to obtain faster development. The initial state and the regeneration rates of resources are all configurable.
Cooperation system is designed for direct cooperation among agents a team. For a single agent, the maximum number of clone balls is 16 (configurable), which causes an agent cannot move fast by splitting indefinitely on its own. At the same time, each clone ball of the agent has 20 seconds cooldown period, which is reset after each split. When the clone ball is its cooldown period, it cannot merge with other clone balls that belong to the same agent. This means that splitting is dangerous because failing to merge quickly can easily lead attacks by other agents. The above two constraints make multiple agents in the same team need to cooperate and transfer size to each other by splitting and ejecting move faster and ensure safety. In addition, the final ranking of
Cooperation:
4

---

Published as a conference paper at ICLR 2023
B
(a) The full the game
(b) The partial view players
Figure 2: Screenshots for the different observation methods of GoBigger. (a) The full view of the game with 4 teams and players in each team. (b) The players in (a) have their observations, and the three figures in each column represent the three players' view from each team.
GoBigger is based on the size of the whole team. Therefore, frequent and reasonable cooperation within the team is an important condition for victory.
Competition system is designed for direct competition among different teams. In GoBigger, the agent loses some size after each timestep. This more obvious when the agent very large. Therefore, the resources to eat other agents to ensure size. Agents can attack other agents by moving and splitting to gain their size. In addition, gathering the size of the team fight against other agents can often have advantage in the competition. GoBigger encourages competition among teams.
Competition:
the map decrease the game progress, the agent needs
4 ENVIRONMENT
GoBigger provides a set of standard game environment configurations. To facilitate access to the current observation and the efficient control of the balls, GoBigger follows the design of gym (Brockman et al. 2016) and develops easy-to-use use interfaces of observation and action.
4.1 OBSERVATION
As mentioned in Section 3 the observation space of GoBigger is quite complex. It consists of GlobalState and PlayerState:
GlobalState is shared by all players. It includes all essential global information, such as map size, total frames of the game, current running frame in the game, and the leaderboard which consists of the current scores and rankings of all teams.
PlayerState is player-wise. It the information that can be observed within the view of each player, making GoBigger a partially observable MDP (POMDP) (Drake) 1962 Sondik 1971). Specifically, the player's view depends on the range of the positions its clone balls. Figure 2 shows the view of different players in game. When the clone balls of player are dispersed enough, a larger view will be applied to this player. Given a view, PlayerState contains the positions and radius of all balls in the view, including food balls, thom balls, spore balls, and clone balls, as well as their owner.
More details can be found in Appendix A.I
4.2
ACTION SPACE
The action space of the clone ball consists of move, eject and split could be shown as:
(ar, y,t)
(1)

---

Published as a conference paper at ICLR 2023
Table 2: GoBigger challenges. Agents Size denotes the number of agents on the map as Table | Map Size denotes the length and width of the map. Food denotes the range of food balls. Thorn denotes the range of thorn balls. Init Size denotes the size of clone balls at their first birth. Limited Frame denotes the number of frames in game.
Name
Agents Size
Map Size
Food
Thom
Init Size
Limited Frame
Small maps
st_tlp1
x 1
32 x 32
[65, 75]
[1, 2]
1000
3600 (3min)
st_tpp
x 2
48 x 48
[130, 150]
[2, 3]
1000
3600 (3min)
st 1_pppp
2
48 48
[130, 150]
[2, 3]
1000
3600 (3min)
st ss_1pp2
2 2
64 64
[260, 300]
[3, []
13000
3600 (3min)
st st_t3p2
3 2
88 88
[500, 560]
[5, 6]
13000
3600 (3min)
Large maps
st_t4p3
x 3
128 x 128
[800, 900]
[9, 12]
1000
14400
(12min)
st_t5p3
x 3
128 x 128
[900, 1000]
[10, 12]
1000
14400 (12min)
st_t5p4
4
X 144
[900, 1000]
[10, 12]
1000
14400 (12min)
st st_16p4
6 4
144 144
[1000, 1100]
[11, 13]
1000
14400 (12min)
Here denotes the action, (x, y) is unit vector for the direction of movement, and the action type for the clone ball to be executed in the current frame. When player applies direction (x, y). enforcement this direction will be applied movement direction to this direction in several frames. The action types consist of moving, ejecting, and splitting. Agents can complete development and attack according to the combination different action types. For more details, please refer to Appendix A.2
all this player's clone balls and smoothly alters their
4.3 REWARD
In Gobigger, the goal of all teams is to become bigger and bigger and finally become the biggest. Thus a natural reward for single agent is to consider its score change of two adjacent steps:
Reward, == Scoree+1 Score.
The above reward can directly reflects the players' growth and induce agents to grow bigger. But can't 't encourage agents to cooperate and confront each other very well. Therefore, GoBigger provides some other kinds of reward functions as follows:
Rapid development must function can be designed as the number of thorn balls eaten by the agent between two adjacent steps.
Example 1:
from eating a large number of thor balls, the reward
The cooperation among teammates consists of ejecting spore balls to each other and cating teammates' clone balls, so the reward function can be designed as the number of spores and the number of clone balls eaten by teammates between two adjacent steps.
Example 2:
Example 3: To improve the offensive performance of the agent, the reward function can be designed as the number of clone balls that the agent eats from other teams between two adjacent steps.
With the help of different kinds of rewards and constraints, an agent with strong cooperation and aggression will be obtained. For more details of reward functions, please refer to Section 5
4.4 SCENARIOS
GoBigger consists of a set of scenarios that aim to evaluate the learning cooperation and competition of different agents in different game settings. The scenarios are carefully designed with different settings including agents size, map size, food number, thorn number, and so on. Most of the scenarios confrontation among over two teams of players. In different scenarios, agents need to learn to acquire resources on the map and fight against other teams through cooperation. Teams are only different in their locations of birth. When the specified time limit is reached, the team with the

---

Published as a conference paper at ICLR 2023
2000
Gee
Toon
140
2006
110
24
2000
1009
244
Figure 3: Agents trained on st_ st_12p2, st__s$$3_0011 13p2, and st st_r4p3 with DQN (the first row) and PPO (the second row). Each column (left to right) denotes the score against built-in bots, the average number of eaten food balls, the average number eaten thorn balls, and the average number of eaten clone balls from teammates.
higher score wins. In the evaluation phase, agents have to battle against other teams of agents in the scenarios. Built-in bots with different levels could be also chosen as competitors. The rule- buse- bots can increase their sizes quickly by eating the food balls and thorn balls and avoid being eaten by other players through the given rules. The complete list of challenges is presented in Table The Init Size of st_12p2 and st_1pp2 13000 to reduce farming time and make the agent more focused on cooperation and competition. According to the map size and the number of agents on the map, the scenarios are simply divided into two parts including small maps and large maps. The large maps may be more difficult its larger observation while playing against more teams of agents.
EXPERIMENTS
In this section, we will present some experiments results of GoBigger, which are based on the most representative scenarios: st_ 12_1222. st t3p2 and st st_14p3. Details of observation encoding and neural network can be found in Appendix B.1 The agents learned by different RL algorithms by fighting against the built-in bots of level 3 A..5 times. The average score will be one of the most important metrics as it stands for the game level of the agents, showing cooperation and competition skills.
evaluated
5.1 SINGLE AGENT ALGORITIMMS
We apply DQN (Miih et al., 2015) and PPO (Schulman al. 2017) on GoBigger, where each agent only use its own observation and learns value function independently. Training details are presented in Appendix B.3 The experiment results different scenarios can be found in Figure The score represents the game level of the agents. Agents can get high scores by eating more food balls and thorn balls in a limited time. It also shows that the popular single-agent algorithms can quickly converge in the given scenarios, inferring that GoBigger is a feasible and challenging environment for multi-agent reinforcement learning research.
5.2 MULTI AGENT ALGORRIHHS
We also apply several state-of-the-an art multi-agent algorithms including QMIX (Rashid et al. 2018). MAPPO Yu et al. 202I). COMA (Foerster et al.) 2018), and VMIX (Su et al. 2021) on GoBigger. The training details are presented in Appendix [B.3] Figure 4 shows the performance of different multi-agent algorithms in GoBigger. Overall MAPPO achieves the highest scores when fighting against the built-in bots and is the best performer in all three scenarios with the fewest environment steps. Additionally, with a fixed number of steps, COMA has the worst performance of all algorithms, demonstrating the on-policy policy gradient methods can not have an advantage over different algorithms. Figure 4 also alsso shows that the number of food balls and thorn balls grows rapidly in the training phase. This is in line with the expectations of the environments, in which the fastest way for an agent farm eat food balls and thorn balls quickly. Besides, after enough steps, the agents
7

---

Published as a conference paper at ICLR 2023
4.P91
Ear Stgss
Skggs
Stegs
Sags
Figure 4: Agents trained on st__sst_p________p_ f22,,ppp st t3p2, and st 14p3 with QMIX (the first row), VMIX (the second row), MtPPO(the third row), and COMA (the fourth row). Each column (left to right) denotes the score against built-in bots, the average number of eaten food balls, eaten thorn balls, and eaten clone balls from teammates.
learn to cooperate with teammates, which is shown from the number of clone clone_team. The growth of clone team means that the agents are trying to merge with teammates.
The results in different scenarios are also shown in Figure 4 st_ st_r2p2 is small map with only two teams and four agents, while st_r4p3 is a large map containing four teams and each consists of three agents. QMIX plays well on small maps including st v_rpp2 and st 13pp2 but plays bad on large map st 14p33. We infer that the map of st 14p3 is too large and too difficult for agents find their teammates, which causes low level of cooperation.
ABLATION STUDY
GoBigger provides small and large maps for training on different scales. Here we explore some factors that have impacts the performance of agent, including the necessity of the observations from teammates, the frequency of acting in the game, and the ladder system.
6.1 DROPPING OBSERVATIONS FROM TEAMMATES
To explore the importance of cooperation in GoBigger, we drop the observations from teammates, making the agents decide their actions only based on their observations. Figure shows that the agents with teammates' observations outperform the agents without observations from the whole team. Agents can not easily find their teammates without the observations sharing especially on large map, i.e.. r4p3, resulting that they could not cooperate with their teammates fight against other teams, which is consistent with our original intention of designing the GoBigger. Appendix C. shows more details.
8

---

Published as a conference paper at ICLR 2023
330300
330000
175100
190000
190000
12111
300000
7500
30000
90300
ul obe Fromt
2000
10M
12M
Em
Figure 5: Comparing agents w/ or w/o observation from teammates. Each picture (left to right) denotes the performance of agents on st f2p2, st 13p2, and st ___4p3..
200000
241900
300000
335000
350000
(1111)
150000
10002
100000
913
200
121
2M
1IM
13M
146
0M
2M
10h1
12M
14M
25
4M
19M
12M
14M
EPY Sago
Figure 6: Comparing agents with different frequencies of actions on st sf_2p2,, st ___3p2, and st sr_143 (left to right). Each line denotes the score against bots with a certain frequency of actions.
6.2 FREQUENCY OF ACTIONS
In most reinforcement learning environments, agents will execute actions immediately after envi- ronments step(Jia et al. 202T). However, it is not necessary in a real-time game. For example, AlphaStar[Vinyaa et 2019) uses a delay head in their network predict delay of the next action. To save computing resources, we explore the impact of the frequency of actions on an agent's perfor- mance. Figure shows that the agents with frequency is because higher frequency will bring more redundant information, which is little more difficult for agents to understand and distinguish.
outperform other settings. We infer this
6.3 LADDER SYSTEM AND LEADERBOARD
In multiplayer games, reward versus some specific opponents is not enough to fully measure agents skills in different situations. So we introduce ladder system based TrueSkill (Herbrich et al. (2066) to evaluate different kinds of agents objectively. This ladder system will make agents continuously play against each other, and then use their final ranks in each game to update their trueskill scores. Figure shows the ladder system including all of our trained agents as well as built-in bot. More details of the ladder system could be found in Appendix C.2
CONCLUSION AND FUTURE WORK
In this paper, GoBigger is presented scalable platform for multi-agent interactive simulation. GoBigger allows M x N game mode that focuses on intra-team cooperation and inter-team competi- tion. We offer diverse set of challenge scenarios in GoBigger for best practices in benchmarking. A reproducible benchmark including several state-of-the-all algorithms under different scenarios is accessible. GoBigger also features a game system configurable with given rule-based built-in bots and visualization tools to make it easier for users evaluate their agents.
the near future, we aim to explore more based on GoBigger. Considering the scenarios in GoBigger now monotonous, we plan to develop more interesting and complex scenarios that require a higher level of coordination amongst agents. We also plan to expand GoBigger as a massive environment to host thousands of agents. With harder multi-agent coordination and competition problems, aim to motivate further research in this domain, particularly in areas such as multi-agent exploration and coordination.

---

Published as a conference paper at ICLR 2023
Acknowledgement. This work was supported by the National Key RD Program of China under Grant STI 2030-00000000 30jo-Major Projects 20212D0201300.
10

---

Published as a conference paper at ICLR 2023
REFERENCES
Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. arXiv preprint 2019.
Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward. Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, al. Deepmind lab. arXiv preprint arXiv:1612.03881, 2016.
Charles Beattie, Thomas Köppe, Edgar A Duéfiez-Guzmán, and Joel Z Leibo. Deepmind lab2d. arXiv preprint 2tt::::21110000222 2020.
Kunal Bhattacharya and Tamás Vicsek. Collective decision making cohesive flocks. New Journal of Physics, 12(9):093019, 2010.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Claudio Castellano, Santo Fortunato, and Vittorio Loreto. Statistical physics of social dynamics. Reviews of modern physics, (2):599 2009.
Yuanpei Chen, Shengjie Wang, Hao Dong, Zongqing Lu,, and Yaodong Yang. Bi-dexhands: Bimanual
dexterous manipulation via reinforcement learning. htttp:://githbe htwpgitttbo.o/.///..eeemeeeeet com cOm/:KU-MAAAKKA
DexterousHandds 2022.
Debashish Chowdhury. Collective effects in intra-cellular molecular motor transport: Coordination, cooperation and competition. Physica A: Statistical Mechanics and its Applications, 372(1):84-9) 2006.
Alvin W Drake. Observation of Markov process through a noisy channel. PhD thesis, Massachusetts Institute of Technology, 1962.
Benjamin Ellis. Smacv2: A new benchmark for cooperative multi-agent reinforcement learning.
https://githuit htwm///wwhi/////c///// 2022.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
Simon Haykin. Neural networks: a comprehensive foundation. Prentice Hall PTR, 1994.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition,
pp. 770-778, 2016.
Dirk Helbing, IlIés Farkas, and Tamas Vicsek. Simulating dynamical features of escape panic. Nature, 40766803):477-490, 2000.
Ralf Herbrich, Tom Minka, and Thore Graepel. TrueskillTM; a bayesian skill rating system. Advances in neural information processing systems, 19, 2006.
Ruonan Jia, Qingming Li, Wenzhen Huang, Junge Zhang, and Xiu Li. Consistency regularization for ensemble model based reinforcement learning. In 18th Pacific Rim International Conference on Artificial Intelligence, Proceedings, Part III 18, pp. 3-16. Springer, 2021.
Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. The malmo platform for artificial intelligence experimentation. In Ijcai, pp. 4246-4247, Citeseer, 2016.
Venkatesan Kanagaraj and Martin Treiber. Self-driven particle model for mixed traffic and other disordered flows. Physica A: Statistical Mechanics and its Applications, 509:1-111 2018.
Karol Kurach, Anton Raichuk, Piotr Stańczyk, Michal Zajac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, et al. Google research football: A novel reinforcement learning environment. arXiv preprint arXiv: /907. 1800 2019.
11

---

Published as a conference paper at ICLR 2023
Trevor McInroe. Repo for the multi-agent pressureplate environment. https://qithu... iue--agentts/presssepppaaaa 2022.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. nature, 518(7540):529-333 2015.
Sharada Mohanty, Erik Nygren, Florian Laurent, Manuel Schneider, Christian Scheller, Nilabha Bhattacharya, Jeremy Watson, Adrian Egli, Christian Eichenberger, Christian Baumberger, Gereon Vienken, Irene Sturm, Guillaume Sartoretti, and Giacomo Spigler. Flatland-rl Multi-agent reinforcement learning trains, 2020.
Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent populations, arXiv preprint arXiv: / 00990, 2017.
John Fredrik Wilhelm Norén. Derk gym environment. https: Sttpss//gymmn www.gy/mn..gaam.....m.. com, 2020.
Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V Albrecht. Comparative evaluation of multi-agent deep reinforcement learning algorithms. arXiv preprint arXiv:2006.07869,
2020.
Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V. Albrecht. Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS), 2021. URL
http://arxiv.org/ wttp:////...bbb/...b//////
Bei Peng, Tabish Rashid, Christian Schroeder de Witt, Pierre-Alexanrre Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Faemac: Factored multi-agent centralised policy gradients. Advances in Neural Information Processing Systems, 34, 2021.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shi- mon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. International conference on machine learning, pp. 4295-4304. PMLR, 2018
Cinjon Resnick, Wes Eldridge, David Ha. Denny Britz, Jakob Foerster, Julian Togelius, Kyunghyun Cho, and Joan Bruna. Pommerman: A multi-agent playground. CoRR, abs/1809.07 2018.
URL
httgg/bb/1188 07124
Miniclip SA. Agar.io https://agar.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint 19244:190. 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy 2017.
optimization algorithms. arXiv preprint
Edward Jay Sondik. The optimal control of partially observable Markov processes. University, 1971.
Stanford
staghuntrpg. Python implement for agar. httppes://stttab...c///sstaatttt
Jianyu Su, Stephen Adams, and Peter Beling. Value-decoomposition multi-agent actor-critics. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11352-1 11360, 2021.
Joseph Suarez, Yilun Du, Clare Zhu, Igor Mordatch, and Phillip Isola. The neural mmo platform for massively multiagent research. arXiv preprint arXiv:21 10.0000000000 10.07594, 2021.
Zhenggang Tang, Chao Yu, Boyuan Chen, Huazhe Xu, Xiaolong Wang, Fei Fang, Simon Du, Yu Wang, and Yi Wu. Discovering diverse multi-agent strategic behavior via reward randomization. arXiv preprint 203.::20030455 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
12

---

Published as a conference paper at ICLR 2023
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czamecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung. David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782)3350-33) 2019.
Ping Xuan, Victor Lesser, and Shlomo Zilberstein. Communication decisions in multi-agent coopera- tion: Model and experiments. In Proceedings of the fifth international conference on Autonomous agents. pp. 316-623,20 2001.
Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative, multi-agent games. arXiv preprint arXiv:2103.01955, 2021.
13

---

Published as a conference paper at ICLR 2023
A GOBIGGER
A. OBSERVATION
The observation GoBigger divided into parts, including Global State and Player States. After each step, GoBigger will return observation for all agents in the environment. Here are the details for Global State. border denotes the map size of the game. votal_frame denotes the number of total frames in the game. rust_frmee.oul denotes the number of frames that have passed. leaderboard denotes the ranks and scores of agents.
giobal_state
"border': : Imap width, map_he sap_height]] I, 'total_frame" total_framee clast_frame_contt faast__rrae_counte 'Leaderboard'; { team_name: team_size
The Player States declare the specific states for each player (or agent) in each team. player _id denotes the identity of the player the game. rectangle denotes the position of partial vision. overlap denotes all kinds of balls in the vision the player, including the position, radius, score, velocity, direction, player id, and team id. team name denotes the team's name of the player. score denotes the score of the player. can can_eject denotes the player can eject at this frame. con_split denotes the player split at this frame.
player_states = player_id:
'rectanglo': [left_top_X, left_top_y, right_botton right_bottom__l.
'overlap':
food' [position.x, position.y, radius, score), ...l,
'thrns' I.position.x, position.y, radius, score, vel.x, vei.yl, ...l,
'spore' lpposition.x, position.y, radius, score, vel.x, vel. owner], ...],
cclone': I[iposition.x, position.y, radius, score, vel.x, Xx vel.y, direction.x, direction.y, player_id, team_id], ...l,
h
-taar__aaaaa name': tean_ sccree player_scon _cc_se, can_efect': bool, cca_sppit'' bool,
12
13
14
15
The original structural observations may be difficult for new users in GoBigger. That's why we modified the observations into a simpler tensor array in Section [.] New users directly use the given observation encoder, while advanced users can build their own observation encoders which are more suitable for their algorithms.
A.2 ACTION
In Section 4.2 we offers a simple action space for the control of agents. And the format of the given action could shown as:
= (2. t)
The value of t is in 0, 2, separately representing move, eject and split. Especially, when t is or the clone balls will eject or split the given direction decided by (æ, y). At most time, a player has more than clone ball. Under this circumstance, the action will be applied all the clone balls of
14

---

Published as a conference paper at ICLR 2023
A
(a)
(b)
(d)
Figure 7: Screenshots for one of the action combinations named middle-ejecting. This action combination will make agents gather their size into one big clone ball. (a) The clone hall is ready to the thorn ball. (b) After eating the thorn ball, the clone ball is forced to split into several pieces. (c) The agent ejects to the center. (d) The clone balls at the center cat the spore and merges their size together.
the player, i.e., all clone balls with sizes exceeding the split threshold will split when t and all clone balls with sizes exceeding the eject threshold will eject spores when t is 2.
A.3 ACTION COMBINATIONS
Though the action space in GoBigger is simple, the combinations of different actions lead quick growth, attack, and escape. Figure shows one of the action combinations named middle-ejecting, which is the combination of moving and ejecting. It is really useful gather most of the size in one clone ball after eating the thorn ball. Besides, agents could use this action combination when they are split into too many pieces, which may help them to avoid being eaten by other agents. In the experiments in Section 5 we find that the agents can perform these action combinations.
A.4 COLLISION DETECTION
The core problem of GoBigger is how detect the collision of the balls in each frame and help the game update the state of the balls. The problem of collision detection could be simplified into scenario where we need get the collided balls from the given gallery balls for each ball in the given query balls. GoBigger has designed four collision detection algorithms as follows:
Exhaustive For each ball in the query, we enumerate each ball in the gallery to determine whether there is collision. This is an exhaustive solution that will take a long time the number of balls in the gallery is large. The complexity of the algorithm could be written in:
On m)
where m denotes the number of balls in the query, and n denotes the number of balls the gallery.
Precision Precision approximation algorithm divides the map into several rows according to the accuracy that has been set, dynamically maintain the row information in each frame and search by row. First, we need to sort the balls in each row according to their ordinate. For the balls query, we abstract the boundary of the ball into rectangle, then traverse each row in the rectangle, and find the first ball covered by the query through dichotomy in each row, and then Enumerate the balls in sequence until the ordinate exceeds the boundary of the query rectangle. The complexity of the algorithm could be written in:
(n * log(n) + Σr log(n) +
(5)
where denotes the number of balls in query, denotes the number of balls in the gallery, & denotes the precision we set, r denotes the number of balls whose position status has changed and p denotes the number of balls that collide with other balls.
TebuildQuaddTre We build Quadtree awo-dimensional plane in every frame based on the positions of balls and search the collisions that happen among the balls in query and gallery according to the Quadtree. With the Quadtree, GoBigger can quickly find the nearby balls of the given query
15

---

Published as a conference paper at ICLR 2023
ball, which helps to search the collisions in relatively small search area. The complexity of the algorithm could be written in:
(n lng(n) + m log(n) + p)
(6)
where m denotes the number of balls in query. n denotes the number of balls the gallery, and denotes the number of balls that collide with other balls.
RemoveQuadTree
This algorithm is based on the RebuildQuadTTeel A.4 and we add the operations
of deleting nodes in the tree with dynamically maintaining the tree. The complexity of the algorithm could be written in:
(r log(n) + log(n) + pp)
where m denotes the number of balls in query, n denotes the number of balls the gallery, r denotes the number of balls whose position status has changed and denotes the number of balls that collide with other balls.
To test the efficiency of the above algorithms, we modify the parameters including the number of balls in query and gallery, the number of changed balls, and the iteration rounds to get a more fair and intuitive result. The data in Table 33 comes from the most representative scenarios. We finally choose Precision A.4 as our default algorithm for collision detection.
Table 3: Comparison of the different algorithms of collision detection on different settings. 7 denotes the number of balls in the gallery. denotes the number of balls in the query. denotes the number of changing balls, which means the number of collisions.
T=3000
T=3000
T=10000
T=10000
T=30000
Q=300
Q=300
Q=1000
Q=2000
Q=600
C=600
C=1500
C=2000
C=5000
C=3000
Exhaustive
688ms
1067ms
8384ms
12426ms
127000ms
Precision
14ms
16ms
61ms
86ms
403ms
Rebuild QuadTree
47ms
50ms
339ms
586ms
5691ms
Remove QuadTTree
48ms
178ms
497ms
2460ms
8419ms
A.5 BUILT-IN BOTS
Built-in bots of different levels help users get started with the environment and perform standard evaluation for algorithm development. Level The bot of level aims to collect neutral resources the map for quick development in size. It can only move and eat the closest food balls in its view. Level 2 We add the exploration of thorn balls in view for the bot of level 2. The bot prefers thorn ball in its view. Level The bot of level 3 can avoid being eaten by other larger players as they will move far away from the larger player in their view.
B EXPERIMENTS
B.1 OBSERVATION TRANSFORM
It is challenging to encode thousands of entities compared to other multi-agent environments. In order to better model the observation information, we transform the original information into Scalar info, Spatial info, Team info. Ball info. As shown in (Figure g Scalar Info models the size of the agent's local vision, the current ranking and score, and the action type at the last moment. Spatial info models the location of different types of balls in vision. Team info models information of teammates in vision. Ball info models the properties of different balls, including relative position, size, and speed. Here more details of the information after transformation.
16

---

Published as a conference paper at ICLR 2023
Raw Data
Feature Extraction
Model Input
Scalar Info
Ball Info
Team Info
Spatial Info
Team
Team2
Food
Step
Rank
Time
View Size
Step (2,,,,,,'''')
Shared
Agent
Agent
Figure 8: The overview of feature encoder.
Scalar info
(view_x, view_y): The positions of the center the observation.
view riew_width: The width of the observation.
score: Score of the agent.
Team score of the agent.
time: Remaining game time.
rank: Rank of the agent.
cest_action_pye type: Action type in the previous step.
Ball info
alliance: The numbering for different types in observation.
Scores for different types of balls.
radius: Radius for different types balls.
rank: Rank for different types of balls.
x.y: position for different types of balls.
next next_x.next_y: The predicted position in the next frame for different types of balls.
ball num: The number the balls(clone and thorn) observation.
Team info
alliance: The teammate's 's identity information.
view how_xx,iew_yy The teammate's 's position information.
player_num: The number of the teammates.
Spatial info
food_x, food_y: The position of food balls.
spore_x, spore_y: The position of spore balls.
ball x, ball_y: The position of clone balls and thorn balls.
food num: The number of food balls.
spore_num: The number of spore balls.
17

---

Published as a conference paper at ICLR 2023
Scalar Info
Encoder
MLP
Ball Info
Encoder
Transformer
MLP
Team Info
Encoder
Transformer
MLP
Feature
Spatial Info
Resnet
MLP
Figure 9: Neural network in experiments.
B.2 NEURAL NETWORK
The neural network in experiments consists of MLP (Haykin 1994). Transformer (Vaswani et al. 2017), and ResNet (He et al.) 2016). The specific structure is shown in Figure 9 Finally, we concatenate all the output embeddings into feature.
B.3
EXPERIMENTAL SETUP
DQN
The batch_size is 512. learning_rate is le-4, replay_buffer_size is 2e4.
PPO
The batch_size is set as 256. clip_range is 0.2, gae_lambda is 0.95, and grad_clip is 0.5.
QMIX
The batch_sizz is set as 512. learning_rad rate is le-4, bufeer_sizee is 2c4.
MAPPO
The batch bateh_size set as 512. clip hinge is 0.2, gae lae_aaabda 0.95, and grad clip is 0.5.
The faach__size is set as 2048, and the data in every step contains all the observations and actions of all the agents. id_lambda is 0.95, and grad_clip is 0.5.
VMIX
COMA The batch size is set as 2048, and the data in every step contains all the observations and actions of all the agents. 1d lambda is 0.95, target target_update appateeinterval interval is 20, and grad_clip is 0.5.
C ABLATION STUDY
C.I DROPPING OBSERVATIONS FROM TEAMMATES
Figure shows the food, thorn, spore, clone clone_self, clone_team, team, and clone clone_other of the experiments in Section 6.1 With the observations from teammates, agents are more likely to eat thorn balls instead of food balls. We can find that the number of caten thorn balls more than that without the observations, as eating thorn balls can provide more scores. A single agent can not eat too many thorn balls, it will split into several pieces after eating a thorn ball. With the cooldown period, an agent can not merge quickly to gather enough size to eat the next thom ball. But when an agent learns to cooperate with teammates, they can gather size quickly and eat the next thorn ball ignoring the cooldown period, which could be found by the number of clone team in the figure. Eating more clone balls from teammates means that the agents can better cooperate with other agents in the same team.
C.2 LADDER SYSTEM AND LEADERBOARD
Our ladder system is based on Trullkill (Herbrich al., 2006), which is skill-based ranking system developed by Microsoft for use with video game matchmaking on Xbox Live. TrueSkill is designed support games with more than two players. In our ladder system, assume that each contestant
18

---

Published as a conference paper at ICLR 2023
250
330
250
150
194
330
50
Toe
Eott
180
254
10M
12M
400
MM
ISM
IMM
12M
Stage
Fex
other
250
200
150
frrm
ehs ISUFT
011
TM
CM
JM
IBM
17M
TM
4M
dM
SM
IBM
17M
(64
SV
PM Dago
100
1754
(a) st_ 12p..
250
200
100
130
the Dro KIREIAIS
obs Ben Saremals
50
obe frim KOREEOOS
09
winth fran ttanmmutce
whw four touremacce
sowiiee ha "EAEETUN
MIM
200
12
150
Suna icssezao
sia Evts, coutsratrr obs Toor VILTITaEES
8-0 Sht from VORTETRRS
(b) st s1_3
300
300
679
200
Tiees Laareaate
231
ebc Even icaneeec.
100
free
1V
7M
+M
IM
FMI
10M
12M
0M
75
4V
AM
IM
ICM
17M
DM
2M
4M
FM
IMM
1IM
Trpe
des JC
6002
12
500
10
30
200
Che flock Narrmaas
100
obs Sues ESRDNCIRR
IC the (ioce NERZEARS
thom Varmuaas
trom scammnatt
17M
22
AM
TBC
J/M
1M
18C
224
451
FM
JIM
186
224
JM
3/M
E.M
(c) st 14p3.
Figure 10: Comparing agents w/ or w/o observation from teammates.
controls whole team. The contestant continuously plays against other contestant and update their scores based on the ranking at the end of the game. The initial score for each contestant is 1000. To ensure the balance of contestants' matches, we select the contestant with the least number of matches each time. SggSssowss the quality we use as the indicator for selecting opponents.
2
(M4: 14))
qualityyran (32, Hi Mjj (i, 0z) =
expl
(8)
2,32 + of o
2(2 + of + 03)
19

---

Published as a conference paper at ICLR 2023
Ngedd Sysen
10,90
1014
VMIX
COMA
1010
BO7
1001
1000
Figure 11: Comparing agents on the ladder system. The scenario is st 12_2..2.
(a)
(b)
(d)
(e)
Figure 12: Screenshots of cooperating eat the thorn ball. (a) The clone ball A and B get closed to each other. (b) The clone ball eject to the A. (c) The clone ball A eat the spore and grow bigger. (d) The clone ball A eat several clone balls from teammates and grow big enough to eat the thorn ball. (e) The clone ball A eat the thorn ball.
(a)
(b)
(c)
Figure 13: Screenshots of competition between different agents. (a) The green clone balls find the purple clone balls nearby. (b) The green clone balls split to eat the purple ones. (c) The purple clone balls gather together to avoid being caten by the green clone balls.
where contestant is assumed to exhibit performance P; N(pi ui, of), contestant is assumed to exhibit a performance pj ~ N(p (3),))) and 8² = (g)
Most of the contestants have played over 1500 games with other contestants. The ladder system ensures that each contestant match opponents with similar game levels much as possible while retaining the possibility of matching opponents with higher or lower game levels.
Figure II shows the ladder system including all the trained agents in Section 5
D PERFORMANCE
In this section, we post some screenshots to show the actual performance of the agent in the game, including the cooperation with teammates and the competition with opponents.
20

---

Published as a conference paper at ICLR 2023
(b)
Figure 14: Screenshots of cooperation between different agents. (a) The clone ball C is ready to eat the thorn ball. (b) The clone balls split into many pieces. (c) The clone balls eject to the center the agents. (d) The clone balls finally merge into a big one.
B
8
(a)
(b)
(c)
Figure 15: Screenshots of competition between different agents. (a) The bule and green agents are competing for the thorn ball. (b) The green clone balls split to eat the thorn ball. (c) The blue clone balls split and cat the green clone balls and win more score.
21


Reviews:
[{"review_id": "jHzdIVYZym", "reviewer": "Reviewer_3xPp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel multi-agent RL environment which extends existing game Agar with multiple teams, thereby enabling M x N style games. A single agent in the game manages up to 16 growing cloned balls, which need to be increased in size to gain more rewards. The resulting action space has a discrete- (type of action - move, eject, split) and continuous component (direction of movement). The approach is evaluated for the different configurations of the environment for both single agent as well as multi-agent RL algorithms. The results show the tendency for cooperation. Lastly, the authors also conduct an ablation study.", "summary_of_the_review": "The proposed environment is interesting. In essence, the game enables the study of cooperation and competion between multiple teams, and has a well-defined logic and resulting game complexity. The comparison to other environments is quite clear. The paper provides a well-structured oview of the main competitors and builds its case on enabling more than two teams to compete and enabling the better study of cooperation. To this end, it is difficult for me to see if the claim about lack of cooperation for main competitor Neural MMO is evident. Is it that Neural MMO is to easy to drive coordination? Or maybe the other extreme, is the logic so difficult to allow for complex behaviour with current algorithms?  The conducted evaluation for the proposed environment shows that teams are coordinating to win and not merely try to fight individual opponents by themselves, which shows that coordination is essential for winning. Is this a strategy evolving over time or occuring rather fast? To sum up, I see that the environment has its merits, but am not (yet) conviced of the novelty.", "strength_and_weaknesses": "Strenghts:\n* A sensible extensions towards multiple teams of an established environment\n* Empirical evaluation for the new environment alone is quite exhaustive, showing that the environment fosters coordination among teams as winning strategies\n\nWeaknesses:\n* The paper does not motivate sufficiently well how/why coordination is not evolving in the main competing environment Neural MMO. The claim should be better explained / backed up with further analyses. If there is a theoretical point to be made, it would fit nicely in the motivation and related work, but pointers to empirical results (showing sufficiently more coordination for the new environment) would be helpful as well.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NjMn8VU4rb", "reviewer": "Reviewer_tWZb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper discusses a new multi-agent simulator that is designed to evaluate cooperative-competitive multi-agent reinforcement learning. The contributions seem to focus on simulating multiple teams with various features (e.g., choices of using local individual observations or team observations).", "summary_of_the_review": "While designing a multi-team multi-agent simulator is important to benchmark MARL methods, novelty of the simulator and its impacts on real-world MARL applications are not well justified.", "strength_and_weaknesses": "Strength: \n\n+ Multi-team multi-agent simulators are important to evaluate RL in cooperative-competitive environments.\n+ The introduced simulator offers various features to customize and interact with the simulations.\n\nWeakness:\n\n- The technical advances or practical novelties are not clear. It seems the main contribution is the design of multi-team simulation. However, why extending a two-team multi-agent simulation to a multi-team simulation is challenging?\n\n- The scenario of the simulation is relatively simple. It seems the simulation only considers the agent moving directions but does not consider the agent velocity or the dynamics of the agent.\n\n- It is not clear how this simulation can benefit real-world multi-agent applications. Other than the multi-player Agar game, what do real-world applications share characteristics similar to this simulation?\n\n- While experiments generally show that this simulator can be used to evaluate multiple RL and MARL methods, how do the experiments demonstrate this simulator is better than the previous MARL simulators?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2cgi00wmBn", "reviewer": "Reviewer_DcyR", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents GoBigger, a platform for studying cooperative and competitive behavior at large scale. The game is inspired by Agar. Even if the design of an environment like that described in the paper might be valuable for the community, it is difficult to identify clear technical contributions in this work. The authors tested the platform using implementations of algorithms, namely DQN and PPO for single-agent algorithms and QMIX, MAPPO, COMA and WMIX for multi-agent environments. The authors showed the results of a comparative analysis of the algorithms but the results appear rather standard and not particularly insightful. It is also worth noting that the implementation of the system itself appears to be rather standard in terms of design and implementation choices.\n", "summary_of_the_review": "Overall, this work makes a contribution to the research community in terms of the platform itself, but its novelty is somehow limited both in terms of the design and implementation of the environment itself and the evaluation of the algorithms presented in the paper. In general, it is difficult to understand the specific insights that it is possible to obtain from adopting it compared to other existing environments.", "strength_and_weaknesses": "Strengths\n+ The authors have developed a platform that might be beneficial for the community at large.\n\nWeaknesses\n- It is difficult to identify a clear novel technical contribution in this work. For sure, the reviewer commends the fact that the authors made a new environment available to the research community. \n- The authors do not clarify and discuss the types of studies that are enabled by this platform. Which questions will the community be able to answer through it? What are the limitations of the platform? What are the specific novel insights that a researcher can derive from adopting this testbed/environment?\n- The goal of the evaluation presented in this work should be clarified. In fact, this appears as a paper presenting a platform, but most of the paper is devoted to the analysis of existing algorithms used for playing the game supported by the environment. The authors should perhaps rethink the scope and the focus of the paper. At the moment, unfortunately it does not appear sufficiently focussed.\n- The authors say that the main contribution is the study of collaborations inside a team and across teams (and competition). This is not clearly discussed in the paper in my opinion. I would also argue that other environments allow for this type for analysis - for example the Hide and Seek game is performed by teams and not by single agents (it is an environment where the agents collaborate and compete at the same time).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster