Multi-Agent Multi-Game Entity Transformer
Authors: Rundong Wang
Weixuan Wang
Xianhan Zeng
Liang Wang
Zhenjie Lian
Yiming Gao
Feiyu Liu
Siqin Li
Xianliang Wang
QIANG FU
Yang Wei
Lanxiao Huang
Longtao Zheng
Zinovi Rabinovich
Bo An
Venue: Submitted to ICLR 2023 (2023)
Keywords: reinforcement learning
multi-agent reinforcement learing
transformer
pretrained model

Abstract: Building large-scale generalist pre-trained models for many tasks is becoming an emerging and potential direction in reinforcement learning (RL). Research such as Gato and Multi-Game Decision Transformer have displayed outstanding performance and generalization capabilities on many games and domains. However, there exists a research blank about developing highly capable and generalist models in multi-agent RL (MARL), which can substantially accelerate progress towards general AI. To fill this gap, we propose Multi-Agent multi-Game ENtity TrAnsformer (MAGENTA) from the entity perspective as an orthogonal research to previous time-sequential modeling. Specifically, to deal with different state/observation spaces in different games, we analogize games as languages, thus training different "tokenizers" for various games. The feature inputs are split according to different entities and tokenized in the same continuous space. Then, two types of transformer-based model are proposed as permutation-invariant architectures to deal with various numbers of entities and capture the attention over different entities. MAGENTA is trained on Honor of Kings, Starcraft II micromanagement, and Neural MMO with a single set of transformer weights. Extensive experiments show that MAGENTA can play games across various categories with arbitrary numbers of agents and increase the efficiency of fine-tuning in new games and scenarios by 50\%-100\%. See our project page at \url{https://sites.google.com/view/rl-magenta}.

Under review as a conference paper at ICLR 2023
MULTI-AGENT MULTI-GAME ENTITY TRANSFORMER
Anonymous authors
Paper under double-blind review
ABSTRACT
Building large-scale generalist pre-trained models for many tasks is becoming an emerging and potential direction in reinforcement learning (RL). Research such as Gato and Multi-Game Decision Transformer have displayed outstanding perfor- mance and generalization capabilities on many games and domains. However, there exists a research blank about developing highly capable and generalist models multi-agent RL (MARL), which can substantially accelerate progress towards gen- eral AI. To fill this gap, we propose Multi-Agent multi-Game ENtity TrAnsformer (MAGENTA) from the entity perspective as orthogonal research to previous tim-sseqeental modeling. Specifically, to deal with different state/observation spaces in different games, amalogize games languages by aligning one single game to one single language, thus training different "tokenizers" and a shared trans- former for various games. The feature inputs are split according to different entities and tokenized in the same continuous space. Then, two types of transformer-bee architectures
model are proposed as permutation-invarti numbers of entities and capture the attention over different entities. MAGENTA is trained on Honor of Kings, Starcraft II micromanagement, and Neural MMO with a single set of transformer weights. Extensive experiments show that MAGENTA can play games across various categories with arbitrary numbers of agents and increase the efficiency of fine-tuning in new games and scenarios by 50%-100% 100%. See our project page at https://aitee //sites. google. com/
deal with various
www/view////cecc/coo...c.......e ht///vie////e/e/eeee///veee x1 21-11-1--- magenta.
INTRODUCTION
In recent years, transformer-based models, as solution to build large-scale generalist models, have made substantial progress in natural language processing (Brown et al., 2020; Devlin al., 2018), computer vision (Dosovitskiy et al., 2020; Bao et al., 2021), graph learning Yun et al. (2019); Rong et al. (2020), Furthermore, they are showing their potential in reinforcement learning (RL) (Reed et al., 2022; Lee et al., 2022; Wen et al., 2022) by modeling and solving sequential problems.
However, there are few inherent challenges in building large-scale general RL agents. First, when the training environment and the test environment are the same, RL is inclined to overfit the training environments, while lacking generalizability to unknown environments. As a result, a model is often needed to be re-trained from scratch for a task. Second, it is challenging for single model to adapt for various environments with differences in the numbers of agents, states, observations, actions, and dynamics. Third, training from scratch normally suffers from expensive computational cost, especially for large-scale RL. For example, AlphaStar requires 16 TPUs to train for 14 days, and Honor of King (HoK) requires 19,600 core CPUs and 168 V100 GPUs to train for nearly half month. Thus, building a general, reusable, and efficient RL model has been an increasingly important task for both industrial and non-industrial research.
To this end, we investigate whether single model, with single set of parameters, can be trained by playing multiple multi-agent games in an online manner, which is blank in current research after Gato (Reed et al., 2022) and MGDT (Lee et al., 2022). We consider training on Honor of Kings (HoK), Starcraft micromanagement (SMAC), and Neural MMO (NMMO), informally asking:
Can models learn some general knowledge about games across various categories?
In this paper, we answer this question by proposing Multi-Agent multi-Game ENtity TrAnsformer (MAGENTA). We consider this problem as a few-shot transfer learning with the hypothesis, where

---

Under review as a conference paper at ICLR 2023
single model is capable of playing many games and can be adapted to never-seen-beforr games or scenarios with fine-tuning. For interpretabilityy we align the design of MAGENTA with the Entity Component System (ECS) architectural pattern of video games and the multilingual transformer. Specifically, we treat different games as different languages by aligning one single game to one single language shown in Fig.3. Different languages have different tokenizers, so do games. Also, expect that the learned representations in different games are similar to the word2vec in NLP. In this case, our transformer can be viewed as a multilingual transformer to capture the common knowledge among different games. We split the feature input according to different entities and tokenize the features in the same continuous space. Unlike existing applications of the transformer as causal time-sequential model, the transformer in MAGENTA serves permutation invarrant architecture to attend to the features of different entities. We propose two types of transformer architecture, Encoder-PPe Poollr- (EP) and Snncoder-Decoder (ED), to build permutation-invarian the output of the model should not change under any permutation of the elements in the input. Lastly, we show the training scheme of MAGENTA about how to train and transfer the pre-trained model.
models. In this way,
As a step towards developing a general model in RL/MARL, our contributions are threefold: First, by aligning games and languages, we show that it is possible to train single generalist agent act across multiple environments in an online manner. Second, we present permutation invvriiat models for adapting various numbers of agents in different games. And third, we find that MAGENTA can achieve rapid fine-tuning in an online fashion to different scenarios within a single game, to a new type of game, and to different numbers of agents. Furthermore, we release the pre-trained models and code to encourage further research in this direction.
PRELIMINARY
2.1
REINFORCHMENT LEARNING AND MULTI-AGENT RL
An RL problem is generally studied as Markov decision process (MDP) (Bellman, 1957), defined by the tuple: MDP- = AS, P, r. 7, T), where S C IR'' is an n-dimensional state space, AA C R" an m-dimensional action space, P
A x S IR. a transition probability function, r a bounded reward function, 7 E (0, 1] discount factor and T a time horizon. In MDP, an agent receives the current state st S from the environment and performs an action o, E A defined by a policy Te S + A parameterized by 0. The objective of the agent is to learn an optimal policy:
TR
Tg- == argmax,, Ene ET= E = s] An MARL problem is formulated as a decentralised partially observable Markov decision process (Dec-POMDP) (Bernstein et al., 2002), which is described as a tuple (n, S. A, P, R, O, Ω. 7), where n represents the number of agents. S, A, P, R all global versions of those in MDP. O [O:)i=1, i- denotes the space of observations of all agents. Each agent i receives private observation og E O, according to the observation function (2(s, i): : S Oz. 2.2 ATTENTION MECHANISM IN TRANSFORMER
One of the most essential components of Transformer (Vaswani et al., 2017) is the attention mecha- nism, which captures the interrelationship of input sequences. Assume that we have n query vectors (corresponding
with n elements) each with dimension d, : Q E Raxd, The attention function is written as Attention Q, K, V) W(QKT)V, which maps queries Q outputs using n, key-value pairs K E Rn=xd,, how similar each pair of query and key vectors is, with weights computed with an activation function V E [gaxd is a weighted sum of V where a value gains more weight if its corresponding key has larger dot product with the query. Furthermore, self-attention refers to cases where Q, K. V share the same set of parameters. Multi-head attention is an extension of the attention by computing attention functions simultaneously and outputing a linear transformation of the concatenation of all attention outputs.
E Rn.xdd_ The pairwise dot product QK" E Rx measures
w. The output w (QK)
2.3 ENTITY IN VIDEO GAME DESIGN
In video game development, the Entity Component System (ECS) is the often used software archi- tecturrl pattern for the representation of game world objects (Bilas, 2002). An ECS has three parts:
We present this paper in the scope of MARL,, and abuse the term "feature" alternttive of observation.
2

---

Under review as a conference paper at ICLR 2023
Entity
Enemy
Player
Bullet
BATTLE
Component
Position
Health
Spawning
System
Render System
Health System
Spawning
System
Figure 1: An Example of ECS of the BattleCity game in Atari 2600.
entities, components, and systems. Entities are composed of data from components, while systems operate on entities' components. For example as shown Fig.1, in BattleCity, the enemy tank, the player, and the bullets can be represented different entities. These entities that cause damage might have a health component. The physics system in this game may query entities whether having mass, velocity, well position components, and do physics calculations the sets of components for each entity.
RELATED WORK
3.1 TRANSFORMER IN RL
Recently, transformer- models have been shown to be a powerful tool for RL to capture global dependencies model long-term squence. The architecture of MAGENTA is closely related to AlphaStar (Jaderberg et al., 2019) and OpenAl Five (OpenAI, 2019), which use transformer- based models embed unit features for each player. Another perspective of using transformer- based models to model an RL problem as a sequence prediction problem that models trajectories autoregressively. such Decision Transformer (DT) (Chen al., 2021). Trajectory Transformer (TT) (Janner et al., 2021), GATO (Reed et al., 2022), and Multi-Game DT (MGDT) (Lee et al., 2022). In addition, Multi-Agent transformer (MAT) (Wen al., 2022) considers the MARL problem sequence prediction problem that generates the optimal action of each agent sequentially.
models study the generality with trajectory sequences in single agent scenarios (Reed et al., 2022; Lee et al., 2022) or focus on single task with the sequences (Jader- berg et al., 2019; OpenAI, 2019; Chen et al., 2021; Janner et al., 2021; Wen al., 2022). addition, they use causal transformer decoders, which use causal masking to discard future information. MA- GENTA fills the current research blank by using a transformer model without positional encoding and exploring generality with entity sequences in multi-agent scenarios. Note that models such as DT, TT, and MGDT utilize the sequence along the time dimension, while MAGENTA along the entity dimension, so MAGENTA is orthogonal to time trajectory sequence modeling methods by viewing MAGENTA as a tokenizer in these methods. We summarize these related works as shown in Table 1.
However, these transformer-
Table 1: Related Work on Transformers in RL
Method
Sequence
Data
Agent
Task
Transfer
Games/Domains
DT,TT
trajectory
offline
single
single
No
Atari, Key-to Door wwwpppiiooncaattooooiiiosooeeoee
Gato
trajectory
offline
single multi
No
MGDT online DT
trajectory
offline
single
multi Yes
Atari
trajectory
online
single
single Yes
OpenAI Gym
MAT
agent
online
multi
single
No
Starcraft II micromanagement
AlphaStar
entity
online
multi
single
No
Starcraft II full game
OpenAl Five
entity
online
multi
single
No
Dota 2
HoK,Neural MMO.
MAGENTA
entity
online
multi
multi Yes
Starcraft II micromanagement
3.2 REUSE TRAINED RL MODELS
Another related research direction to reuse trained RL models, which is an important subfield of RL. Some approaches directly the trained models. For example, avoid restarting from scratch after changes in code and environment, OpenAl Five (OpenAI, 2019) used the "surgery" approach to convert trained model to certain larger architectures with customized weight initialization.
3

---

Under review as a conference paper at ICLR 2023
tokenizer
output
spatial feature
CNN
Transformer
Value
entity feature
tokens
heroes
Cp Pg. Pg.... Eiz
tokenizer
FC
Action
FC
Figure 2: (Left) Overall framework of MAGENTA. (Right) An example tokenizer for HoK. Spatial features are as images and entity features as vectors.
MGDT (Lee et al., 2022) and Gato (Reed et al., 2022) follow the manner of few-shot transfer learning by using a single set of weights play many games or handle tasks in multiple domains simultaneously. Furthermore, some approaches implicitly exploit trained models, such as behavior cloning (Gao al., 2018; Hester et al., 2018), distillation (Rusu et al., 2015; Parisotto et al., 2015), and the use of the teacher-student framework (Chang et al., 2015; Schmitt et al., 2018; Uchendu et al., 2022). Considering that all these methods are based on well rell-pretrained model weights, MAGENTA follows the fashion of direct usage and provides our code and trained agents. Interested researchers can use the provided models for further exploration on how to reuse such models.
MULTI-AGENT MULTI-GAME ENTITY TRANSFORMER
In building generic multi-agent Al for different games, the following challenges encountered: (1) different feature inputs (i.e., different observations), different action spaces, and different rewards across different classes of games; (2) the number of entities varies across different classes of games.
Motivated by multilingual models (Devlin et al., 2018; 8; Liu et al., 2020; Xue et al., 2021) in the NLP community, we treat different games as different languages to deal with different inputs of features. Although words or sentences in different languages have different representations, such as "/ love you" in English, and "Je i' t'aime" in French; however, the meaning behind those words the same. We also find the analogy in the ECS game design pattern. Games are designed based on physical rules in the world. Basically, entities in most games can move, collide, and even attack. The components or attributes of these entities are changing according to the "virtual rules", which are mimics of the physical rules. The RL policy can be considered as neural network system to approximate such rules. So divide such neural network system into three parts: tokenizor, transformer, and output. Tokenziors are trained for different games, just as they are trained for different languages in NLP. analogy to the multilingual transformer. While output
Transformer is multi-gama game transformer, are finetuned according to the down-streaming tasks.
4.1 GAMES AS LANGUAGES
Tokenizor is used to obtain initial embeddings of words in the NLP community. When we treat games languages, different features of entities can be analogized to different "words". Such "words" typically composed of images and vectors. Therefore, we need to divide the features into "words" and then map them into the same continuous space.
A good tokenization for different games should be feature-space-ggnosic.. Different games have different features, such as images or vectors. Furthermore, the features of different entities have different shapes, dimensions, and meanings. Therefore, good tokenization is supposed to handle different feature spaces and split the entire feature into individual elements, such as the tokenizer in NLP splits sentences into words.
We categorize the feature input into two types: images and vectors. When playing games, we can have a screenshot as an image. Typically, we call them frames as shown in (Mnih et al., 2015). The numerical attributes of the entities can be vectors. For example, a vector composed of the health point, the mana point, the speed, the attack, and defense, etc. As shown in Fig.2, to process images, we first use convolution neural networks to extract the latent representation and then linearly project the representation into the embedding space. For processing vectors, we divide the vectors into many inputs that correspond to different entities. Then we use a fully connected neural network to get tokens for different entities. The same types of entities share the same parameters in neural networks.
4

---

Under review as a conference paper at ICLR 2023
observations
sentences
R2 1K
love you
Je t'aime
split & tokenize
split tokenize
tokens
tokens
Games as Languages. (Left) Tokenize different observations from different games. (Right) Tokenize different sentences from different languages.
Figure 3:
We chose our tokenization scheme with simplicity in mind, but many other schemes possible. For example, more sophisticated methods, such as VAE (Kingma & Welling, 2013) can be used to learn a more effective token representation.
4.2 TRANSFORMER IN MAGENTA
In many game scenarios, the number of entities to be modeled changes over time (e.g., deaths of soldiers in Starcraft II, revival of enemies in Neural MMO), and although some empty spaces can be set aside in advance from network compatibility perspective, these spaces are not always suitable for new entities. In addition, the order of the entities should not affect the RL model training. Unlike the order of words in sentences, the order dependence in RL leads to inability to handle the input of a varying number of entities.
model for this issue should satisfy two critical requirements. First, the model should be able process input of any size. The number of input tokens depends on the number of entities. Second, should be order-agnostic or permutation-invariinn The output of the model should change under any permutation of the elements in the input. Classical feedforward neural networks violate both requirements, and RNNs are sensitive input order. To this end, use the transformer architecture as the core of MAGENTA. We modify the vanilla transformer architecture by removing the positional encoding to satisfy these two requirements.
Furthermore, to force our model to be order agnostic, we design two types of the transformer architecture, as shown in Figure Encoder-Poolin (EP) and Encoder-Decoder (ED). Detailed despcritption can be found in Appendix. The encoder is used to obtain the mapping between the token representations of entities (c',...,ee" e") and the output of the attentive features (e (et...., E where is the number of all entities. The key idea is that each agent can only affect the related entities, and thus should pay more attention to them.
For EP. we the pooling function to post-process the embeddings. A network that performs pooling over embeddings extracted from the elements set. More formally, net (f e²}) = (pool (f (e.).... ()) Here, operations like mean, summation, max, or similar can be used as the pooling function aggregate the embeddings. Zaheer al. have proven that all permutation-invarte functions can be represented as this inline equation when pool is the sum operator and 6, p any continuous functions, thus satisfying the permutation- -in.aaaaiiiinnnnaiannneeee requirement.
L L En
Transformer Encoder
Pooling
Transformer Decoder
e
e, e2,e2
é
én
Figure 4: The transformer architecture in MAGENTA. (Left) The transformer encoder. (Mid) Pooling. (Right) The transformer decoder. EP consists of the left and mid parts, while ED consists of the left and right parts. We denote the the token representations of entities by (e'....,e² e²) and the output of the attentive features by (é),...,,") We unify from different architectures as the permutation invaaiant representation.
5

---

Under review as a conference paper at ICLR 2023
For ED, we design decoder to post-process the embeddings. Specifically, instead of using self- attention, the decoder uses the original attention mechanism and outputs Attention(Q K, V) E R gdx where Q is the embedding è' of the current agent i (an agent also an entity), while K, V are the embeddings of all entities. In this way, the output of ED is only one single vector that does not change under any permutation of the elements in the input.
4.3 TRAINING AND IMPLEMENTATION DETAILS
Training scheme. We focus MARL scenarios and choose three representative environments from the industrial and academic domains: Honor of Kings (HoK), Neural MMO (NMMO), and Starcraft II mi- cromanagement (SMAC). We design our three-stage training scheme as shown in Fig.5.
0
Performance
000 Random initialized model"
100
-
0
Stage From the random initialized model M train expert models with MAGENTA on each individual game and obtain expert models M) M N, M in different games, where rep- resents HoK, N represents NMMO, S represents SMAC.
101 M1 expert models dift games
101 MZ
000
transfer models games
final model all games
Figure 5: The training scheme of MAGENTA. The dotted lines represent transfers to new game or a new scenario. Each cell repre- sents tokenizer, transformer and output, re- spectively. Different colors represent differ- ent games.
Stage 2. We transfer M to new games and get M2 H.N My.s. The tokenizer and output are trained from scratch in the new games. Note that HoK, with a state and action space of magnitude 1020000 et al., 2020), is much more complex. So we transfer the trained HoK transformer to NMMO and SMAC.
We start from M and train the transformer M -M..N. in selected scenarios in these three environments. The tokenizer and output are from M2. Then we check whether A/3 handle never-scen-before scenarios in these games with few-shot transfer.
Stage 3.
Policy updates. In order to avoid training instability in our large-scale distributed environment, we use the dual-clip PPO method for each agent. Unlike the original algorithm, we introduce the policy mo (a, (x)) and the estimate of advantages A, (a;, o;). When To (a, ox) >> T0a. (as o:) ra(a)lo)) and A, < 0, the ratio (+) fuu (az]o:) is huge, causing the large and unbounded variance since
(,))), < 0. Dual-clip PPO introduces another clipping parameter that indicates the lower bound
when A, < 0. The new objective is defined as:
policy (0) = E, max (cAr, min (110))
(1)
where is the original clip parameter in PPO.
Value updates. We use the multi-head value mechanism for different rewards sources. Specifically, the values of achieving one reward are regarded as head. Therefore, the value loss is defined as:
I. value (0)
E.
Σ
R v.t)
Viccal
Σ
Eu wa (os)
(2)
Lhead
head
where Wk is the learnable weight of the k-th head and V. (o:) is the k-th value.
EXPERIMENTS
We conduct our experiments to answer a number of questions:
How is the performance of the entity transformer in single game?
How effective is MAGENTA in transfer games?

---

Under review as a conference paper at ICLR 2023
Figure 6: Games where MAGENTA trains. (Left) Neural MMO. (Mid) Honor of Kings. (Right)
Starcraft II micromanagement (SMAC). See detailed description about these games in Appendix.
How effective is MAGENTA in transfer to new scenarios?
How does MAGENTA scale with model size?
What does MAGENTA attend to? (See Appendix E.2 due to page limits.)
What the learned embedding in games looks like? (See Appendix E.3)
5.1 SETUP
Models used in the experiments. We list the different versions of /name and the corresponding transfer experiments in Table 2.
EP
ED
TL
model
M
HOK winrate (%)
H,S.N
H,S,N
H
60
(baselines)
40
M
H->
H->
(new games)
S.N
S.N
20
M
H+S+N-> H+S+N->
(new scenarios)
S,N
S.N
50
75
Training
100
125
150
Table 2: Different Models and Transfer Experi- ments.
Figure 7: The performance of different variants of MAGENTA in HoK 3v323
Training and fine-tuning. We train MAGENTA models on Hok with an ADAM optimizer of 2 10- learning rate, 31 0.9 and B2 0.999, without weight decay, gradient clip 0.5, and batch size 2048. And we train MAGENTA on SMAC and NMMO with an ADAM optimizer of 1 10-4 learning rate, 81 0.9 and Bg 0.999, no weight decay, gradient clip 0.3, and batch size 4096. Detailed hyperparameter are described in the Appendix F.
Metrics. We measure the performance of MAGENTA in HoK by the win rate and SMAC/NMMO by cumulative rewards. Due to prohibitively long training times, we only evaluated one training seed HoK and three training seeds SMAC/MMM..
5.2 How IS THE PERFORMANCE OF THE ENTITY TRANSFORMER?
In this subsection, we mainly evaluate M MAGENTA in HoK 3v3 scenario. We trained 3 different variants of MAGENTA: EP, ED, TL (EP followed by an LSTM network), and train FC+LSTM (FC layers as replacement of the transformer) as the opponent to calculate the win rate with three variants. All variants have similar model sizes. We introduce the TL model to show that MAGENTA is orthogonal to the time-sequential modeling methods. At each point, two models with the same training time compete with each other.
omit the performance of the FC+LSTV model, since it a 50% line. We use time-axis since the training is in distributed manner. It is difficult to show the exact number of samples.

---

Under review as a conference paper at ICLR 2023
3mleesy) reward
Brijeayy) neward
6m9mthhrd) reward
16
model
1.8
18
14
142
25minarc)
14
142
16
10
0.0
1.0
0.0
0.2
0.4
0.6
0.0
0.5
1.0
0.0
1:0
2.0
Mil Step
Mil Step
Mil Step
Mil Step
3ebzlebbate hard) roward
27m30miss-he (aard reward
5mmmis-haa hardh reward
GhSxtc hard) reward
9.0
9
16
8.5
8
14
8.0
7-
8
12
7.5
0.0
1.0 Step
2.0
0.0
1.0 Mil Step
2.0
0.0
1.0
2.0
0.0
1.0
2.6
Mil Step
Mil Step
Figure 8: Transfer performance in SMAC. M trained HoK, M in HoK, SMAC 5m_vs_6n _m_m scenario, NMMO 8 agent scenario, and M2 in HoK, SMAC 8m_vs_9m, NMMO agent scenario.
In Fig.7, we can see that after 160 hours of training, all variants of MAGENTA outperform the FC+LSTM model. It shows the benefit of using the transformer architecture. An interesting finding that, at the beginning, MAGENTA hardly won the FC+LSTM model. The transformer can increase model capacity and expressiveness while needing more training data compared to the FC model.
5.3 HOW EFFECTIVE ARE MAGENTA AT TRANSFER TO NEW GAMES?
We want evaluate whether MAGENTA can adapt the the novel games. To do this, transfer the trained HoK transformer NMMO and SMAC. We show the training pro- cess of M 2 and 3 In Fig.8. we show the result of transfer to SMAC. Before this transfer, 2 is already trained in HoK, M² in HoK, SMAC 5m_vs ____v___m________ 6m scenario. NMMO agent scenario, and MY HoK, SMAC Lin_vs_9m NMMO 8 agent scenario. For M2, the tokenizer and the output are trained from scratch with the pre-trained transformer. For M all modules in MAGENTA are pre-trained. In this subsection, we focus on M and M since 2 is trained in the games. We scenarios, including easy, hard, and super hard scenarios. In Fig.9, we show the result of transfer to NMMO. We show the performance in the agent scenario and the agent scenario. We can clearly see that 2 can outperform M'. The results verifies our hypothesis that pre- -raitraining in other games should indeed help with rapid learning of a new game.
2.0
M²
0.8
Bagentrewar reward
M²
0.6
M
Aagent
Sagent
0.4
1.0
0.2
0.0
2.0
0.0
0.2
0.5
Mil Stop
Stop
Figure 9: Transfer performance in NMMO.
that 2 can outperform M in most SMAC
5.4 How EFFECTIVE IS MAGENTA IN TRANSFER TO NEW SCENARIOS?
We devise our own evaluation setup by transferring to the never-seen-befont scenarios in HoK, SMAC, and NMMO. HoK has three scenarios: 1v1, 3v3, and 5v5. In HoK, we have two experiments to answer whether we can transfer to more or fewer agents.
In Fig. 10a, show the training performance of model M MP.1 M2 and M2.5 4. The result shows that the models transferred from 3v3 and 5v5 to 1v1 outperform the model trained 1v1 from scratch. And there is slight performance difference between M3 and M In Fig. 10b, the model transferred from 1v1 to 5v5 outperforms the model trained in 5v5 from scratch. The experiments show that MAGENTA learned some transferable knowledge which can be used in the eever-seen-befor scenarios, even in the single agent scenario. Then we look back at Fig.8 8 and Fig.9. We can see that 3 can outperform 2 in the held-out scenarios. The results in this subsection highlight the benefit of MAGENTA's rapid fine-tuning to new scenarios across different tasks and number agents or entities.
We omit other games in the subscript.
8

---

Under review as a conference paper at ICLR 2023
100
HOK 1v1 winaate (5a
HOK elo Score
80
1500
60
model
3000
40
M'
model
20
M313v3)
500
M
M*(5v5)
[1v1)
20
40
50
60
70
25
75
100
125
150
175
Training hours
Training hours
(a) From 5v5 and 3v3 Ivl.
(b) From Ivl 5v5.
Figure 10: The transfer in HoK. The win rate is against the FC+LSTM model.
5.5 HOw DOES MAGENTA SCALE WITH MODEL SIZE?
180%
We obtain the final performance M on all games and choose M as the baseline to compute the normal- ized score. Fig.
Normal loedalized Score
HoK
SMAC
NMMO
shows that when the size of the model increases, in the complex game, i.c., HoK, MA- GENTA has better performance. In less complex games, MAGENTA shows flat decreasing performance with increasing model size. We interpret this result as overpa- rameterization, where richer model is fitting than essary. In fact, we conducted a confirmatory experiment using just few FC layers in SMAC and NMMO. It show performance similar to M
60
120M
180M
240M
Number Model Parameters
Figure 11: Normalized scores for all training games with different model sizes.
We also conduct an ablation study HoK on which parts of MAGENTA are transferable in Fig. 12. We transfer different parts of MAGENTAto the new scenarios. It shows that all transfers benefit the adapt to the new scenarios and the transfer of the whole model weights achieves the best performance.
10C
2000
80
HOX winut
3500
8
model
model
8
1000
M20k3 p2)
500
20
20
30
60
20
30
70
20
20
60
Traningg
(a) From 3v3 to Ivl
(b) From 5v5 to Ivl
(c) From 5v5 3v3
Figure 12: The ablation of transferable parts of MAGENTA in HoK. P1 represents the transfer of tokenizer. P2 represents the transfer of tokenizer and transfomer. P3 represents the transfer of the whole model weights.
CONCLUSION
In this paper, we have made an attempt to develop highly capable generalist agents. Namely, we provide perspective: viewing different games as different languages. When different tokenizers trained for various games, entities are split into tokens. The utilization of transformer models can deal with different feature inputs and various number of entities. The attention mechanism in the transformer also encourages the agent to focus on highly related entities. Our results exhibit clear benefit using large transformer-basee models from entity perspective in multi-agent multi-gama domains. We believe the trends suggest clear paths for future work, that with larger models and larger suites of tasks.
Limitations. We acknowledge some limitations of our conclusions. First, our results are largely based on performance in battle games, such as real-time strategy games, MOBA games, and MMO games, where the rules are similar to each other. Second, the tokenizer is not general for all games. We still need to train specific tokenizers to handle different entities in various games. Third, we still follow the online fashion, which remains the question whether offline datasets training efficiency. Lastly, it remains unclear whether we can observe other forms of generalization, such zero-shot adaptation, well whether conclusions hold for other settings.
help and boost the

---

Under review as a conference paper at ICLR 2023
REFERENCES
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2/06. 08254,, 2021.
Richard Bellman.
Markovian, Journal of Mathematics and Mechanics, pp. 679-684, 1957.
Daniel S Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilberstein. The complexity of decentralized control of Markov decision processes. Mathematics of Operations Research, 27()): 819-840, 2002.
Scott Bilas. data-driven game object system. In Game Developers Conference PritiiceePPrreeeeee Proceedings, 2002.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877-1901, 2020.
Kai-Wei Chang, Akshay Krishnamurthy, Alekh Agarwal, Hal Daumé III, and John Langford. Learning to search better than your teacher. In International Conference on Machine Learning, pp. 2058- 2066. PMLR, 2015.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in Neural Information Processing Systems, 34:1544::44 15044-15097, 2021.
Jacob Devlin, Ming-Wei Wing-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 18.ivv:880.040
2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 / 1929, 2020.
Yang Gao, Huazhe Xu. Ji Lin, Fisher Yu, Sergey Levine, and Trevor Darrell. Reinforcement learning from imperfect demonstrations. arXiv preprint arXiv:1802.05313. 2018.
Yiming Gao, Bei Shi, Xueying Du, Liang Wang, Guangwei Chen, Zhenjie Lian, Fuhao Qiu, Guoan Han, Weixuan Wang, Deheng Ye, al. Learning diverse policies in moba games via macro-goals.
Advances in Neural Information Processing Systems, 34: 34:16171-16182. 2021.
William H Guss, Cayden Codel, Katja Hofmann, Brandon Houghton, Noburu Kuno, Stephanie Milani, Sharada Prasanna Mohanty, Diego Perez Liebana, Ruslan Salakhutdinov, Nicholay Topin, et al. The minerl competition on sample efficient reinforcement learning using human priors. 2019.
Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
Max Jaderberg, Wojciech M Czarnecki. lain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, et al. Human- level performance in 3d multiplayer games with population-based reinforcement learning. Science, 364(6443):859 865, 2019.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in Neural Information Processing Systems, 14:1273-1286. 2021.
Michal Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech Jaskowski. Viz. Doom: Doom-based Al research platform for visual reinforcement learning. In IEEE Conference on Computational Intelligence and Games, pp. 341-348, Santorini, Greece, Sep 2016. IEEE, URL http:/ http://axvvvv... org/ abs/ /1005.6...0 02097. The best paper award.
Diederik P Kingma and Max Welling. arXiv:1312.66 2013.
Auto-encoding variational bayes.
arXiv preprint
10

---

Under review as a conference paper at ICLR 2023
Kuang- Huen Lee, Ofir Nachum, Mengjia Yang. Lisa Lee, Daniel Freeman, Winnie Xu, Sergio Guadarrama, lan Fischer, Eric Jang, Henryk Michalewski, et al. Multi-game decision transformers. arXiv preprint arXiv:2205.15244 2022.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zeitlemoyer. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726-742, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 510 518(7540):529-53) 2015.
OpenAl. OpenAl Five. https: https://opeeaa..comm/lo///////// March 4, 2019.
five/, 2019. Accessed
Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning, arXiv preprint urXiv:1511.063 2015.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning, pp. 2778-2787. PMLR, 2017.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.066 022200006775. 17, 2022.
Microsoft Research. Minerl: Towards ai in minecraft. https::/ / readtheeoss to/ en/ htt//aaess///ttt/.lt index. html, 2019. Accessed July 4, 2022.
Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Self-supervised graph transformer on large-scale molecular data. Advances in Neural Information Processing Systems, 13:12259-12577 2020.
Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirk- patrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distillation. arXiv preprint 17Xiv:151 2015.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. Rudner, Chia-Man Hung, Philiph H. Torr, Jakob Foerster, and Shimon Whiteson. The StarCraft Multi-Agent Challenge. CoRR, abs/1902.04043. 2019.
Simon Schmitt, Jonathan Hudson, Augustin Zidek, Simon Osindero, Carl Doersch, Wojeiech M Czarnecki, Joel Z Leibo, Heinrich Kuttler, Andrew Zisserman, Karen Simonyan, et al. Kickstarting deep reinforcement learning. arXiv preprint /803 2018.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwieserr, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7787):484 489, 2016.
Joseph Suarez, Yilun Du, Clare Zhu, Igor Mordatch, and Phillip Isola. The neural mmo platform for massively multiagent research. In J. Vanschoren and S. Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021. URL https: datasets seseettt--eeencce benchmarks proctes-coooeeeeoeeeeoe neurips cc httpp/pper///000 Paper roundl pdf.
Ikechukwu Uchendu, Ted Xiao, Yao Lu, Banghua Zhu, Mengyuan Yan, Joséphine Simon, Matthew Bennice, Chuyuan Fu, Cong Ma, Jiantao Jiao, et Jump-start reinforcement learning. arXiv preprint arXiv:2204.02372. 2022.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
11

---

Under review as a conference paper at ICLR 2023
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017.
Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schrittwieser, et al. StarCraft II: A new challenge for reinforcement learning. arXiv preprint arXiv:1708.04782. 2017.
Hua Wei, Jingxiao Chen, Xiyang Hongyang Qin, Minwen Deng, Siqin Li, Liang Wang. Weinan Zhang. Yong Yu, Lin Liu, et al. Honor of kings arena: an environment for generalization in competitive reinforcement learning. arXiv preprini urXiv:2209. 08483, 2022.
Muning Wen, Jakub Grudzien Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, and Yaodong Yang. Multi-agent reinforcement learning is sequence modeling problem. arXiv preprint arXiv:2205. 14953, 2022.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5: massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisticx; Human Language Technologies, pp. 483-498, June 2021.
Deheng Ye, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang, Xipeng Wu, Qingwei Guo, al. Mastering complex control in moba games with deep reinforce- ment learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 6672-6679, 2020.
Seongjun Yun, Minbyul Jeong, Rachyun Kim, Jaewoo Kang, and Hyunwoo Kim. Graph transformer networks. Advances in Neural Information Processing Systems, 32, 2019.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander Smola. Deep sets. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30, 2017.
12

---

Under review as a conference paper at ICLR 2023
A RELATED WORK ABOUT RL IN VIDEO GAMES
Games have been a focus of artificial intelligence research for decades as a stepping stone towards more general applications. Recently, (dcep) RL has been widely used in Game AI, for example, Go (Silver et al., 2016), Atari (Mnih et al., 2015), Super Mario Bros (Pathak et al., 2017), Quake III Arena Capture the Flag (Jaderberg et al., 2019), VizDoom (Kempka et al., 2016), Minecraft (Research, 2019; Guss et al., 2019), Neural MMO (Suarez et al., 2021), Honor of Kings (Wei et al., 2022; Ye al., 2020; Gao et al., 2021), StarCraft II (Jaderberg et al., 2019; Samvelyan al., 2019; Vinyals et al., 2017), and Dota 2 (OpenAI, 2019). In this work, we mainly focus on using a single set of weights play mceti-scenario multi-agent games, i.c. HoK, Neural MMO, and StarCraft II.
B HONER OF KINGS
Honor of Kings is one of the most popular MOBA games worldwide. The gameplay is to divide ten players into two camps to compete on the same symmetrical map. Players of each camp compete for resources through online confrontation, team collaboration, etc., and finally win the game by destroying the enemy's 's crystal. The behaviors performed by players in the game can be divided into two categories: macro-strategies and micro-operations. Macro-strategy that is, long-distance scheduling or collaborating with teammates for quick resource competition, such as long- distance support for teammates, collaborating to compete for monster resources, etc. Micro-operation that is, the real-time behavior adopted by each player in various scenarios, such as skill combo release, evading enemy skills, etc. Complicated game maps, diverse hero combinations, diverse equipment combinations, and diverse player tactics make MOBA games extremely complex and exploratory.
Figure 13 shows the UI interface of Honor of Kings. For fair comparisons, all experiments in this paper were carried out using a fixed released gamecore version (Version 3.73 series) of Honor of Kings.
Main Hero
214
C.1
C.3
C.2
Figure 13: The UI interface of Honor of Kings. The hero controlled by the player is called Main Hero. The player controls the hero's 's movement through the bottom-left wheel (C.1) and releases the hero's skills through the bottom-right buttons (C.2, C.3). The player can observe the local view via the screen, observe the global view via the top-left mini-map (A), and obtain game states via the top-right dashboard (B).
Feature Design. See Table 3.
Agent Action. Table 4 shows the action space of agents.
Reward Design. Table 5 demonstrates the details of the designed environment reward.
C SMAC
SMAC benchmark is a challenging set of cooperative StarCraft II maps for micromanagement developed by (Samvelyan et al., 2019) built on DeepMind's PySC2 (Vinyals et al., 2017). We
13

---

Under review as a conference paper at ICLR 2023
Table 3: Feature details.
Feature Class
Field
Description
Dimension
1. Unit feature
Scalar
Inclodes heroes, minions, monsters, and turrets
3946
mana, speed, level, gold, KDA.
Heroes
Status
and magical attack and defense, etc.
1562
Position
Current coordinates
Status
Current HP, speed, visibility, killing income, etc.
920
Minions
Position
Current coordinates
Monsters
Status
Prxition
Current HIP, speed, visibility, killing income, elc.
Current 2D coordinates
728
56
targets, atlack speed, etc.
540
Thrrets
Position
Current 2D coordinates
40
In-game stats feature
Scalar
Real-time statistics of the game
104
Time
Current game time
57
Camp
Types camps
Static statistics
Alive heroes
Number alive heroes of twa camps
Kill
Kill number of each camp
Alive turrets
Number of alive turrets of two camps
Alive heroes diil'
Alive heroes difference between camps
11
Comparative statistics
Kill diff
Kill difference between two camps
Alive turrers diff
Alive turrets difference between camps
Table 4: The action space of agents.
Action
Detail
Description
Illegal action
Placeholder.
None action
Executing nothing stopping continuous action. Moving to a certain direction determined by move and move y.
Move
Normal Attack
Executing normal attack enemy unit.
SkillI
Executing the first skill.
Skill2
Executing
skill.
What
SkilL3
Executing the third skill. Executing fourth (only
have Skill4).
Summoner ability
An additionnl skill choosing before the game bagins (10 to choose).
Return (Recal)))
Returning to spring, should he continuuslly executed.
Item skill
Some items enable additional skill player's hero.
Restore
Blood recovering continuously in 10s, can be disturbed. given by special ally heroes.
Collaborative
Move X
The x axis offket of moving direction.
Move Y
The y-axis offict of moving direction.
Skill X
The x-axis offsct of skill.
The y-axis offict skill.
Who
Turgel unit
The game unil(s) chosen to atlack.
introduce states and observations, action space and rewards of SMAC, and environmental settings of MAGENTA below.
States and Observations. At each time step, agents receive local observations within their field of view, which contains information (distance, relative relative y, health, shield, and unit type) about the map within circular area for both allied and enemy units and makes the environment partially observable for each agent. The global state is composed of the joint observations, which could be used during training. All features, both the global state and in individual observations of agents, normalized by their maximum values.
Action Space. actions are in the discrete space. Agents are allowed to make moveldirection), attack[enmmy id], stop and no-op. The no-op action is only for dead agents and it is the only legal action for them. Agents can only move in four directions: north, south, east, or west. The shooting range is set to for all agents. Having a larger sight range than shooting range allows agents make use of the move commands before starting to fire. The automatical built-in behavior of agents is also disabled for training.
14

---

Under review as a conference paper at ICLR 2023
Table 5: The details of the environment reward.
Head
Reward Item
Weight
Type
Deseription
Gold
0.005
Dense
The gold gained.
Experience
0.001
Dense
The experience gained.
Farming Related
Mana
0.05
Dense
The rate mana fourth power).
No-ap
-0,000001
Dense
Stop and do nothing.
Attack monster
Spurse
Attack monster.
Kill
I
Sparse
Kill enemy hero.
Death
-1
Sparse
Being killed.
Assist
1
Sparse
Assists.
KDA Related
Tyrant
Spurse
buff killing lyrant, lyrant, storm lyrant.
Overlord haff
1.5
Sparse
Get huff' of killing the overlord.
Expose invisible enemy
Sparse
Get visions of enemy heroes.
Last hit
0.2
Sparse
Last hitting an encmy minion.
Health point
The bealth point
(to fourth power).
Damage Related
Hurt to hero
0.3
Sparse
Attack enemy heroes.
Pushing Related
Attack Attack crystal
1
Sparse
Attack
1
Sparse
Attack cacmy home base.
Win/Lose Related
Destroy home base
2.5
Sparse
Destroy enemy home hase.
Rewards. At each time step, the agents receive a joint reward equal to the total damage dealt on the enemy agents. In addition, agents receive bonus of 10 points after killing each opponent, and 200 points after killing all opponents for winning the battle. The rewards are scaled so that the maximum cumulative reward achievable in each scenario is around 20.
Environmental Settings of MAGENTA. The difficulty level of the built-in game AI we use in our experiments is level 7 (very difficult) by default as many previous works did. The used scenarios shown in Table 6. We present the table of all scenarios in SMAC in Table 6. The Ally Units are agents trained by MARL methods and Enemy Units are built-in game bots. For example, 5m vs vs_6m indicates that the number of MARL agent is 5 while the number of the opponent 6. The agent (unit) type is marine". This asymmetric setting is hard for MARL methods.
Table 6: SMAC Environments
Name
Ally Units
Enemy Units
Type
3m
3 Marines
3 Marines
homogeneous & symmetric
8m
Marines
8 Marines
homogeneous & symmetric
25m
25 Marines
25 Marines
homogeneous & symmetric
5m_vs_ 66
5 Marines
6 Marines
homogeneous & asymmetric
8m_vs_9m
8 Marines
9 Marines
homogeneous & asymmetric
27m vs. 30m
27 Marines
30 Marines
homogeneous & asymmetric
Stalkers &
3 Stalkers & 6 Zealots
$$$5__s_3$62
heterogeneous asymmetric
Zealots
6h_ sh_vs8
Hydralisks
Zealots
micro-trick: focus fire micro-trick: wall off
corridor
6 Zealots
24 Zerglings
D NEURAL MMO
Neural MMO is a platform inspired by Massively Multiplayer Online games, genre that sim- ulates persistent worlds with large player populations and diverse gameplay objectives. It fea- tures game systems configurable to research both individual aspects of intelligence (e.g. navigation, robustness, collaboration) and on combinations thereof. Support spans to 1024 agents and minute- hours-long time horizons. Our enviroment code heavily based
'A type of unit (agent) in StarCraft II. Readers can refer to https://1ice ped net./ starcraft2/Marin ne_ of veid) for more information
15

---

Under review as a conference paper at ICLR 2023
the JJCA12022-Neum MMO PvE challenge: (jcai-2022--0 satte-neura-mmooo
Observations and Actions: Neural MMO agents observe sets of objects parameterized by discrete and continuous attributes and submit lists actions parameterized by lists of discrete and object- valued arguments. This parameterization is flexible enough to avoid major constraints environment development and amenable to efficient serialization (see documentation) to avoid bottlenecking simu- lation. Each observation includes 1) fixed crop of tile objects around the given agent parameterized by position and material and 2) the other agents occupying those tiles parameterized by around a dozen properties including current health, food, water, and position. Agents submit move and attack actions on each timestep. The move action takes a single direction argument with fixed values of north, south, east, and west. The attack action takes two arguments: style and target. The style argument has fixed values of melee, range, and mage. The agents in the current observation are valid targer argument values. Encoding/decoding layers are required to project the hierarchical observation space to fixed length vector and the flat network hidden state to multiple actions. We also provide reusable PyTorch subnetworks for these tasks.
Neural MMO tasks are defined by reward function on particular environment configuration (as per above). Users may create their own reward functions with full access to game state, including the abilitt to define per-agent reward functions. We also provide two default options: simple survival reward (-1 for dying, 0 otherwise) and a more detailed achievement system. Users may select between self-contained and tournament evaluation modes, depending their research agenda.
This reward function is based on gameplay milestones. For example, agents may receive small reward for obtaining their first piece of armor, medium reward for defeating three other players, and large reward for traversing the entire map. The tasks and point values themselves are clearly domain speciif,,,,, but we believe this achievement system has several advantages compared to traditional reward shaping. First, agents cannot farm reward- in contrast to traditional reward signals, each task may be achieved only once per episode. Second, this property should make the achievement system less sensitive to the exact point tuning. Finally, attaining a high achievement score somewhat guarantees complex behavior since tasks are chosen based upon difficulty of completion. We are currently running a public challenge that requires users to optimize this metric.
Achievement system:
E ADDITIONAL EXPERIMENTAL RESULTS
E. Raw SCORES OF M3 IN HOK, SMAC, AND NMMO
We show the raw scores of M3 in HoK, SMAC,, and NMMO. In SMAC, we compute the mean reward over 8 environments and NMMO over 2 environments.
HoK (win rate)
SMAC (mean reward over 8 envs)
NMMO (mean reward over 2 envs)
78M
69.3%
15.6
1,14
157M
87.5%
16.4
1.19
236M
97.2%
13.9
1.25
E.2 WHAT DOES MAGENTA ATTEND TO?
We find that MAGENTA consistently attends to related entities. Fig. 14 visualizes the selected attention heads and layers for HoK 3v3 scenarios. By fixing one frame in running competition in HoK, we find that heads attend to entities such as monster, enemy, and friend, which are the entities highly related to the current agent's state. The visualization of MAGENTA in SMAC shown in Appendix 15. We find that heads attend to allies.
E.3 WHAT THE LEARNED EMBEDDING GAMES LOOK LIKE?
In this subsection, visualize the embedded entities in HoK and SMAC using t-SNE (Van der Maaten & Hinton, 2008). We run MAGENTA for one episode in HoK and collect the embedding of entities for 6000 game frames. Also, we run MAGENTA for 50 episodes in SMAC and collect
16

---

Under review as a conference paper at ICLR 2023
CV.
4.0
Figure 14: The attention the entity transformer on 3v3
main agent
Figure 15: The visualization of attention weights in SMAC.
200
100
100
50
0-
-50
amy here
soldter
-100
ony soldier
ally tower
emy tower
-20
-200
-100
(a) HoK.
(b) SMAC.
Figure 16: The visualization of entity embedding in HoK and SMAC.
the embedding of the entities. We can see clear separation between allies' embedding and ene- mies' embedding in both HoK and SMAC. Even though HoK and SMAC are two different games, MAGENTA can still learn some general knowledge across different games.
F IMPLEMENTATION DETAILS
First, we introduce the EP and EP architecture. The parameters of the network architecture are: Transformer Encoder layers: 3, Decoder layers: 3, HeadNum: 4, HeadDim: 256. 2 vector with dimension of d. d, is the dimension of the value matrix in the attention module. Since the output of the encoder is (n, d,), where n the number of entities, the pooling merges along the entities and the decoder uses e; query to extract information. Here z; denote the
17

---

Under review as a conference paper at ICLR 2023
permutation invariian representation and do not differentiate from different models. We modify the main text according to this point for clarity.
q.k.
Transformer Incoder/Decoder
Main Info
FC
Action
Transformet Encoder
Average Pooling
Esead
Ally Info
FC
X.
Softmax
Matmul
Emy Info
FC
FC
Value
Unner
Unner
quiery
Figure 17: The architecture of EP.
G.A.v
Transformer Encoder/ Decoder
Main Info
FC
FC
Action
Transformrr Encoder
Transformer Decader
Uncar
Uncar
Ally Info
Matmul
Matmul
Emy Info
Vallun
Ureat
Lroar
&x
query
key
vollad
Figure 18: The architecture of ED.
Here, we describe the framework. For HoK, we use the self-developed RL framework for train- ing (Ye et al., 2020; Gao al., 2021). For SMAC and NMMO, we use the open-source Ray RLlib implementation of Proximal Policy Optimization (PPO), which scales out using multiple workers for experience collection. This allows us to use a large amount of rollouts from parallel workers during training to ameliorate high variance and aid exploration. We do multiple rollouts in parallel with distributed workers and use parameter sharing for each agent. The trainer broadcasts new weights to the workers after their synchronous sampling. Now turn attention environment-specife settings.
Table 7: MAGENTA hyper-parameters used in HoK.
Name
Value
Discount rate
0.99
GAE parameter
1.0
KL coefficient
0.2
Rollout fragment length
1000
Training batch size
100000
SGD minibatch size
10000
# of SGD iterations
60
Learning rate
le-4
Entropy coefficient
0.0
Clip parameter
0.3
Value function clip parameter
10.0
18

---

Under review as a conference paper at ICLR 2023
Table 8: MAGENTA hyper-parameters used in SMAC.
Name
Value
Discount rate
0.99
GAE parameter
1.0
KL coefficient
0.2
Rollout fragment length
1000
Training batch size
100000
SGD minibatch size
10000
# of SGD iterations
60
Learning rate
lc-4
Entropy coefficient
0.0
Clip parameter
0.3
Value function clip parameter
10.0
Table 9: MAGENTA hyper-parameters used in NMMO.
Name
Value
Discount rate
0.99
GAE parameter
1.0
KL coefficient
0.2
Rollout fragment length
1000
Training batch size
100000
SGD minibatch size # of SGD iterations
10000
60
Learning rate
le-4
Entropy coefficient
0.0
Clip parameter
0.3
Value function clip parameter
10.0
19


Reviews:
[{"review_id": "i9kkUnVhtY", "reviewer": "Reviewer_5qFD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces MAGENTA, which is a variant of PPO with a transformer architecture. Based on experiments in large cooperative video games, MAGENTA appears to be able to transfer knowledge across games. \n", "summary_of_the_review": "I think this paper is the first to show what many might have suspected: that large models transfer and generalize in cooperative multi-agent environments. This is interesting and I think future work can build off of this fact. ", "strength_and_weaknesses": "I think this is a nice paper. Although similar research has been done in language, offline RL, and multi-game RL, this is the first to show that the same trend of scale helping generalization also works in cooperative multiagent RL.\nThe experiments are on very large video games, and it is not obvious that so much would transfer to different video games, especially because the representation is not the part that gets transferred, but the transformer that decides which actions to take. \nOne thing I would have liked to seen would be better small-scale experiments where comparisons with many baselines could be tested. \nThe results are not striking, but I think they clearly show the benefit of MAGENTA\nI think this paper will have some impact in the related literature on cooperative RL and transformers for decision making\nFigure 7 should be clearer, I think this is head to head against the FC+LSTM\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "hdDCfPozLLx", "reviewer": "Reviewer_D4zs", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors proposed a novel transformer architecture to solve multi-agent games. The model contains two major parts (1) game specific tokenizers and output layers which encode different game observations into the same token space and output the right actions (2) the main transformer model with ​​permutation invariant pooling and encoder-decoder modifications. With a multi-stages training, the proposed method is able to fast adapt to novel games or new game settings. \n", "summary_of_the_review": "While the paper is mostly clearly written, the novelty of the paper is limited.  ", "strength_and_weaknesses": "The strengthnesses of the paper:\n1) Proposed a single transformer architecture to solve multiple different battle games.\n2) Ablation studies to test the performance of the model(s) at different training stages.\n\nThe weaknesses of the paper:\n1) The clarity of the paper can be improved. For example in section 4.2 authors discuss the encoder-pooling and the modified encoder-decoder architectures, but leave many missing details: the latent variable z_i seems to be a scalar with encoder pooling due to the symmetry requirements, which looks rather stranger (or is that the case?).  \n2) Need more details on the architecture, i.e. how many encoders/decoders/attention heads are used, etc. \n3) Also, the performance comparison plots use different x-values: some use samples and some use time. This is a bit confusing. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "gzwe3Zb-9i1", "reviewer": "Reviewer_gmYJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper describes a tranformer-based architecture for policies for multi-agent (video) games, that tokenises image-based inputs plus entity-component-system-based inputs (representing data of entities in video games), allowing the architecture to process inputs from various different games in a single and consistent manner. Policies are trained in three different video games (Honor of Kings, Neural MMO, and StarCraft II Micromanagement), some of which have multiple different scenarios. Several experiments show that it is effective to train in some games, transfer to others, and then finetune in the other games.", "summary_of_the_review": "The paper is interesting and has some interesting and potentially significant contributions, but in its current form has too many issues in terms of clarity.\n\n**Note after discussion with authors**: I feel that the paper has been substantially improved during the discussion phase and have updated my score accordingly.", "strength_and_weaknesses": "## Strengths:\n\nThe primary strength of the paper is that the empirical results look convincing, and sufficiently extensive with some ablations. To the best of my knowledge, the proposed architecture is also novel, and it has some interesting insights.\n\n## Weaknesses:\n\nThe primary weakness is that several parts of the paper are unclear, and some statements even seem wrong. See below for detailed comments related to this.\n\n---\n\n**Note after discussion with authors**: I feel that the paper has been substantially improved during the discussion phase and have updated my score accordingly.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "O7EYyG_s78", "reviewer": "Reviewer_9Jjr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a learning framework, MAGENTA, that can tackle multiple multi-agent environments with one transformer agent. The authors focused on real-time strategy games in this work, including Honor of Kings (HoK), Starcraft II micromanagement (SMAC), and Neural MMO (NMMO). They treat agent entities as tokens and learn separated tokenizers, a unified transformer body, and separated output layers. It shows that it is possible to train generalist agents for multiple multi-agent environments in an online manner (using PPO), and shows that such online generalist agents can rapid adapt to never-seen-before games or scenarios with fine-tuning. The results empirically show that the proposed model learn common knowledge about multi-agent games across various categories.", "summary_of_the_review": "I'm not able to interpret the results given the current manuscript, so couldn't recommend acceptance now. I hope the authors would be able to fix this.", "strength_and_weaknesses": "Strengths\n* It is a good proof of existence of generalist agents for multiple multi-agent environments. I think showing the community how to learn such agent is a valuable empirical contribution.\n* The transfer-to-unseen-scenario results and transfer-to-new-game results are convincing, showing that training the pre-trained model with more games are usually beneficial.\n\nWeaknesses\n* I found Sec. 5.5 (scaling and comparison to single-game agents) confusing. With only scores normalized by single-agent performance at each size tick. I think it doesn't have enough information for me to interpret the results and the conclusion \"We interpret this result as overparameterization, where a richer model is fitting than necessary\" is kinda hand-wavy. I would suggest adding raw scores, which could help the reviewers to inspect the claim and help reader to understand better.\n* The learning curriculum (Figure 5) is adhoc, and there is no ablation on the curriculum. The current curriculum basically uses Honor of Kings as an anchor. I think even just explaining what could happen otherwise would be helpful. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject