Transformer-based Working Memory for Multiagent Reinforcement Learning with Action Parsing
Authors: Yaodong Yang
Guangyong Chen
Weixun Wang
Xiaotian Hao
Jianye HAO
Pheng-Ann Heng
Venue: NeurIPS 2022 Accept (2022)
Keywords: multiagent system
deep reinforcement learning
action parsing
working memory

Abstract: Learning in real-world multiagent tasks is challenging due to the usual partial observability of each agent. Previous efforts alleviate the partial observability by historical hidden states with Recurrent Neural Networks, however, they do not consider the multiagent characters that either the multiagent observation consists of a number of object entities or the action space shows clear entity interactions. To tackle these issues, we propose the Agent Transformer Memory (ATM) network with a transformer-based memory. First, ATM utilizes the transformer to enable the unified processing of the factored environmental entities and memory. Inspired by the human’s working memory process where a limited capacity of information temporarily held in mind can effectively guide the decision-making, ATM updates its fixed-capacity memory with the working memory updating schema. Second, as agents' each action has its particular interaction entities in the environment, ATM parses the action space to introduce this action’s semantic inductive bias by binding each action with its specified involving entity to predict the state-action value or logit. Extensive experiments on the challenging SMAC and Level-Based Foraging environments validate that ATM could boost existing multiagent RL algorithms with impressive learning acceleration and performance improvement.

Transformer-based Working Memory for Multiagent Reinforcement Learning with Action Parsing
Yaodong Yang', Guangyong Chen Weixun Wang", Xiaotian Hao", Jianye Hao', Pheng Ann Heng
DDpaarmeett of Computer Science and Engineering, The Chinese University of Hong Kong 7Zejjang Lab 7Zeejiang University *Tanjin University Instittte of Medical Intelligence and XR, The Chinese University of Hong Kong yydapple@gmail com, gychen@zheeiangeee com.
[wxwang, xiaotianhao, jianye. tt edu. cn, phengêcse. cuhk. edu. hk
Abstract
Learning in real-world multiagent tasks is challenging due to the usual partial observability of each agent. Previous efforts alleviate the partial observability by historical hidden states with Recurrent Neural Networks, however, they do not consider the multiagent characters that either the multiagent observation consists of number of object entities or the action space shows clear entity interactions. To tackle these issues, we propose the Agent Transformer Memory (ATM) network with transformer-bet based memory. First, ATM utilizes the transformer to enable the unified processing of the factored environmental entities and memory. Inspired by the human's 's working memory process where limited capacity of information temporarily held in mind can effectively guide the decision-making ATM updates its fixed-capacity memory with the working memory updating schema. Second, agents' each action has its particular interaction entities in the environment, ATM parses the action space introduce this action's 's semantic inductive bias by binding each action with its specified involving entity to predict the state-action value or logit. Extensive experiments on the challenging SMAC and Level-Based Foraging environments validate that ATM could boost existing multiagent RL algorithms with impressive learning acceleration and performance improvement.
Introduction
Multiagent reinforcement learning (MARL) way of learning how make sequential decisions when multiple agents interact with the environment given the constraints of partial observability 25 There are many representative real-world applications of MARL including wireless network optimization [7], autonomous driving
[5] and energy distribution (2)]. Partial observability has been long-standing challenge for reinforcement learning (RL) since the real environment state can be partially hidden from the agent 22,, and this issue becomes even worse for MARL as the involving dynamics more complex than single-agent RL with teammates concurrently exploring.
Typically, to tackle the partial observability, most previous efforts propose to summarize the historical observation trajectories to provide additional information for the current local observation. For instance, in a navigation task, the only way to distinguish two T-junctions that look identical is to remember the past key observations before entering into either T-junction (2). Among them, gated Recurrent Neural Networks (RNNs) 33 serve as the most popular mechanism to represent memory for RL 19 10, 13 and does for MARL [26] 31). Given the superior performance achieved by the transformer on sequential modeling tasks over RNNs 34 some pioneer efforts have been
CCoressponiigg author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

contributed to replace RNNs with transformer as the memory structure in single-agent RL, resulting in superior performance especially on tasks that require the long horizon reasoning ability with sequential memory 17
However, the transformer-basee methods remain limited because the computational complexity increases quadratically with the length of historical observation trajectories. Meanwhile, these RNN- -ased and transformer-basee based works seldom consider the multiagent system characters that the multiagent observation consists of a number of object entities and the action space contains meaningful and clear entity interactions. In this paper, make the first attempt to leverage the transformer as a memory processing structure to facilitate MARL algorithms under the partially observable setting through carefully considering the above multiagent system characters.
28 3
Our first contribution is to introduce the transformer- based working memory [17 23 into MARL. For the first time, we develop unique transformer- based multiagent memory structure with the help of the working memory updating mechanism by explicitly considering the allied agents for the factorized multiagent observation space. Specifically, we dynamically maintain a fixed-capacity memory for each agent and update with the imitated human working memory process where a small amount of information is stored in mind while working with it for effective decision making [22]. Furthermore, considering the factored multiagent environment, we leverage the transformer to generate new memory by uniformly processing the sequential memory slots and observed environment entities. Additionally, with the fixed-capacity memory. the computational complexity of the transformer keeps the same when agents continuously interact within the environment.
Our second contribution is introduce an action semantic inductive bias and implement with a technique named Entity-Bouu Bound Action Layer. The main insight behind the Entity-Bound Action Layer is that each action in the action space interacts with particular involved environment entities according to its semantic meaning [36]. For example, in one battle scenario, an attack action mainly causes impacts on the targeted specific enemy. Thus, we are motivated to parse the action space to explicitly model the interaction, so that each action is uniquely bound to its targeted entity to predict the state-action value or logit. Entity-Bound Action Layer brings lots of benefits, including increase the computational flexibility of the policy network and making the agent's 's action more explainable.
Furthermore, we find that the above contributions can be seamlessly integrated into a compact model, what we term as Agent Transformer Memory (ATM) network, with simultaneously processing the sequential working memory and the spatial entities, and binding unique entity embeddings to actions by the action semantic inductive bias. Extensive experiments on the challenging SMAC and Level- Based Foraging environments validate that ATM could be easily plugged in many existing MARL algorithms with impressive learning acceleration and performance improvement across various tasks.
2 Background
2.1 Markov Games
Markov games are a multi-agent extension of Markov Decision Processes [6] They are defined by state transition function, 7' S x A x ... A + P(S), which defines the probability distribution over all possible next states, P(S), given the current global state S and the action A' produced by the i-th agent. Note that the reward is usually given based on the global state and actions of all agents Ri: : S A X x A + R. If all agents receive the same rewards, i.c. R = R Markov games are fully- coopeccoopea a best-interest action of one agent is also a best-interest action of others [88.
We use the partially observable Markov games as our settings, in which each agent i receives a local observation o' iZ( + O'. Thus, each agent learns policy x' O + P(A'), which maps each agent's 's observation to a distribution over its action set, to maximize its expected discounted returns, I' (xx'') =E, Fagl='' "_ ""....."."""" ...T EE I (, ...,"a M) with y E [0. 1] as the discounted factor. To help address the partial observability issue, we record the historical experience of agent i's action- observation r' to provide extra information. Traditionally, an RNN such as GRU is used in the agent's 's individual policy or Q-value function network for abstracting its historical experience (30]
2.2 Transformer
Harnessing the transformer to handle the partial observability has been investigated in RL 132 17
28 The core of the transformer is the self-attention mechanism [4] The self-attention adopts
three matrices, Q Tin W K Ti, w* V W" representing set of queries, keys and values

---

respectively where The E R is the input of transformer and W E R4*4, W E RR4d4 and W I2 xd, are learnable weight matrices. n, the number of input entities while des d2, d, denote the dimensions of input entities, keys (or queries) and values. The attention output is computed as
QK?
Air K. V) softmax(
V dd
where dg also behaves as scaling factor. In practice, multi-head structure employed to allow the model to focus on different representation sub-spaces. The multi-heat attention in transformer uses H different sets of learned projections as
mhu V, = concat (head),..., WMA
where head; = (Ir K, W; W W?).
WALHA is the learnable matrix for the concatenated vectors from each attention head. After the multi-head self-attention, two Add Norm operations with residual layers are applied to compute the transformer's output Tows.
Tres = (T + Tm):
(3)
Tour (Ts + (LinerrrrT)TTrea))
where LN is the layer normalization and ReLU represents the ReLU activation function. Linear is linear layer. The transformer block outputs Tou with inputting Tin.
Agent Transformer Memory Network
In this section, we give detailed description of the ATM framework shown in Figure [ To handle the partial observability, ATM provides each agent with a slot-based memory to store the past key information in rf, which is updated by the working memory mechanism. To restrict the computational complexity of applying the transformer on the whole trajectories, ATM uses a fixed- capacity memory buffer M. Next, ATM embeds all the spatial entities and memory slots with relative positional embeddings as Tin and inputs them into the transformer block. Using spatio-temporal embeddings Tow outputted from transformer, ATM calculates individual Q-values or policy logits with Entity-Bound Action Layer by semantically binding unique entity embeddings in Tour to actions.
3.1 Agent Transformer Memory
Here we give the detailed design of the ATM network with the compartmentalized blot-based memory mechanism. Formally, each agent maintains its own memory buffer matrix M E Rauxdyy Each row of M is memory slot m with size dm where j E [1,2....,Mv), which stores the agent's 's memory on the past timesteps. There are fixed нм slots in M and the memory buffer is updated at each timestep.
When applying the transformer to handle the various observation space, we take advantage of the rich-entity character of the multiagent environment. As shown in Figure I left part, we divide entities in the agent observation space into three spatial entity sets: "self" Gally and o'.... The self entity of o'ayy contains agent i's individual features and is embedded as e'ye via an embedding layer. Similarly, the entities of other allied agents duty contain agent i's observation each other agents and are embedded as elaty via another embedding layer. Then the remaining entities (such as enemies environmental objects) oe embedded as e via another embedding layer. Besides feeding these spatial entities into the transformer block, to enable the agent memory, we also regard agent i's memory entities and input them into the transformer block
sequential memory slots m². !!. after being embedded into e/, via an embedding layer. With the embedded sequential entities from memory and spatial entities from the local observation, the transformer block's 's input is built as
T lé Euty e e
After the transformer block updating entity embeddings, the resulting outputting embeddings are
T TRM(TH)) le Lond Lowe Lout ally Sent e
levers
(5)
3

---

Memory Updating
Temporal Memoory
XL
Spatial Observation
Memory
Entity- Action Layer
Mill:: Fead Albetton
Self
Add Nam
Feed Forvard
Addi Nerm
('(tte) or of (+fr)
nbky
Allies
MLP
Entities
Relative Positional Embedding
Transforor Block
Tour
Figure 1: Agent Transformer Memory Network. The left part illustrates an example of an agent's local observation with the sequential memory M and spatial entities including itself o'ayy allies oally and enemies o'... The right part is the network architecture of ATM. All memory slots and observed entities are first encoded with relative sequential spatial positions (grey squares) and then embedded via a multi-layer perceptron (MLP). For memory or duplicated self entities, the grey square is one-hot ID. For allies or other entities (such as enemies), the grey square is a relative distance between the central self entity and the allies or other entities. Next, all entity embeddings denoted Tin are inputted into the transformer block. Then the outputting entity embeddings Towr used to compute the Q-values or action probabilities with the Entity-Bound Action Layer. Meanwhile, the working memory mechanism continuously upgrades the agent's 's memory with the transformer's output at each timestep. "xL" means that there are layers for the transformer block.
where 7 TRM indicates the standard transformer block of Eq. I. and (3. Then each agent's 's Tow is used update its memory buffer M and compute the state-action value or policy.
As the transformer processes these embedded memory and spatial entities as a set, the spatial or sequential order information of entities would be lost if do not encode it into the entity features. Therefore, we treat the agent's self entity as the central entity and encode the relative positional embeddings into all other entities related with it. For the sequential memory entities, we perform the one-hot embedding M = [] where is the identity matrix to indicate the passing timesteps compared with the current timestep 1. For the spatial entities, we encode the relative distances (i.e.. the x- x-coordinate distance and y-coordinate distance) with the central self entity into the entity features. Next, we elaborate the working memory updating schema with Tour of each agent.
3.2 Working Memory Updating Schema
To facilitate agents to hold past information, we imitate the working memory process in cognitive science, which describes cognitive system with limited capacity that can hold information temporarily [21 [22]. Holding information in mind and manipulating it, such a theoretical concept has been utilized in single-agent deep reinforcement learning to alleviate the partial observability problem and has shown to surprisingly increase the reasoning ability of agents [17,23) [3). In detail, working memory drops the oldest memory slot m, and add the newest memory slot updated from the transformer block's output. The new memory M,41 is updated as
m = tanh(h, WM bb))
(6)
M [m:+:: M, -]]]
where h, = Tou [0:] is the self entity embedding outputted by the transformer and fanh is the tanh activation. Wa is weight matrix and by bias vector. The working memory mechanism behaves first-in-firsee fir queue manage the agent's memory of the past timesteps.

---

There are two benefits of ATM compared with RNN-based networks. First, ATM effectively utilizes the factorized structure existing the agent observation space and employs the powerful transformer block to handle these various meaningful entities to extract internal concurrent relations such as coordination of teammates. Second, ATM's multi-slot memory mechanism could provide the short- cut recurrence to focus on longer sequential information, replacing gated RNN's single path of information flow with a network of shorter self-attention paths [17.
3.3 Parsing Action by Entity-Bound Action Layer
After ATM outputting Towr we need to calculate individual state-action values O or the action probabilitiess of the agent's 's policy x'. One common way is using linear layers map the self entity embedding h = Tour [0:] to all action nodes' values [17 (similar for the policy probability logits) of the output layer such as
e' Q'=
However, mapping the self entity to all actions does not effectively utilize the aggregated spatio- temporal embeddings from transformer as different actions may require different information. When designing the action space, each action is given its own semantic meaning to interact with different involved entities the environment. Such an inductive bias of actions naturally requires to bind the involved relevant entity embedding from Tow the semantically corresponding actions such that
='= Q'aaaa)) a),
(8)
where ha E Tour and Ra, + ha, if #C
To realize Eq. (8), we propose the Entity-Bound Action Layer to uniquely bind the outputting entity embeddings T with actions. For each action, we map the most relevant entity embedding to by parsing the action (e.g- mapping the enemy entity embedding in Tow to the corresponding attacking action in A). Then we use the bound entity embedding to compute each action's 's state-action value or probability logit via linear layers. When computing. all used entity embeddings ha E Tow share the same linear layers. If one entity maps to more than one action, we replicate this entity with one-hot ID in Ti to make sure that each action has one unique outputting entity embedding in Towr Figure I shows an example of applying the Entity-Bound Action Layer on four moving actions related to the agent itself. The corresponding self entity in Ti is duplicated four times with ID make sure that each moving action could receive its unique entity embedding from Tou Then the uniquely bound entity embeddings in Tow used calculate the final state-action values or probabilities via the shared linear layers. Using the unique entity embeddings to calculate the corresponding state-action values or probabilities allows each action to flexibly focus the different inputting entities such as the memory slots or the allies via self-attention, thus improving the computational flexibility compared with using the same outputting embedding = Tour (0:] compute all actions' values.
The Entity-Bound Action Layer is based on ATM's factorized observation space and the compart- mentalized memory slots, which not be trivially implemented with RNNs concatenated observations. For the working memory updating schema in Section 3.2. if the self entity et" is duplicated f times in Tin to map different actions, the new memory slot is computed by averaging all the output embeddings originated from the self entity. Then the memory is updated
h mean (Tur 10 Bleerl))
m 1 tanh(h, WM + by). M = [m++;:M[[
(9)
4 Experiment
4.1 StarCraft Multi-Agent Challenge
In this section, we evaluate our method in the StarCraft II decentralized micromanagement tasks and use StarCraft Multi-Agent Challenge (SMAC) environment [I]] as our testbed, which has become a commonly-use -uee benchmark for evaluating state-of-the-a MARL approaches. At the beginning of each episode, the enemy units are going to attack the allies. We train multiple agents to control
"Code is available at /tt.pp/s///ttttttooo//o/o/t////t wwt./////OOOAA///erI1111111111
wwt
5

---

4m 5m
5m vs 6m
0.7
atm--o/ix
groegplex
0.0
gru apiix
Test Win Rata
gra-ggmix
Win
0.4
0.2
0.0
0.2
0.0
0.4
Stops
Steep
(a) 4m, 55
(b) 5m, 6in
6h vs 8z
corridor
stm-splex
atm--gpmi
gra-qplex
gru-qplex
Test Win Rate
gru-qmax
Test Win Rate
gmn
0.4
0.3
0.2
1.0 SSteps
Staps
(c) 6h_vs__8 8z
(d) corridor
Figure 2: Results different StarCraft Multi-Agent Challenge scenarios.
allied units respectively to beat the enemy, while built-in handcrafted Al controls the enemy units. Training and evaluation schedules such as the testing episode number and training hyper-parameters are kept unchanged. The version of StarCraft II is 4.6.2. We build ATM based on the pymarl [3] and ATM is used as the agent network when plugged into MARL algorithms.
First, describe the features of each entity type. In the SMAC scenarios, the exact observation of agent is set of entities e where E {o {osely Jenmyy } and d(e, < d'ight while d(e, is the distance between the entity and agent and d'ighr is agent's sight range. The self entity d'etr contains each agent's own health feature, movable features, and the possible type and shield features. Each ally entity dally contains features such as the health, the distance, relative x coordinate, and relative coordinate between this ally and the agent itself. The other entities here are the enemies, which have similar features as the ally entities. The inca-innformtiinn such as the agent id and last action is added into the agent's 's self entity. Agents cannot share information such as their first-person observations among themselves as we assume there is no communication. At the same time, if one entity is in both agent and agent j's sight ranges, then agents and agent j's own local observations include the same seen entity. The discrete set of actions that agents are allowed to take consists of move(direction), attack[enemy id], stop and no-op. Here we assign actions of move oove(direction,, stop and no-op the self entity while actions of attack[enemy id] to corresponding enemy entities. For the memory part, we use three memory slots and four attention heads with each head having 16 hidden dimensions. Then the embedding layers are of 64 hidden dimensions. For mapping the entity embeddings in Tour to state-action values, we use one linear layer shared by all actions' bound entity embeddings and set the output dimension to More details are provided the Appendix.
We test ATM with QMIX (30] and QPLEX (35, two of the most representative algorithms on SMAC. We compare ATM with GRU, which is widely adopted in MARL algorithms in pymarl [3]] and is validated to obtain consistent advantages over MLP in SMAC (30 We perform the experiments on four maps: 4m _s______ 2m, 5m_vs vs_ 3m, 6h vs 8z, and corridor. The last two maps are almost the most difficult maps where previous methods fail to learn [II] Results are averaged over 6 independent training runs with different random seeds and the resulting plots include the median performance as well as the 25-75% percentiles. As shown in Figure we can see that ATM consistently improves the performance of QMIX and QPLEX on these maps. Surprisingly, ATM makes the breakthrough on 6h vs. 8z and corridor where GRU-based QMIX and QPLEX fail. The learned strategies with ATM

---

exhibit the delicate micro-mann moccroomaipplttion. For example, in corridor, agents stand in circle formation maximize their attacking damage while avoiding being attacked from behind.
4.1.1
Ablation Study of ATM
No king memory
ne omal memory
Test Win Rata
0.8
tm
Wio mem
0.8
gF+
amr!
W/ mcn
1000
Test Witn
0.00
0.4
0.4
0.00
00000
88.00
200G
19:00
19:00
00000
D000
0.0
0.0
00:00
00.00
2000
5000
4m V5_ 5m
5mLvs, 6m
4mLys, 5m
5m, vs, 6m
(a) ATM components.
(b) Memory updating schomes.
Figure 3: Ablation Study of ATM.
We also conducted the ablation study of each part of ATM, which is shown in Figure 3(a) All the results are averaged over 6 independent training runs with different random seeds and show the median test win rate. The 'woo mem' is ATM without any memory on QMIX by setting the memory slot number O. The 'w/o EBA' is ATM without Entity-Bound Action Layer on QMIX, which instead uses linear layers to map the self entity embedding to all action nodes' values. The 'trm' means that ATM degenerates to the basic transformer without memory Entity-Bound Action Layer on QMIX. We could see that, on this map, the memory contributes a lot to the algorithm performance while the Entity-Booud
Action Layer is also essential technique for improving the performance. without the two key parts of ATM, the model degenerates to the basic transformer and gets the lowest test win rate, which validates the effectiveness of each component ATM.
Meanwhile, we study the effect of the memory updating schema. Besides the working memory, we introduce another two state-of-the-e of-the-art memory updating schemes from the single-agent RI. (details in Section A.1.2). The first one is relational memory [32). which is first proposed for tasks that require complex sequential reasoning ability. The most important idea of relational memory to allow memory slots to interact with each other to perform complex relational reasoning with the information they remember. The second one AMRL 4 which uses standard memory module to summarize short-term context, and then aggregate all prior states from the standard model without respect to order to provide advantages both in terms of gradient decay and aggal-to-noiss. ratio over time. Figure 3(b) shows the results of these memory updating schemes. The working memory performs better than others while memory updating methods substantially exceed the ATM without memory.
4m 5m
5m 6m
Episodle Raturrs
Episodic
siet
3
5961
sitt
0.0
0.2
0.8
0.6
0.8
1.5
1.2
3.4
1.5
1.0
2.8
0.0
0.2
0.1
0.6
0.8
3.0
1.2
1.A
3.6
1.0
2.0
Steps
14 Stopps
(a) Memory slot number on 4ot_vs 5m.
(b) Memory slot number on v5m6.
Figure Memory slot number study.
4.1.2 Memory Slot Number
Next, we study the impacts the memory slot number. We test ATM with memory slot numbers ranging from 0 to 5. The results are shown in Figure We could that when the number of

---

memory slots is larger than O which means with memory, ATM could obtain similar performance and achieves best when the slot number is 3. Indeed, if the number of the memory slots increases to the length of the past timesteps, ATM becomes transformer over the whole sequential observation trajectories. But Figure 4 indicates that truncated observation trajectories such as timesteps are enough while the computational complexity of the transformer increases quadratically with the length of the historical trajectory. We set slot number at 3 for all tasks in both SMAC and LBF.
4.1.3 Attention Illustration of ATM
Memory Heatmap
Attack Heatmap
Move Attention Heatmap
Agre1
Agesss
Agess
Agesc
Agait
Agent
Agon13
Ageit
Agerc
Apents
Rome
Agons 2
Agme
(a) Memory Attention
(b) Attack Action Attention
(c) Move Action Attention
Figure 5: Attention heatmaps, 's' is agent itself, 'c' are enemies, 'a' are allies, and 'm' is memory.
ATM utilizes the structure information in multiagent environment with self-attention, then the attention weights of entities could be illustrated to indicate the inner dynamics of ATM. Here we show how the new memory is generated by attending over entities and how the bound entities of actions attends over entities. As shown in Figure 5(a) the new generated memory at each timestep absorbs the enemy entities and ally entities most as the agent need to remember them. On the contrary, the self entity is less attended to be absorbed into memory as it is never lost in the sight range. Figure ((b) indicates that the attack actions attend over the self, ally and enemy entities at the same time, which is consistent with the key strategy of focused fire. Meanwhile, Figure ] attend most on enemies and allies, which means that the move actions also needs coordination. Besides, we also provide a more detailed object-orienter translation of the agent's decision process with entities in Appendix C
4.2 Level-Based Foraging
We further test ATM on the classical Level-Based Foraging (LBF) tasks [. where agents collect foods scattered randomly in grid world. Agents and foods are assigned levels, such that a group of one or more agents can collect the food if the sum of their levels is greater or equal to the food's level. Agents can move in four directions, and have an action that attempts to load an adjacent food (the action will succeed depending on the levels of agents attempting to load the particular food). When one or more agents load food, the food level is rewarded to the agents by the agent level. We test ATM on three distinct partially observable LBF tasks with variable agents and foods. For example, "15x15_ 5_3p__ff" means that the grid has a size 15x15 and there are agents and foods on the grid. The agent's 's observation space includes self entity, ally entities and food entities. We test on three challenging maps: "15x15_3 5____5ff" "15x15_44 35x15__4_550 4p_"5"" and "15x15_4pp __4p_6f"" and set the sight range of agents at 7 to introduce partial observability while ensuring that agents observe some object entities.
For the configuration of ATM, the memory slot number is and 4 attention heads are used with each head having 16 hidden dimensions. The embedding layers for the agent, allies, and food entities are layer linear networks of 64 hidden dimensions. The action space of LBF [None, North, South, West, East, Load] and we duplicate the self entity to map each action. To map the action entities in Tour to action probability logits, we use a one-layer linear network shared by all action entities and set the hidden dimension to 1. Other configurations are kept the same as the best parameters reported in the epymarl (27] after a grid search and could be referred to in Appendix. As MAPPO and MAA2C exhibit excellent performance on almost all tasks, we plug ATM into MAPPO and MAA2C to test ATM on the two on-policy algorithms while comparing with GaU-based and MLP-based ones, All the results are averaged over
such as training and testing
8

---

5 independent runs with different random seeds for 5 million steps and the resulting plots include the median episodic return plus 25-75% percentiles while the highest episodic return is normalized 1.
15x15 3p_5f
15x15 4p_5f Sf
15x15.4 4p. 6f
Retures
Epissdde
022
6.6 05 1.0
15
3,0
6.0 Staps
255
25
25
45
5.8
0.
65
10
1.5
24
25
2.0
25
4.5 45
5.0
0.0
0.5
2.0
15
2.0
2:5
3.5
2.5
4.0
45
5.0
Stops
Stops
(a) 15x15-3p-51
(b) 15x15-4p-51
(c) 15x15-4p-6f
Figure 6: Results different Level-Baset Foraging scenarios with sight range.
Figure shows the results of the LBF tasks. We could see that, with the help of ATM, the learning speeds of MAPPO and MAA2C are improved by a large margin compared with GRU and MLP. Notably, GRU-based MAPPO and MLP-based MAPPO fail on the "15x15_4p 6f". 4_____46""______ However, with ATM, MAPPO easily learns from scratch. Meanwhile, ATM-MAA2C exhibits the best performance in all the scenarios. In the LBF environment where agents have a partial sight window, ATM exhibits superior learning speed and final performance when plugged into MAPPO and MAA2C.
5 Related Work
One of the most relevant classes of work is the single-agent RL algorithms with recurrent memory to solve the partially observable Markov decision process (POMDP). For deep reinforcement learning (DRL), DRQN 19) and RDPG [T] are the first two works introduce RNNs into DRL provide memory for agents. R2D2 [3] then studies the representational drift and recurrent state staleness phenomena while using LSTM in RL and proposes a "burn-in" updating method to better restore the initial state. At the same time, the attention mechanism is exploited to help update the memory. FRMQN 24 utilizes the key-value memory perform the writing operation and the soft attention to perform reading operation with the spatio-temporal context vector. As an adapted version of FRMQN, DCRAC 23 maintains the separate external value and key memory blocks and uses the hidden state from the LSTM and query to aggregate relevant information from memory. Furthermore, RMC [32] employs the multi-head self-attention to let each memory attend over all of the other memories, and update its content based on the attended information to perform sequential relational reasoning. Next, AMRL uses LSTM to summarize short-term context, and then aggregate all prior states over time from LSTM to provide advantages both in gradient decay and signal-to-nowe ratio.
Some works are also explored to build agent memory with transformer. For example, WMG [17 uses the transformer to reason over dynamic set of vectors representing observed and recurrent states in single-agent DRL. At the same time, GTrXL 22 found that the standard transformer architecture is difficult to optimize especially pronounced with RL objectives. Then they propose the modified GTrXL with a gating technique that substantially improves the stability and learning speed of the original transformer. Following GTrXL, CoBERL 3 and HCAM [14] use integrated or hierarchical memory structure to help organize the internal state with more than 10 millions of parameters. Besides online RI., transformers are also investigated in offline RL by treating RL. as sequence modeling problem such as Decision Transformer (6] and Trajectory Transformer [12.
On the other side, transformers are introduced into MARL for offline and transfer learning. Based on Decision Transformer, Multiagent Decision Transformer (20) studies the paradigm of offline pre-training with online fine-tuning in MARL on the SMAC platform. UPDeT [1] utilizes a transformer- -based model to enable multiple tasks transferring in MARL through the transformer's strong generalization abilities. Unlike previous works in MARL using the transformer, here we focus on designing a transformer- based memory structure for agents with local observations to boost online MARL algorithms in the challenging partially observable environments. There are also some efforts focusing on the partial observability in the board multiagent domain. For instance, Subramanian et al. [8] extend the mean-filed method into the partially observable settings specially for the large-

---

scale multiagent system by maintaining a distribution over the mean action parameter. In [15]. the SPARTA Monte Carlo search procedure is proposed improve the agreed-upor policy for Hanabi, cooperative partially observable card game. Besides, model-based IPOMDP-net is designed for multi-agent planning under partial observability with an interactive belief update mechanism [9]
Conclusion
In this work, we propose the Agent Transformer Memory network to handle partial observability in MARL by specially considering the multiagent system characters. First, the transformer uniformly aggregates the agent's factorized observed entities and sequential memory slots to generate new internal memory and updates memory with the working memory updating schema. Second, we utilize Entity-Bound Action Layer to uniquely bind predefined entity with corresponding actions to compute the state-action values or probability logits. Experiments demonstrate ATM's excellent ability to speed up the learning and improve the performance of MARL algorithms on various tasks.
For future works, on the one hand. it is interesting to extend the idea of the Entity-Boune -Bonnd Action Layer learn how to automatically map the ATM's outputting entity embeddings to the unique actions which are difficult to be configured manually or by simple heuristic rules. On the other hand, is promising to introduce more advanced natural language processing techniques into the MARL framework to boost the performance of MARL algorithms by utilizing the rich semantic meanings which naturally exist in MARL environments.
ACKNOWLEDGMENTS
The work was supported by National Key R&D Program of China (2022FF00200000) National Natural Science Foundation of China (Project No. 62006219), Natural Science Foundation of Guangdong Province (2022A151501157)) 1579), and Hong Kong Innovation and Technology Fund under Project No. ITS/170/20 and Project No. GHP/080/20SZ.
References
[1] Stefano V. Albrecht and Subramanian Ramamoorthy, A Game-Theoretic Model and Best- Response Learning Method for Ad Hoc Coordination in Multiagent Systems. In Proceedings of the 2013 International Conference on Autonomous Agents and Multi-Agent Systems, AAMAS '13, pages 1155-1156, Richland, SC, 2013. International Foundation for Autonomous Agents and Multiagent Systems.
[2] Bram Bakker. Reinforcement Learning with Long Short-Term Memory. In Advances in Neural Information Processing Systems, volume 14. MIT Press, 2002.
[3] Andrea Banino, Adrià Puidomenech Badia, Jacob Walker, Tim Scholtes, Jovana Mitro- vic, and Charles Blundell. CoBERL: Contrastive BERT for Reinforcement Learning. In arXiv:2107.05331 [cs], July 2021.
[4] Jacob Beck, Kamil Ciosek, Sam Devlin, Sebastian Tschiatschek, Cheng Zhang, and Katja Hofmann. Amrl: Aggregated memory for reinforcement learning, In International Conference on Learning Representations, 2020.
[5] Yongcan Cao, Wenwu Yu, Wei Ren. and Guanrong Chen. An overview of recent progress in the study of distributed multi-agent coordination. IEEE Trunsactions on Industrial Informatics, 9(1 91)):427-438 2012.
[6] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision Transformer: Reinforcement Learning via Sequence Modeling. In Advances Neural Information Processing Systems, 2021.
[7] Farnaz Derakhshan and Shamim Yousefi. A review on the applications of multiagent sys- tems in wireless sensor networks. International Journal of Distributed Sensor Networks, 15(5):5500147771995077 May 2019.
10

---

[8] Sriram Ganapathi Subramanian, Matthew E. Taylor, Mark Crowley, and Pascal Poupart. Partially Observable Mean Field Reinforcement Learning. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '21, pages 537-545, Richland, SC, 2021. International Foundation for Autonomous Agents and Multiagent Systems.
[9] Yanlin Han and Piotr Gmytrasiewicz. IPOMDP-Ner A Deep Neural Network for Partially Observable Multi-Agent Planning Using Interactive POMDPs. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pages 6062-6069, July 2019.
[10] Nicolas Heess, Jonathan J. Hunt, Timothy P. Lillicrap, and David Silver. Memory-based control with recurrent neural networks. In arXiv: 1512.04455 [cs], December 2015.
[11] Siyi Hu, Fengda Zhu, Xiaojun Chang, and Xiaodan Liang. UPDeT: Universal Multi-agent RL via Policy Decoupling with Transformers. In International Conference on Learning Representa- fions, 2021.
[12] Michael Janner, Qiyang Li, and Sergey Levine. Offline Reinforcement Learning as One Big Sequence Modeling Problem. In Advances Neural Information Processing Systems, 2021.
[13] Steven Kapturowski, Georg Ostrovski, Will Dabney, John Quan, and Remi Munos. Recurrent Experience Replay in Distributed Reinforcement Learning. In International Conference on Learning Representations, 2019.
[14] Andrew Kyle Lampinen, Stephanie C. Y. Chan, Andrea Banino, and Felix Hill. Towards mental time travel: hierarchical memory for reinforcement learning agents. In arXiv:2105.14039 [cs]. December 2021.
[15] Adam Lerer, Hengyuan Hu, Jakob Foerster, and Noam Brown. Improving Policies via Search in Cooperative Partially Observable Games. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 7187-7194, April 2020.
[16] Michael L. Littman. Markov games framework for multi-agent reinforcement learning. In Machine Learning Proceedings, pages 157-163. Elsevier, 1994.
[17] Ricky Loynd, Roland Fernandez, Asli Celikyilmaz, Adith Swaminathan, and Matthew Hausknecht. Working Memory Graphs. In Proceedings of the 37th International Confer- ence on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 6404-6414. PMLR, July 2020.
[18] Laetitia Matignon, Guillaume J. Laurent, and Nadine Le Fort-Piat. Independent reinforce- ment learners cooperative Markov games: survey regarding coordination problems. The Knowledge Engineering Review, 27(1):1-31, 2012.
[19] Matthew Hausknecht and Peter Stone. Deep Recurrent Q-Learning for Partially Observable MDPs. In AAAI Fall Symposium Series; 2015 AAAI Fall Symposium Series, 2015.
[20] Linghui Meng, Muning Wen, Yaodong Yang, Chenyang Le, Xiyun Li, Weinan Zhang, Ying Wen, Haifeng Zhang, Jun Wang, and Bo Xu. Offline Pre-trained Multi-Agent Decision Transformer: One Big Sequence Model Tackles All SMAC Tasks. In arXiv:21
[cs], December 2021.
[21] George A. Miller. The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63(2):81-7, March 1956.
[22] Akira Miyake and Priti Shah, editors. Models of Working Memory: Mechanisms of Active Maintenance and Executive Control. Cambridge University Press, edition, April 1999.
[23] Xiaodong Nian, Athirai A. Irissappane, and Diederik Roijers. DCRAC:
Deep Conditioned Recurrent Actor- Critic for Multi-Objective Partially Observable Environments. Proceedings of the 19th International Conference on AutoRomous Agents and MultiAgent Systems, AAMAS '20, pages 931-938, Richland, SC, 2020. International Foundation for Autonomous Agents and Multiagent Systems.
11

---

[24] Junhyuk Oh, Valliappa Chockalingam, Satinder Singh, and Honglak Lee. Control of Memory. Active Perception, and Action in Minecraft. In Proceedings of the 33rd International Conference on International Conference on Machine Learning- Volume 48, ICMI. 16, pages 2790-2799. .org 2016.
[25] Frans A. Olichock and Christopher Amato. Concise Introduction to Decentralized POMDPs. SpringerBriefs in Intelligent Systems. Springer International Publishing, Cham, 2016.
[26] Shayegan Omidshafiei, Jason Pazis, Christopher Amato, Jonathan P. How, and John Vian. Deep Decentralized Multi-Tass Multi- Reinforcement Learning under Partial Observability. In Proceedings of the 34th International Conference Machine Learning Volume 70, ICML'17, pages 2681-2690. JMLR.org, 2017.
[27] Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V. Albrecht. Bench- marking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks. In Proceedings of the 35th Conference on Neural Information Processing Systems (NeurlPS 2021) Track on Datasets and Benchmarks, November 2021.
[28] Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayaku- mar, Max Jaderberg, Raphaël Lopez Kaufman, Aidan Clark, Seb Noury, Matthew Botvinick, Nicolas Heess, and Raia Hadsell. Stabilizing Transformers for Reinforcement Learning. In Pro- ceedings of the 37th International Conference on Machine Learning. volume 119 of Proceedings of Machine Learning Research, pages 7487-7498. PMLR, July 2020.
[29] Gauthier Picard. Designing Marketplace for the Trading and Distribution of Energy in the Smart Grid. In Proceedings of the 14th International Conference on Autonomous Agents and Multiagent Systems, pages 1293, 2015.
[30] Tabish Rashid, Mikayel Samvelyan, Christian Schröder de Witt, Gregory Farquhar, Jakob N. Foerster, and Shimon Whiteson. QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning. In Proceedings of the 35th International Conference on Machine Learning, pages 4292-422252 4301, 2018.
[31] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung. Philip H. S. Torr, Jakob Foerster, and Shimon Whiteson. The StarCraft Multi-Agent Challenge. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '19, pages 2186-2185 Richland, SC, 2019. International Foundation for Autonomous Agents and Multiagent Systems.
[32] Adam Santoro, Ryan Faulkner, David Raposo, Jack Rae, Mike Chrzanowski, Théophane Weber, Daan Wierstra, Oriol Vinyals, Razvan Pascanu, and Timothy Lillicrap. Relational Recurrent Neural Networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, pages 7310-7321, Red Hook, NY, USA, 2018
[33] Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to Sequence Learning with Neural Net- works. In Proceedings of the 27th International Conference on Neural Information Processing Systems Volume 2, NIPS' 14, pages 3104-33 3112, Cambridge, MA, USA, 2014. MIT Press.
[34] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you Need. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 1998-6000 Curran Associates, Inc., 2017.
[35] Jianhao Wang. Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent leeaanning In International Conference Learning Representations, 2021.
[36] Weixun Wang, Tianpei Yang. Yong Liu, Jianye Hao, Xiaotian Hao, Yujing Hu, Yingfeng Chen, Changjie Fan, and Yang Gao. Action Semanties Network: Considering the Effects of Actions in Multiagent Systems. In International Conference Learning Representations, 2020.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] claim that we propose a transformer-based memory mechanism for MARL with considering the multiagent characters of the factored multiagent observation and the rich-interaction action space.
(b) Did you describe the limitations of your work? [Yes] We provide the limitations of our work in Appendix
(c) Did you discuss any potential negative societal impacts of your work? [Yes] The discussion is in Appendix D
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as URL)? [No] We provide the environment details and method configurations such the network structure to reproduce our results. We implement our method and compare with baselines using open-sourced pymarl and epymarl framework and will release code once it is accepted.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] We give the details of the environments and the network structure configurations both in the Section and the Appendix.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] In both the Figure 2 and the Figure 6 we show the error bars of the learning curves.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We cite SMAC [3], pymarl [31] and epymarl (27), etc.
(b) Did you mention the license of the assets? [Yes] We mention the licenses of SMAC, pymarl and epymarl in the Appendix.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'l re using/curating? [Yes] We give the URLs of SMAC, pymarl and epymarl in the Appendix.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "pi8bo1VM453", "reviewer": "Reviewer_mbJQ", "summary": "This paper proposes a new Multiagent Reinforcement Learning (MARL) method using Agent Transformer Memory (ATM). ATM consists of 4 parts; memory, self, allies and entities. Especially, for self, it duplicates the token to use it for Entity-Bound Action (EBA) layer which affects the performance clearly. For memory, ATM uses working memory concept, and other memory types are also evaluated as an ablation study. ATM-based MARL agent outperforms other methods for StarCraft Multi-Agent Challenge (SMAC) and its architecture (EBA and working memory) shows better performance than alternatives (e.g., relational memory or without EBA).\n\n=================================\n\nWhat happening if they apply appending style memory to their model is interesting and they didn't show that, but except that, they handled lots of concerns through author-reviewer discussion phase. I sustain my score.", "questions": "- Could you share the plots in Figure 2 and 6 by comparing in wall-clock time? I think ATM should be slower than GRU, but want to know how much slower. If ATM is not slower than GRU, then it is interesting also.\n- Did you try to apply appending style memory or TrXL style memory? Those one can be computationally expensive, but useful from the perspective that it can support to attend the past explicitly.", "limitations": "I think that one of the limitations of ATM is computationally more expensive than recurrent module. I don't want to say \"so we don't use ATM\". I want to discuss the comparison in the aspect of computation or time to get the action from the agents. Even though ATM is more expensive, sometimes we must pick ATM to solve some tasks requiring to infer huge interactions. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- It proposes a new memory module, ATM which can infer the interaction between the agent, allies, entities and memory.\n- It evaluates for SMAC, ATM-based agent outperforms GRU-based one which is usually used.\n- It reports the ablation studies for EBA and memory types and the number of memory slots.\n\nWeaknesses:\n- As shown in [1], the agent with Transformer requires more computations, even though it can benefit for sample efficiency [1,2] or interactions between entities like this paper. For example, in Figure 2 (a) and (b), ATM-based models outperform GRU-based models in steps, but when comparing in wall-clock time, ATM-based model could be much slower than recurrent module-based models.\n- The memory following the working memory mechanism is interesting, but it cannot attend the past explicitly. Other transformer-based agents [2,3] get the advantages from attending the past directly, while it cannot do that through ATM architecture.\n\n[1] Parisotto, Emilio, and Ruslan Salakhutdinov. \"Efficient transformers in reinforcement learning using actor-learner distillation.\" arXiv preprint arXiv:2104.01655 (2021).\n\n[2] Parisotto, Emilio, et al. \"Stabilizing transformers for reinforcement learning.\" International conference on machine learning. PMLR, 2020.\n\n[3] Chen, Chang, et al. \"Transdreamer: Reinforcement learning with transformer world models.\" arXiv preprint arXiv:2202.09481 (2022).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "pTm1O8s_k4h", "reviewer": "Reviewer_KrWT", "summary": "This paper presents an approach for modeling partially-observable multi-agent MDPs. The paper proposes two modifications on top of a general transformer for modeling the relationship between observation and action: first a memory module that maintains information from previous observations associated with all agents, and second a prediction layer that filters information for predicting actions by the arguments of that action (e.g., predicting to attack enemy i is conditioned on the observation information about enemy i). They evaluate on two MARL environments and show improvements over existing methodologies for modeling memory over time, including RNN-based methods.", "questions": "* What would alternatives to the entity-bound action layer be? Using only memory of o_self to predict actions, or using some combination (e.g., concatenation) of all observations (which has a drawback of many parameters, some of which may be unnecessary)? I'm a bit confused what it means to ablate EBA in the experiments, and whether you experimented with all three settings (EBA, self-only, and all observations).\n* Did you perform any interpretation of the memory? E.g., if memory/observation can be interpreted as an image.", "limitations": "The entity-bound action layer seems specific to the action spaces available in the two environments. What if the action space is much lower-level, such that there aren't semantic relationships between a particular action and a particular entity in the environment (e.g., instead of \"attack enemy i\", an agent may have to move to a strategic position and rotation with multiple navigation actions, then \"attack\" by firing some weapon in the direction it is pointed)? Would the entity-bound action layer still generalize to this?", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThis is an interesting problem and an interesting approach to the problem. The paper is relatively clear and experiments on both environments are mostly comprehensive and performance over compared methods is impressive.\n\n--- \n\nWeaknesses:\n\nThe way that transformer-based working memory is introduced in 42-48 makes it seem as though the main contribution is applying an existing method to an existing problem. I'm not familiar enough with related work to make this judgment, however.\n\nI was confused about what an observation includes: is o_ally the observation *of* an ally, or the ally's observation? If the latter, does this also hold for enemies, and if so, it seems as though an agent shouldn't have access to the enemy's observation. If the former, is there information sharing between agents (i.e., sharing their first-person observations, which may not overlap with another ally's)? How would this affect performance? If would help to have a formal definition of the task, including what exactly observations look like (are they images?). ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "P4Tr0D--pFK", "reviewer": "Reviewer_9Aou", "summary": "This paper is about designing a new architecture, Agent Transformer Memory (ATM), for multiagent reinforcement learning where each agent is equipped with its own working memory and action space. Specifically, the authors propose to use a transformer-based architecture to process both the moving entities in the environment and the recent memory slots. The output of the transformer consists of an updated memory slot (to be pushed to the limited-capacity working memory) and the encoded representations for each entity. Then, each agent selects an action (from their respective action space) based on their most recent memory slot and the representations of their surrounding entities. Experiments were conducted on the well-established StarCraft Multi-Agent Challenge (SMAC) and Level-Based Foraging (LBF) environments. The authors test the proposed agent transformer memory with QMIX and QPLEX algorithms commonly used on SMAC, and with MAPPO and MAA2C for experiments on LBF. Empirically, it was shown that agents with memory and personalized action space perform better. The authors also investigate different memory updating schemes and show that using ATM works best.", "questions": "- Based on Eq. 4 and 5, should $T_{in}$ and $T_{out}$ be indexed by $i$, i.e., $T_{in}^i$ and $T_{out}^i$?\n- In Figure 5, why is Agent 5 heat map incomplete? Was it taken out of action around time step 20?\n- From Figure 5, it looks like the memory is not very useful to attend to after the first few couples of steps. Why is that?\n- Did the authors try making the trained agents with different architectures play against each other (instead of against the built-in AI)?", "limitations": "The authors discussed the limitations and social impacts in the Appendix. Notably, how the learning process requires some exploration that could lead to unsafe situations for both the agents and humans, and also how the Entity-Bound Action Layer requires expert knowledge to manually configure it.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**What I like about this paper**\n- Each agent having their own point-of-view of the environment, own working memory, and own action space, while using the same model (same weights, different input $T_{in}$).\n- The ablation study of the proposed architecture ATM. It does seem like having a working memory is important (even if only one slot).\n\n**Potential weaknesses**\n - Lack of coordination between the agents. If I understood correctly, all agents predict their next action at the same time. That said, it seems to be a common approach in the MARL literature.\n - Even though this work takes place in a partially observable setting, the proposed technique requires to know in advance how many entities (and their attributes) there will be in the game. Since the proposed architecture requires the relative spatial positions between the self entity and all other entities (even if they are out of the field of view?), it is not clear to me what remains \"partially observable\".\n - It is not clear how the proposed model performs when playing against other learning agents.\n\n**Originality, quality, clarity, and significance**\n\nAs pointed out in the related work, the concept of having a transformer-based working memory is not new. The paper would benefit from stating clearly how the proposed technique differs from each related work. The main thing I could see is the application of a transformer encoder in the multi-agent RL setting with agent-centric observations and working memory. It wasn't clear to me how the Entity-Bound Action Layer related to any previous work (if any). I found the paper technically sound and the ablation study is backing up the proposed agent transformer memory component.\n\nI found the paper well-written and well-organized for the most part. In Figure 1, it is not clear what the gray squares represent, is it the spatial/temporal embeddings, or the entity/memory IDs? While it is unclear how the proposed model compares to the actual state-of-the-art on tested environments, the empirical results suggest the ATM architecture would be useful to others in the community.\n\nOverall, I tend to recommend this paper for acceptance.\n\n**Minor**\n- p.2 (l.83): should $d_o$ be $d_e$, if not what is $d_o$?\n- p.5 (l.164): \"for four times\" -> \"four times\"\n- p.7 (l.243): \"...agent need remember them.\" -> \"..agent needs to ...\"\n- p.7 (l.244): \"...sight field\" -> \"..sight range\"\n- p.7 (l.246): \"...focusing fire\" -> \"..focused fire\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept