Offline Communication Learning with Multi-source Datasets
Authors: Yihuan Mao
Rui Hu
Lulu Zheng
Jianhao Wang
Chongjie Zhang
Venue: Submitted to ICLR 2023 (2023)

Abstract: Scalability and partial observability are two major challenges faced by multi-agent reinforcement learning. Recently researchers propose offline MARL algorithms to improve scalability by reducing online exploration cost, while the problem of partial observability is often ignored in the offline MARL setting. Communication is a promising approach to alleviate the miscoordination caused by partially observability, thus in this paper we focus on offline communication learning where agents learn from an fixed dataset. We find out that learning communications in an end-to-end manner from a given offline dateset without communication information is intractable, since the correct communication protocol space is too sparse compared with the exponentially growing joint state-action space when the number of agents increases. Besides, unlike offline policy learning which can be guided by reward signals, offline communication learning is struggling since communication messages implicitly impact the reward. Moreover, in real-world applications, offline MARL datasets are often collected from multi-source, leaving offline MARL communication learning more challenging. Therefore, we present a new benchmark which contains a diverse set of challenging offline MARL communication tasks with single/multi-source datasets, and propose a novel Multi-Head structure for Communication Imitation learning (MHCI) algorithm that automatically adapts to the distribution of the dataset. Empirical result shows the effectiveness of our method on various tasks of the new offline communication learning benchmark.

Under review as a conference paper at ICLR 2023
OFFLINE COMMUNICATION LEARNING WITH MULTI- SOURCE DATASETS
Anonymous authors Paper under double-blind review
ABSTRACT
Scalability and partial observability are two major challenges faced by multi-agent reinforcement learning. Recently researchers propose offline MARL algorithms to improve scalability by reducing online exploration cost, while the problem of partial observability is often ignored in the offline MARL setting. Communication is a promising approach to alleviate the miscoordination caused by partially ob- servability, thus this paper we focus on offline communication learning where agents learn from an fixed dataset. We find out that learning communications in an end-to-end manner from a given offline dateset without communication in- formation is intractable, since the correct communication protocol space is too sparse compared with the exponentially growing joint state-action space when the number of agents increases. Besides, unlike offline policy learning which can be guided by reward signals, offline communication learning is struggling since communication messages implicitly impact the reward. Moreover, in real-world applications, offline MARL datasets are often collected from multi-source, leaving offline MARL communication learning more challenging. Therefore, we present a new benchmark which contains a diverse set of challenging offline MARI. com- munication tasks with singlc/multtisource datasets, and propose novel Multi- Head structure for Communication Imitation learning (MHCI) algorithm that au- tomatically adapts to the distribution of the dataset. Empirical result shows the effectiveness of our method on various tasks of the new offline communication learning benchmark.
INTRODUCTION
Cooperative multi-agent reinforcement learning is essential for many real-world tasks where multi- ple agents must coordinate to achieve joint goal. However, the problems of scalability and partial observability limit the effectiveness of online MARL algorithms. The large joint state-action space makes exploration costly especially when the number of agents increases. On the other hand, the partial observability requires communication among agents make better decisions. Plenty of previous works in MARL try to find solutions for these two challenges, with the hope to make co- operative MARL applicable to more complicated real-world tasks (Sukhbaatar al. (2016); Singh et al. 2019); Yang et al. (202I)).
Recently, emerging researches apply offline RL to cooperative multi-agent RL in order to avoid costly exploration across the joint acteonctionn space, thus the scalability is improved. Offline RL defined as learning from fixed dataset instead of online interactions. In the context of single- agent offline RI,, the main challenge is the distributional shift issue, which means that the learned policy reaches unseen state-action pairs which are not correctly estimated. By constraining the policy on the behavioral policy, offline RL has gained success diverse single-agent offline tasks like locomotion (Fu et al. (2020)) and planning without expensive online exploration(Fimn Levine (207)).
As the problem of scalability can be promisingly alleviated by utilizing offline datasets, another challenge of MARL, i.e., the partial observability, be addressed by introducing communications during coordination. Communication allows agents to share information and work as a team. A wide range of multi-agent systems benefit from effective communication. including electronic games like

---

Under review as a conference paper at ICLR 2023
StarCraft II (Rashid et al. (2018) and soccer (Huang et al. (2021) well as real-world applications, e.g., autonomous driving
Shwartz et al. (2016)) and traffic control (Das et al. (2019))
Although several offline MARL algorithms (Yang et al. 2021 Pan et al. (2022) have been pro- posed to tackle with the scalability challenge recently, how to deal with the partial observability in the offline MARL setting. has not received much attention. Unluckily, simply adopting com- munication mechanism offline MARL, i.e., learning communication in an end-to-end manner by offline datasets is still problematic. Finding effective communication protocols without any guid- ance can be the bottleneck especially when the task scale increases. It may converge to sub-optimal communication protocols that influence the downstream policy learning. To handle this problem, in this paper, we investigate the new area of offline communication learning, where the multiple agents learn communication protocols from static offline dataset containing extra communication information. We denote this kind of dataset as "communication-bann -basee dataset" to distinguish it from single-agent offline dataset. In real-world applications, communication- based dataset may be collected by variety of existing communication protocols, like handcraft protocols designed by experts, or hidden protocols leaned by other agents. Therefore, communication -based dataset can be established in offline MARL learning boost the performance of the downstream tasks.
Previous offline RL works focus on eliminating the problem of distributional shift, while offline MARL communication learning faces different challenges. Unlike policy learning which is directly guided by reward signals since actions influence the expected return, it is hard to evaluate commu- nication learning since communication serves an implicit impact between agents. What's worse, it is likely that the offline dataset is multi-source in real world, thus trajectories may be sampled by different communication protocols as well as policies. The multi-source property introduces extra challenges as we cannot simply imitate the dataset communication. Offline communication learning algorithms need to distinguish the source of each trajectories before learning from them. In this paper. We propose Multi-head Communication Imitation (MHCI) that accomplishes multi-source classification and message imitation at the same time. To our best knowledge, MHCI is the first to learn composite communication from multi-source communication- based dataset. We also provide theoretical explanation on its optimality under the dataset.
To better evaluate the effectiveness of our algorithm well as for further study, we propose an offline communication learning benchmark, including environments from previous works and ad- ditional environments that require sophisticated communication. The empirical results show that Multi-heaa Communication Imitation (MHCI) successfully combines and refines information in the communication-basel dataset, thus obtains outperformance in diverse challenging tasks of the of- fline communication learning benchmark.
Our main contributions two-folds: 1) analyze the new challenges in offline communica- tion learning, and introduce a benchmark of offline communication learning which contains diverse tasks.; 2) we propose an effective algorithm, Multi-head Communication Imitation (MHCI), which aims to address the problem of learning from single-source or multi-source datasets, and our method shows superior outperformance in various environments of our benchmark.
2 RELATED WORKS
MARL with communication Multi-agent reinforcement learning has attracted great attention in recent years. (Tampuu al. (017): Matignon et al. (2012); Mordatch & Abbeel (2017); Wen
et al. 2 In MARL, the framework of centralized traming and decentralized execution has been widely adopted (Kraemer Banerjee (2016) Lowe al. (2017)) For cooperative scenarios under this framework, COMA (Foerster et al. (2018) assigns credits to different agents based on a central- ized critic and counter-factual advantage functions, while another series of works, including VDN (Sunehag et al. (2018), QMIX (Rashid al. (2018)) and QTRAN (Son et al. (2019)), achieve this by applying value-function factorization. These MARL algorithms show remarkable empirical re- sults when tested on the popular StarCraft unit micromanagement benchmark (SMAC) (Samvelyan
et al. (2019)).
CommNet (Sukhbattar et al. (016)), RIAL and DIAL (Foerster et al. (016 are seminal works that allow agents to learn how to communicate with each other in MARL. CommNet and DIAL design the communication structure in differentiable way enable end-to-end training, and RIAL trains
2

---

Under review as a conference paper at ICLR 2023
communication with RL algorithms by constraining communication to be discrete and regarding as another kind of actions. To make communication more effective and efficient, IC3Net (Singh et al.
(2019)), ) Gated-ACML (Mao al. (0206)), and I2C (Ding et al. (2020) utilize a gate mechanism to decide when and who to communicate with. TarMAC (Das et al. (2019) and DAACMP (Mao et al. 2020aa) achieve targeted communication by introducing attention mechanism. NDQ (Wang* regularizers to minimize communication. MAIC (Yuan 2022 realizes sparse and effective communication by modeling teammates.
et al.
2020)) uses ceaemaaiiontthoo---ae
al.
Offiine MARL Offline RI. is hot topic in the last few years. It focuses on RI. given a static dataset. The problem of distributional shift critical for offline RI (Fujimoto et al. (2019) ) and typical solutions include constraining the policy (Kumar et al. (2019): Wu et al. (2020)
the Q
value (Kumar et al. (202h)) Recently, offline MARL is also attracting interests. As a combination of two areas, offline RL and MARL, it faces new challenges of larger state-action space. Yang al. (2021 proposes a even more conservative method compared to single-agent offline RL. Pan et al. 2022 finds that conservative offline RL algorithms converge to local optima in the multi-agent setting, thus proposes better optimization methods.
Multi-source fusion Works in other fields also concern about multi- source fusion. For example, Sun et al. (2021) use an attention mechainsm to fuse multiple word features for downstream tasks.
3 BACKGROUND
In a fully cooperative multi-agent RL, the environment is generally modelled as a Dec-POMDP Oliehock Amato (2016). Adding communication, Dec-POMDP with communication is defined as G << 1,S, A, P, S2, O, 7, M >. 1 21, ... , .n) stands for the set of the agents. 8 S S,a; E A,o; E Ω are the state, action and observation of a certain agent, and the observa- tion of agent o; is calculated by the observation function o O(s, i). P(s'ls. a), R(s, a) are the transition function and the reward function, and y E [0,1)
1) is the discount factor. When communica- tion is added to this model, my is denoted message conveyed from agent to agent j. In practice, we take s = o, by default. The offline dataset is denoted as DD
Based on the basic definitions of POMDP with communication, action-observatton history T; E T = (I2 x A)", and a policy n, (m.,, T,) is defined on the history. In order unify the notation of communication protocols proposed by previous works, we first give general form of communication
can further denote the
mij = C.j(os), = T, oj Tr) (Vi,j E II,
(I)
where Cij is arbitrary communication function, and is the abbreviation of all the agents. Al- though 05 is included in Tj. here we abuse the notation for convenience. In Equation each message from to j is generated from the communication function, and an action is determined through the policy function π. by taking as input all the received messages, individual observation, and indi- vidual history if needed. In Appendix A we list the expression of the communication protocols in previous works. Generally speaking, they take different inductive bias into account and can all be viewed as modified protocols from Equation
4 A MOTIVATIVE EXAMPLE
Before digging into the specific method of offline communication learning, we first illustrate why learning communication directly from communication-basee dataset is important, rather than learn- ing an end-to-end manner, i.e., learning communication from scratch. this section, we analyze the influence of the number of agents n and the dimension of states p. by comparing the performance in an imaginary communication game with tunable n, p. The number of agents corresponds to the scalability of task, and the dimension of states reflects the complexity of information sources (e.g. images are complex, while velocity and acceleration information isn't 't).
The imaginary communication game consists of agents, with state dimension p and observation dimension 9. And 2q (not q) dimensions of the state space are related to the policy of each agent in (4), therefore besides the q dimensions that can be directly observed defined in (3). the other q
3

---

Under review as a conference paper at ICLR 2023
dimensions not observed thus come from communication. The max horizon T = and each of the p-dimension states is randomly initialized from -1, 1. Details are in Appendix
= 1, 8 E (-1,1)
(2)
0 (s indea(t), ( index " index (+)
(3)
I1 undens)
aj 0
(a is the optimal action of agent i)
(4)
otherwise
Σ I(a, = a,)).
(5)
1=
Figure shows how the performance decays with the increasing n and p. general, learning communication from dataset performs better than learning communication from scratch, especially when the task becomes harder with larger n or The basic reason that, in challenging tasks where the whole state space is enourmous, pure offline MARL methods still have difficulty in finding unobserved information related to an optimal policy, even hard exploration in the state space is no longer problem. A randomly initialized communication function may converge to sub-optimal, that confuses the downstream policy to some extent. For example, in multi-agent navigation task with image inputs, the agent needs communicate whether the goal is in sight. However, there's much redundant information in the image, so the communication tends to converge to easier patterns like the color background of the goal, which are misleading downstream policy learning.
sid mm
Figure I: Evaluation accuracy with different number of agents and different dimension of the state
space p. comparing fitting the dataset communication and learning communication from scratch.
5 METHOD
In this work, we propose Multi-head Communication Imitation (MHCT) for offline communication learning as shown in Figure In Scction[5.] we provide the complete definitions of the multi-source communication in the offline setting, and derive the offline optimality according to the communi- cation the dataset. Finally in Section 5.2 the full structure of our Multi-head Communication Imitation (MHCI) is introduced, which learns composite communication protocol that supports the optimal policy according the offline dataset.
5.1 UNIVERSAL LOCAL COMMUNICATION THE OFFLINE SETTING
In order to introduce the concept of Universal and Local Communication, we first define how the multi-source dataset is split. Denote communication function C(o.) Q + Rpn. (p is the dimension of each message mij), which specifically
C((or,... ,n)) = ((2(())) (Cin(or), Cni (C,. Can (on)).
(6)
Assume that the communication in dataset D can be discriminated by source into |G] groups, with state domain and communication function
G ((S(2) (L) (S(2) (()))))))))) ((1G))

---

Under review as a conference paper at ICLR 2023
Communication Network
Policy Network
C(R)
mi
C33
Linear Fusion
m
RNN
mij
(00
(R)
m
Agent i
pe.ias
Agent
Communication Network
Policy Network
hy
mij
(122
mij
o
C33
Linear Fusion
mi.
RNN
aj
A f-1
mf
Figure 2: The pipeline of offline communication learning, breaking into parts, the communica- tion network and the policy network. Our work focuses on offline communication learning using multi-head structure shown in the communication network part. The messages generated by first computing the messages in all the heads, and then linear fusion is applied on them.
name each communication function C(,, i E [[G]] as one Local Communication (LC), There are |G' Local Communications in the dataset in total.
Definition 1. The relation that (()) dominates (2) is defined as
3f, Vj,k f (oz)) cf (oz).
(8)
Definition 2. Define the Universal Communication (UC) the communication function that con- tains all of dataset communication function.
UC + TR"",
3f1 ... fla Vi [[G]),j,k ( = C/2 (o,)
(9)
i.e.
Vi E [[G]) UC dominates (++)
Compared to the Local Communication, the Universal communication includes the information of all the Local Communications C(), (.)),.))....... c(,),,...,C, C(GG))
After introducing the concept of Universal and Local Communication, prove that the Universal Communication sufficient for obtaining policies that match or outperform the dataset in The- orem For simplicity, first denote the optimal policy and the optimal value function under Dec- POMPPOMDP (c(M) (transformed from the real MDP M by the communication function C(-)) (M)' V Vic((M)
Theorem 1. The optimal expected return based on the Universal Communication is greater than or equal to that of dataset communications and policies, i.e.
LS,,C, T) Gr, (Tve(Ma) (Sint) (ILc(MM)) (Nint)))
(10)
Theorem 1 means that the optimal policy given UC(-) is always equal or better than that of the dataset, under the dataset MDP MD as defined in (Fujmoto et al. (2019h). The whole proof is in appendix B
5

---

Under review as a conference paper at ICLR 2023
In fact, the provably true Universal Communication is at least the concatenation of all Local Com- munications, because without any additional assumptions on communication, the local communi- cations may include distinct information. As result, we actually approximating the Universal Communication in the following algorithm design.
5.2 MULTII-HEAD COMMUNICATION IMITATION
In section 5.1 we have shown that in order to guarantee optimality compared with the performance of dataset policies, combining all the information contained by dataset communication essential for the policy training afterward. To this end, we propose Multi-head Communication Imitation (MHCI) pipeline that learns communication from multi- datasets. It simultaneously optimizes the predicted category of each communication and imitates the dataset communication. The learned multi-head communication can be viewed as Local Communication mentioned above. Besides train- ing, adopt a Linear Fusion module to fuse the learned communications in different categories. The fusing phase can be understood as approximating the Universal Communication. With MHCI and policy learner, we're able to master the communication and policy in challenging tasks with multi-source offline datasets.
Multi-heat structure We design multi head stucture to classify the true category. The structure inspired by the widely used attention module, where query and key are first computed from the input, and the weight of each element is obtained from the inner-product of key and query. In our multi-heat structure, we simply compute the key from dataset communication, and the query from the observation. After taking the inner-product and applying the softmax operator, we get prediction probability of each category, which used in the overall imitation loss Equation [I In practice, the communication heads (()) layers for efficiency.
((k) share most networks except the last two
k
Loss Immitation
prob; MSE(C (OD), mD))
(11)
(g/33)
key
C
key
Og
C
19,00
key
M (k)
key
m2p
query
Dot product
(SSE(mp. mn mn) (SE(mg, m(al)
Softmax
MSE(mo, (999)
MSE((m,. ((N)))
prodye
Det product
Imitation Loss
Figure 3: The schematics of the network structure and the loss computation in Equation 11
Linear fusion Although the Multi-head Communication Imitation successfully learns the commu- nication function of each category, we still need to fuse all those categories of generated messages in the testing phase. A trivial fusion method is to concatenate them all, but it is bandwidth-consun as the dimension of all the received messages my is le p. To fuse different categories of message output, long as not adding extra burden on computation,
Linear fusion which is enough

---

Under review as a conference paper at ICLR 2023
in most cases empirically. It can be viewed approximation of the concatenated messages. We fuse the received messages in the following way.
Mffusin fusion A (m) m
(8)
(12)
where (ml(), (2(2),... ,... (1)) are the learned messages with dimension p. k is the total number of heads in MHCL. Further investigations Linear Fusion in the ablation study in Section 6.3
In practice, the fusion matrix Akpxp is learned by the downstream RL loss. Since it's probable that the dataset communication isn' optimal, we also assign additional learned head of communica- tion. So the optimization of the linear fusion as follows
A, ar orgm Loss RL a, ((m m( ,.. mk)), c(++1) (o))
(13)
A
EXPERIMENT
In order to evaluate offline communication learning algorithms, we introduce a benchmark that con- sists of many kinds of communication-intensss in previous researches are investigated in our work, and we also create new tasks that require so- phisticated communication. The details of the benchmark are included in Section 6. All the environments except Room (an added environment introduced in Section 6.1) are built based on StarCraft II. It is a widely used platform
cooperative multi-agent tasks. Tasks introduced
variety researches on MARL and communication
(Samvelyan et al.] (2019); Rashid et al.] (2018); Wang* et al.] (2020)). The implementation details including hyperparameters and dataset generating scheme are Appendix D.I
Besides the part of communication learning, we use Implicit Constraint Q-larning (ICQ, Yang et al. (22TT) as the downstream offline policy learning algorithm our experiments. is designed to trust only state-action pairs that appears in the dataset. Our experiment results in Section 6.2 are based on ICQ combined with relative offline communication learning methods. By keeping the downstream offline MARL algorithm all the same among comparison, empirically show how communication learning influences the overall performance.
6.1 OFFLINE COMMUNICATION LEARNING BENCHMARK
Figure 4: The environment Room
Figure The environment 6olb_vs in StarCraft II
Besides environments used in previous researches, the first added environment is Room. Each agent is corresponding to one of the goals (diamonds in Figure and acquires reward only when it arrives the proper goal of the same color. The goals are initialized randomly in the upper row, while the agents are initialized randomly in the lower row. The challenge comes from the fact that, each agent has no idea where the corresponding goal is. So the optimal joint policy first going up to check the color of that diamond, and then sending the coordinate message the agent with the same color. In this way, every agent aware of its proper destination, and is able to head for the proper goal with reward.
We also come up with a coordination task in StarCraft called 6olb vs Ir. In this environment, there are 6 observers who can't move, and agent (between the observers on the top) needs

---

Under review as a conference paper at ICLR 2023
to reach the enemy's position and kill it. The 6 observers and 1 agent initiallized in the fixed position, while the enemy is randomly initialized the lower-left or lower-right corner. However, each agent has limited sight range. So the message is transferred from the lowest observer to upper observer, and finally transferred to the agent that can move to the enemy.
During designing environments with sophisticated communication, we conclude the following patterns that make the task really challenging.
Inconsistent communication means that the communication is inconsistent among dif- ferent receivers. Unlike broadcasting part of one's observation to everyone, sending the necessary message without redundant information a more effective way.
Delayed-effect communication Consider special case that the optimal policy is based on the received message in the past. Optimizing in environment with such communication pattern requires recursive backpropogating along the historic policy network (e.g. RNN), adding to additional difficulty in convergence.
In conclusion, there are different kinds of environments in the benchmark. Some come from previ- ous online cooperative MARL researches, and others are new environments where a sophisticated communication is essential for solving these tasks. Specifically, we assign moderate-size datasets in order to investigate how offline communication learning helps downstream policy learning in chal- lenging tasks. Although real-world tasks involve larger multi-agent systems and more complicated observations than experimental settings, we hope the offline communication learning benchmark can bridge the gap between experimental RL tasks and more challenging real-works applications.
6.2
MAIN RESULTS
MHCI (Ours)
Learning communication
Pure imitation
from scratch
3b Do.vv_1ll Ihmm-meduum--
38.1 + 7.6
16.3 + 4.8
25.7 + 15.5
vs__r-medium-rrndom
52.2 +: 13.4
35.3 + 1.9
60.4 + 6.3
ros.uullmeddum--rndooo
47.9
25.0 10.2
50.0 13.5
MMM-medium-randoo
17.6 2.9
5.7
10.0 1.2
1o2r vs At-medium-randoo
59.7 29.5
65.3 8.4
63.1+
82.5 6.3
3.5
95.4
Koom-expert-eepee Room- Kcoom-exper--xxperr-eeddd medimt
86.3 + 5.8
5.5
69.4 + 6.7
59.7 = 1.5
2.5
Room-medium--random
67.2 + 6.4
49.9 + 1.7
61.0 + 1.2
bol.b_s..--medduu-rradde
45.0 + 5.7
44.0 + 6.5
44.0 + 1.4
The performance comparison of our method MHCI, learning communication from scratch and pure imitation, normalized by the score of the expert (from 0 to 100). The results are averaged three random seeds with standard deviation.
Table 1:
As shown in Table 1 we compare our algorithm with two other communication learning strategies. Learning communication from scratch in the second column means that instead of using the dataset communication, the communication function is trained together with the policy in an end-to-end manner. Pure imitation means that without discriminating different sources, the communication is learned by minimizing the MSE loss between dataset and learned communication. The table is split into two parts, including environments introduced in previous works and added environments respectively.
In the first part (the first rows of results), we cope with StarCraft tasks introduced in Wang* et al.] (2020). From the results, we can conclude that by learning from communication- baased datasets, both MHCI and pure imitation perform better in Among the 3 methods, MHCI takes the property of multi-source into account and achieves higher scores than Pure imitation. In lo10b_vs. Is--ru-md random, pure imitation performs better than MHCI and learning from scratch, which is probably because the large number of agents (11 agents total) makes multi-head learning difficult. In 1o2r vs dt-medium-randon all the three strategies actually perform similarly.
52.vs_lul.
medium-random and MMM-medium-randonm
8

---

Under review as a conference paper at ICLR 2023
The comparison in the added environments described in Section 6.1 is included in the second part (the last rows of results). In the single-source dataset Noom-medium pure imitation obtains the highest score, and MHCI is the second. We also compare under three mixing schemes in Room. The results show that our algorithm outperforms learning communication from scratch by using dataset communication,
well pure imitation, which neglects the multi-source property. In 6olb_vs__ where a cascading communication is required, all the three methods fail to have good scores. We look forward to future algorithms that can handle 6olb_ bb_.v___rrr._ee... wwt.vss.--mmdum--t-- Ir-medium-random
In conclusion, we conduct comparison among the three methods, MHCI, learning communication from scratch and pure imitation, on the created benchmark of offline communication learning. In this way, we show that by utilizing dataset communication, both MHCI and pure imitation work better than learning communication from scratch. And many datasets, dealing with the multi- source property gives MHCI an additional advantage over pure imitation. Therefore, we can draw the conclusion that MHCI boosts downstream policy learning in offline communication learning with multi-source datasets.
6.3 ABLATION STUDY
In this section, look into the effectness of each module in MHCL. As introduced in Section our method includes two main parts, the multi-head structure for communication imitation learning, and the way of fusing the messages from different sources. Therefore, respectively compare the modules with other alternatives.
MHCI (ours)
closest
Room-expert-expert Iwm-mmdmmmmmmmmmmmmmm
86.3 + 5.8
79.2 + 7.3
38.1 + 7.6
1.7 1.6
Learned linear fusion (Ours) 86.3 + 5.8
PCA-based fusion
Hidden
Foom-cxpert-exxpe
66.5 + 11.8 28.7 11.7
55.3 + 2.9
3b nin-meddum--andee
38.1 + 7.6
8.1 + 4.7
Table 2: Ablation study including alternative classifying methods (the upper part), and alternative techniques of message fusion (the lower part).
The effectiveness of the multi-head structure In MHCI, network to predict the categories of the dataset communications. An alternative is to classify each communication to its closest category argmnin dist(m ()). We compare the performances using the two classification methods in the upper part of Table 2 drawing the conclusion that our multi-head structure outperforms the method of finding the closest category and acts more robustly.
attention-like multi-head
The effectiveness of línear fusion We compare between learned linear fusion, PCA-based fusion and directly using the hidden variable shared by all communication heads. PCA-based fusion means that the fusion matrix obtained from calculating the compression matrix given the concatentated And the results in the lower part of Table show that learned
multi-head messages linear fusion performs better than PCA-based fusion.
CONCLUSION
Partial observablity and Scalability are two main challenges in cooperative multi-agent RL, we look into the new area offline communication learning that hopefully addresses the two problems to achieve higher group intelligence. However, addtional challenges come from the multi-source property offline dataset, which directly training communication in the end-to-end manner isn't tt effective enough. Therefore, we propose Multi-head Communication Imitation, aiming to combine the information in the dataset communication to boost downstream policy learning. Empirical results show the effectiveness of our algorithm under the offline communication benchmark.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Abhishek Das. Théophile Gervet, Joshua Romoff, Dhruy Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. TarMAC: Targeted multi-agent communication. In Kamalika Chaudhuri and Rus- lan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learn- ing, volume 97 of Proceedings of Machine Learning Research, pp. 1538-1546. PMLR, 09-15 Jun 2019. SttLhtts:://m htttp:::/ppoceeddinnnne wwwpeaiiinnmmm......mmmmm.mm. htt....ree//vv//ddssss 9a 19tmmml
Ziluo Ding, Tiejun Huang, and Zongqing Lu. Learning individually inferred communication for multi-agent cooperation. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 22069 22079. Cur- ran Associates, Inc., 2020, URL /tt//tttss////rreeeeee necrrips wwtpsarrisscccpppppeeee wctpppppeppp../////////// wwtpb/occc533oof33ooeee oft.eeoeobbeeeaccccccfae22333333 Paw.ee...2pppppppp...
Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 2786-2793, 2017. doi: 10. 1109/CCRA.2017.7999222
Jakob N. Foerster, Yannis M. Assael, Nando de Freitas, and Shimon Whiteson. Learning com- municate with deep multi-agent reinforcement learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, pp. 2145-2153, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510835 10838819.
Jakob N. Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon White- son. Counterfactual multi-agent policy gradients. In Proceedings of the Thirty-Secone AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelli- gence Conference and Eighth AAAI Symposium on Educational Advances Artificial Intelli- gence, AAAI 18/IAAI 18/EAAI' 18. AAAI Press, 2018. ISBN 978-1-57735-800--
Karl Pearson F.R.S. Liii on lines and planes of closest to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 2(11):559-572, 1901. doi: 10.1080/1 (118001178644010946656
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine D4RL: datasets for deep data-driven reinforcement learning- CoRR, 00220, 2020. URL https:
org/aba 2004. 07219
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.). Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2052-2062. PMLR, 09-15 Jun 2019.
Shiyu Huang, Wenze Chen, Longfei Zhang, Ziyang Li, Fengming Zhu, Deheng Ye, Ting Chen, and Jun Zhu. Tikick: Towards playing multi-agent football full games from single-agent demonstra- tions, 2021.
Landon Kraemer and Bikramjit Banerjee. Multi-agent reinforcement learning as a rehearsal for decentralized planning, Neurocomputing, 190:82 94, 2016. ISSN 0925-2312. doi: https://doi.or 10.10166j.eeucom.201..0111 1031. URL
10.101ele/0111000000 atticlee/11/809222311
www. http://wwwwccneeeeieeeeeeee com science/
783
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Sys- tems, volume 32. Curran Associates, Inc., 2019. URL https: httttp::///proceeenns..... .neurip.. cc/ppppe//211////e///22222 335
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.),
Advances in Neural Information Processing Systems, volume 33, pp. 1179 1191. Curran Asso- ciates, Inc., /ttp:p/cpppp/ppppceeasss.sssss http://ccccaaaaaaoeorece///rp/or 0d2b2061 00f5
10

---

Under review as a conference paper at ICLR 2023
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor- critic for mixed rooperatiee-oompeittee environments. In Proceedings of the 31st International Conference Neural Information Processing Systems. NIPS' 17, pp. 4382-6393, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 97815 510860944.
Hangyu Mao, Zhengehao Zhang, Zhen Xiao, Zhibo Gong, and Yan Ni. Learning multi-agent com- munication with double attentional deep reinforcement learning. Autonomous Agents and Multi- Agent Systems, 34, 03 2020a. doi: 10. 10.10077ss4458--000-095555
Hangyu Mao, Zhengehao Zhang. Zhen Xiao, Zhibo Gong, and Yan Ni. Learning agent commu- nication under limited bandwidth by message pruning. Proceedings of the AAAI Conference Artificial Intelligence, 34:5142-5149, 2020b. doi: 10. 111669/aaa..334044.....
Lactitia Matignon, Guillaume Laurent, and Nadine Fort-Piat. Independent reinforcement learners in cooperative markov games: A survey regarding coordination problems. The Knowledge Engi- neering Review, 27:1 31, 2012. doi: 10. (B017//0226888812000005
Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent populations, Proceedings of the AAAI Conference on Artificial Intelligence, 32, 03 2017. doi: 10.1609/aaai 14992.
Frans A. Oliehoek and Chris Amato. A concise introduction decentralized pomdps. In Springer- Briefs in Intelligent Systems, 2016.
Ling Pan, Longbo Huang, Tengyu Ma, and Huazhe Xu. Plan better amid conservatism: Offline multi-agent reinforcement learning with actor rectification. In Kamalika Chaudhuri. Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.). Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 17221-17237. PMLR, 17-23 Jul 2022. URL. https: //pp:s/e//pcceeeeeee ngs mopssmm... mllr press/ v 62 http:/an22a.hhh. htmll
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. QMIX: Monotonic value function factorisation for deep multi-agent reinforce- ment learning. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. PMLR, 10-15 Jul 2018. URL https: / http::///cccceeeng..com ml wwt.m.......s..../mmm
Bashid18a.htr
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philip H. S. Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. In Proceedings of the 18th International Con- ference on Autonomous Agents and MultiAgent Systems, AAMAS '19 pp. 2186-2188, Rich- land, SC, 2019. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9781450363099.
Shai
Shaked Shammah, and Amnon Shashua. Safe, multi-agent, reinforcement learning for autonomous driving. CoRR, abs/1610.03295, 2016. URL Uttp:ttp://axxv orqq/
abs/ 0as/1610.. 03295
Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar. Individualized controlled continuous communication model for multiagent cooperative and competitive tasks. In International Confer- on Learning Representations, 2019. URL https: http:://ppppevveeeeeee httttts::///peeeeiiii net. wwt.et/f.ee.t.tmeeeeeee forum?
rye7knCqK7
Kyunghwan Son, Daewoo Kim, Wan Kang, David Earl Hostallero, and Yung Yi. QTRAN: Learn- ing to factorize with transformation for cooperative multi-agent reinforcement learning. In Ka- malika Chaudhuri and Ruslan Salakhutdinov (eds.). Proceedings of the 36th International Con- ference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5887-5896 PMLR, 09-15 Jun 2019. URL https: https///ccceeeinnga mir. r. ptwss//v7/
1on 9a ht.ml

---

Under review as a conference paper at ICLR 2023
Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus. Learning multiagent communication with backpropagation. Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16. pp. 2252 2260, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819.
Jianxin Sun, Qi Li, Weining Wang, Jian Zhao, and Zhenan Sun. Multi-caption text-to-face syn- thesis: Dataset and algorithm. In Proceedings of the 29th ACM International Conference on Multimedia, MM '21, pp. 2290-2298, New York, NY. USA. 2021. Association for Com- puting Machinery. ISBN 9781450386517. doi: 10.1 10.114555444008334755991 URL https:
//doi.org/10. 1eee11114444440 3475391
Peter Sunehag, Guy Lever, Audrunas Gruslys. Wojeiech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, and Thore Graepel. Value- decomposition networks for cooperative multi-agent learning based on team reward. In Proceed- ings of the 17th International Conference on Autonomous Agents and MulriAgent Systems, AA- MAS 18, pp. 2085-2087, Richland, SC, 2018. International Foundation for Autonomous Agents and Multiagent Systems.
Ardi Tampuu, Tambet Matiisen, Dorian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, and Raul Vicente. Multiagent cooperation and competition with deep reinforcement learning- PloS one, 12(4):c0172395, 2017.
Tonghan Wang". Jianhao Wang*. Chongyi Zheng, and Chongjie Zhang. Learning nearly decompos- able value functions via communication minimization. In International Conference on Learning Representations, 2020. URL https:
http://ppeeeviveeeeeee https:://ptpeeeeeeeeeee net. mett:forrumm1--h11----ggr
Ying Wen, Yaodong Yang. Rui Luo, Jun Wang, and Wei Pan. Probabilistic recursive reasoning for multi-agent reinforcement learning. In International Conference on Learning Representations, 2019. [RL/Lttts://ppeeeer//
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. 2020. Uttptttss///ppeereve ew. net/ tett:/forrmmm.dddJggg1011/
Yiqin Yang, Xiaoteng Ma, Li Chenghao, Zewu Zheng, Qiyuan Zhang, Gao Huang, Jun Yang. and Qianchuan Zhao. Believe what you see: Implicit constraint approach for offline multi- agent reinforcement learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 10299-10312. Curran Associates, Inc., 2021. U/L/attpss/pproceednngss mew.eeee...nnn...nn..s........
Lei Yuan, Jianhao Wang, Fuxiang Zhang, Chenghe Wang, Zongzhang Zhang, Yang Yu, and Chongjie Zhang. Multi-agent incentive communication via decentralized teammate modeling. In AAAI, 2022.
Sai Qian Zhang, Qi Zhang, and Jieyu Lin. Efficient Communication in Multi-Agent Reinforcement Learning via Variance Based Control. Curran Associates Inc., Red Hook, NY, USA, 2019.
12

---

generally used all kinds of Under review as a conference paper at ICLR 2023
A DIFFERENT COMMUNICATION PROTOCOLS IN PREVIOUS WORKS
In Section the general form of how communication works multi-agent RI. is summarized in Equation We further analyze that previous works communication can be viewed as applying different modifications on the general form.
Informative communication
myj E M Rd. d is the dimension of message. differentiable communication learning.
Broadcasting communication
C Ca1 C22
Cin Vi E 1 (Sukhbaatar et al. (2016)).
Discrete communication (action communication)
Mij. E M = Am (1.2 |Aml)- Each agent i sends the messages all receivers (Foerster et (2016).
Targeted communication
The message follows the broadcasting communication protocol, while the policy function includes an attention module. m, C(o.)), n (Attention(mm, 0j); (Das et al. (2019)).
Incentive communication
The message directly influence the Q value of cach receiver. my M RR/A ag. arg max {Q(oj, T2) +m;} (Yuan et al. 2022
B SUPPLEMENTARY PROOF OF THEOREM
In general, the proof Theorem is based on Lemma which includes more efforts. In this section, first give proof of Theorem I using Lemma And the residual part all proof of Le
Theorem 1.
The optimal expected return based on the Universal Communication is greater than or equal to that of dataset communications and policies.
(C, E Gg, (To)(Mo) (Sintt))) (Sinit) 2 (TLc(MM)),((ims))
(14)
Proof. According to Definition I and LC, Gx Cvo dominates CLC. Therefore by Lemma I we have
Vioc (Simit) 2 Vi (ic(Ma)
And apparently V (i (8iist) (M (Simit). Therefore
Viao (MM)) (Simit) 2 (io (Binis) 2 VTo(Mo)) (sinst)
Lemma 1. If commumication function C1 dominates Cz. have Vie, (Simat))))) 2 Vieg(saa (24) (Sinst).
(15)
The following is the proof of Lemma Recall that the observation function in POMDP is S S O.
Definition 3. The relation that O, contains more information than O2 is defined as 3f S + O,Ys E S(E (() O2(s).
can be understood as the fact that O contains strictly more information than O2.
Denote history = (81, 01 82, 02, 81-1, Tt (a] = (81, 02, "., 81 at-1) (Vt), 7, E Te. the probability p(r)
Denote the history under observation function 0(.) as O(Tz) = (0(s1). a1, 0(2)), O(Ti) O(T,), the probability p(T??) = Eo()) p(T))
Denote the infostate obtained from the history 1S O(T)) + ASS), (1)()))(s) =
p(T)/p(77)
13

---

Under review as a conference paper at ICLR 2023
Lemma 2. O(Ti) is split of Te. If O₂ contains more information than O2. O2(Ti) is split of O1(Ti)
Proof.
1. (T)) c IS(O2(T)))
2. Probability adds up to 1. (By taking expectation)
Therefore define the Splito, (O2(T))) ((Oi(Ti))
Lemma 3. Suppose
(0),)),"" 00-1, 0,-), 7(() = (01, ,1-1, if
E SS Qo, (Tz ai) Qo, (Te (Te= at). we have E.0. -Splitiooo, (T) 2o, (5. (a aa)
2o (T( (T(a):02 at)
Proof.
LHS= E
E
(T
01 , (plito, 01)
2 E
(Te
0
RHS
Theorem 2. Vt, O₂ dominates O2 have E - (T 2o, (T ar) 2 Qo, (T)2,
Proof. Prove literately. Without generality, suppose all trajectories have the same length maan, and the inequality naturally holds for the ending state.
E oi Splito, (1 2o (Te)) a(+1) 2 Qo, (1 a.+1).
Co, (T+
(0,1a,) ar)
Φ1 ~ Splito, (7
max Qo, (T++1 a4+1)
(n).0,
(Denote T+11
(a),0: Concat (T.), (T.),a)))
01 Splito, (7)
ast1
max
E
To, (T4T) (T .0411)
(a),0;
act v. Sptitoo (
max
Co, (a),0z a(41)
(:: Lemma
aill
-T,
Oz
C DETAILS OF THE IMAGINARY COMMUNICATION GAME
The formulation of the imaginary communication included in Section It is named as an imag- inary communication game, because it doesn't it include any real-world correspondence. However, in this way we can concretely measure the influence of scalability, state space complexity and the advantage from learning communication from datasets.
The number of agents n.. The state space is of p dimension, while the indiviual observation space is g. Natually, p > q, i.e., each agent itself doesn't 't observe all the information. In the implemen- tation, we guarantee nq > p and that all the concatenated observations information of the state. The partial observability lies in the fact that for each agent i, there 2q dimensions that relate to the optimal policy, but only q dimensions are included in its own observa- tion. Therefore, communication mechanism is required to make up this lost dimensions. The
all agents include all the
14

---

Under review as a conference paper at ICLR 2023
two experiments share the same network structure, but still, learning communication from scratch fails to converge to optimal when the the task become more challenging.
The network structure involves communication module and policy module, and both modules 3-layer MLPs. The communication module of agent takes the individual observation o, as input and outputs the message ms: each receivers. The policy module of agent j takes oj, m.j input and outputs an action. All the network doesn't share parameters.
Since this environment is of horizon T and the reward belongs to 0, 1, it can be modeled classification problem. We use BCE loss for optimization, with the data size 1000, batch size 1000 step covers epoch) and learning rate 0.1 in SGD optimizer. In learning communication from scratch, we optimize both the communication and policy module by the downstream BCE Loss. In the fitting dataset communication, we optimize the communication module by minimizing the MSE loss using dataset communication, while optimize the policy module by the downstream BCE loss. For the sake of fairness, we use the rule of early stopping that stops training the evaluation accuracy doesn't 't increase in 30 steps.
Figure D the average evaluation accuracy over 100 experiments at each value of n, with = 2. In every experiment, the environment and dataset are re-initialized. In appendix, we also compare under different of = in Figure 6 and different data size 10000 in Figure
Artuaay
to Sua mm
Stata
Symees
aganes
17
apents
Figure 6: Evaluation accuracy with different n, p. given q = (size 1000 by default).
scratch
dacaser communicot
Gate
Stds
Eppents
Sagert
Figure Evaluation accuracy with different n, p. given size 10000 (q by default).
D IMPLEMENTATION DETAILS
D.1 MHCI IMPLEMENTATION
Hyperparameters of MHCI and the data size the offline communication learning benchmark are respectively presented in Table 3 The implementation details of the downstream policy learning follows the same manner in ICQ's original implementation (Yang et al. (202T).
In all the experiments, the state the observation concatenation by default.
15

---

Under review as a conference paper at ICLR 2023
Hyperparameter Communication dimensison
Value
Number of attention heads
Batch size
16
Learning rate
0.0005
Table MHCI hyperparameters table.
D.2 BENCHMARK
Dataset
Size
Room-expert
500
Roon-exxprr-expert-mcciii
750
Room-mediun--andon
500
All the StarCraft II tasks (medium-randonm
100
Table 4: The table of data size in offline communication learning benchmark.
For convenience, we expert represent manually designed datasets, which optimal. Medium indicates that the dataset is collected by a learned policy and communication. And random is just trajectories collected by randomly intialized policy and communication. We mix datasets from different sources and name as expert-medium for example, meaning the mixture of a expert dataset and medium dataset.
Besides, for easier understanding, we mention in this paragraph the general challenge of StarCraft II tasks included in the benchmark. In previous communication-rellan works, the sight range each agent is assigned to small value, making the tasks partially observable. We also use this criterion the added task 6olb_vs_Ir.
Dataset generation details In StarCraft II related tasks (3b_vs_lhh Ihlm, 1010b_vs.Ir, 5z_vs_ Iul, MMM, 1o2r_vs_4r and 6olb_vs_1rg the medium dataset is generated by collecting the trajectories after 2e7 steps training of the NDQ algorithm (Wang* et al. (2020)) which means that the policy trained to convergence. And the random dataset is collected at the first step of training, meaning that the policy is only randomly initialized. The state required by the mixing network is the concate- nation of all agents' observation, rather than a predefined low-dimension state as in previous works. Because the latter requires domain knowledge which is unfeasible in the general CTDE assumption.
In the Room environment, we manually designed expert datasets, in which all the agents act per- fectetly (First transmit the location the correct agents, then move towards its own goal, de- scribed in Section 6. And the communication function is designed as my (1 + location) 10 i observes j's goal, otherwise And in datasets where two different expert sources are included, they'r re m = (1 + location) * 10, m = - 11 +1+ location) * 10 respectively. The medium dataset generated by offline training (training communication from scratch) on abundant expert data. And the random dataset is also collected from the trajectories of randomly initialized policies.
E EFFECTIVENESS OF MULTI-HEAD STRUCTURE
In this section, we look into the classification performance of multi-head structure communication learning, and show how it improves the downstream performance of the policy.
From the head classification results shown Table MHCI successfully classifies different cat- egories of dataset communication into different heads in most datasets, which allows for accurate communication learning. And eventually the overall performance of the downstream policy is in- creased significantly. On contrast, in those datasets that communication classification doesn't work well (e.g. 1o10b_vs.1r 5z.vs. uul and Room- meddum), MHCI doesn't surpass pure imitation.
(Since the second version in rebuttal updates the way of linear fusion, need to clarify that the tendency shown Table 5 similar both in the old version of random linear fusion and the new version of learned linear fusion.)
16

---

Under review as a conference paper at ICLR 2023
Multi-head
Single-head
Head
Head 40.8%
Head 3.0%
Head
Head 2
Head 3
Cate Cate
56.2%
Cate
100%
/
/
3b_vs. lhlm
100%
0%
0%
Cate
100%
/
Score: 38.1 1 + 7.6
Score: 25.7
+ 15.5
Head
Head
Head 100%
Head
Head 2
Head 3
Cate 0%
0%
Cate
100%
/
/
1o10b_vs_1r
Cate
0%
0%
100%
Cate
100%
/
Score: 52.2 + 13.4
Score: 60.4 + 6.3
Head
Head
Head 0%
Head
Head 2
Head 3
Cate
100%
0%
Cate Cate 2
100%
/
_z_vs_lul
Cate 2
99.8%
0.2%
0%
100%
/
Score: 47.9 + 23.0
Score: 50.0
13.5
Head
Head 2
Head 3
Head
Head 2
Head 3
Cate Cate 2
0%
41.3%
58.7%
Cate
100%
/
MMM
0%
0%
100%
Cate 2
100%
/
Score:
17.6 + 2.9
Score: 10.0
+ 1.2
Head 5.1%
Head
Head
Head 1
Head 2 Head 3
Cate Cate 2
94.9%
0%
Cate Cate
100%
/
/
1o2r_vs.4r
0%
100%
0%
100%
/
Score: 59.7 + 29.5
Score: 63.1
+ 15.2
Head 0%
Head
Head 100%
Head I
Head 2
Head 3
Room-m
Cate
0%
Cate
100%
/
/
Score: 88. 2 = 4.2
Score: 95.4
3.6
Head 0%
Head
Head 100%
Head 100%
Head 2 /
Head 3
Cate Cate
0%
Cate 1
/
Room-ee
0%
100%
0%
Cate
100%
/
Score: 84.5 + 3.8
Score: 58.7 + 0.5
Head
Head
Head 100%
Head
Head 2
Head 3
Cate 0%
0%
Cate
100%
/
/
Room-eem
Cate
0%
100%
0%
Cate 2
100%
Cate
20%
80%
0%
Cate
100%
/
Score: 66.1 1 + 4.7
Score: 55.7 + 2.5
Head
Head
Head 100%
Head
Head 2 Head 3
Cate 0%
0%
Cate
100%
/
/
Room-mr
Cate
32.0%
68.0%
0%
Cate 2
100%
/
Score: 63.9 + 6.5
Score: 61.0 + 1.2
Head
Head
Head 0%
Head
Head 2
Head 3
Cate Cate 2
16.5%
83.5%
Cate
100%
/
/
6o
0%
100%
0%
Cate 2
100%
/
Score: 47.6 + 10.7
Score: 44.0 E 1.4
Table 5: The predicted head of each category is shown in this table, well the average of related methods. Multi-head stands for out method MHCI, and single-head means pure imitation. The dataset Room-ce is the abbreviation of Room-expert-expert The predicted head probability is collected under one experiment after 2e5 steps, where the probability represents the proportion of communication that is classified into each head.
F ABLATION ON THE LEARNED LINEAR FUSION
The comparison among random fusion, learned fusion, learned fusion with learned head (used in MHCI) is shown in Table 6 Compared with the ablation study in the main text in Section 6.3 the influence of different ways of linear fusion isn't the main influencing factor of performance. Using MHCI with arbitrary linear fusion, we can obtain satisfying performance in the offline communica- tion learning benchmark.
17

---

Under review as a conference paper at ICLR 2023
Random matrix
Learned matrix
Learned matrix
learned head (MHCT)
3b vs. -.lh-mmmeduum-rdddm
45.4 3.7
38.1 3.0
38.1 = 7.6
lo10b .........a......... It-medium-raadom
34.6 + 10.0
48.2 4.2
52.2 + 13.4
_7_vs_lut
sul-medium--anddon
62.5 22.2
52.1 + 17.9
47.9 + 23.0
29.0
20.0 5.3
17.6
1o2r_vs_4r-man -a----dduu--randee
65.3 13.9
56,3 10.4
59.7 29.5
Room-medium
88.2
1.2
$2.5
Room-expert-exper
3.85
81.7
86.3
Room--xxpet--xpee--meeee
66.1
64.8
6.7
63.9 = 6.5
62.8 5.0
67.2
6olb vs Ir-medium mmnediu--random
47.6 = 10.7
39.0 = 4.9
45.0 = 5.7
Table The performance comparison of different ways of linear fusion, normalized by the score of the expert (from 0 to 100). The results averaged over three random seeds with standard deviation.
G THE EFFECT OF COMBINING DIFFERENT COMMUNICATION PROTOCOLS
We look into how MHCI combines different communication protocols and improve the overall per- formance. We specifically design the dataset including multiple sources that are valuable in different aspects, and therefore combining all of them, rather than learning one of them, essential for good performance.
detail, we modify the Room-medine dataset by masking half of the senders' messages, and leaving the trajectories the same. In 50% data, mask the first 4 agents' sent messages, and in the remaining 50% data, we mask the last agents' sent messages (8 agents all). Therefore, the communication learning algorithm needs combine the experiences of how the first 4 agents should communicate, and also how the last 4 agents should communicate. The experiment results shows that, under such dataset that requires combining experiences, MHCI gains advantage by combining different communication protocols, and exceeds learning from scratch.
50% mask 1-4
Learn communication from scratch
100% mask 1-4
Not masked
50% mask 5-8
82.0 11.4
69.5 2.3
88.3 + 1.2
74.2 3.5
Table 7: The experiment results when the optimal communication in the dataset is masked partially, showing that MHCI combines experience from different sources communication protocols.
H COMPARISON WITH PREVIOUS ONLINE COMMUNICATION LEARNING ALGORITHMS
We compare MHCI with VBC (Zhang al. (2099) under 3b_vs. vs. thim-mediim--aadon in Figure VBC fails to converge because it isn't 't designed for the offline setting.
18

---

Under review as a conference paper at ICLR 2023
ABC
IN p. / pro
Siccess roze
0.3
We ful /ylg yma
0.1
0.0
tirestep
Figure 8: Learning curve on 3b Lb_vs_ www.:hhlmmmmdiim-aaddoooo
19


Reviews:
[{"review_id": "fo1eeUyYg6F", "reviewer": "Reviewer_zXRw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work investigates learning the communication strategy in MARL from offline communication datasets. The authors proposed a multi-head attention method to encode the messages to support learning data from multiple sources. The proposed method is evaluated using a custom window environment and a custom StarCraft environment. ", "summary_of_the_review": "This work is a looking at the communication problem in offline MARL. It tries to learn from communication data that is multi-sourced. The proposed method seems to be effective and the problem is relatively new; however, the challenge of the research problem seems limited which limits the novelty of the proposed method. ", "strength_and_weaknesses": "**Strength**\n- This paper is relatively easy to follow.\n- The proposed network seems to have better performance compared to the chosen baselines.\n\n**Weakness and Questions**\n- How are the different datasets generated (e.g. export, medium, etc)? Does the model still work well if the communication protocol changes completely? More details should be provided. \n- The method should also be evaluated under more combinations of communication messages to understand the performance related to the difference in the multi-source data. How do combining different trajectories in different datasets and different communication protocol affects the performance of the proposed algorithm? Can the algorithm learn to use communication strategy learned from some trajectories to a different trajectory to improve performance? More ablation would be helpful here.\n- In the evaluation section, it would be a good idea to provide the performance of the different benchmarks under the online learning case. For example, comparing the proposed method to online techniques (e.g. I2C) in MMM. \n- In the motivating example, more details should be provided in terms of how the two models (learning communication from scratch and feed dataset communication) are trained. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MoM_MN8s6g", "reviewer": "Reviewer_UfVH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors analyze the new challenges in offline communication learning, and introduce a benchmark of offline communication learning which contains diverse tasks. The author propose an effective algorithm, Multi-head Communication Imitation (MHCI), which aims to address the problem of learning from single-source or multi-source datasets, and our method shows superior outperformance in various environments of our benchmark.", "summary_of_the_review": "See Strength And Weaknesses.", "strength_and_weaknesses": "Pros: \n1. The motivation is clear. \n2. The paper is well-written and organized. \nCons: \n1. More results on more challenging datasets are needed to verify the superiority of the proposed method. \n2. Some related works are missing, e.g., Multi-caption Text-to-Face Synthesis: Dataset and Algorithm.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FOT-2kybUu", "reviewer": "Reviewer_gcz9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper mainly considers communication in offline multi-agent reinforcement learning (MARL). The authors propose a new benchmark that contains a set of offline MARL communication tasks with single/multi-source datasets. The “multi-source” means that the messages are from different communication protocols. The authors show that there exists a universal communication protocol that can represent all sources. Also, the authors propose a multi-head structure for communication Imitation learning. Experiments show a good performance in a few scenarios in SMAC and Room.", "summary_of_the_review": "I like the idea of incorporating communication into offline MARL. However, I think the authors should improve the current version.", "strength_and_weaknesses": "Pros:\n\n- This paper takes on an interesting problem in MARL: the combination of communication and offline MARL.\n- The experiments are completed.\n\nHowever, I have several concerns here:\n- To my understanding, this paper is mainly to learn a universal communication protocol to achieve higher accuracy. This point is interesting from the perspective of MARL communication but is not adequate. Since we can just view this problem as simple supervised learning. End-to-end communication offline learning is more attractive.\n- Why do we need an offline MARL dataset with communication messages? The objective of offline RL/MARL is to obtain good policies from the dataset for direct deployment. We can have an offline MARL dataset by running communication but discarding communication messages. Since the messages are mainly from the observations of each agent. Will agent communication during deployment? If not, we can mimic such a dataset for deployment. If so, we can also have a communication mechanism inside the policy learning without mimicking the multi-source messages.\n- In Fig.1, Left = Mid + Right. No need to plot redundant figures.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AZeuSqsNu7", "reviewer": "Reviewer_311E", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper is about how to learn offline environments that can communicate between agents in multi-agent environments. Each agent learns a function that generates messages from their observations, based on the messages in the datasets. The communication function is learned with a multi-head attention structure, and in the evaluation, agents transfer messages derived from a linearly random mixing of messages learned from each head. Experiments have been performed in environments that require intensive communication, based on Room and StarCraft 2. The proposed structure shows higher performance than learning communication without message information in datasets or imitation learning without a multi-head attention structure.", "summary_of_the_review": "This paper is meaningful in that it attempts to approach a new kind of multi-agent offline RL environment. However, the paper is poor at showing the process of generating the datasets, showing the performance of the proposed method(more evaluation and ablation that shows the advantages of the structure should be provided), and the lack of explanation and analysis of linear fusion.", "strength_and_weaknesses": "Strengths\n\nThis paper presents a new problem of how to learn the offline multi-agent RL datasets with communication and proposes an algorithm based on a multi-head attention structure to solve it.\nThe paper shows the performance of the proposed method by creating a new benchmark, and it shows higher performance than the existing offline RL algorithm.\nMathematical concepts explain well how the proposed algorithm can get a high return in an environment where datasets are generated from various distributions.\n\nWeaknesses\n\nIt seems more necessary to analyze whether the proposed multi-head structure is suitable for problem-solving. When learning based on various datasets, it is necessary to analyze whether each head is learned diversely as intended and to provide evidence of performance increase. \n\nThere is also a lack of explanation on how to obtain the dataset. There will be a variety of ways to design the communication function. Since the created message is included in the dataset, the algorithm performance may change significantly depending on the algorithm that created the dataset.\n\nSince it is an algorithm designed to learn datasets made from various distributions well, it is also effective to show the performance change according to the number of distributions included in the dataset.\n\nThe explanation and backgrounds for linear fusion are poor. The multi-head attention structure is designed to learn messages from various distributions separately, but it is too naive to randomly mix all heads in actual use. In the process of randomly mixing multi-heads, the basis of the structure seems to disappear. The possibility of a structure that uses learned multi-head more efficiently is open.\nThe explanation of the base algorithm ICQ seems to be poor, and it is necessary to explain why this algorithm is chosen.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject