Towards Effective and Interpretable Human-Agent Collaboration in MOBA Games: A Communication Perspective
Authors: Yiming Gao
Feiyu Liu
Liang Wang
Zhenjie Lian
Weixuan Wang
Siqin Li
Xianliang Wang
Xianhan Zeng
Rundong Wang
jiawei wang
QIANG FU
Yang Wei
Lanxiao Huang
Wei Liu
Venue: ICLR 2023 notable top 25% (2023)
Keywords: game playing
multi-agent
human-ai communication
human-ai collaboration
deep reinforcement learning

Abstract: MOBA games, e.g., Dota2 and Honor of Kings, have been actively used as the testbed for the recent AI research on games, and various AI systems have been developed at the human level so far. However, these AI systems mainly focus on how to compete with humans, less on exploring how to collaborate with humans. To this end, this paper makes the first attempt to investigate human-agent collaboration in MOBA games. In this paper, we propose to enable humans and agents to collaborate through explicit communication by designing an efficient and interpretable Meta-Command Communication-based framework, dubbed MCC, for accomplishing effective human-agent collaboration in MOBA games. The MCC framework consists of two pivotal modules: 1) an interpretable communication protocol, i.e., the Meta-Command, to bridge the communication gap between humans and agents; 2) a meta-command value estimator, i.e., the Meta-Command Selector, to select a valuable meta-command for each agent to achieve effective human-agent collaboration. Experimental results in Honor of Kings demonstrate that MCC agents can collaborate reasonably well with human teammates and even generalize to collaborate with different levels and numbers of human teammates. Videos are available at https://sites.google.com/view/mcc-demo.

Published as a conference paper at ICLR 2023
TOWARDS EFFECTIVE AND INTERPRETABLE HUMAN-AGENT COLLABORATION IN MOBA GAMES: A COMMUNICATION PERSPECTIVE
Yiming Gao-* Feiyu Liu Liang Wang!
Zhenjie Lian' Weixuan Wang!
Siqin Li Xianliang Wangb Xianhan Zeng' Rundong Wang! Jiawei Wang! Qiang Fu? Wei Yang' Lanxiao Huang² Wei Liu'
Tenneet AI Lab, Shenzhen, China TTencent TiMi L1 Studio, Chengdu, China (yatminggao, marsxlinn iu, enginewang, leolian, wawaaanwwaiiiiinn waihinwang gracesqli lemxlwang, ri ritzeng, rundongwang, donut leonfu, willyang, jackiehuang) etenent...... com;
edu
ABSTRACT
MOBA games, e.g., Dota2 and Honor of Kings, have been actively used as the testbed for the recent AI research games, and various AI systems have been developed at the human level so far. However, these AI systems mainly focus on how compete with humans, less exploring how collaborate with humans. To this end, this paper makes the first attempt to investigate human-agent collabo- ration in MOBA games. In this paper, we propose enable humans and agents to collaborate through explicit communication by designing an efficient and inter- pretable Meta Ceta-Coommnd Communication -aaeed framework, dubbed MCC, for accomplishing effective human- buman-agent collaboration in MOBA games. The MCC framework consists of two pivotal modules: 1) an interpretable communication pro- tocol, i.e., the Meta-Command, to bridge the communication gap between humans and agents; 2) meta-command value estimator, i.e., the Meta-Command Selector, to select valuable meta-command for each agent to achieve effective human- -agett collaboration. Experimental results in Honor of Kings demonstrate that MCC agents can collaborate reasonably well with human teammates and even generalize to collaborate with different levels and numbers of human teammates. Videos are available https: //stt: google. cow coo/vieee /tcc//ccco//// demo
INTRODUCTION
Games, the microcosm of real-world problems, have been widely used as testbeds to evaluate the performance of Artificial Intelligence (AI) techniques for decades. Recently, many researchers focus on developing various human-level Al systems for complex games, such as board games like Go (iilver et al. 2016; 2017), Real-Time Strategy (RTS) games like StarCraft 2 (Vinyals et al. 2019), and Multi- Online Battle Arena (MOBA) games like Dota 2 (OpenAl al.) 2019). However, these AI systems mainly focus on how to compete instead of collaborating with humans, leaving Aluman-Agent Collaboration (HAC) in complex environments still be investigated. this paper, we study the HAC problem in complex MOBA games (Silva & Chaimowicz 2017). which is characterized by multi-agen cooperation and competition mechanisms, long time horizons, enormous state-action spaces (1020000) ) and imperfect information (OpenAl et al. 2019 Ye et al. 2020a)..
HAC requires the agent to collaborate reasonably with various human partners (Dafoe et al.) (2020). One straightforward approach is to improve the generalization of agents, that is, to collaborate with a sufficiently diverse population of teammates during training. Recently, some population-bail based methods proposed to improve the generalization of agents by constructing a diverse population of partners in different ways, succeeding in video games (Jaderberg et al., 2017 2019 Carroll et al.) Strouse et al. 2021) and card games (Hu et al. 2020: Andrei et al. (021). Furthermore, to better evaluate HAC agents, several objective as well as subjective metrics have been proposed (Du
2019
al. 2020 Siu al. 2021 McKee al. 2022). However, the policy space in complex MOBA
'These authors contributed equally this work.

---

Published as a conference paper at ICLR 2023
Strateyy
Gome Environment
Cleas Up Tog--Lame Monions
Whene: Come To The Tip-lane
What: Clean Mintons
How long; Within 20 Secondss
Where
How long
L,, E, T me
What
C1
A2
Where: Come To The Dragon
What: Arack The Dragon
Signaling System
MizroOperation
Signaling System
Covee Andi Deagon
How long' Usni The Dragon Krlled
(a) Key elements in Honor of Kings
(b) Example of Meta-Command
Figure I: Introduction of Honor of Kings. (a) Key elements in Honor of Kings, including the game environment, muleooooperatin buttons, examples of strato-traatey,. and the signaling system. (b) Exampic of collaboration via meta-commands The Come And Kill The Dragon is more valuable for humans A and B and agent D to collaborate, while the Clean Up Top-Lane Minions is more valuable for human C and agent E to collaborate.
games is enormous Gao et al. 2021 and requires massive computing resources to build a sufficiently diverse population of agents, posing big obstacle the scalability of these methods.
The communication ability to explicitly share information with others is important for agents to collab- orate effectively with humans (Daloe al., 2020). In Multi-Agent Reinforcement Learning (MARL), communication is often used to improve inter-agem collaboration. Previous work (Sukhbaatar et al.
2016] Foerster Foerster et al., 2016 Lazaridou et al.) 2016 Peng et al.] 2017,) Mordatch & Abbeel 2018 Singh et al. 2018 Das et al. 2019: Wang et al. (220 mainly focused on exploring communication protocols between multiple agents. Other work (Ghavamzadeh & Mahadevan 20044] Jiang & Lu 2018: Kim et al. 2019) proposed to model the value of multi-agent communication for effective collaboration. However, these methods all model communication in latent spaces without consider- ing the human-interpretable common ground (Clark & Brennan 1991; Stalnaker 2002) or lingua franca Kambhampati et al. (2022). making themselves less interpretable to humans. Explicit com- munication dominated by natural language is often considered in human-robot interaction (Kartoun) et al.) 2010 Liu et al. 2019, Shafti et al. 2020: Gupta et al. [021]. However, these studies are mainly limited collaboration between a robot and human through one-way communication, i.e., humans give robots orders. Therefore, there is still large room to study RL with the participation of humans.
Success in MOBA games requires subtle individual micro-operations and excellent communication and collaboration among teammates on macro-strategies, i.c., long-term intentions (Wu) 2019 Gao al. 2021). The micro-operation ability of the existing State-Of-Thh-AA (SOTA) MOBA agents has exceeded the high-level (top I 1)) humans (Ye et al.) (222aa). However, these agents' macro-strategies are deterministic and quite different from those of humans (Ye et al., (020a)). Moreover, all existing SOTA MOBA Al systems lack bridges for explicit communication between agents and humans on macro-strategies. These result in the agent's 's behavior not being understood immediately by humans (Ye al. 2020a) and not performing well when collaborating with humans (see Section 4.3
To this end, we propose an efficient and interpretable Command Communication- human- agent collaboration framework, dubbed MCC, to achieve effecctive HAC in MOBA games through explicit communication. First, we design an interpretable communication protocol, i.e., the Meta- Command, general representation of macro-strategier to bridge the communication gap between agents and humans. Both macro-strategie sent by humans and messages outputted by agents can be converted into unified meta-commands (see Figure (b)). Second, following Gao et al. (2021) construct a hierarchical model that includes the command encoding network (macro-strategy layer) and the meta-command conditioned action network (micro-action layer), used for agents generate and execute meta-commann commands. respectively. Third, we propose a meta-command value estimator, i.c., Selector, to select the optimal meta-command for each agent to execute. The
the Meta-Comman training process of the MCC agent consists of three phases. We first train the command encoding network to ensure that the agent learns the distribution of meta-commands sent by humans. Afterward, we train the meta-command conditioned action network to ensure that the agent learns to execute meta-commann commands. Finally, we train the meta-command selector to ensure that the agent learns to select the optimal meta-commands to execute. We train and evaluate the agent in Honor of Kings' 5v5 mode with a full hero pool (over 100 heroes). Experimental results demonstrate the effectiveness of the MCC framework. In general, our contributions
follows:
2

---

Published as a conference paper at ICLR 2023
To the best of our knowledge, we are the first investigate the HAC problem in MOBA games. We propose the MCC framework to achieve effective HAC in MOBA games.
We design the Meea-Commmand to bridge the communication gap between humans and agents. We also propose the Meta-Command Selector model the agent's value system for meta-commands.
We introduce the training process of the MCC agent in typical MOBA game Honor of Kings and evaluate in practical human-agent collaboration tests. Experimental results show that the MCC agent can reasonably collaborate with different levels and numbers of human teammates.
BACKGROUND
2.1
MOBA GAMES
MOBA games have recently received much attention from researchers, especially Honor of Kings (Wul 2019; Ye et al. 2020a blc Gao et al. 2021), one the most popular MOBA games worldwide. The gameplay is to divide ten players into two camps compete on the same map. The game environment is shown in Figure a). Each camp competes for resources through individual micro-operations macro-stratege strategies (B), and finally wins the game by destroying
(A1, A2) and team collaboration the enemy's s crystal. Players can communicate and collaborate with teammates through the in-game signaling system. Particularly, players can send macro-strategies by dragging signal buttons (CI, C2) to the corresponding locations in the mini-map (D), and these signals display to teammates in the mini-map (E). See Appendix foro detailed game introductions.
2.2 HUMAN-AGENT COLLABORATION
We consider an interpretable communicative human-agent collaboration task, which can be extended from the Partially Observable Markov Decision Process (POMDP) and formulated as tuple < N. H. S, A A O, M, r, P. y where N and И represent the numbers of agents and humans, respectively. S is the space of global states. AN the spaces of actions of N agents and H humans, respectively. O {O:) denotes the space of observations of N agents and H humans. M represents the space of interpretable messages. P S X AN X A + S and S x AN x A + I denote the shared state transition probability function and reward function of agents, respectively. Note that r includes both individual rewards and team rewards. 7 [0, denotes the discount factor. For each agent in state 8, E S, it receives an observation of E O, and a selected message cj E M, and then outputs message mi+1 To(o)) M, where mo and no
J.=1..... and A I {A, ).1,...., denote
an action a; mo (oi, ct) E AN and action network and message encoding network, respectively. A message selector of (= Ct) is C M.
introduced to select a message of from message set C.
We divide the HAC problem in MOBA games into the Human-to-Agent (H2A) and the Agent-to- Human (A2H) scenarios. The H2A Scenario: Humans send their macro-strategies as messages to agents, and agents select the optimal one to collaborate with humans based on their value systems. The A2H Scenario: Agents send their messages as macro- strateist to humans, and humans select the optimal one to collaborate with agents based on their value systems. The goal of both scenarios that agents and humans communicate macro- strateis with pre-defined communication protocols and then select valuable macro-strategies for effective collaboration win the game.
3 META-COMMANU COMMUNICATION
FRAMEWORK
In this section, present the proposed MCC framework in detail. We first briefly describe three key stages the MCC framework (Section 3.I). Then we introduce its two pivotal modules: I) an interpretable communication protocol, i.e., the Meta-Command, as a general representation of macro-strategies to bridge the communication gap between agents and humans (Section 3.2); 2) a meta-command value estimator, i.e., the Meta-Command Selector, model the agent's value system for meta-commands to achieve effective HAC in MOBA games (Section 3.3
3.1 OVERVIEW
The process of the MCC framework consists of three stages: (I) the Meta-Command Conversion Stage, (II) the Meta-Command Communication Stage, and (III) the Human-Agent Collaboration
3

---

Published as a conference paper at ICLR 2023
Meta- Command Conversion Stage
Communication Step:
pm
Command Comoerter yor
Time Step:
1++11
7
Command Ercoding Notwork S
[mif) .mm]
(i]) Meta- Command Communication Stage
① m
C
Meta Command Sefector
IR:
(HI Ituman-Agent Collaboration Stage
Ocal
Weta Commard Conditione Action Networkktt
Environment
(A)
(b)
Figure The MCC framework. (a) Three key stages MCC: (I) the mela-command conversion stage, (II) the meta-command communication stage, and (III) the human-agent collaboration stage. (b) The temporal process of MCC. Stage I and II are executed at each communication step. Stage III is executed at each time step.
Stage, as shown in Figure 2 Notably, Stage I and II are executed at each communication step. and Stage III is executed at each time step. At Stage I, the MCC framework converts humans' explicit messages and agents' implicit messages into unified meta-commands m my, respectively, and broadcasts them to all agents and humans. At Stage II, the MCC framework estimates the values of all received meta-commands C₁ and selects the optimal one Q E C. for each agent to execute. The selected meta-command will remain unchanged between each two communication steps (c.g. within [t. + 7Tc)) time steps). At stage III, the MCC framework predicts a sequence of actions for each agent to perform based on its selected meta-command c₁. In each game, humans and agents collaborate multiple times, that is, execute the three stages multiple times, to win the game.
3.2 META-COMMAND
We divide macro-strategy into three components: where to go, what to do, and how long. For example, macro-strategy can be Come And Kill The Dragon, which consists of Come To The Dragon Location (where to go), Attack The Dragon (what to do), and Unril The Dragon Is Killed (how long). Thus, a general representation of macro-strane strattgg, i.e., the Meta-Command, can be formulated as tuple < T., E, T >, as shown in Figure [b), where T. is the Location to go, E is the Event to do after reaching T., and T' is the Time Limit for executing the meta-command.
Meta-Command Conversion. To realize bidirectional interpretable human- agent communication, the MCC framework converts humans' explicit messages and agents' implicit messages into unified meta- commands. To achieve the former, we use the Command Converter function f (Appendix (B..1 to extract corresponding location L and event E# from explicit messages sent by humans in the in game signaling system. To achieve the latter, use command encoder network (CEN) n& (m) to generate and E based on the agent's 's observation o. The CEN is trained via supervised learning (SL) with the goal of learning the distribution of meta-commands sent by humans (Appendix (B.6.2).
In MOBA game settings, we use a common location description, i.e., divide L. of meta-commands in the map into 144 grids. Since the macro-strategy space is enormous (Gao et al. 2021). customizing corresponding rewards for each specific event to train the agent is not conducive to generalization and even impossible. Instead, train a micro-action network learn do optimal event E location L, just as humans do optimal chiiroooperations at location L based on their own value systems. We also do not specify a precise T for the execution of each specific meta-command. Instead, we set Tmee to how long takes a human to complete macro-strategy in MOBA games. Usually, 20 seconds corresponds to an 80% completion rate, based on our statistics (Appendix B.6.2). Thus, the MCC framework converts humans' explicit messages into meta-commands m =< " EE TT >, generates moma-ns m =< L, E", T'me > for agents based on their observations (Figure 2)). and then broadcasts them to all agents and humans.
Mleta-Command Execution. After receiving a meta-commant candidate set, agents can select one meta- cotmmommmmd from it execute. Note that the MCC framework will replace E with E* when the agent selects a meta-command from humans. We adopt the MCCAN (a)o, m) for agents to perform actions based on the selected meta-command, as shown in Figure 3l (II). The MCCAN is trained via
4

---

Published as a conference paper at ICLR 2023
Command Encociing Network (CIN)
(it) Meta- Command Conditoone Action Networkk (MCCAM)
Oberratton
m
m
m
Vi
CKN Festure
vo-ttained
RL
G:
Expert data < m, >
G,
NL
(11) Command Selector (CS)
Map Gata
Gate
Sharet MEPS
Bating
0
argmax
m,
Q.
softmax
exery
qaery
02
me
6
Qf
Fand Entte
100
pre-trained
C
RL
pre-trained
Waker
Vi Gl
(b)
Figure 3: The training process and model structure of MCC. (a) The training process is divided into three phases: we first (I) train the CEN via supervised learning (SL), then (II) train the MCCAN via goal-conditioned RL, and finally (III) train the CS via RL. Among them, the dashed box represents the frozen model. (b) The detailed CS model structure, including CNN feature extraction, gating mechanism, target attention module, etc.
self-play RL with the goal of achieving a high completion rate for the meta-commands while ensuring that the win rate is not reduced. To achieve this, we introduce extrinsic rewards (including individual and team rewards) and intrinsic rewards rf ri" my: St+1) = =I where f extracts the agent's location from state 8,, and l (8+)) me| is the distance between the agent's 's location and the meta-command's ' location at time step Intuitively, the intrinsic rewards adopted to guide the agent to reach L of the meta-command and stay at to do some event E. The extrinsic rewards event E" at L. Overall, the optimization objective is maximizing the expectation over extrinsic and intrinsic discounted total rewards G. = Reg am-
me| m (s++1) - me)-
adopted guide the agent perform optimal actions to reach L. and do optimal
C + a Σj= 7' where (s) = lim; P (s: = s 5o, л)) is the probability when following и for t time steps from sp. We use to weigh the intrinsic and extrinsic rewards.
After training the CEN and the MCCAN, we can achieve HAC by simply setting an agent to randomly select meta-command derived from humans execute. However, such collaboration is non- intelligent and can even be a disaster for game victory because agents have no mechanism to model meta-commands values and cannot choose the optimal meta-command to execute. While humans usually choose the optimal one based on their value systems for achieving effective collahoration to win the game. Thus, we propose a meta-command value estimator to model the agent's value systems for meta-commands, as described in the following subsection.
3.3 META-COMMAND SELECTOR
In MOBA games, the same macro-stratee strrtest often has different values for different humans in different situations. For example, macro- strategy can be Come And Kill The Dragon, as shown in Figure [b). It is more valuable for humans A and B and agent D to collaborate. However, another macro-strategy Clean Up Top-Lane Minions more valuable for human C and agent than the others. Therefore, agents must select the most valuable meta-command from the received meta-command candidate set C' to achieve effective human-agent collaboration. We propose a meta-command value estimator, i.c., the
Selector (CS) Tu (o, C), to estimate the values of all received meta-commands and select the most valuable one for each agent to execute.
CS Optimization Objective. Typically, executing a meta-command involves reaching location L and doing event E, of which the latter is more important to the value of the meta-command. For example, for the meta-command Come And Kill The Dragon, if the event Kill The Dragon cannot be done within T'me time steps, then it is pointless Come The Dragon. Thus, the optimization objective of CS is to select the optimal meta-command m Tw (ot, C.) for each agent to maximize the expected discounted meta-command execution return G
Tb
3
G
E Eunning minngraNTT
i D Time T-mo
R
rtti
==0
T=
(I)
(II)
5

---

Published as a conference paper at ICLR 2023
where or O, C2 is the meta-command candidate set in state st, V [0, 1) is the discount factor, and R, is a generalized total extrinsic rewards before reaching location L and doing event E, respectively. T - Tm is the time for reaching I, and 3 > 1 is a trade-off parameter representing the relative importance of F.
execution reward function. For R, (1) and (II) are the
CS Training Process. We construct self-play training environment for CS, where agents can send messages to each other, as shown in Figure (II) Specifically, three tricks are adopted to increase the sample efficiency while ensuring efficient exploration. First, each meta-command m is sampled with the argmax rule from the results outputted by the pre-trained CEN. Second, each agent sends its meta-command with a probability p every T time steps. Finally, each agent selects the final meta-command e sampled with the softmax rule from its CS output results and hands it over to the pre-trained MCCAN for execution. We use the multi-head value mechanism (Ye et al. 2020a) to model the value of the meta neta-command execution, which can be formulated as:
I, (w) = Eo,c.
Σ Ready
|G V (O,
where (S, C') is the value of the k-th head. For DQN- methods (Mnih al. 2015 al. 2016 Wang et al. 2016). the Q loss is:
Van Hasselt
I (w) Eo,C,M ||Geooall et(0,,,
Grotal
Σ
W&GF
heady
where we is the weight of the k-th head and C is the Temporal Difference (TD) estimated value error Rg + Tme vbh (0', C*) - ( C),
CS Model Structure. We design a general network structure for CS, as shown in Figure 3b). In MOBA games, the meta-commands in adjacent regions have similar values. Thus, we divide the meta- cotaa-oo in the map into grids, common location description for MOBA games, and use the shared Convolutional Neural Network (CNN) to extract region-related information improve the generalization of CS to adjacent meta-commands. Then, the map embeddings of all received meta-commands are integrated into a map set embedding by max-pooling. Besides, we use the gating mechanism (Liu et al., (021) ) to fuse the map set embedding and the state embedding of the observation information. Finally, to directly construct the relationship between the observation information and each meta-command, we introduce target attention module, where the query is the fused embedding and the key is the map embedding of each fed into the subsequent state-action value network (o,, C, m) and state value network V(o, C') of CS. In this way, we can also easily convert the state-action value network Q(o, C, m) to the policy network π (m|o, C). Thus, the CS model structure be easily applied to the most popular RL. algorithms, such as PPO (Schulman et al. 2017), DQN (Mnih et al., 2015). etc.
The fused embedding is
EXPERIMENTS
In this section, we evaluate the proposed MCC framework by performing both agent-only and human- agent experiments in Honor of Kings. All experiments were conducted in the 5v5 mode with full hero pool (over 100 heroes, see Appendix A.4).
Train Environment
Agent-Only Environment
Human AAgett Environment
Test
Test
Broadiastt Channel
Browdasst Channel
Broadcast
VS
Simulated Human
35
VS
Opponents
Fixed Opponents
Figure 4: Communication environments in the experiment. The orange arrows indicate sending meta- commands, and the blue arrows indicate receiving meta-commands. The dashed line denotes sending meta- commands with probability p.

---

Published as a conference paper at ICLR 2023
4.1 EXPERIMENTAL SETUP
Due to the complexity of MOBA games and limited resources, train the CEN, the MCCAN, and the CS sequentially instead of training the MCC framework jointly. Specifically, we first train the CEN via SL until it converges for 26 hours using 8 NVIDIA P40 GPUs. The batch size of each GPU is set to 512. Then, we train the MCCAN by fine-tuning the pre-trained WuKong model (Ye) 2020a) conditioned on the meta-command sampled from the pre-trained CEN. The MCCAN is trained until it converges for 48 hours using physical computer cluster with 63,000 CPUs and 560 NVIDIA V100 GPUs. The batch size of each GPU is set 256. The parameter a is set to 16. After that, we train the CS via self-plas until it converges for 24 hours using a physical computer cluster with 70,000 CPUs and 680 NVIDIA V100 GPUs. The batch size each GPU is set to 256. The parameter 3 is set to 2. Each agent sends meta-command with a probability p of 0.8 and an interval T of 20s, as shown in Figure 4 a). For the entire training process of the MCC framework, the location L of meta-commands in the game map is divided into 144 grids, and the time limit T' for the meta-command execution is set to 20s. Finally, we obtain the trained MCC agent that can receive meta-commands from other agents and humans and select the most valuable one to execute.
al.
To evaluate the performance of the MCC agent, we conduct both agent-only and human-agent experiments and compare the MCC agent with three different types of agents: the MC-Base agent (agent only executes its own meta-commant without communication), the MC-Rand agent (agent randomly selects meta-command execute), and the MC-Rule agent (agent selects the nearest meta-command to execute). Note that the MC-Base agent can be considered as a State-Of-The-Al (SOTA) in Honor of Kings since it maintains the same capabilities as the WuKong agent (Ye et al. 2020a) (Appendix B.6.3. Results are reported over five random seeds.
4.2 AGENT-ONLY COLLABORATION
Directly evaluating agents with humans is expensive, which is not conducive to model selection and iteration. Instead, we built two agent-only testing environments, Test and Test II, to evaluate agents, as shown in Figure b).. Test is complex environment where all agent teammates can send and receive meta-commands simultaneously with an interval of 20s. Test evaluates the agents' performance under complex situations. Test II is a simple environment to simulate most practical game scenarios. where at most one human can send his/her macro-strategy at time step. Thus, in Test II, only one agent is randomly selected to send its meta-command with an interval of 20s, and the other agents only receive meta-commands. See the detailed experimental results of the CEN and MCCAN in Appendixes B..622 and B.6. respectively.
Finding 1: MCC outperforms all baselines.
We first evaluate the capabilities of the MCC agent and baselines to examine the effectiveness of CS. Figure 5 (a and (b) show the win rates (WRs) of four types of agent teams who play against each other for 600 matches in Test I and Test II, respectively. Figure (jc) demonstrates the final Elo scores (Coulom| 2008) of these agents. We see that the MCC agent team significantly outperforms the MC-Rand and MC-Rule agent teams. This indicates that compared to selecting meta-commands randomly or by specified rules, the CS can select valuable meta-commands for agents to execute, resulting in effective collaboration. Such collaboration manners in the MCC agent team can even be conducive to winning the game, bringing about 10% WR improvement against the MC-Base agent team. On the contrary, the unreasonable collaboration manners in the MC-Rand and MC-Rule agent teams can hurt performance, leading to significant decreases in the WR against the MC-Base agent team. Note that the MC-Base agent has the same capabilities as the WuKong agent Ye et al. 2020a). the SOTA in Honor of Kings. Overall, the MCC agent achieves the highest Elo scores compared to all baselines in both testing environments, validating the effectiveness of CS. Notably, we also find that the WRs of the MCC agent in Test I and Test Il are close, suggesting that the MCC agent can generalize to different numbers of meta- meta-commands, We also investigate the influence of different components, including CNN feature extraction with the gating mechanism (Liu et al. 2021). target attention module, and optimization algorithms on the performance of CS (Appendix B.6.4)
4.3 HUMAN- AGENT COLLABORATION
In this section, we conduct an online experiment to evaluate the MCC agent and baselines in collaborating with humans, as shown in Figure c). We contacted the game provider and got
7

---

Published as a conference paper at ICLR 2023
MC Base MC-Rtand MC-iuuls
MCC
Elo Score
80
59.3
41.5
MC--aae
76.8
71
41.3
80%
1200
1121
70%
1062
1003
1076
NC-
20
24.5
14.3
23.2
42.5
15.5
90%
1000
50%
57%
900
813
862
NC-Rul
40.7
75.5
34
NC Rull-
29
57.5
21.8
800
700
58.5
85.7
66
-3%%
MCC
58.7
84.5
78.2
30%
27%
MC-Base
Test
# MC-Rand
MC-Rule
Test
MCC
(a) Win rate (row vs. col) Test I. (b) Win rate (row vs. col) Test
(c) Elo scores of different agents,
Figure 5: AT performance in the testing environments. (a) and (b) show the win rate maps of four types of agent teams who play against each other. (c) shows the final Elo scores of these agents.
test authorization. The game provider helped us recruit 30 experienced participants with personal information stripped, including 15 high-level (top1%) and 15 general-level (top30%) participants. design: m Human + n Agent (mH + nλ) team mode evaluate the
We used a within-particippe performance of agents teaming up with different numbers of participants, where m + n = 5. This design allowed us to evaluate both objective performances as well as subjective preferences.
All participants read detailed guidelines and provided informed consent before the testing. Participants tested 20 matches for the IH + 4A team mode. High-level participants tested additional 10 matches for the 2/ + 3A and the 3H + 2A team modes, respectively. After each test, participants reported their preference over the agent teammates. For fair comparisons, participants were not told the type of their agent teammates. The MC-Base agent team was adopted as the fixed opponent for all tests. To eliminate the effects of collaboration between agents, we prohibit communication between agents. Thus the agents can only communicate with their human teammates. See Appendix [ for additional experimental details, including experimental design, result analysis, and ethical review.
Finding 1: Human-MCC team achieves the highest WR across team modes and human levels.
We first compare the human-agedn team objective performance metrics supported by the MCC agent and baselines, shown in Table D We see that the human-MCC team significantly outperforms all other human-agent teams across different team modes and human levels. This indicates that the MCC agent can generalize different levels and numbers of human teammates. Note that the SOTA agent can easily beat the high-level human players (aar 2019 Chen 20 So as the number of participants increases, the WRs of all human-agent teams decrease. Surprisingly, the WR increased significantly when the participants teamed up with the MCC agent. We suspect that the human-MCC team has also achieved effective communication and collaboration on macro-strated strate-ss To verify this, we count the Response Rates (RRs) of agents and participants to the meta-commands sent from their teammates, as shown in Table 2 We find that the RRs of the MCC agents to high-level participants (73.05%) and the high-level participants to the MCC agents (78.5% are close to the RR of high- participants themselves (74.91%) %).. This suggests that the CS is close to the value system of high-level humans. Besides, the RRs of participants to the MCC agents (73.43% and 78.5%) are higher than those of the MC-Rand agents (41 .07% and 35.69%). indicating that participants collaborated with the MCC agents more often and more effectively.
Table I:
The win rates (WRs) of different human- agent teams against the fixed opponent. Each human- agent team consists of different types (MC-Base, MC- Rand, and MCC) of agents and different levels (General and High) and numbers ( (IH, 2H, and 3H) of participants.
Table 2: The response rates (RRs) of Receiver to the meta-commands sent by Sender, including the RRs of agents to participants (H2A scenarios), the RRs of participants to agents (A2H scenarios), and the RRs of participants themselves.
IH + 4A
2H +1+3A 3A
3H 2A
Sender' Reocivr General
High
MCC
Agent),Teat Team
General
High
High
High
MC-Rand 41.07%
35.69%
34.03%
MC-Base
23%
42%
26%
8%
General
72.34%
61.17%
MC-Rand
5%
28%
18%
3%
High
74.91
73.05%
MCC
37%
54%
39%
18%
MCC
73.43%
78.50%
Finding 2: Participants prefer MCC over all baselines.
We then compare the subjective preference metrics, i.c., the Reasonableness of H2A, the Reason- ableness of A2H, and the Overall Preference, reported by participants over their agent teammates, as
8

---

Published as a conference paper at ICLR 2023
MC-Base
MC-Rend
MCC
Preference
3
2
General-leve
High-level
General-level
High-level
Ceneral-level
Highilevel
(a) Reasonableness H2A (b) Reasonableness of A2H
[c] Overall Preference
Figure 6: Participants' preference over their agent teammates. (a) Reasonableness of H2A: how well the agents respond to your meta-commands. (b) Reasonableness of A2H: how reasonable the meta-commands sent by agents (c) Overall Preference: your overall preference for the agent teammates. Participants scored (1: Terrible, Poor. Normal, Good, Perfect) these metries after each game test. Error bars represent 95% confidence intervals, calculated over games. See Appendix C.2.3 for detailed wording and scale descriptions.
shown in F Participants believed that the MCC agent responded more reasonably and gave the highest score the Reasonableness of the H2A metric Figure(]ja) Besides, participants also believed that the meta-commands sent by the MCC agent are more aligned with their own value system and rated the MCC agent much better than the MC-Rand agent in the Reasonableness of A2H metric (Figure b))) In general, participants were satisfied with the MCC agent over the other agents and gave the highest score in the Overall Preference metric (Figure((c)) The results of these subjective preference metrics are also consistent with the results of objective performance metrics.
4.4 COLLABORATIVE INTERPRETABILI ANALYSIS
To better understand how the MCC agents and humans can collaborate effectively and interpretably. We visualize the comparison CS and high-level participants' value systems on game scene with three meta-commands existing, shown in Figure 7 We see that the CS selects the meta-command for the two heroes in the red dashed box to collaborate, selects the meta-commant C for the two heroes in the purple dashed box collaborate, and selects the meta-command A for the remaining hero to execute alone. The CS selection results are consistent with the ranking results of high-level participants, confirming the effectiveness of the collaboration behaviors between the MCC agents and humans. Such collaboration enhances the human interpretability of agent behavior.
Q-Value
Average Rank of lege-level Participants
5
B
#
Figure 7: Case study on the value estimation of CS and average rank of high-level participants.
5 CONCLUSION
In this work, we proposed an efficient and interpretable Cota-Command Communication- framework, dubbed MCC, to achieve effective human-agent collaboration in MOBA games. To bridge the communication gap between humans and agents, we designed the Meta-Command common ground between humans and agents for bidirectional communication. To achieve effective collaboration, we constructed the Meta-Command Selector a value estimator for agents to select valuable nota-commands to collaborate with humans. Experimental results in Honor of Kings demonstrate that MCC significantly outperforms all baselines in both objective performance and subjective preferences, and it could generalize different levels and numbers of humans.

---

Published as a conference paper at ICLR 2023
REFERENCES
Lupu Andrei, Cui Brandon, Hu Hengyuan, and Foerster Jakob N. Trajectory diversity for zero-shot coordination. 139:7204-7213, 2021.
Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-ai coordination. Advances in Neural Information Processing Systems, 32, 2019.
Hongyu Chen. Tencent shows off artificial intelligence with honor of kings ai competition 2021. URL
http:///www.pprttoosuseeeeeeeoos |t http:///aeaaaaaoaoaaaoooooo///oe /Iechnology/2221/// /ttp:///22221/1/--------aaeeeeee awwp:
Herbert H Clark and Susan E Brennan. Grounding in communication. 1991.
Rémi Coulom. Whole-history rating: A bayesian rating system for players of time-varying strength. In Proceedings of the International Conference on Computers and Games, pp. 113-124, 2008.
Allan Dafoe, Edward Hughes, Yoram Bachrach, Tantum Collins, Kevin McKee, Joel Z Leibo, Kate Larson, and Thore Graepel. Open problems in cooperative ai. arXiv preprint arXiv:2012.08630, 2020.
Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. Tarmac: Targeted multi-agent communication. In Proceedings of the International Conference on Machine Learning, pp. 1538 1546, 2019.
Yuqing Du, Stas Tiomkin, Emre Kiciman, Daniel Polani, Pieter Abbeel, and Anea Dragan. Ave: Assistance via empowerment. Advances in Neural Information Processing Systems, 33:4560-4571, 2020.
Jakob Foerster, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agen reinforcement learning. Advances in Neural Information Processing Systems, 29, 2016.
Yiming Gao, Bei Shi, Xueying Du, Liang Wang, Guangwei Chen, Zhenjie Lian, Fuhao Qiu, Guoan Han, Weixuan Wang. Deheng Ye, et al. Learning diverse policies in moba games via macro macro-goals. Advances in Neural Information Processing Systems, 34, 2021.
Mohammad Ghavamzadeh and Sridhar Mahadevan. Learning communicate and act using hierar- chical reinforcement learning. Computer Science Department Faculty Publication Series, pp. 172, 2004.
Abhinav Gupta, Marc Lanetot, and Angeliki Lazaridou. Dynamic population-based meta-learning for multi-agent communication with natural language. Advances in Neural Information Processing Systems, 34:16899-16912 2021.
Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. "other-play" for zero-shot coordination. In Proceedings of the International Conference on Machine Learning, pp. 4319-4410, 2020.
Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojeiech M Czamecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, et al. Population based training 2017.
of neural networks. arXiv preprint
Max Jaderberg, Wojciech M Czarnecki, Iain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, et al. Human- level performance in 3D multiplayer games with population-based reinforcement learning. Science, 364(6443):859-865, 2019.
Jiechuan Jiang and Zongqing Lu. Learning attentional communication for multi-agent cooperation. Advances in Neural Information Processing Systems, 31, 2018.
Subbarao Kambhampati, Sarath Sreedharan, Mudit Verma, Yantian Zha, and Lin Guan. Symbols as a lingua franca for bridging human-aa chasm for explainable and advisable ai systems. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 2222-122677 2022.
10

---

Published as a conference paper at ICLR 2023
Uri Kartoun, Helman Stern, and Yacl Edan. A human-robot collaborative reinforcement learning algorithm. Journal of Intelligent Robotic Systems, 60(2):217-239, 2010.
Daewoo Kim, Sangwoo Moon, David Hostallero, Wan Ju Kang, Taeyoung Lee, Kyunghwan Son, and Yung Yi. Learning schedule communication in multi-agent reinforcement learning. arXiv preprint arXiv:1902.0f 025444 2019.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-agent cooperation and the emergence of (natural) language. arXiv preprint arXiv:1612.07182. 2016.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision, pp-
2980-2988, 2017.
Hanxiao Liu, Zihang Dai, David So, and Quoc V Le. Pay attention to mlps. Advances in Neural Information Processing Systems, 34:9204-9211 9215, 2021.
Quan Liu, Zhihao Liu, Wenjun Xu, Quan Tang, Zude Zhou, and Duc Truong Pham. Stuman-robo collaboration in disassembly for sustainable manufacturing. International Journal of Production Research, 57(12):4027-4044 4044, 2019.
Kevin McKee, Xuechunzi Bai, and Susan Fiske. Warmth and competence in human-agent cooperation. arXiv preprint arXiv:2201.13445 2022.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattic, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529 533, 2015.
Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi agentt populations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Sharmila Nair. Tencent's ai beats human players to win honour of kings mobile game, 2019, htw.://ttttrr..ooommmctcccccccce URL http:///wwww..tttsasee....ooooea
E039s hettsaaiabastts---maaaaaaaaeeeee wtwp:pppwwaaaraoooooooooooooooo-
mobi g-gamm)
OpenAl, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with large scale deep reinforcement learning. arXiv preprint arviv: 1tt.:::9/2.66000 12.06800 2019.
Peng Peng, Ying Wen, Yaodong Yang, Quan Yuan, Zhenkun Tang, Haitao Long, and Jun Wang. Multiagent iidireetiinalll-coorbiie coordinated nets: Emergence of human-level coordination in learning to play starcraft combat games. arXiv preprint 1703. 2017.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shi- mon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In Proceedings of International Conference Machine Learning, pp. 4295-4304, 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 17X7v:1707.06344 2017.
Ali Shafti, Jonas Tjomsland, William Dudley, and Aldo Faisal. weal-world human robott collabora- tive reinforcement learning. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 11161-11166 2020.
Victor do Nascimento Silva and Luiz Chaimowicz. Moba: new arena for game ai. arXiv preprint arXiv: /705. 1004. 2017.
11

---

Published as a conference paper at ICLR 2023
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):484 484 2016.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of Go without human knowledge. Nature, 550(7676):354 359, 2017.
Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar. Learning when to communicate at scale in multi-agent cooperative and competitive tasks. arXiv preprint 1rXiv::812.00755. 2018.
Ho Chit Siu, Jaime Peña, Edenna Chen, Yutai Zhou, Victor Lopez, Kyle Palko, Kimberlee Chang, and Ross Allen. Evaluation of human-ai teams for learned and rule-based agents hanabi. Advances in Neural Information Processing Systems, 34:16183-161999 2021.
Robert Stalnaker. Common ground. Linguistics and philosophy. 25(516):701-211, 2002.
DJ Strouse, Kevin McKee, Matt Botvinick, Edward Hughes, and Richard Everett. Collaborating with humans without human data. Advances in Neural Information Processing Systems, 34, 2021.
Sainbayar Sukhbaatar, Rob Fergus, et al. Learning multi-agent communication with backpropagation. Advances in Neural Information Processing Systems, 29, 2016.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czamecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in StarCraft using multi-agent reinforcement learning. Nature, 575(7782):500-35, 2019.
Rundong Wang, Xu He, Runsheng Yu, Wei Qiu, Bo An, and Zinovi Rabinovich. Learning effi- cient multi-agent communication: An information bottleneck approach. In Proceedings of the International Conference on Machine Learning, pp. 9908-999 2020.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Mare Lanctot, and Nando Freitas. Ducling network architectures for deep reinforcement learning. In Proceedings of the International Conference on Machine Learning, pp. 1995-2003, 2016.
Bin Wu. Hierarchical macro strategy model for moba game ai. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 1206-1213, 2019.
Deheng Ye, Guibin Chen, Wen Zhang, Sheng Chen, Bo Yuan, Bo Liu, Jia Chen, Zhao Liu, Fuhao Qiu, Hongsheng Yu, et al. Towards playing full moba games with deep reinforeement learning. Advances in Neural Information Processing Systems, 33:621-634 2020a.
Deheng Ye, Guibin Chen, Peilin Zhao, Fuhao Qiu, Bo Yuan, Wen Zhang, Sheng Chen, Mingfei Sun, Xiaoqian Li, Siqin Li, al. Supervised learning achieves human-level performance in moba games: A case study of honor of kings. IEEE Transactions on Neural Networks and Learning Systems, 2020b.
Deheng Ye, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang, Xipeng Wu, Qingwei Guo, al. Mastering complex control in moba games with deep reinforce- ment learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 6672-6679, 2020c.
12

---

Published as a conference paper at ICLR 2023
A ENVIRONMENT DETAILS
A.1 GAME INTRODUCTION
Honor of Kings is one of the most popular MOBA games worldwide. The gameplay divide players into two camps to compete on the same symmetrical map. Players of each camp compete for resources through online confrontation, team collaboration, etc., and finally win the game by destroying the enemy's crystal. The behaviors performed by players in the game can be divided into two categories: macro-strategies and micro-operations. Macro-strategy is long-distance scheduling collaborating with teammates for quick resource competition, such long long-distance support for teammates, collaborating to compete for monster resources, etc. Micro-operation is the real-tima behavior adopted by each player in various scenarios, such skill combo release, evading enemy skills, etc. Complicated game maps, diverse hero combinations, diverse equipment combinations, and diverse player tacties make MOBA games extremely complex and exploratory.
A.2 GAME ENVIRONMENT
Figure X shows the UI interface of Honor of Kings. For fair comparisons, all experiments in this paper were earried out using fixed released game engine version (Version 3.73 series) of Honor of Kings.
Main Hero
C,3
C.1
C.2
C.3
Figure 8: The UI interface of Honor of Kings. The hero controlled by the player is called Main Hero. The player controls the hero's movement through the bottom-left wheel (C.1) and releases the hero's skills through the bottom-righ buttons (C.2. C.3). The player can observe local view via the screen, observe global view via the top-left mini-map (A), and ohtain game states via the top-right dashboard (B).
A.3 IN-GAME SIGNALIGG SYSTEM
A.1
Player Come And Kil The Drodcn
C
A.2
B.1
A.1
Main Hero
B.2
B.3
Figure 9: The in-game signaling system of Honor of Kings. Players can send their macro-strategies by dragging signal buttons (A.2) to the corresponding locations (A.1) ) in the mini-map. The sent result is displayed as yellow circle (D). The buttons in C are the convenience signals representing attack, retreat, and assembly, respectively. Voice (B.2) and (B.1 and B.3) are two other forms communication.
13

---

Published as a conference paper at ICLR 2023
Figure demonstrates the in in-game signaling system of Honor of Kings. Players communicate and collaborate with teammates through the in in-game signaling system. In the Human-Agent Collaboration Test, humans send macro-strategies agents through signals like A in Figure g and these signals are displayed to teammates in the form of D. The MCC framework converts these explicit messages, i.e., signals, into meta-commands by the hand-crafted command converter function f and broadcasts them to all agent teammates, Moreover, the MCC framework can also convert the meta-commands sent by agents into signals by the inverse for and broadcast them all human teammates.
Voice (B.2) and text (B.1 and B.3) are two other forms of communication. the future, will consider introducing general ceta-co encoding model that can handle all forms of explicit messages (signals, voice, and text).
A.4
HERO POOL
Table 3 shows the full hero pool and 20 hero pool used in Experiments, Each match involves two camps playing against each other, and each camp consists of five randomly picked heroes.
Table 3: Hero pool used in Experiments.
Lian Po, Xiao Qiao, Zhao Yun. Mo 2z, Da Ji, Ying Zheag, Sun Shangxiang, Luban Qihao, Zhuang Zhou, Liu Clan Gao Jianli. Ke, Zhong Wuyan, Sun Bin, Bian Que, Bai Qi, Mi Yue. Ly Bu, Zhou Yu, Yuan Ge
Xia Hondun, Zhen Ji, Cao Cao, Dian Wei, Gongben Wacang, Li Bai, Make Bolun, Di Renjie, Da Mo, Xiang Yu Wa Zetion, Si Mayi, Lao Fazi, Guan Diao Chan, An Qila, Cheng Yaojin, Lu No. Jinng Ziya. Lio Bong
Han Xin. Wang Zhaojun. Lan Lingwang, Hua Mulan, Ai Lin. Zhang Liang. Buzhi Huowu, Nake Lolu, Ju Youjing. Ya Se Yoanfang. Yu Zhong Kui, Yang
Full Hero pool
Wukong. Niu Mo, Yang Jian, Ny Wa, Ne Zha, Ganjiang Moye, Ya Dianna, Cai Wenji. Taiyi Zhenren, Donghuang Taiyi, Gui Guzi, Zhu Geliang Da Qiao, Huang Zhong, Kai, Su Lie, Baili Xuance, Baili Shouyue, Yi Xing, Mong Qi, Gong Sunli, Shen Mengxi Xin, Jia Luo, Dun Shan, Sun Ce Zha Rajie, Shangguan Waner, Ma Chao, Dong Fangyan, Xi Shi, Meng Ya. Lahan Dashi, Pan Gu, Chang F, Meng Tian Jing. Guduo,
Liu Bei, Zhang
Chengii Sihan
Ming Shiyin, Pei Qinhu, Kaang Tie. Mi Laidi, Yao, Yun Zhongjun.
Lan, Sikong Zhen Erin. Yun ying. Jin Chan, Sang Qi
Jing. Pan Gu, Zhao Yun. Ju Youjing, Doughuang Talyi, Zhang Fei, Gui Guzi, Da Qiao, Sun Shangxiang, Luban Qihao, Chengji Sihan, Huany Zhong, Zhunng Zhou, Lian Liu Bang Zhong Wuyan, Yi Xing, Zhou Yu, Shi. Zhang Liung
20 Hero Pool
B FRAMEWORK DETAILS
B. INFRASTRUCTURE DESIGN
Inference Server
Fra
parameter sync
Network forward prediction
Sample management
Server with Env
RL L earner
self play
Policy network
agents
Ens
agents
Value network
Memory Pool
aggnnts
agents
with Reduce
Figure 10: The designed infrastructure.
Figure 10 shows the infrastructure of the training system (Ye al. 2020a), which consists of four key components: AI Server, Inference Server, RL Learner, and Memory Pool. The AI Server (the actor) covers the interaction logic between the agents and the environment. The Inference Server is used for the centralized batch inference the GPU side. The RL Learner (the learner) is distributed training environment for RL models. And the Memory Pool is used for storing the experience, implemented memory efffccettf circular queue.
Training complex game AI systems often require a large number of computing resources, such as AlphaGo Lee Sedol (280 GPUs), OpenAl Five Final (1920 GPUs), and AlphaStar Final (3072 TPUv3
14

---

Published as a conference paper at ICLR 2023
cores). We also hundreds of GPUs for training the agents. A worthy research direction is to improve resource utilization using fewer computing resources.
B.2
REWARD DESIGN
Table 4 demonstrates the details of the designed environment reward.
Table 4: The details of the environment reward.
Head
Reward Item
Weight
Type
Description
Gold
0.005
Dense
The gold gained.
Experience
experience gained.
Farming Related
Mana
0.05
Dense
The rate of mana (to the fourth power).
No-op
-0.00001
Dense
Stop and nothing.
Allack monster
Atlack monster.
Spurse
Kill enemy
Death
-1
Sparse
Being killed.
Assist
Sparsc
Assists.
KDA Related
Tyrant bull
1
Sparse
Get bull of killing tyrant, dark tyrant,
Overlord
1.5
Spurse
Get buff' killing overlord.
Expose invisible enemy
0.3
Sparse
Get visions of enemy heroes.
Last hit
0.2
Sparse
Last hitting an eacmy minion.
Damage Related
Health point
3
Dense
The bealth point the hero (to the fourth power).
Hurt hero
Spurse
Attack enemy herves.
Pushing Related
Attack turrets
Sparse
Attack turees..
Attack crystal
Sparse
Attack enemy home base.
Win/l ase Related
Destry home hase
2.5
Sparse
Destray enemy home hase.
B.3
FEATURE DESIGN
B.3.1
CEN
See Table
Table 5: Feature details of CEN.
Feature Class
Field
Description
Dimension
Type
1. Unit feature
Scalar
Includes beroes, minions, monsters, and turrets
3946
Current HP, mana, speed, level, gold, KDA, and magical attack and defense, etc.
Status
1562
(onc-hot, normalized float)
Heroes
Carrent courdinates
(normalized float)
Minions
Status
Position
Current HP, speed, visihility, killing income, etc.
Current 2D coordinates
920
(one hot, normalized float) (normalized float)
Status
Current HP, speed, visibility, killing income, etc.
728
(onc-hot, normalized float) (normalized float)
Monsters
Position
Current 2D coordinates
56
Status
Current HP. locked targets, attack speod, Current 2D conrdinates
540
(onc-hot, normalized float)
Turrets
Position
40
(normalized float)
2. In-game stats feature
Scalar
Real Real-time statistics of gume
104
game
(one-hot)
Camp
Types of two camps
I
(onc-hot)
Statio statisties
Alive heroes
Number of alive heroes two camps
(ooc-hot)
Kill
Number of Kill each camp (Segment representaaionn)
6
(one-hur hol)
Alive turrets
Number of alive turrets each camp
8
(one hot)
Alive heroes diff
Alive heroes difference between twoo camps
II
(one-hot)
Comparative statistics
Kill difference between two camps
(ooc-hot)
Alive turrets dilf
Alive turrets difference between two camps
6
(one-hot)
15

---

Published as a conference paper at ICLR 2023
B.3.2 MCCAN
See Table
Table Feature details MCCAN.
Feature Class
Field
Description
Dimension
Unit fenture
Scalar
heroes, minions, monsters, turrets
Current HP, mana, speed, level, gold, KDA, bull.
Status
bed states, orientation, visibility,
1842
(onc-hor, nocmalized float)
Position
Current 2D coordinates
20
(normalized float)
Heroes
Atrihate
and defense, magical attack and defense, etc.
main
not,
camp (leam), job, physical attack
1330
(one-hou normalized float)
Skill cool down time, usability, level,
Skills
2005
(one-hot, noemalized float)
range. bulf effects, bad effects, Current item lists
Item
(ane hol)
Status
Current HP, speed, visibility, killing income, etc.
1160
(one-hod normalized float) (normalized float)
Position
Current coordinates
Minions
Attribute
Camp (team)
(one-hot)
Type minions (melee creep, ranged creep.
Type
200
(onc-hot)
siege creep, super croep, etc.)
Status
Current HP, speed, visibility, killing income, etc.
(one-hot, normalized float) (normalized float)
Monsters
Position
Current 2D coordinates
56
Type
Type of monsters (normal. blue. red, lyrant, overlord, etc.)
168
(one-hot)
Status
Current HP, locked targets, attack speed, etc.
$20
(ane hot, normalized float)
Position
Current coordinates
(normalized float)
Type
Type of turrets (tower, high tower, crystal, etc.)
80
(onc-hot)
In in-game feature
Scalar
Real-time
of the game
Time
Current game time
5
(one hol)
Gold
Golds of camps
12
(normalized float)
Static statistics
Alive heroes
Number of alive heroes of two camps
10
(one-hot)
Kill
Kill number of camp (Sogment representation)
(one-hot)
Alive turrets
Number of alive turrets of two camps
8
(une hot)
Gold diff
Gold difference between camps (Segment representation)
5
(ane hol)
Alive heroes diff
Alive heroes difference between two camps
(one-hot)
Comparative statistics
Kill diff
Kill difference between camps
5
(onc-hot)
Alive turrets diff
Alive turrets difference between two camps
6
(one-hot)
3. Invisible opponent information
Scalar
Invisible information used for the value net
560
Opponent
Position
distancees,
(ncrmalized float)
Current 2D coordinates of all non player characters,
NPC
Position
440
(normalized float)
including minions, monsters, and turrets
Spatial feature
Spatial
image-like, extracted in channels for convolution
6x17x17
Region
Priential damage regions of ally and enemy skills Bullets ally and enemy skills
2x17x17
Skills
Bullet
2x17x17
Obstacles
Region
Forbiddea region for heroes to move
1x17x17
Bushes
Region
Bush region for heroes to hide
1x17x17
8. Meta- Command feature
Spatial
Flattened Meta-Command
144
(onc-hot)
16

---

Published as a conference paper at ICLR 2023
B.3.3 CS
See Table 7
Table 7:
Feature details of CS.
Class
Field
Description
Dimension
Type
1. Unit feature
Scalar
Includes heroes, minions, monsters, and turrets
3946
Current HP, mana, speed, level, gold, KDA, and magical atlack and defense, etc.
Status
1562
(one-bot, normalized float)
Heroes
Poxition
Current 2D coordinates
20
(normalized float)
Status
speed, visibility, killing incomee
normalized
Minions
Position
Current 2D coordinates
80
(normalized float)
Status
Current HP, speed, visibility, killing income,
728
(one-hor, normalized float) (Bormalized float)
Monsters
Position
Current 2D coordinates
56
Status
Current HP. locked targets, attack speed, etc.
540
(one-bot, normalized float)
Position
Current 2D coordinates
40
(normalized float)
2. In-game stats feature
Scalar
Real-time statistics of the game
104
Time
Current time
57
hot)
Camp
Types of two camps
(one-hat)
Static statistics
Alive heroes
Number alive heroes of camps
10
(onc-hot)
Kill
Kill numher of each camp
6
nate-hot
Alive turrets
Number alive tirrets two camps
(one-bot)
Alive heroes diff
Alive heroes di fifference between two camps
I1
(one- naee-ha)
Comparative statistics
Kill diff
Kill difference between camps
(one-hot)
Alive turrets diff
Alive turrets difference between two camps
(one hot)
3. Invisible opponent information
Scalar
Invisible information used for the value net
560
Opponent
Position
coordinates, distances,
(normalized float)
NPC
Position
including monsters, and turrets
Current 2D coordinates of non-player characters,
440
(normalized float)
Metn-Command
Spatial
image- like,
convolution
1x12x12
Meta-Commands
Spatial
All reseived Meta-Commaadss in the team
5x12x12
(one-bot)
B.4 AGENT ACTION
Table slsows the action space of agents.
Table 8: The action space of agents.
Action
Detail
Description
Illegal action
Placeholder.
None action
Executing nothing stopping continuous action.
Move
Moving to a certain direction determinedd by move and move y.
Normal Attack
Executing normal attack enemy unit.
Skilll
Executing the first skill.
Skill2
Executing
skill.
What
Skill3
Executing the third skill.
Skill4
Exccuting fourth skill (only heroes have Skill4), An additional skill choosing before the gume beyins (10 choose).
Summoner ahility
Return home(Recall)
Returning to spring, should be continuuslly executed. Some can euable additional skill player's Blood recovering continouusly in 10s, can be disturbed.
skill
Restore
Collaborative
given by special ally
Move X
The x-axis offiet of moving direction.
Move Y
The y-axis offict of moving dircction.
How
Skill X
The x-axis offset of skill.
The y
Who
Target unit
The game unit(s) chosen to attack.
17

---

Published as a conference paper at ICLR 2023
B.5
NETWORK ARCHITECTURE
B.5.1
CEN
Figure I1 Ihowws the detailed model structure of CEN. The CEN network extracts game stats features and serval unit features from the observation o. Each unit feature is encoded by shared MLP layers to obtain serval unit embeddings, and the game stats features are encoded by MLP layers obtain the game stats embedding. Finally, we concatenate the unit embeddings and the game stats embedding. and output the probability distribution of the meta- contaadcmmaass... The outputted meta-command indicates the macro-strategy for future T time steps.
Game Stats Feature
MLPs
Meta Command
Unit Feature
Sample
Menszers
Rule
Turnets
Shared
Figure 11: The CEN model structure.
B.5.2 MCCAN
shows the detailed model structure of MCCAN. The MCCAN predicts a sequence of actions for each agent based on its observation and the meta-command sampled from the Top-k Softmax distribution of CEN. The observations are processed by deep LSTM, which maintains memory among steps. We the target attention mechanism to improve the model prediction accuracy, and we design the action mask module to climinate unnecessary actions for efficient exploration. To manage the uncertain value of state-action in the game, introduce the multi-heat value estimation (Ye et al.] (020a) into the MCCAN by grouping the extrinsic rewards in Table 4 value head and estimate using separate value network.
Figure 12
We also treat the intrinsic rewards Besides, we introduce a value mixer module (Rashid et al. 2018) to model team value to improve the accuracy of the value estimation. All value networks consist of FC layer with LSTM outputs as input and output a single value, respectively. Finally, following Ye et al. (2020a) and Gao et al. (202T), we adopt hicrarchical heads of actions, including three parts: 1) What action take; 2) who to target; 3) how to act.
Network Parameter Details. We develop channels of spatial features read from the game engine with resolution 6* 17*17, and then use 5*5 and 3*3 CNN to sequentially extract features. The LSTM unit sizes are 1024 and the time steps are 16. The k value the CEN sampling is set to 20.
Gome State Feature
Seature
Herot
Actiona
Now
Oftitxx
Mive Allzk
Sputial Posture
Eebbad EroNcs
1 Gunedattt Pto
Muln stout
Comupi Taland
NAY
Wirdss Fdlite
CEN
Mota Cormond
Opponentiido
Figure 12: The MCCAN model structure.
B.5.3 CS
Figure 1 shows the detailed model structure of CS. For each agent, the CS predicts the value Q C. m. or probability ग Tmm o. C) each meta-command m based on its observations o and all received meta-commands C. First, all received meta-commands (m from human and from
18

---

Published as a conference paper at ICLR 2023
the CEN) is reshaped 2D image (12*12*1). Then, related information from meta-commandss i.e., z =CNN(m") and 2m =CNN(m). Besides, integrated into map set embedding
shared CNN to extract the region-
the map embeddings of all received meta-commands by max-pooling, i.e., ims = -Max-poolinge =m ). After that, we use the Gating Mechanism to fuse the map set embedding and the state embedding 2, MM))(() of the observation information For the Gating Mechanism, we use the state embedding and map set embedding 2ma to calculate the attention each other, that is, the gating Ims =MLP(zo) for 7ms and the gating 9o =MLP(2m. for 20. Gating will be used to retain attentional information, as well as fused them, i.e. 2 concat( 9o + Zot 2ms 9ms + 2ms]. Finally, the fused embedding 2 and map embeds { 2m will be used as key ky and query {qm {m respectively, and input the Target Attention module to predict the state-action value (Q-value) or the probability. We also use the fused embedding to estimate the state value (V-value) from observations o and all meta-commands C.
Observation
Meta-Commande
[mn m
CNN Feature Extraction
m
Shared CNNs
ZiL
State Embed
Map Set Embed
Max Poaling
H
Map Embed
Map Embed
Zm
Shared MLPs
Ims
Map Gate
State Gate
9o
Gating
Target Attention
query
4m
query
Mechanism
kr
Fused Embed
MLP
key
MLP
@ Co or mmm"
(o.
(o, C, m) or (m)o,
Figure 13: The CS model structure.
B.6 DETAILS AND ANALYSIS COMPONENTS
B.6.1 META-COMMAND
Command Converter Function fce The fcr extracts the corresponding location and event informa- tion from human messages. Specifically, the game engine converts rough messages from the in-game signaling system into predefined protocol buffers (PB), and then the MCC framework uses the foo to extract the location and event information from the received PB to construct the corresponding meta-command.
Command Extraction Function fee The fer directly extracts the agent's location from the current state (the position feature Table 6), which used obtain the intrinsic rewards by calculating the "distance" from the agent to the currently executed meta-command.
B.6.2 CEN
Training Data. We extract meta-commands from expert game replay authorized by the game provider, which consist of high-level (top 1% player) license game data without identity information. The input features CEN are shown Table 5 The game replay consists multiple frames, and the information of each frame is shown in Figure g We divide the location L of meta-commands in the map into 144 grids. And we set the event to be all units (103 in total) in the game. For setting Tme counted the player's completion time for meta-commands from expert game replay, and the
19

---

Published as a conference paper at ICLR 2023
results shown in Figure. [4 We can that 80% neeaaco can be completed within the time of 20 seconds in Honor of Kings. Thus, T set to 300 time steps (20 seconds).
100
Cumulative Percentage (%)
20
Frequency
Frequency (%)
15
Cumulative Percentage
66
10
40
5
0
10
20
30
40
50
Completion Time (s)
Figure 14: Time statistics of humans completing mcta-commands in real games.
Given a state 8 in the trajectory, we first extract the observation or for each hero. Then, we use hand-crafted command extraction function f to extract the meta-command m, = (st+rmm)) corresponding the future state mt+T By setting up labels in this way, we expect the CEN (mmm)) to learn the mapping from the observation to its corresponding meta-command my.
The detailed training data extraction process follows:
First, extract the trajectory (5o: ................................ St. St..... St+Tme, SN) from the game replay, where N is the total number frames.
Second, randomly sample some frames {t)t {0, .,..,,,.. N]) from the trajectory т.
Third, extract feature o, from state s, for each frame {tlt {0, (0,1....,N))
Fourth, extract the location L, and the event E, from the state ST4The in frame t + Tme
Fifth, use L, and Eg to construct the label m, =< L,, E,, T >.
Finally, < 0,, my > formed into a pair sample in the training data.
Optimization Objective. After obtaining the dataset {< o, }}. train the CEN T (mlo) via supervised learning (SL). Due to the imbalance of samples at different locations of the meta- commands, we use the focal loss (Lin et al. 2017) alleviate this problem. Thus, the optimization objective is:
SL (0) = Eo.M [-am(1 - no (o)) log(=s(o)) - (1- (1)) (o) log(1 =T(())]
-
where 0.75 is the balanced weighting factor for positive class (m 1) and 2 is the tunable focusing parameter. Adam (Kingma & Ba 2014) is adopted as the optimizer with an initial learning rate of 0.0001. Especially, We compute the focal loss for and E, respectively.
Experimental Results. Figure 15 shows the meta-command distributions the initial CEN, the converged CEN, and high-level humans. We that the meta-commands predicted by the CEN gradually converge from chaos to the meta-commands with important positions. The Kullback-Leibler (KL) divergence of the meta-command distribution between the CEN and high-level humans decreases from 4.96 to 0,44 as training converges. The distribution of the converged CEN in Figure [5(b) is close to the distribution of high-level humans in Figure 15 suggesting that the CEN simulate in real games.
the generation of human
B.6.3 MCCAN
Training Environment. We use a similar training environment as the WuKong agent and the OpenAI Five agent, i.e., the agent interacts with the environment in self-play manner. Specifically, for
20

---

Published as a conference paper at ICLR 2023
2013
500
6004
(
(b)
(c)
Figure 15: The meta-command distributions of CEN and high-level humans. (a) The meta neta-command distribution of the initial CEN. (b) The neta-command distribution of the converged CEN. (c) The meta- command distribution of high-level humans.
the MCCAN training, each of the 10 agents always executes its own meta-command generated by the CEN. And for each agent, every T'm time steps, the pre-trained CEN Te (s(m))) generates meta-command my based on observation og- The meta-command my then kept for T' time steps and sent to the MCCAN continuously. During the interval [t, TTc), the MCCAN predicts sequence actions based on o, and m, for the agent to perform.
Optimization Objective. The MCCAN is trained with the goal of achieving a higher completion rate for the meta-commands generated by the pre- pre-trained CEN while ensuring that the win rate not reduced. To achieve this, introduce extrinsic rewards (including individual and team rewards) and intrinsic rewards
int (Sc, M, s(+1) = -1 me| - -1500 (st+1) - mz
where / extracts the agent's 's location from state 8,, and l /ee (x²) - mel is the distance between the agent's location and the meta-command''s ' location in time step Intuitively, the intrinsic rewards adopted to guide the agent to the location L of the meta-command and stay at L to do some event E. The extrinsic rewards adopted to guide the agent perform optimal actions reach and do the optimal event at Overall, the optimization objective maximizing the expectation over extrinsic and intrinsic discounted total rewards G, Rug-da
Di= 7 + E 7 where
d (s) lim;++00 P (s: so, π the probability when following т for steps from 8o- We use to weigh the intrinsic rewards and the extrinsic rewards. The experimental results about the influence a on the performance the MCCAN are shown in Figure 16
Training Process. The MCCAN is trained by fine-tuning pre-trained WuKong model Ye et al. 2020a) conditioned on the model is the State-Or Ofe-----Thhrete (SOTA) model in Honor of Kings, which can easily beat the high-level human players
sampled from the pre-trained CEN. Note that the WuKong
We also modified the Dual-clip PPO algorithm (Ye al. 2020a)
introduce the meta-command m into the policy To (arlo mc) and the advantage estimation A, = A(a,, ot, m.). The Dual-clip PPO algorithm introduces another clipping parameter e to construct lower bound for rx(0)
mo) when A, < 0 and rz (0) > 0. Thus, the policy loss is:
Te m.)
(7) E,, m (cA, min(clip(ri (0), -,,1,11 A),,,
where 7 is the original clip parameter in PPO. And the multi-head value loss is:
LV (())) = E..l
Σ (G Vi Vo (ot, mc)], Veotal
Σ w.Va
headu
heads
where wa is the weight of the k-th head and V/ (or, m.) is the k-th value.
'Wukong Al beats human players win Honour Of Kings mobile game,
https::/ww.tthstttet
m. n..m//tech/tech-neeee http:/chheeaeaaaaa1/////////aaee
hanour-of-kingg mobi mo-iee-gaaeeeeeeeeeeeeeeem-----e
'Honor Kings Wukong AT defeated the human team,
http:://ww..porttassiieee
21

---

Published as a conference paper at ICLR 2023
70.00%
50.00%
a-0
Win Rate
40.00%
am4
30.00%
Q--8
20.00%
a=16
10.00%
0.00%
20.00%
40.00%
60.00%
80.00%
100.00%
Completion Rate
Figure 16: The win rate and completion rate of MCCAN with different a. The opponent is the pre-trained action network, i.c., MCCAN with o=0.
Experimental Results. We conducted experiments explore the influence of the extrinsic and intrinsic reward trade-off parameter a on the performance of MCCAN. The win rate and completion rate results are shown in Figure I6 As a increases, the completion rate of MCCAN gradually increases, and the win rate of MCCAN first increases and then decreases rapidly. Notably, we can train an agent with completion rate close to 100% by increasing a, but this will significantly reduce the win rate because the meta-command executed is not necessarily optimal and may result in the death of agents.
When = 16, the completion rate of the trained agent for meta-commands is 82%, which is close to the completion rate of humans (80%). And the win rate of the trained agent against the SOTA agents Ye et al. 2020a Gao al. 2021 is close to 50%. Thus, we finally set = 16.
B.6.4 CS
Additional Results in Agent-Only Collaboration Testing. To eliminate the transitivity of WR, we added additional experiments compare the MCC agent and baseline agents to the WuKong (SOTA) agent for each 600 matches Test Environment. The WRs (PI versus P2) are shown in Table 9 We that the effective CS mechanism in the MCC agent enables the agents to communicate and collaborate effectively, which improves the WR compared to the WuKong agent (SOTA).
Table 9: The WRs of the MCC agent and haseline agents against the WuKong agent (P1 versus P2).
P2IP1
MC-Basc
MC-Rand
MC-Rule
MCC
WuKong
49.2%
18.5%
39.5%
57.8%
Ablation Studies. We further investigate the influence of different components on the performance of CS, including CNN feature extraction with the gating mechanism (w/o CNN-GM), target attention module (w/o TA), and PPO optimization algorithm (MCC-PPO). We conduct ablation studies in Test with a 20 hero pool. In practical games, meta-commands with adjacent regions often have similar intentions and values. Thus the response rate of the agent to adjacent meta-commands should be close as possible. Besides, the higher the agent's response rate meta-commands, the more collaborative the agent's 's behaviors. Thus we expect the response rate of CS to be as high as possible. Generally, we expect CS's Response Rate (RR) be as high as possible while ensuring that the Win Rate (WR) maintains.
Figure a) demonstrates the WR of different CS ablation versions during the training process, and Figure 17 b) shows the converged WR-RR results. We see that after ablating the TA module, the WR and RR CS reduce significantly, indicating that the TA module can improve the accuracy of CS to meta-commands. Besides, after ablating the CNN-GM module, the RR of CS is most affected, which is reduced by 20%. It indicates that without the CNN-GN module, the value estimation of CS adjacent meta-commands is not accurate enough. resulting in missing some highly valuable commands. We notice that the MCC and MCC- ACC-PPO both metrics are close, confirming the versatility of the CS model structure.
22

---

Published as a conference paper at ICLR 2023
70.0%
70.0%
69.0%
60.0%
66.0%
40.0%
64.0%
30.0%
MDC
62.0%
MCC-PPO
w/o CNN- GM
MCC
20.0%
60.0%
MC--Radd
wo GNN-GM
10.0%
Rule
wo
Training Time (h)
Response Rate (RR)
(a)
(b)
Figure 17: Results ablation studies. (a) The training curves different CS ablation versions. (b) The
converged WR-RR results of different CS ablation versions. The shadow indicates the standard deviation.
C DETAILS OF HUMAN-AGENT COLLABORATION TEST
C.I ETHICAL REVIEW
The ethics committee of third-party organization, Tencent (the parent company of Honor of Kings), conducted ethical review of our project. They reviewed our experimental procedures and risk avoidance methods (see Appendix C.1.2). They believed that our project complics with the "New Generation of AI Ethics Code of the country which the participants belonged (China), they approved our study. In addition, all participants consented to the experiment and provided informed consent (see Appendix C.1.1) for the study.
C.1.1 INFORMED CONSENT
All participants were told the following experiment guidelines before testing:
This experiment is to study human-agent collaboration technology in MOBA games.
Your identity information will not be disclosed to anyone.
All game statistics are only used for academic research.
You will be invited into matches where your opponents and teammates are agents.
Your goal is to win the game as much as possible by collaborating with agent teammates.
You can communicate and collaborate with agent teammates through the in-game signaling system.
Agent teammates will also send you signals representing their macro-strategies, and you can judge whether execute them based on your value system.
After each test, you can report your preference over the agent teammates.
Each game lasts 10-20 minutes.
You may voluntarily choose whether to take the test. You can terminate the test at any time if you feel unwell during the test.
After all tests are complete, you may voluntarily fill out debrief questionnaire to tell us your open-ended feedback on the experiment.
At any time, you want to delete your data, you can contact the game provider directly to delete
If participants volunteer to take the test, they will first provide written informed consent, then we will provide them with the equipment and game account, and the test will begin.
https: https::/www.de www.qui dance. com/n ews/ ch na 1.sues wwweeegnneeaiiioo on- eethi caccede
CChina MOST issues New Generation of AT Ethics Code,
23

---

Published as a conference paper at ICLR 2023
C.1.2 POTENTIAL PARTICIPANT RISKS
First, we analyze the risks of this experiment the participants. The potential participant risks of the experiment mainly include the leakage of identity information and the time cost. And we have taken series of measures minimize these risks.
Identity Information. A series of measures have been taken to avoid this risk:
All participants will be recruited with the help of a third party (the game provider of Honor of Kings), and we do not have access to participants' identities.
We make a risk statement for participants and sign an identity information confidentiality agreement under the supervision of a third party.
We only unidentifiable game statistics in research, which are obtained from third parties.
Special equipment and game accounts are provided to the participants to prevent equipment and account information leakage.
The identity information of all participants is not disclosed to the public.
Time Cost. We will pay participants to compensate for their time costs. Participants receive $5 at the end of each game lest, and the winner will receive an additional S2. Each game test takes approximately 10 to 20 minutes, and participants can get about an average of $20 an hour.
C.2 EXPERIMENTAL DETAILS
C.2.1 PARTICIPANT DETAILS
We contacted the game provider and got a test authorization. The game provider helped us recruit 30 experienced participants with personal information stripped, including 15 high-level (top1 %) and 15 general-level (top30%) participants. All participants have more than three years of experience in Honor of Kings and promise to be familiar with all mechanies in the game, including the in-game signaling system in Figure
And special equipment and game accounts are provided to each participant prevent equipment and account information leakage. The game statistics we collect are only for experimental purposes and are not disclosed to the public.
C.2.2 EXPERIMENTAL DESIGN
We used a within-participge design: m Human + Agent (mH + nÅ) team mode to evaluate the performance of agents teaming up with different numbers of participants, where m + n Each participant is asked to randomly team up with three different types of agents, including the MC-Base agents, the MC-Rand agents, and the MCC agents. For fair comparisons, participants were not told the type of their agent teammates. The MC-Base agent team was adopted the fixed opponent for all tests. Participants tested 20 matches for the 1H + 4A team mode. High-level participants tested additional 10 matches for the 2H/ 3A and the 3// 2A team modes, respectively. After each game test, participants reported their preference over the agent teammates.
We prohibit communication between agents to eliminate the effects of collaboration. Thus the agents can only communicate with their human teammates. In each game test, humans can send the converted meta-commands whenever they think their macro-strategies to make the agent behave like humans (at most one human sends his/her meta-command at a time step), we restricted agents from sending their meta coeeaddsmmanss. i.e., only one agent sends a valuable meta-command to human teammates from the agents' current meta-commands. Specifically, this process consists of several steps: (1) The MCC framework randomly chooses human teammate (note that human and agent observations are shared); (2) The MCC framework uses his/her observation and all agents' contaddmmaads as the input of the CS, and obtained the estimated values these meta-commands; (3) The MCC framework selects the meta-command with the highest value; (4) The MCC framework selects the corresponding agent to send the meta-command. The above process is executed at interval of 20 seconds.
important. Furthermore,
In addition, as mentioned in Ye et al. (2020a) Gao et al. (2021), the response time of agents usually set to 193ms, including observation delay (133ms) and response delay (60ms). The average APM of
24

---

Published as a conference paper at ICLR 2023
agents and top e-sport players are usually comparable (80.5 and 80.3, respectively). To make test results more accurate, we adjusted the agents' capability to match high-level humans' performance by increasing the observation delay (from 133ms to 200ms) and response delay (from 60ms to 120 ms).
C.2.3 PREFERENCE DESCRIPTION
After each test, participants gave scores several subjective preference metrics to evaluate their agent leammates, including the Reasonableness of H2A (how well agents respond to the mela-commands sent by participants), the Reasonableness of A2H (how reasonable the meta-commands sent by agents are), and the Overall Preference for agent teammates.
For each metric, provide detailed problem description and a description of the reference scale for the score. Participants rated their agent teammates based on how well their subjective feelings matched the descriptions in the test. The different metries described follows:
For the Reasonableness of H2A, "Do you think the agent teammates respond reasonably your commands? Please evaluate the reasonableness according to the following scales."
1) Terrible: No response, or totally unreasonable.
2) Poor:
Little response, mostly unreasonable.
3) Normal: Response, but some unreasonable.
4) Good:
Response, mostly reasonable.
5) Perfect:
Response, and perfect.
For the Reasonableness A2H, "Do you think the commands sent by the agent teammates are reasonable? Please evaluate the reasonableness according to the following scales. Note that if you don't receive any commands, please ignore this question."
1) Terrible: Totally unreasonable.
2) Poor: Low reasonable.
3) Normal: Some reasonable.
4) Good: Mostly reasonable.
5) Perfect: Totally reasonable.
For the Overall Preference, "What is your overall preference for the agent teammates collaborating with you? Please rate the following words according your subjective feelings: 1) Terrible; 2)
Poor; 3) Normal; 4) Good; 5) Perfect".
C.2.4 ADDITIONAL SUBJECTIVE PREFERENCE RESULTS
Detailed subjective preference statistics are presented Table Compared with no collaboration (MC-Base) and random collaboration (MC-Rand), the participants preferred reasonable collaboration (MCC).
Reasonableness of H2A. Participants prefer MC-Rand MC-Base, suggesting that they expect the agent to respond more to their commands. Nonetheless, the score of MCC is much higher than MC-Rand, indicating that participants prefer their commands to get reasonable rather than incorrect responses.
Reasonableness of A2HI Participants express significant preference for MCC over MC-Rand, demonstrating that they agree with MCC's commands and are more willing to collaborate. The results are consistent with Figure 7 where participants believe the commands sent from MCC align more with their own value system. Note that the non-collaborative setting prohibits MC-Base from sending any commands, so we made no statisties.
Overall Preference. Participants are satisfied with the MCC agent over other agents and give the highest score. By comparing MC-Base and MCC, we can observe that human-agent collaboration based meta-command communication can bring a better impression of the human-agent team in the MOBA game. However, while MC-Rand is higher than MC-Base in the Reasonableness of H2A metric, it is lower than MC-Base in the Overall Preference metric. Therefore, collaboration is important but not as necessary as winning. This metric further confirms the reasonableness of MCC in human-agent collaboration.
25

---

Published as a conference paper at ICLR 2023
Table 10: The subjective preference results of all participants in the Human-Agent Collaboration Test, including mean scores variances (in parentheses).
Participant Preference Metrics (from terrible to perfect, I~5)
Type of Agent
Human Teammate
MC-Base
MC-Rand
MCC
General-level
2.3 (0.38)
2.7 (0.24)
4.0 (0.60)
Reasonableness of H2A
High-level
2.2 (0.21)
2.5 (0.41)
4.1 (0.55)
Reasonableness of A2H General-level
1.9 (0.35)
4.3 (0.31)
High-level
1.7 (0.24)
4.4 (0.35)
General-level
2.7 (0.41)
1.3 (0.27)
4.3 (0.40)
Overall Preference
High-level
2.5 (0.21)
1.2 (0.17)
4.5 (0.41)
D DISCUSSION
Limitations and future work. First, the training process of the MCC agent consumes vast computing resources like other SOTA MOBA agents. Thus, we will optimize the training process of existing MOBA agents, aiming to lower the threshold for researchers to study and reproduce work in MOBA games. Second, the meta-commands we proposed are generic to MOBA games and cannot be directly extended to other types of games, such as FPS and Massively Multiplayer Online (MMO). We will design a more general meta-command representation, such as natural language, and extend the MCC framework to other games. Third, we will apply the MCC agents to the friendly bots in the teaching mode of Honor of Kings. All in all, we would hope that this work can not only offer new ideas to researchers but also bring a better human-agent collaboration experience in practical applications.
26


Reviews:
[{"review_id": "wwTlmMww_s", "reviewer": "Reviewer_Lfxb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work investigates human-agent collaboration in Multiplayer Online Battle Arena (MOBA) games and presents a human interpretable communication framework through which humans and agents can communicate their respective strategies during the game. It introduces the concept of meta commands which acts as the common ground for communication of macro-strategies between the humans and the agents. Further, the framework includes a meta command selector which estimates the value of the meta commands and selects a reasonable meta command to follow. The results show that using such a Meta Command Communication (MCC) framework improves the overall team performance and beats the current SOTA in MOBA games. \n\nOn a broader level, MOBA games present an interesting set of challenges for Game AI, one of which is human-agent collaboration. This paper addresses this challenge and shows that having an interpretable communication framework between humans and agents can achieve effective human-agent collaboration. They evaluate the MCC agent on agent-only environments and on online experiments with humans of different levels. ", "summary_of_the_review": "Overall, this work provides a novel instantiation of a human-interpretable common ground in MOBA games. The results show that such a framework improves human-agent collaboration and thereby the overall team performance. The framework on the other hand seems to be specific to the game used as the test-bed, Honor of Kings. But the promising objective and subjective evaluation of the framework could potentially inspire people in the AI community to have a human-interpretable communication framework for human-agent collaboration. Hence, this paper can be accepted to ICLR 2023.", "strength_and_weaknesses": "Strengths/Interesting aspects\n1. Communication through a human-interpretable common ground: The MCC framework establishes a human interpretable communication framework where irrespective of what the agent’s internal representations are, the communication happens in a symbolic form which is interpretable to humans in the loop. This also inclines with having symbols as the lingua-franca [1] with the meta-commands being the symbolic information that is used as a common ground in human-agent interaction. The winrates from the online experiments show the effectiveness of the MCC framework with different types of agents and levels.\n\n2. Human subject study for the performance of MCC teams: Even though the results show that as more humans are present in the team the win rate reduces against an all-agent SOTA method, in comparison to the other methods, human teams with MCC agents improve the overall team performance. Further, the response rates are also high in MCC-human teams.\n\n3. Comparison of Command selector and high-level participant’s value systems - Although it doesn’t give a complete picture, it provides a preliminary insight into how the CS meta command selections are consistent with the ranking results\n\nWeaknesses/Clarifications\n\n1. Meta command is defined as a three element tuple where the event E (what to do ) is also part of it. The paper states the following “To achieve the former, a hand-crafted command converter function f cc is used to generate L of meta-commands by extracting the location from explicit messages, such as signals, sent by humans. To achieve the latter, we use a Command Encoding Network (CEN) πϕ(m|o) to generate L of meta-commands.” It is not clear as to how the event E in a meta command is incorporated (if needed) in the meta command execution if in the command conversion stage the output is just the locations extracted/generated from the explicit human commands/observations. Clarification on this aspect should be provided in the paper.\n\n2. As much as it extends the SOTA in Honor of Kings, the common ground here is game-specific. The meta commands seem to be specifically designed for Honor of Kings. It is not clear if the meta commands or techniques are generic to other MOBA games either. \n\n\n[1] Kambhampati, Subbarao, et al. \"Symbols as a lingua franca for bridging human-ai chasm for explainable and advisable ai systems.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 36. No. 11. 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "benKbn4Idp", "reviewer": "Reviewer_peTd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "I found this paper fairly hard to understand, so I will write a summary that’s fairly different from the author’s presentation that would have been easier for me to understand, and the authors can tell me if my summary is inaccurate.\n\nThere have been several agents that play multiplayer online battle arena (MOBA) games, such as OpenAI Five (Dota), AlphaStar (StarCraft), and the WuKong model (Honor of Kings). These agents play _competitively_ against humans: 5 copies of the agent control the 5 heroes on the team. However, we would also like to have AI agents _collaborate_ with humans: we can test this in MOBA games by having the team of 5 heroes be controlled by a mixture of humans and agents.\n\nA key challenge in human-AI collaboration (HAC) is how to facilitate communication between the humans and the AIs. The core contribution of this paper is to propose such a method and demonstrate its utility in Honor of Kings.\n\nFirst, we design a structured message format that humans can understand, which is a triple <L, E, T>. L is the location, which is one of 144 grid squares on the map. E is the task to perform, such as “kill the dragon”; I assume it is selected from a predesigned set of possible actions, though the paper does not say. T is an integer that specifies a number of timesteps in which to complete the action. The overall message <L, E, T> is to be interpreted as “in the next T timesteps, go to L and complete action E”. Only these instruction messages are supported; there is no other kind of communication.\n\nThen, for a single timestep, our agents work as follows:\n\n1. From their input observation o (and history, perhaps), generate a candidate message <L, E, T> to send to the other team players (whether AI or human). This is done by a Command Encoding Network (CEN) π_ϕ(m|o) (where m is the generated message / meta-commands). \n2. Collect messages { <L, E, T> } from other agents. Select which of the messages to follow from amongst these messages and the message you generated yourself. This is done by a Meta-Command Selector (CS) π_ω(o, C) (where C is the set of received messages / meta-commands).\n3. Select an action to execute based on the chosen message and the input observation. This is done by the Meta-Command Conditioned Action Network (MCCAN) π_θ(a|o, m), where m is the message / meta-command chosen in the previous step, and a is the action.\n\nThe human players play as normal, but are only shown one of the generated meta-commands. This is done by the CS but I am not clear how (see Q5 in my list of questions).\n\nTraining for these 3 models works as follows:\n\n1. The CEN π_ϕ(m|o) is trained using supervised learning on an expert dataset that is automatically extracted from a dataset of expert gameplay using a hand-crafted function.\n2. I don’t understand how the MCCAN is trained (see questions in the Strengths and Weaknesses section).\n3. The CS is trained after the CEN and MCCAN are already trained. We simply use the full agents in a self-play setting, and train the CS using deep RL. Instead of using the environment reward function over the entire game, we instead use only the rewards during the execution of the current meta-command, and we weight the rewards obtained after reaching the location L twice as highly as the rewards obtained prior. (We also use multi-head value functions that predict different aspects of the reward, e.g. gold obtained vs. enemies killed, as done in prior work.)\n\nThe resulting MCC agent is compared against various ablations that show the value of communicating meta-commands between agents in an all-AI setting. The agent and various baselines are also paired with humans; the MCC agent achieves the highest winrate and is also most preferred by the human expert players.", "summary_of_the_review": "I want to like this paper: it tackles an important problem, it proposes a high-level design that is relatively simple and has a clear reason why it would help, and it shows strong experimental results in a complex, high-dimensional setting.\n\nHowever, as the paper is currently written, (a) it was quite difficult for me to understand even the high-level design, though I did eventually figure it out, and (b) even after that I cannot understand the details of how the method actually works. This is a key requirement for a scientific paper, and so unfortunately I would recommend against acceptance.\n\n(Due to the lack of clarity I am also reducing my score for correctness, as there were some claims in the paper that I could not evaluate because the necessary information wasn’t present.)\n\nI also have some other qualms (weaknesses 2 and 3 above) but they are minor in comparison and I would recommend acceptance if those were the only weaknesses.\n\nUPDATE: After the author response many of the necessary details have been added, and so I am raising my score. However, I still find the paper relatively hard to understand.", "strength_and_weaknesses": "Strengths:\n1. The paper tackles an important and significant problem domain – interpretable communication with humans in the service of human-AI collaboration.\n2. The experimental results are strong, showing both improvements on win rate for the game, as well as improvements in human preference over which agents to play alongside.\n\nWeaknesses:\n1. The paper is hard to understand; despite poring over the details I would not know how to reproduce the paper’s results. See the list of questions below.\n2. The authors claim that the MC-Base baseline (where meta-commands are not communicated) can be considered SOTA since it has the same level of performance as the SOTA WuKong agent, citing an appendix which is not present. It is not clear why this claim is true – did the authors evaluate the win rate between these agents? If so, why not also evaluate the win rate between MCC and WuKong, if you have access to the WuKong model? Even if you have evaluated the win rate of MC-Base vs WuKong, it is still good to compare MCC to WuKong, as chances of winning are not necessarily transitive (i.e. even if MCC beats MC-Base and MC-Base beats WuKong, it could still be that MCC loses to WuKong, though it is not likely).\n3. The proposed method requires some domain-specific hardcoding (e.g. to define the message protocol, to define the set of possible events E, etc).\n\nNotes on clarity and questions for the authors:\n\n1. Which parts of my summary are incorrect?\n\n2. In a message <L, E, T>, does the action E have to be selected from some predefined list of actions? If not, what are the legal values of E?\n\n3. When describing the message <L, E, T>, in the example you suggest that T could be “until the dragon is killed”. However, in the experiments it sounds like T is always set to be some integer. Is it possible to have T be something like “until the dragon is killed” or does it have to be an integer? If it is possible, how is that implemented?\n\n4. Is it possible for an agent to choose not to send any message?\n\n5. In Figure 4(c), it appears that the AI agents must coordinate to send a single meta-command to the humans, which is done by using the meta-command selector π_ω(o, C). What do you use for the input o in this case, given that there are multiple AI observations but the CS only takes one observation?\n\n6. I would assume that the CS should choose which meta-command to execute based on the probability of executing the meta-command successfully and the value of success for the rest of the game. This means that when training the CS, we need to give it a reward signal that captures how valuable it is to successfully complete the meta-command, which you would get from the environment rewards *after* the meta-command is complete. However, the CS optimization objective explicitly considers rewards only up to the end of the meta-command, rather than considering rewards after the completion of the meta-command. Why do you choose this instead of considering rewards up to the end of the game (or up to some fixed horizon length after the end of the meta-command)?\n\n7. For the MCCAN training, the authors say\n\n> We adopt an intrinsic reward r^{int}_t(s_t, m_t, s_{t+1}) $= |f^{ce}(s_t) − m_t| − |f^{ce}(s_{t+1}) − m_t|$ to guide the process of executing the meta-command $m_t$, where $f^{ce}$ is a hand-crafted command extraction function.\n\nI am confused about what is happening here. Presumably this intrinsic reward somehow guides the policy to take actions that would achieve the meta-command $m_t$, since nothing else in the MCCAN training gives it that incentive. But how does it do this? My best guess is:\n\n- We create $f^{ce}$, a hand-crafted function that “guesses” which meta-command is being executed based on the current state / observation\n- We want the agent to take actions that cause $f^{ce}$ to make the correct “guess”, since that corresponds to taking actions that complete the meta-command, so we want the agent to minimize the distance between the guess from $f^{ce}$ and the actual meta-command at the end of the meta-command time interval, and which suggests a reward of $- |f^{ce}(s_{T^{mc}}) - m|$\n- We apply potential shaping [1] over the time length of the meta-command to get the intrinsic reward above. (Though if this were the case there should be a discount factor in the reward.)\n\nIs this correct? If so, why don’t you have the discount factor? Also, wouldn’t this incentivize the agent to complete the meta-command on the last timestep $T^{mc}$, rather than completing it as fast as possible?\n\n8. For the MCCAN training, what is the training environment? The CS has not yet been trained so it cannot be used. Are you using a self-play setting where each of the 10 agents always executes its own meta-command generated by the CEN?\n\nOther notes:\n\nSection 4.4 (Collaborative Interpretability Analysis) would be improved by some quantitative metrics: for example, you could ask players to choose key moments during the game, and then report how often the Q-value ordering agrees with the expert player rank ordering.\n\n\n[1] Ng, Andrew Y., Daishi Harada, and Stuart Russell. \"Policy invariance under reward transformations: Theory and application to reward shaping.\" Icml. Vol. 99. 1999.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OR6RK2RWxq", "reviewer": "Reviewer_Usav", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a framework for human-agent collaboration in MOBA games. The framework consists in two main parts: i) a protocol of commands that human and agents use to communicate with each other, and that can be expressed as a tuple of \"what\" the sender asks the receiver to do, for \"how long\" has to be done, and \"where\" the action has to happen; and ii) a hierarchical RL approach the agents follow to learn what commands to send, and to which commands they should respond - the approach is hierarchical because the learning process is done sequentially in three independent stages: first, message generation is leant by cloning the behaviour of expert data; second, the low level actions required to accomplish a command are learnt by fine tuning a pretrained model, such that the agents learn to complete the tasks described by the commands without decreasing the win rate; third, the agents learn to select which message among all the messages received (e.g., one per human player). Experiments show the proposed framework results in increased win rates with respect to agents that do not communicate with other players (i.e., they only execute their own commands), and that human players prefer the proposed agents with respect to agents that do not communicate.\n", "summary_of_the_review": "**Edit after the authors rebuttal**\n\nThe authors have addressed my concerns as discussed in the rebuttal thread. Although the proposed approach is MOBA specific, I think it might inspire extensions to other game and real scenarios - see my ethical concern below. Moreover, the results in terms of winning rate and subjective preference are impressive. I am raising my score to acceptance. However, as noted in the rebuttal thread, I agree with other reviewers there are some issues to be discussed.\n\n**Original review**\n\nThis is an interesting paper that tackles an important problem in a rather sophisticated manner. It has good ideas and can likely inspire future research. \n\nThe results are illustrative but not strong. Fortunately, win rates of 5A teams and preference over playing with other human players are strong baselines that would shed light on the performance of the proposed solution.\n\nIn my opinion, a major contribution of this work is to actually make this thing work, which includes multiple machine learning engineering decisions and engineering expertise. However, although some high-level ideas are given, many crucial details are missing, which could prevent reproducing the results, and hence reduce the potential impact of the work. \n\nIf the authors tackle these two concerns: stronger baselines and reproducibility, I will increase my recommendation score.\n\n", "strength_and_weaknesses": "### Strengths\n+ The proposed framework is conceptually simple and might be extended to other scenarios (beyond MOBA or even beyond gaming) where humans and agents communicate at the strategic level.\n+ The whole design is full of interesting design choices that will likely inspire future research, like the reward function to fine tuning the low-level actions network, the command representation as a grid that it can be learnt with a CNN, or the command selector architecture that combines the gating mechanism with attention\n+ The paper establishes a baseline for a challenging and relevant problem.\n\n### Weaknesses\n- Many implementation details are missing (see comments on reproducibility below).\n- Clarity could be improved (see comments below) ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%