Towards Effective and Interpretable Human-AI Collaboration in MOBA Games
Authors: Yiming Gao
Feiyu Liu
Liang Wang
Zhenjie Lian
Weixuan Wang
Siqin Li
Xianliang Wang
Xianhan Zeng
Rundong Wang
jiawei wang
QIANG FU
Yang Wei
Lanxiao Huang
Wei Liu
Venue: NeurIPS 2022 Submitted (2022)
Keywords: Human-AI Collaboration
Game Playing
Deep Reinforcement Learning

Abstract: MOBA games, e.g., Dota2 and Honor of Kings, have been actively used as the testbed for the recent AI research on games, and various AI systems have been developed at the human level so far. However, these AI systems merely focus on how to compete with humans, less exploring how to collaborate with humans. To this end, this paper makes the first attempt to investigate human-AI collaboration in MOBA games. In this paper, we propose to enable humans and agents to collaborate through explicit communications by designing an efficient and interpretable Meta-Command Communication-based framework, dubbed MCC, for accomplishing effective human-AI collaboration in MOBA games. The MCC framework consists of two pivotal modules: 1) an interpretable communication protocol, i.e., the Meta-Command, to bridge the communication gap between humans and agents; 2) a meta-command value estimation model, i.e., the Meta-Command Selector, to select a valuable meta-command for each agent to achieve effective human-AI collaboration. Experimental results in Honor of Kings demonstrate that MCC agents can collaborate reasonably well with human teammates and even generalize to collaborate with different levels and numbers of human teammates. Videos are available at https://sites.google.com/view/mcc-demo.

Towards Effective and Interpretable Human-AI Collaboration in MOBA Games
Anonymous Author(s)
Affiliation
Address
email
Abstract
MOBA games, e.g., Dota2 and Honor of Kings, have been actively used as the testbed for the recent AI research on games, and various Al systems have been developed at the human level so far. However, these Al systems merely focus on how to compete with humans, less exploring how to collaborate with humans. To this end, this paper makes the first attempt to investigate human-Al collaboration in MOBA games. In this paper, we propose enable humans and agents to collaborate through explicit communications by designing efficient and interpretable Meta- Command Communication- based framework, dubbed MCC, for accomplishing effective human-Al collaboration in MOBA games. The MCC framework consists two pivotal modules: 1) an interpretable communication protocol, i.e., the Meta-Cone Command, bridge the communication gap between humans and agents; 2) meta comta-d value estimation model, i.c., the Coma-Co Selector, select valuable meta-command for each agent to achieve effective human-Al collaboration. Experimental results in Honor of Kings demonstrate that MCC agents can collaborate reasonably well with human teammates and even generalize to collaborate with different levels and numbers of human teammates. Videos are available
2
6
10
12
14
16
17
Introduction
18
19 Games, as the microcosm of real-world problems, have been widely used as testbeds evaluate
20 the performance of Artificial Intelligence (AI) techniques for decades. Recently, many researchers
21 focus on developing various human-level AI systems for complex games, such as board games like
22 Go
(27 28), First-Person Shooting (FPS) games like ViZDoom [14]. Real-Time Strategy (RTS) 23 games like StarCraft (34, and Multi-player Online Battle Arena (MOBA) games like Dota [22].
24 However, these Al systems focus merely on how to compete instead of collaborating with humans,
25
leaving Human-Al Collaboration (HAC) in complex environments still to be investigated.
26 In this paper, we study the HAC problem complex MOBA games, which characterized by multi- agent cooperation and competition mechanisms. long time horizons, enormous state-action spaces (1020000), ), and imperfect information [22 26 38) HAC requires the agent to collaborate reasonably 29 with various human teammates. One straightforward approach is improve the generalization of agents, that is, to collaborate with an enough diverse population of teammates during training. There
27
28
30
31 are some Population- -asee Training (PBT) based algorithms and learning systems 2
32
31 41 proposed to improve the generalization of agents in video games by constructing diverse
33
population of agents in different ways. However, this approach requires vast amount of diverse data and massive computing resources, posing big computational obstacle for complex MOBA games.
34
Human team success in MOBA games requires not only subtle individual micro-operations but also excellent communications and collaborations among teammates on macro-strategies, i.c., long-term intentions 8 377. Consequently, focus on enabling humans and agents collaborate through
38
37
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

State & Micro-Operation
Macro-Strategy
Explicit Message
Meta-Cormaand
Cleas Up ZopeLave Minios
Where Couse Tn The Tipelaw
What: Cless Up Mislos
DotaZ
long: Mirkeo
How long
L, E, T
Honar of Kings
What
Where: Come To The Desgre
What: Artet The Oraggn
Coee
long: The Dragon Killed
(h)
Figure 1: MOBA game-related introduction. (a) Key elements of MOBA games such as Dota 2, Honor of Kings, etc. Players observe from the state of the environment, make micro-operations and macro-strulegies decisions, and collaborate through explicit messages (e.g..text and signals). (b) Example of collaboration via meta-commands. The Come And Kill The Dragon is more valuable for humans A and and agent D to collaborate, while the Clean Up Top-Lane Minions is more valuable for human and agent E to collaborate.
38
explicit communications and propose an efficient and interpretable based human-Al collaboration framework, dubbed MCC, to solve the HAC problem in MOBA a
Communication-
39
40
games. First, we design an interpretable communication protocol, i.e., the general representation of macro- strategiss to bridge the communication gap between agents and humans. Both macro-stratede strategies sent by humans and messages outputted by agents can be converted into unified meta-commands (see Figure D. Second, following Gao et al. 8 construct hierarchical model that includes the command encoding network (macro-strategy layer) and the meta-command conditioned action network (micro-action layer), used for agents to generate and exccute meta-commands. respectively. Third, we propose a meta-command value estimation model, ie., the Ceta-Commaan Selector, to select the optimal meta-command for each agent to execute. The training process the MCC framework consists of three phases. We first train the command encoding network to learn the distribution of meta-commands sent from humans. Afterward, we train the meta-command conditioned action network to ensure that the agent has the near- huma-h completion rate for commands. Finally, we train the meta-command selector ensure that the agent can select a valuable meta-command to achieve effective collaboration. We train and evaluate the agent in Honor of Kings 5v5 mode with full hero pool (over 100 heroes). Experimental results demonstrate the effectiveness the MCC framework. In general, our contributions are as follows:
41
42
43
44
46
48
50
52
53
54
To the best of our knowledge, we the first investigate the HAC problem in MOBA games. We propose an efficient and interpretable Meta-Commant Communication based framework dubbed MCC achieve effective human-AI collaboration in MOBA games.
56
57
58
We design an interpretable communication protocol to bridge the communication gap between humans and agents. In addition, propose meta-co value estimation model to select valuable meta-comea meta--oommnnd for each agent to achieve effective human-Al collaboration.
60
61
We introduce the training process of the MCC framework in typical MOBA game Honor of Kings and evaluate it in practical human-Al game tests. Experimental results show that MCC agents can reasonably collaborate with different levels and numbers of human teammates.
62
63
64
2
Related Work
85
2.1
MOBA Games AI Research
66
MOBA games, such as Dota 2 and Honor of Kings, have attracted much attention from Al researchers due to their multi-agent cooperative and competitive mechanics, long time horizons, partial observa- tion, and enormous state-action spaces
67
68
Recently, OpenAl et al. 22 introduced an Al system named OpenAI-Five that defeated professional players in Dota 2 5v5 mode under the condition of limited heroes. Ye et al. [38 39 40 proposed another learning system named WuKong that can
22 38
69
71 surpass top e-sport players in Honor of Kings with a full hero pool. Further, Wu (37 and Gao et
al. g proposed learning systems that enable the agent to learn human strategies achieve policy
73 diversity. However, these Al systems can only defeat human players but cannot collaborate well due
to the communication gap between agents and humans, see Table I] In most real-world scenarios, the
75 excellent collaboration between humans and agents may make more sense than the competition.

---

78
2.2
Human-AI Collaboration
PBT is considered one way to solve the HAC problem 4 Most PBT-based methods are devoted to training an agent which can be compatible with unseen partners by maintaining a population of agents with diverse behaviors in different ways I 2 [O
78
19 20 30), These methods have 80 been validated on both objective and subjective metrics in video games Overcooked and Capture the Flag and card game Hanabi. However, the main difference between these games and MOBA games
81
82 is that these games do not provide explicit communication mechanics for collaboration on macro-
89 strategies between agents and humans. Besides, MOBA AI agents usually need to learn billions of network parameters to cope with the enormous state-action spaces (1 (1020000) 38 which constitutes prohibitive computational burden for learning. As a more realistic topic of HAC, human-robot interaction in manufacturing also attracts much attention 13 17 25). However, these studies are mainly limited collaboration between robot and human through one-way communication, i.e., humans give robots orders. Therefore, there is still a large room to study RL with the participation of humans. This work can be a stepping stone for broader wolld applications.
84
85
86
87
88
89
90
2.3
Multi-Agent Communication
91
Communication is often used in Multi-Agent Reinforcement Learning (MARL) to improve inter- MARL methods are mainly focused on exploring communication protocols between multiple agents with an end-to-end RL framework [5] 7 9 23
agent collaboration. Most communication-
95
94
Jiang and Lu [12] and Kim el al. [15] proposed to model the value of multi-agent communication for effective collaboration. Unfortunately, these methods all model communications
29
2 36)-
95
96 in latent space without considering human-AI interactions, making it less interpretable to humans. Instead, we focus on enabling humans and agents to collaborate through explicit communications.
97
3 Auman-AI Collaboration
98
99
We consider an interpretable communicative human-Al collaboration task, which can be ex- tended from Partially Observable Markov Decision Process (POMDP) and formulated as tuple < N, H, S, A A A HI O, M, r, 7, >, where N and H represent the numbers of agents and humans, respectively. S is the space of global states. AN {AA Ji=1....N and A H {A")==,..., denote the spaces of actions of N agents and H humans, respectively. O = [O.) denotes the space of observations of N agents and H humans. M represents the space of interpretable messages, that is, the Meta-Commands in the MCC framework. P : S x AN x A + S and
101
102
103
104
105
S x A x A + IR denote the shared state transition probability function and reward function of N agents, respectively. Note that, r includes both individual reward and team reward. 7 E [0. 1) denotes the discount factor. For each agent in state 8t E S. it receives an observation 0j O and a selected message of E M. and then outputs an action of Ia (of, ct) E A₁ and a new message mi+1 no (of) E M, where Te and Th are action network and message encoding net-
106
107
108
109
110
)
work, respectively. message selector cj = Th (of, Ct) is introduced to receive a message set
111
C. = (m) C M from all agents and humans and select the optimal one to execute.
112
113
We divide the HAC problem in MOBA games into the Human-to-Al (H2A) and the -uman (A2H) scenarios. The H2A Scenario: Humans send macro- strategies as messages to agent teammates,
114
macro-strategies
115
and agents combine them with their own messages select the optimal one based on their message selector to execute, achieving effective collaboration with humans. The A2H Scenario: Agents send messages as macro-strategies to human teammates, and humans combine them with their own macro-strategies to select the optimal one based on their own value systems to execute, achieving effective collaboration with agents. The goal of both tasks is that agents and humans communicate macro- strrtesst with pre-defined communication protocols, and then select valuable strro-strategies for effective collaboration to win the game.
116
117
118
119
120
121
4
Meta-Command Communication-Basel Framework
122
123
In this section, we present the proposed MCC framework in detail. We first briefly describe three key stages of the MCC framework (see Section 4. Then we introduce the two pivotal modules in the
124
125
MCC framework: 1) an interpretable communication protocol, i.e., the Meta-Command, as general representation of macro-strategies to bridge the communication gap between agents and humans (see
127
Section 4.2): 2) meta-command value estimation model, i.e., the Ceta-Command Selector, to select
valuable meta-command for each agent to achieve effective HAC in MOBA games(see Section 4.3)
3

---

Step:
(1 Conversion Stage
T+n
Command
mf
Converter
Soxt
sanal
Command Encoding
Network.T
(mm m.) Meta Command Communication Stage
Cy = [mp..m) mc)
Meta Commard
So ectorr
[i]] Human- All Collaboration Stage
*.*
Mets Command Conditioee Artion Network Ey
G2
Environment
Figure 2: The temporal process of the MCC framework. For each communication step and T'), MCC first (I) converts messages from humans and agents into meta-commands, then (II) selects the optimal meta-command for each agent to execute, and (III) finally predicts a sequence of actions for each agent to perform. The selected meta-command is retained and executed for n time steps. This process is repeated until the end of a game.
4.1
Overview
130
The flow of the MCC framework can be divided into three stages: the cema-co conversion stage. the meta-command communication stage, and the human-Al Al collaboration stage, as plotted in Figure At the Meta-Commant Conversion Stage, the MCC framework converts the maaro-strategies sent by humans and the messages outputted by the command encoding network of agents into unified meta commands and then broadcasts them to all agents and humans. At the Meta-Command Communication Stage, the MCC framework uses the meta-command selector to estimate the values of all received meta-commands and select the optimal one for each agent to execute. Note that humans also select the optimal meta-command based on their value systems. At the Human-AT Collaboration Stage, the MCC framework adopts the meta-command conditioned action network to predict a sequence of actions for each agent to perform based on its selected meta-command. For each game, humans and agents have to collaborate multiple times, that is, they need to perform the above three stages multiple times to win the game.
131
135
135
137
139
141
142
4.2
Meta-Co
143
In MOBA games, we propose that macro-strategy consists of three components: where go, what to do, and how long. For example, a macro-strategy can be Come And Kill The Dragon, which consists of Come To The Dragon (where to go). Attack The Dragon (what do), and Until The Dragon Is Killed (how long). Thus, we propose general representation of macro- strot-sstategiss i.c.. the Meta-Commann as an interpretable communication protocol to bridge the communication gap between agents and humans.
144
145
148
147
148
149
Meta- Command Definition. We formulate the Peeta-oomaand as a tuple < L, Em >, as shown in Figure [[), where L is the Location to go, E' is the Event to do after reaching L, and T is the Time Limit for executing the meta-command. Among them, L is the key to the meta-commant which contains the intention of the macro-strategy. E can be thought of human micro-operation. which is implemented through a pre-trained micro-action network Ty in the MCC framework. T can be set to how long it normally takes human to complete macro-strategy in MOBA games, usually 20 seconds corresponds to 80% completion rate for meta- commmaddssdd,,, see Appendix 1.1.
151
153
155
156
Meta- Mela-Command Conversion. To realize interpretable human-AI communication, we convert the explicit messages from humans and the implicit messages from agents into unified command.s..ds. To achieve the former, a hand-crafted command converter function f is used to generate of meta- commands by extracting the location from explicit messages, such as text and signals, sent by humans.
157
158
159
160
To achieve the latter, we use a Command Encoding Network (CEN) n (m]o) to generate of meta- commands. The CEN is trained via supervised learning (SL. with the goal of learning the distribution of meta-commands sent from humans, as shown in Figure 3 (T). The training dataset {< o, m >} obtained by extracting the observation and its corresponding meta-command m from expert data.
182

---

Command Encoding Network (D6N)
(II)
Mute Cormmnn Conditiondd Action Network
(MCCAN)
Observation
Mets- Command
Jangli
Ty(iiinn))
(i
ralned
14" (Pp
CNN Feature
Eerrction
(p))(t))
MUP
Shaind Chin
Expert Ny.
.xy(ts/ss.m))
State Entied
May Emiked
Mai Eritee
Map Eribed
(11))
Coremnnd Selector
Maari MLl
State Cate
softmax C. (.))
Mechaniam
0/4
mnf"
Target Attantion
+20
a/2
(o, G,,m)")
pro Daineddddd
Rd.
pre-traned
V(A)
Vase
Oorx
(a)
(h)
Figure 3:
The training process and model structure of MCC. (a) The training process is divided into three phases: we first (I) train the CEN via supervised learning (SL), then (II) train the MCCAN via goal-conditioned RI, and finally (III) train the CS via RI. Among them, the dashed box represents the frozen model. (b) The detailed CS model structure, including CNN feature extraction, gating mechanism, target attention module, etc.
After converting all messages into unified meta-commands, the MCC framework broadcasts them to all agents and humans. Then, agents and humans receive an identical meta-command candidate set. 165
166
Execution. After receiving meta-command candidate set, agents se- lect one meta-command from it to execute. We adopt a Meta-Command Conditioned Ac- tion Network (MCCAN) Ty ( m) for agents to perform actions based on the selected meta- command, as shown in Figure (II).... The MCCAN is trained via goal-conditioned RL with the goal of achieving near-human completion rate for the meta-commands generated by the pre-trained CEN while ensuring that the win rate is not reduced. We adopt an intrinsic reward rt (st, my St41) = =()) - my - (s(()))) me| to guide the process of executing the meta- command m,, where foe is hand-crafted command extraction function. We train the MCCAN with the objective of maximizing the expectation over extrinsic and intrinsic discounted total re-
167
188
189
170
172
173
174
[++] where is trade-off parameter and d (s) lim P (st (s1=s s Su, π) is the probability when following for steps from 8g-
wards G. E Egndy aawso E I'riti Σド= )
178
177
After training the CEN and MCCAN, we can achieve HAC by simply setting an agent to randomly select meta-command derived from humans to execute. However, such collaboration is non- intelligent and can even be a disaster for game victory because agents have no mechanism to model the values of meta-commands and cannot choose the optimal meta-command to execute.
179
180
181
While humans usually choose the optimal one based on their value systems for achieving effective collaboration to win the game. Thus, we further propose meta-command value estimation model to
182
183
select a valuable meta-command for each agent, as described in the following subsection.
184
4.3
Meta-Command Selector
185
In real-world MOBA games, the same macro-strategy often has different values for different humans different situations. For example, a macro-strategy can be Come And Kill The Dragon, as shown in F It is more valuable for humans A and B to collaborate. While another macro-strategy can be Clean Up Top-Lane Minions, which is more valuable for human rather than humans and B. Therefore, it is important to select the most valuable meta-command from the received meta-command candidate set to achieve effective human-Al AI collaboration. We propose a meta-command value estimation model, i.e., the Meta-Command Selector (CS) (o, C'), to estimate the values of all current meta- commands and select the most valuable one for each agent to execute.
186
187
188
190
192
193
CS Optimization Objective. Typically, the execution of meta-command involves reaching location L and doing event E, of which the latter is more important to the value of the meta-command. For example, for the meta-command Come And Kill The Dragon, if Kill The Dragon event cannot
184
195
196
be done within Te time steps, then it is pointless to Come To The Dragon. Thus. the long-term reward me for executing a meta-command can be expressed as the total rewards within T time
197
steps by interacting with the environment: Iime ET' ==0 lits + B Σ}=r= Titj where TL < Tm is the time for reaching L and 3 > is trade-off parameter. Note that the reward function
5

---

200
includes both individual rewards and team rewards. The optimization objective of CS is to select the optimal meta nema-co mi (o,, C₁) for each agent to maximize the expected discounted
201
meta-command execution return G Eg-day m am aa Zi=o Time ( where Oz E O,
202
C is the meta-command candidate set in state 8,, and T'me E [0, 1) is the discount factor.
204
CS Training Process. We construct a self-play training environment for CS where agents can send messages to each other. Specifically, three tricks in Figure (II) are adopted to increase the sample efficiency while ensuring efficient exploration. First, each sent meta-command m is sampled with the argmax rule from the results predicted by the pre-trained CEN. Second, each agent sends its mema-co with a probability p every Tmo time steps. Finally, each agent selects the final meta-command sampled with the softmax rule from its CS output results and hands it over to the pre-trained MCCAN for execution. We use the multi-head value mechanism [38] to model the value of the meta-command execution, and the corresponding value loss can be formulated as:
205
206
207
208
210
L (w) Es.c
Σ
|G VO, C)l
(S
houdy
212 213
where (S, C)) is the value of the k-th head. For DQN-based methods 21 351, I (w) Es.C..
the Q loss is:
[|GGeoaal (S, M)] GLotal
Σ hends
21d
where wi is the weight of the k-th head and G is the Temporal Difference (TD) estimated value error Reg + Tme Vi (S", C²) CS, 216
CS Model Structure. We design a general network structure for CS towards MOBA games, as shown in Figure b]). In MOBA games, the meta-commands corresponding to adjacent regions usually have similar values. Thus, divide the meta-commands in the map into grids, common location description for MOBA games, and use the shared Convolutional Neural Network (CNN) to extract region-related information from the meta-commands to improve the generalization of CS to adjacent meta- wneaa-ommaans. Besides, we use the gating mechanism [8] to fuse the map embedding of all received meta-commands and the state embedding of the observation information. Finally, to directly construct the relationship between the observation information and each meta-comman introduce target attention module, where the query is the fused embedding i and the key is the map embedding mí' of each meta-command. The fused embedding / is used as the input into the subsequent Q network Q(h, m²) and V network V(h) network of CS. In this way, the Q network can also be easily converted to the policy network π (m]] m²). Thus, the CS model structure can be casily applied to most popular RL algorithms, such as PPO 24 DQN [1], etc.
218
220
222
224
we
226
228
229
Experiments
230
231
We evaluate the proposed MCC framework in Honor of Kings, one of the most popular MOBA games worldwide, which has been actively used as the testbed for recent game Al research 88 37 We conduct all experiments in Honor of Kings 5v5 mode with full hero pool (over 100 heroes), except ablation studies with 20 hero pool for exploring the influence of different model components more sufficiently and efficiently.
232
233
234
235
236
5.1
Experimental Setup
237
5.1.1
Training Setup
238
Due to the complexity of MOBA games and limited resources, instead of training jointly, train the CEN, MCCAN, and CS sequentially. For all model training, the location I, of meta-commands in the
239
240
map is divided into 144 grids. The time limit T for the meta-command execution is set to 20s.
241
CEN Training Settings. We train the CEN via SL until it converges for 26 hours using 8 NVIDIA
242
P40 GPUs The batch size of each GPU set to 512. Adam [6] is adopted as the optimizer with an initial learning rate of 0.0001.
244
MCCAN Training Settings. We train the MCCAN by finetuning a pre-trained micro-action net- work (38), the state-of-the-art (SOTA) model in Honor of Kings, which is conditioned the meta-command sampled from the pre-trained CEN. The MCCAN is trained until converges for 48
246
Detailed parameter settings for all training processes be found in the Appendix.

---

Train Environment
Agent-Only Environment
Human-Al Game Test
Test
Broadkast Channel
Broadoast
Broadrast Channel
Broadcast Channel
LUGHT
VS
Simulated
VS
VS
Human
VS
LBU
LOFH
BB
Broadcast Channel
Opponents
Fixed Opponents
Figure 4: Communication environments in the experiment. The orange arrows indicate sending meta- commands, and the blue arrows indicate receiving meta comaacommands The dashed line denotes sending meta- commands with probability p.
NC-Caae MC Sandd MC-Ffude
MDC
MOC
Elo Score
80
59.3
41.5
VCC Gase
76.8
71
41.3
80%
1121
1143
70%
1062
1003
1076
MC--andd
24.5
14.3
23.2
42.5
15.5
40%
862
44%
813
MC Rule-
40.7
75.5
34
48%
57.5
21.8
40%
800
700
MCC-
58.5
85.7
66
30%
84.5
78.2
38%
Test
Test
(a) Win rate (row vs. col) in Test (b) Win rate (row vs. col) in Test II.
(c) Elo scores of different agents.
Figure 5: AI performance in the testing environments. (a) and (b) show the win rate maps of different agents who play against each other. (c) shows the final Elo scores of these agents.
247
hours using a physical computer cluster with 63,000 CPUs and 560 NVIDIA V100 GPUs. The batch size of each GPU is set to 256.
249
CS Training Settings. We train the CS via self-play until it converges for 24 hours using a physical computer cluster with 70,000 CPUs and 680 NVIDIA V100 GPUs. The batch size of each GPU is set to 256. The parameter set to 2. Each agent sends meta-command with probability p of 0.8 and an interval T'me of 20s, as shown in Figure
250
252
253
5.1.2
Evaluating Setup
Our primary concern is whether the agents trained with the MCC framework, briefly called the MCC agents, can collaborate with humans well. However, evaluating agents with humans is expensive, which is not conducive model selection and iteration. Therefore, we built two agent-only testing
255
257
environments: Test and Test II, for the model selection and iteration process, as shown in Figure We also evaluate the MCC agents in practical human-AI game tests to examine the performance of collaborating with humans, as shown in Figure c)..
258
259
260
Compared Agents. We compare the MCC agent with three different types of agents: the MC-Base agent (agent only executes its own meta-command without communication), the MC-Rand agent (agent randomly selects meta-command to execute), and the MC-Rule agent (agent selects the nearest meta-command to execute). We adopt the MC-Base agent-only team as the opponent for all tests. Note that the MC-Base agent-only team has the ability of the SOTA and is more stable than human-only team. Results are reported over five random seeds.
262
254
286
Agent-Only Environmental Settings. Test is the most complex environment where all agent teammates send and receive meta-commands simultaneously with an interval of 20s. Test I is used to evaluate the agents' performance under extremely complex situations as well as in ablation studies. Test II simple environment simulate practical game scenarios, where most one human sends his macro- saarrestrattgy at time step. Thus, in Test II, only one agent is randomly selected send its meta-command with an interval of 20s, and the other agents only receive meta-commands.
287
288
269
270
271
Human-AI Game Testing Settings. We had different types of agents team up with different levels and numbers of humans, including 15 strong humans (top1 %) and 15 average humans (top30%), in m AI Human mode, where m + 5. For fair comparisons, each tester was not told the
273
7

---

275
type of agent teammates. To eliminate the effects of collaboration between agents, we prohibit agents from receiving meta-commands from their agent teammates, and the agent can only receive
278
277
meta-commands from humans. In each game test, humans can send the converted meta-commands
278
whenever they think their macro-strategies are important. To make the agent behave like humans
279
(at most one human sends his macro-strategy at a time step), we restrict agents from sending their
280
mnta-commmadss We randomly choose a human teammate and use his observation and all agents'
meta-commands as the CS input and select the final output of CS to send with an interval of 20s.
282
5.2
Results in Agent-Only Environment
283
5.2.1
AI Performance
The Kullback-Leibler (KL) divergence of the meta-command distribution between the CEN and
285
humans decreased from 4.96 to 0.44 as training converges. The MCCAN is trained with the parameter
286
equal to 16. The win rate of the trained agent against the SOTA agent 8 38 is close to 50%. The average completion rates of the trained agent and humans for meta-commands are 82% and 80%, respectively. Notably, we can train an agent with higher completion rate by increasing a, but
287
288
289
this will significantly reduce the win rate because the meta-commant executed is not necessarily optimal and may result in the death of agents. We put the detailed experimental results of the CEN and MCCAN in the Appendix A.10.1 and A.10.2 due to space limitations.
290
291
Figure 5 a) and (b) show the win rates of four types of agents who play against each other for 600 matches in Test and Test II, respectively. We see that the MCC agent achieves the highest win rate against all the other agents in both testing environments, indicating that the CS can select valuable meta-command for each agent to collaborate, and such reasonable collaboration is conducive to winning the game. The MC-Rand and MC-Rule agents are worse than the MC-Base agent, confirming that agents executing low-value meta-commands can hurt performance. Notably, we find that the win rates of the MCC agent in Test I and Test II are close, suggesting that the MCC agent can generalize
293
285
286
287
298
to different numbers of meta- meta-commands. Figure[5[c) demonstrates the final Elo scores ( of these agents. clearly shows the effectiveness of CS in agent-only collaboration scenarios.
299
300
301
5.2.2
Ablation Studies
302
We further investigate
31.5%
303
the influence of different
60.0%
304
components,
including CNN feature extraction
Vin Raate (400)
50.0%
Win Rawe (492)
05.0%
40/6
307
306
with the gating mechanism
(w/o CNN-GM), target
3mmm
MCC
MCG-FD
62.3%
308
attention module (w/o
2000%
MC--tand MC Pale
80.3%
wia ChNN-CN
309
TA). and PPO optimization (MCC-PPO). on the performance of
1000%
via T4
310
algorithm
Treniin Time (h)
34.0%
38.0%
38.2%
Resporree Rate (RR)
40.0%
13.2%
0%
311
(b)
312 CS. We conduct ablation
Figure 6: Results of ablation studies. (a) The training curves of different CS ablation versions. (b) The converged WR-RR results of different CS ablation versions. The shadow indicates the standard deviation.
313
studies in Test I with a
314
20 hero pool. In practical
315
games, meta-com commacommands
316
with adjacent regions often have similar intentions and values. Thus the response rate of the agent to adjacent meta-commands should be as close as possible. Besides, the higher the agent's response rate to noma-com the more collaborative behaviors of the agent, thus we expect the response rate of CS as high as possible. Generally, we expect the Response Rate (RR) of CS as high as possible while ensuring that the Win Rate (WR) is not reduced.
317
319
321
Figure a(a) demonstrates the WR of different CS ablation versions during the training process, and Figure blb shows the converged WR-RR results. We see that after ablating the TA module, the WR and RR of CS are greatly reduced, indicating that the TA module can improve the accuracy of CS Besides, after ablating the CNN-GM module, the RR of CS is most affected, which is reduced by 20%. It indicates that without the CNN-GM module, the value estimation of CS to adjacent meta-commands is not accurate enough, resulting in missing some actual high valuable meta-commands, We notice that the MCC and MCC-PPO in both metrics are close, confirming the versatility of the CS model structure.
322
323
324
to meta-commande
325
327
8

---

Table The WR of different human-Al teams against MC-Base agents Al + Human mode.
Table 2: The RR of humans and agents teammates.
Sender\ Receiver
Averuge Human
Sirong Human
MCC
Type of Agent
MC-Rand
41.07%
35.69%
34.03%
Teammate
MC- Base
MC-Rand
MCC
Average Human
72.34%
61.17%
Average Human
23%
5%
37 %
Strong Human
74.91 % 78.50% %
73.05%
Strong Human
42%
28%
54%
MCC
73.43%
Q-Value of CS
Average Rank of Strong Human
B
B
Figure 7: Case study the value estimation of CS.
329
5.3
Results in Human-Al AT Game Test
330
Due to space limitations, we only show the objective results in 4 + Human mode. Other modes results and the subjective preference results of testers can be found in the Appendix A.10.3 and A. shows the WR of different human-Al AI teams who play against the MC-Base agent-only team. We see that the MCC agent significantly outperforms other agents, regardless of whether they pair with a strong or average human. To explain why humans have higher WR when paired with the
331
332
333
334
335
MCC agents, we count the RR of agents to the meta-commands sent from human teammates (H2A scenarios) and the RR of humans to the meta-commands sent from agent teammates (A2H scenarios), respectively, as shown in Table 2 H2A scenarios, the RRs of the MCC agents to average humans and strong humans are 61.17% and 73.05%, respectively, indicating that the MCC agents are more willing to respond to valuable meta-commands sent from strong humans. We also notice that the RR of the MCC agents to strong humans (73.05%) is very close to the RR of strong humans themselves (74.91%) 91).. suggesting that the CS close to the value system of strong humans. In A2H scenarios, the RRs of average humans and strong humans to the MCC agents are 73.43% and 78.5%, respectively, which is significantly higher than that of MC-Rand agents (41.07% and 35.69%), indicating that the commandss sent from the MCC agents more valuable and reasonable humans. Note that the RR the MCC agents to the MC-Rand agents is 34.03%, which close to that of strong humans (35. 69%), once again confirming that the CS is close the value system of strong humans.
336
337
339
341
343
345
346
347
also visualize the comparison of CS and strong human value systems on game scene with three meta-commands existing, as shown in Figure 7 We see that the CS selects the meta-command B for the two heroes in the red dashed box to collaborate, selects the meta-command C for the two heroes in the purple dashed box to collaborate, and selects the meta-command A for the remaining hero to execute alone. The CS selection results are consistent with the ranking results of strong humans, confirming the effectiveness of CS and the interpretability of the collaboration behavior between MCC agents and humans.
349
351
353
Conclusion
354
355
In this paper, we proposed an efficient and interpretable Meta-Command Communication -based framework, dubbed MCC, achieve effective human-AI collaboration MOBA games. To bridge
356
357
the communication gap between humans and agents, we designed an interpretable communication protocol, i.e., the Meta-Command, to convert the explicit messages from humans and the implicit messages from agents into unified creta-commads. To achieve effective collaboration, we constructed a meta- command value estimation model, i.e., the Meta- Peta-Coomand Selector, to select valuable meta-commane for each agent to execute. Finally, we introduced the training process of the MCC framework and conducted practical human-Al game tests the typical MOBA game Honor of
358
359
360
362
Kings. The experimental results show that the MCC agents can collaborate reasonably with human teammates and even generalize to collaborate with different levels and numbers of human teammates.
364
We expect this work can be foundation for future HAC research in complex environments.

---

References
386
367
[1] Lupu Andrei, Cui Brandon, Hu Hengyuan, and Foerster Jakob N. Trajectory diversity for zero-shot coordination. 139:7204-7213, 2021.
388
389
[2] Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Picter Abbeel, and Anca Dragan. On the utility of learning about humans for human-ai coordination. Advances in Neural Information Processing Systems, 32, 2019.
370
372
[3] Rémi Coulom. Whole-history rating: bayesian rating system for players of time-varying strength. In Proceedings of the International Conference on Computers and Games, pages 113 124, 2008.
373
374
[4] Allan Dafoe, Edward Hughes, Yoram Bachrach, Tantum Collins, Kevin R McKee, Joel Z Leibo, Kate Larson, and Thore Graepel. Open problems cooperative ai. arXiv preprint arXiv:20/2.08630, 2020.
376
378
[5] Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. Tarmac: Targeted multi-agent communication. In Proceedings of the International Conference on Machine Learning, pages 1538-1546, 2019.
380
[6] Yuging Du, Stas Tiomkin, Emre Kiciman, Daniel Polani, Pieter Abbeel, and Anca Dragan. Ave: Assistance via empowerment. Advances in Neural Information Processing Systems, 33:4560-4571, 2020.
382
384
[7] Jakob Foerster, loannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning communicate with deep multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 29, 2016.
386
387
Yiming Gao, Bei Shi, Xueying Du Liang Wang, Guangwei Chen, Zhenjie Lian, Fuhao Qiu, Guoan Han, Weixuan Wang, Deheng Ye, et al. Learning diverse policies in moba games via macro-goals. Advances Neural Information Processing Systems, 34, 2021.
388
389
99] Mohammad Ghavamzadeh and Sridhar Mahadevan. Learning to communicate and act using hierarchical reinforcement learning. Computer Science Department Faculty Publication Series, page 172, 2004.
391
392
393
[10] Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. "other-play" for zero- shot coordination. In Proceedings of the International Conference Machine Learning, pages 4399-4410, 2020.
395
[1]] Max Jaderberg, Wojciech M Czamecki, Iain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, et al. Human- level performance in 3D multiplayer games with copulation-basee reinforcement learning. Science, 364(6443):859 865, 2019.
397
399
[12] Jiechuan Jiang and Zongqing Lu. Learning attentional communication for multi-agent coopera- tion. Advances in Neural Information Processing Systems, 31, 2018.
401
402
[13] Uri Kartoun, Helman Stern, and Yael Edan. A human-robot algorithm. Journal of Intelligent & Robotic Systems, 60(2):217-239, 2010.
reinforcement learning
403
404
[14] Michat Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojeiech Jaškowski. Vizdoom: A doom- boood- ai research platform for visual reinforcement learning. In 2016 IEEE Conference on Computational Intelligence and Games, pages 1-8, 2016.
405
406
407
[15] Daewoo Kim, Sangwoo Moon, David Hostallero, Wan Ju Kang, Taeyoung Lee, Kyunghwan Son, and Yung Yi. Learning to schedule communication in multi-agent reinforcement learning. arXiv preprint arXiv: 1902.01554, 2019.
408
410
[16] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6966 2014.
10

---

412
[17] Quan Liu, Zhihao Liu, Wenjun Xu, Quan Tang, Zude Zhou, and Duc Truong Pham. Human- robot collaboration in disassembly for sustainable manufacturing. International Journal of Production Research, 57(12):4027-4044 2019.
413
415
[18] Hanxiao Liu, Zihang Dai, David So, and Quoe V Le. Pay attention mlps. Advances in Neural Information Processing Systems, 34:9204-92 92155, 2021.
417
[19] Kevin R McKee, Xuechunzi Bai, and Susan T Fiske. Warmth and competence in human-agent cooperation. arXiv preprint arXiv:2201. 13448, 2022.
418
419
[20] Kevin R McKee, Joel Z Leibo, Charlie Beattie, and Richard Everett. Quantifying the effects of environment and population diversity in multi-agent reinforcement learning. Autonomous Agents and Multi-Agent Systems, 36(1):1 2022.
420
421
422
[21] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529 -339 2015.
423
424
425
426
427
[22] OpenAI, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with large scale deep reinforcement learning. arXiv preprini arXiv: 101.9992.98888 2019.
428
429
431
435
[23] Peng Peng, Ying Wen, Yaodong Yang, Quan Yuan, Zhenkun Tang, Haitao Long, and Jun Wang. Multiagent bidirectionally-coodiiii to play starcraft combat games. arXiv preprint 1703. 10069, 2017.
nets: Emergence of human-level coordination in learning
435
[24] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint
437
2017.
[25]
Ali Shafti, Jonas Tjomsland, William Dudley, and A Aldo Faisal. Real-world human-robot collaborative reinforcement learning. 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 11161-11166, 2020.
439
441
266] Victor do Nascimento Silva and Luiz Chaimowicz. Moba: A new arena for game ai. arXiv preprint arXiv: 10443, 2017.
443
[27]
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schritwieserr, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mas- tering the game of Go with deep neural networks and tree search. Nature, 529(7587):484 489, 2016.
445
447
[28] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of Go without human knowledge. Nature, 550(76766:354-3399 2017.
449
[29] Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar. Learning when to communicate at scale in multi-agent cooperative and competitive tasks. arXiv preprint arXiv:1812.0975: 2018.
451
[30] Ho Chit Siu, Jaime Peña, Edenna Chen, Yutai Zhou, Victor Lopez, Kyle Palko, Kimberlee Chang, and Ross Allen, Evaluation of human-ai teams for learned and rule-based agents in hanabi. Advances in Neural Information Processing Systems, 14:16183-16195, 2021.
453
455
[31] DJ Strouse, Kevin McKee, Matt Botvinick, Edward Hughes, and Richard Everett. Collaborating with humans without human data. Advances in Neural Information Processing Systems, 34, 2021.
457 [32] Sainbayar Sukhbaatar, Rob Fergus, et al. Learning multi-agent communication with backpropa- gation. Advances in Neural Information Processing Systems, 29, 2016.

---

459
[33] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
460
461
[34] Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575(7782))3300-35444 2019.
462
463
[35] Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In Proceedings of the International Conference on Machine Learning, pages 1995-2003, 2016.
465
467
[36] Rundong Wang, Xu He, Runsheng Yu, Wei Qiu, Bo An, and Zinovi Rabinovich. Learning efficient aglti-agent communication: An information bottleneck approach. In Proceedings of the International Conference on Machine Learning, pages 9908-99999 99188999111111 2020.
468
469
470
[37] Bin Wu. Hierarchical macro strategy model for moba game ai. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 1206-1213, 2019.
472
[38] Deheng Ye, Guibin Chen, Wen Zhang, Sheng Chen, Bo Yuan, Bo Liu, Jia Chen, Zhao Liu, Fuhao Qiu, Hongsheng Yu, et al. Towards playing full moba games with deep reinforcement learning. Advances in Neural Information Processing Systems, 33:621-632, 2020.
473
474
475
[39] Deheng Ye, Guibin Chen, Peilin Zhao, Fuhao Qiu, Bo Yuan, Wen Zhang, Sheng Chen, Mingfei Sun, Xiaoqian Li, Siqin Li, et al. Supervised learning achieves human-level performance in moba games: A case study of honor of kings. IEEE Transactions on Neural Networks and Learning Systems, 2020.
478
477
478
479
[40] Deheng Ye, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang, Xipeng Wu, Qingwei Guo, al. Mastering complex control in moba games with deep rein- forcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34,
481
pages 6679, 2020.
483
[41] Rui Zhao, Jinming Song, Haifeng Hu, Yang Gao, Yi Wu, Zhongqian Sun, and Wei Yang. Maximum entropy population based training for zero-shot human-ai ai coordination. arXiv preprint 1771 2021.
484
485
Checklist
486
487
1. For all authors...
488
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See the abstract and the contributions in Section
489
490
(b) Did you describe the limitations of your work? [Yes] For margin reasons, we will discuss limitations here. We have currently only verified the effectiveness of the MCC framework in MOBA games, and we will explore in more types of complex games, such First-Person Shooting (FPS) and Massively Multiplayer Online (MMO) in the future.
432
494
485
(c) Did you discuss any potential negative societal impacts of your work? [Yes] The purpose of our method is only for academic research based on game environments, e.g., investigating the MOBA game-playing problems. Like AlphaGo or AlphaStar, the potential negative societal impacts of our work will be limited to the development of gaming AI applications. However, our research will contribute to the research community, the game industry, and the e-sports community.
486
497
498
499
500
501
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
504
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
12

---

507
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [No] The code and the data are proprietary.
508
509
(b) Did you specify all the training details (e.g., data splits, hyperparameters. how they were chosen)? [Yes] See the full experimental setup in Sectionn511.1 and Appendix.
511
512
(c) Did you report error bars (c.g., with respect to the random seed after running exper- iments multiple times)? [Yes] All results are reported over five random seeds (see Section (.1.2). For the convenience of presentation, the median is shown in AI Perfor- mance, while detailed error bars are presented in Ablation Studies. Due to the high cost of Human-AI Game Test, we only used the median model for testing.
513
514
515
516
(d) Did you include the total amount of compute and the type of resources used (c.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] See the resources in Section[5.1.1
518
519
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
520
(a) If your work uses existing assets, did you cite the creators? [Yes]
521
(b) Did you mention the license of the assets? [Yes]
522
(c) Did you include any new assets either in the supplemental material or as a URL? [No]
523
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] Honor of Kings is released and recognized "testbed" for problems [40] 39 38 R 37.. We contact the relevant author and obtain authorization from the game provider.
524
525
526
527
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] See Appendix. All data in this paper is game- related and has nothing to do with identity information.
528
529
530
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [Yes] See Appendix. In the Human-AT Game Test, all participants have more than three years of experience in Honor of Kings and are familiar with all the information in the game. Before the test, we also inform the participants of the detailed test instructions, and the participants voluntarily choose whether participate in the test.
532
538
534
535
536
537
We have detailed ethics descriptions in Appendix A.9. As stated in Section A.9.2, participants were given instructions before testing. As mentioned in point 5 (Line 87-88), participants' game statistics will be only used for academic research, and participants choose whether to participate not.
539
541
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [Yes] We had performed process similar to IRB before the test was conducted in this paper. The institution and all participants have approved our research. First, we analyze the risks of these experiments to the participants. The risks mainly include the leakage of identity information and the time cost. Then, a series of measures are implemented to prevent these risks in Appendix A.9.2. We make a risk statement for participants and sign an identity information confidentiality agreement. We only use information related to the game state in our research without identity information. In addition, special equipment and accounts are provided to the participants to prevent leakage of equipment and account information during the test. The identity information of all participants is not disclosed to the public.
542
543
544
545
546
547
549
551
553
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [Yes] In the Human-AI Game Test, participants can get dollars for each match, and each match is about 10 to 20 minutes.
554
555
13


Reviews:
[{"review_id": "z7s2guLJxy", "reviewer": "Reviewer_WAvp", "summary": "The paper introduces an approach for effective  Human-AI collaboration in the MOBA game  Honor of Kings. This is achieved by developing the interpretable Meta-Command Communication (MCC) based framework as a bridge for effective and interpretable  Human - AI communication within the game. ", "questions": "* What is the benefit the presented research contributes to the overall AI community given the lack of reproducibility?\n* Do you plan to apply the method in parallel to some open environment and open-source the results? Like Dota2, for example, Dota2 is presented on Fig.1 , but the method has not been evaluated in the setting of Dota2, which is a bit confusing. \n* Can the hand-crafted command extraction function be replaced with a learned function? to transfer the approach to the setting of different games.\n* As I understand the evaluation of AI agents collaborating with Humans has been done based on the win rate. Are there any results evaluating the human player impression from pairing up with AI team-mates, i.e., is it easy to recognize a human team-mate from the AI team-mate?", "limitations": "I do not think that the limitations associated with the reproducibility of the research have been properly addressed. This includes access to the game training environment, trained checkpoints, etc..", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The work follows the line of new AI approaches development within the setting of MOBA computer games. The particular MOBA - the Honor of Kings is used as the environment in this work, HoK along with DOTA II appeared in a few preceding AI studies.  \n\nThe paper presents the first Human-AI interaction study within a significantly more complex MOBA game environment than the preceding HAI state-of-the-art including the Hanabi, Overcooked, and Capture the Flag environments. To achieve the studied goal, the paper introduces an interpretable protocol for H2AI and AI2H communication - the meta-command communication engine. The experimental evaluation is extensive and well documented. The paper is generally well written, with many figures describing their approach, results from the experiment, and an ablation study. \n\nOn one hand it is hard to criticise the empirical novelty of the presented work, on the other hand, I object the clear lack of reproducibility of the work : the training/evaluation environment is not available, the datasets of the games used for meta-command training are not shared, agent checkpoints are not available as well,  and the overall training procedure required a formidable amount of compute.  Moreover, the method has been tested in the restricted setting of a single environment (Honor of Kings) with a pool of handcrafted meta-commands . I can see clear advantages of the work for the HoK player community, especially if the bot is deployed online, and available for online play/ practice. However, I have a hard time noticing clear contributions of the work to the overall (multi-agent) AI community.\n \nI would like the authors to think about and present their point of view on what overall benefit their research provides for the overall AI community, which goes beyond the community of HoK players in light of the apparent lack of reproducibility of the method.   \n\nIdeally, as the environment and source code are proprietary, I would like to see the method being applied in parallel to an open environment, e.g. Dota II open API, so other researchers can test it and reproduce it in their own research. \n\nTo conclude, I'm on the fence about accepting the paper, with the main disadvantage being the lack of reproducibility of the whole approach. If the AC confirms that it is fine to present the results obtained using proprietary code, without the possibility of reproducing I will be fine accepting the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "hMICWPuRQN", "reviewer": "Reviewer_xzzg", "summary": "The paper describes a system which is able to collaborate with human players in MOBA for the communication, selection, and execution of mid-level tasks (20s). They propose a simple communication method which is used by agents and human players, which is translated to an internal representation, evaluated in comparison with other goals, and executed using 3 different systems. Few details of the training sets are provided. The system is evaluated in comparison with other weaker strategies and is only about 10% better than an agent which does not communicate (MC-Base). In tests with human players, insufficiently described, it shows a good performance when paired with strong humans.", "questions": "1. Where does the training data come from? How much data? Were the users informed of the use of the data for training an AI system? Did they consent?\n2. Did you access the ability level of the gamers provided by the game company? \n3. Why did you not compare human-human results with human-ai results?\n4. Why did MC-Base performed so well, and why this was not properly discussed in the paper?", "limitations": "As I mentioned in the S&W section, the authors tend to overstate their claims, especially considering that: (1) they are only considering mid-level instructions; (2) they did not compare to a human baseline; (3) they do not know if real human players would use their MCC framework in real-stake games; and (4) there is no evidence that the populations they tested are representative.", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe paper is ambitions and, to a certain extent, complete, in the sense of aiming to solve an entire problem which is complex and, to some extent, hard to define, develop, and evaluate adequately.\n\nThe results of the paper (the ones which are acceptable, see Weaknesses) are promising, although since there is no baseline with other human players, it is hard to evaluate the actual performance level of the system which was developed.\n\nThe most important contribution of this paper, in my view, is providing some evidence that massive use of NNs and GPUs, in supervised learning, may be able to create collaborative AI players which understand mid-level instructions in complex game scenarios.\n\n\nWeaknesses:\n\nThe paper is hard to read, mostly because there is an excessive effort to include everything in the main text. It comprises significant amount of work and hardware training, but it takes a very careful examination and reading to understand it. The paper is not clear about what is in the appendix (for instance, the details of the human experiment), so the reader has to mine things as it reads along. It was a frustrating experience to read this paper.\n\nA key problem of the paper is that it exaggerates its claims. For instance, the MCC framework is very limited as a communication strategy, since it takes care only of mid-level strategy (20s). Collaboration ranges from quick commands to complex strategies which cover the whole game. That issue is never discussed in the game, and the proposal of the MCC framework seems to have never considered how human players actually communicate and interact in a real game. MCC seems to have been designed for easy of translation.\n\nThe authors also use the term interpretable in a way that is confusing with the current use in the field. While the Machine Learning field uses the term often to convey ML models whose behavior can be understood by human beings, in this case it means commands, issues by either machines or humans, in a subset of human language. At best, it is a confusing use, at worst, it is misleading.\n\n\nThere are some key methodological flaws in the paper. The main one, a very important one, is related to where the data used to train the system comes from, how much data is used, and etc.\n\nIt is not clear why the authors did not compare with the WR of only human teams, which would provide some sort of baseline to the results. Also, the analysis of the Agent-Only tests is too limited and, in many ways, misleading. Figure 6 shows that the MC-Base system is almost as good as MCC, which raises questions of how effective and important communication is in the evaluation task used in the paper. In Figure 5, we see that clearly in the under-60% win rate against MC-Base. This is an important result which is ignored in the paper, misrepresenting, at some extent, how good the results are. Papers should be as clear about what did not work as about what worked, which is clearly not this case.\n\nMoreover, there is so little information about the players used in the test which is hard to trust the results. The authors blindly accepted the game makers' classification and abilities of the players, which could be simply wrong. \n\nThis paper also raises some ethical concerns. The authors never described where the training data comes from. Is it from real users' data? In this case, where they informed and consented to have their data used this way? Also, collaborative and communicative AI players raise concerns of cheating and of commercial exploitation of money-based platforms. They authors ignore those issues.\n\nFinally, I found it problematic when the authors claim that they are the first to investigate the HAC problem in MOBA games, especially considering that this kind of research dates back to the 90s. In particular, the authors should look into David Chapman's pioneering work in the field:\n* Vision, Instruction, and Action by David Chapman , The MIT Press 1991\n* Agre, Philip E., and David Chapman. \"What are plans for?.\" Robotics and autonomous systems 6.1-2 (1990): 17-34.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Inadequate Data and Algorithm Evaluation", "Inappropriate Potential Applications & Impact  (e.g., human rights concerns)"]}, "extra_text": {}}, {"review_id": "alc5QzpqJ7T", "reviewer": "Reviewer_PjFP", "summary": "This submission takes on the challenge of human-agent collaboration in Honor of Kings, a MOBA (multi-player online battle arena) strategy game. It proposes a particular approach to human-agent collaboration based on an explicit communication framework (“meta-commands”) and a hierarchical approach to collaboration. The submission evaluates Meta-Command Communication, its proposed approach, in several multi-agent and human-agent experiments.", "questions": "Did participants provide informed consent for participating in this research and providing their data for research purposes?\n\nHow do state-of-the art agents such as WuKong perform in human-agent teams in Honor of Kings?\n\nDo participants prefer playing on a team with the MCC agent over state-of-the-art agents (e.g., WuKong)?\n\nCan the authors explain why human-agent collaboration methods developed for Overcooked (Carroll et al., 2019; Strouse et al., 2021) would fail in this setting?\n\nSee also the revision recommendations in the \"Strengths and Weaknesses\" section.", "limitations": "I didn’t see any coverage of technical or conceptual limitations (e.g., how training sequentially might affect performance or whether the proposed method would work in non-MOBA collaborative domains). I would recommend that the authors discuss those areas, and reflect a bit on any potential negative societal impact. If they believe there is no risk, they should mention that.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**First, an ethics flag: Did participants provide informed consent for their involvement in this research and for the use of their data toward this project?** The checklist states that the authors engaged a \"process similar to IRB\" and that participants signed a \"risk statement\" and \"identity information confidentiality agreement\", but it's not clear to me that they actually provided informed consent. Were they fully informed about the purpose of the research, did they confirm their approval for involvement, were they told about how to withdraw their data, could they report any misconduct or unexpected harms, etc.?\n\nI appreciate the opportunity to review this work; it’s apparent that the authors put in a lot of effort to develop this human-agent collaboration framework and test it with human players.\n\nI see several prominent strengths for this paper. It takes an interesting approach to human-agent collaboration, focusing on the role of explicit (bounded) strategic communication to help synchronize human and agent policies. I appreciate the use of both multi-agent and human-agent experiments to evaluate the effectiveness of the approach. And it’s terrific that the authors provide details for their experiments and approach in the supplemental information.\n\nOn balance, however, I think this work is not quite ready for acceptance. The paper is primarily weakened by insufficiencies in the experimental evaluations and description of the method. The latter may be somewhat straightforward to fix, but I imagine fixing the former may require some additional work. \n\nThe most pressing issues I can see lie with the agent evaluation. It’s great that the experiments include several baselines against which the MCC agent is compared. However, the baselines only include three variants of the MCC agent. This is a pretty weak benchmarking approach, since it effectively tests whether ablations decrease performance. The gold standard approach would be to test against other standard agents from this domain, including some baselines that operate completely independently from the meta-command framework. As mentioned at the beginning of the paper, prior research has developed state-of-the-art agents for Honor of Kings (e.g., WuKong; Ye et al., 2020a, 2020b). How do such state-of-the-art agents---even if developed for competition against human teams---perform when placed on teams with humans?\n\nRelatedly, for space reasons, the main text reports results for only 4 agents-1 human games. Unfortunately, these are the least informative games for evaluating the human compatibility of agents, since win rate and other team metrics can be driven entirely by agents collaborating with each other. (Again, running human-agent experiments with state-of-the-art, non-MCC agents developed for all-agent team play would help establish useful reference points for the MCC agent’s performance here.)\n\nFinally, recent human-agent interaction work emphasizes the importance of evaluating interactive agents on both objective performance metrics and subjective metrics like satisfaction. See Du et al. (2020, NeurIPS), Strouse et al. (2021, NeurIPS), and McKee et al. (2022, AAMAS). As discussed in these papers, recent conversations around research ethics and responsible development of AI substantially increase the importance of considering human autonomy and preferences in designing interactive agents. Can we really claim that an agent is “better” (or human-compatible at all) if people dislike interacting with it?\n\nAside from the evaluation approach, a few scattered observations on other issues:\n- With regard to the field of human-agent collaboration, the paper claims more novelty than is warranted. In particular, it ignores work demonstrating effective human-agent collaboration in Overcooked from prior years at this conference: Carroll et al., (2019, NeurIPS) and Strouse et al. (2021, NeurIPS).\n- Related to the prior point on baselines, the paper claims on page 3 that state-of-the-art AI agents “can only defeat human players but cannot collaborate well due to the communication gap between agents and humans”. Can the authors supply any evidence for the claim that OpenAI-Five, WuKong, etc. would fail with human teammates? (In particular, because of the communication gap between humans and agents?)\n- It is unclear to me how several parameters are chosen (e.g., $T_{mc}$).\n- I don’t understand the justification for the reward formula on line 196. Why is there a trade-off parameter? Why not just sum reward without that multiplication factor?\n- Overall, there are some recurring issues with the manuscript style, particularly around statements of fact/necessity and unevidenced claims.\n   - On the former, the manuscript often uses language implying that a claim is a fact or a necessary statement, rather than a proposal (“We propose [...]”). I’d recommend the authors be a bit more deferential toward alternative approaches to their methods. For example, one section of the paper claims that “in MOBA games, a macro-strategy comprises three components: where to go, what to do, and how long” (p. 4). If this is a definition inherited from another paper (“X et al. (2020) introduce the concept of “macro-strategies”, which they describe as…”) or that this paper is introducing (“We propose that a _macro-strategy_ consists of [...]”), that should be stated more directly so that readers can engage with those arguments. Otherwise, I would argue that this is an insufficient definition: What about _who_ is involved in the strategy? What about conditional strategies, which are only triggered by certain conditions or have branching paths? Similar problems occur elsewhere, such as the section on page 3 that asserts “the HAC problem in MOBA games can generally be divided in the Human-to-AI (H2A) and the AI-to-Human (A2H) scenarios”. What about symmetric communication scenarios, or scenarios where there is no communication at all? Again, the manuscript is a bit too assertive in making general claims.\n   - Relatedly, there are a number of unevidenced and unsubstantiated claims throughout the manuscript. I’d gently encourage the authors to consider the duty we have to our readers and any researchers who will build from our work: we should reference prior work or offer empirical evidence when we make claims. What evidence supports the following claims?\n     - “However, these AI systems can only defeat human players but cannot collaborate well due to the communication gap between agents and humans” (p. 3)\n     - “However, these tasks do not consider long-term macro-strategies planning and collaborating” (p. 3)\n     - “The HAC problem in MOBA games can generally be divided into the Human-to-AI (H2A) and the AI-to-Human (A2H) scenarios” (p. 3)\n     - “It normally takes a human [...] 300 time steps (20 seconds)” “to complete a macro-strategy in MOBA games” (p. 4)\n     - “Besides, humans always want their meta-commands to be responded to as much as possible” (p. 8)\n     - A final question: the introduction suggests that human-agent collaboration is harder or distinct from multi-agent collaboration. Can the authors explain why this is the case?\n\nI’d like to thank the authors again for their contributions to these research problems. The authors claim that this research is a first for the field of human-agent collaboration: if that’s the case, I think addressing these issues would do the field a service, setting solid groundwork for others who will follow in this area.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Responsible Research Practice (e.g., IRB, documentation, research ethics)"]}, "extra_text": {}}, {"review_id": "VPLK0un5dQ5", "reviewer": "Reviewer_8yYn", "summary": "In this paper the authors introduce an approach to allow for human-AI collaboration (HAC) in the game Honor of Kings. This is made possible by devising a meta-command communication (MCC)-based framework with three components. First, messages (human-agent or agent-agent) are converted to this meta-command representation, then the optimal meta-command is chosen, and finally a sequence of actions are selected conditioned on the optimal meta-command. The paper introduces this framework, how it is trained, and then gives the results of agent-agent and agent-human experiments. ", "questions": "1. What is the authors’ motivation for this work?\n2. How would the authors relate Hanabi hint tokens to their MCC framework?\n3. How did the authors extract the training data for CEN?\n4. Why didn’t the author’s collect data on human experience? Or if they did, why not report them?", "limitations": "As discussed above, there’s a number of technical limitations not discussed by the authors. Ethically, everything seems to be fine. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "In terms of strengths, the paper introduces a novel and detailed approach to human-AI collaboration/human-AI teaming that should extend to at least other MOBA games but could potentially serve as the basis for a general communication framework in domains with similar constraints. The paper itself, outside of some small language issues, is overall very well-written, and the results are fairly convincing. I have minor concerns but overall this seems to be a promising approach. \n\nIn terms of weaknesses, as I mentioned above, I have a number of minor concerns. First, there’s a surprising lack of discussion motivating the work presented in this paper. Essentially all readers are given is “We expect this work can be a foundation for future HAC research in complex environments.”. Some indication of planned use cases/applications would be appreciated. Are the authors thinking of using these agents as friendly bots in Honor for Kings? Or is this purely theoretical and meant to develop HAC approaches?\n\nSecond, while I largely agree with the author’s summary of related work, I think they undersell the extent to which Hanabi requires a simplified version of their MCC framework [1]. Given how human and AI agents most communicate using the “hint tokens”, there’s actually a fair amount of overlap between the frameworks. Some acknowledgement of this in the paper would be beneficial. \n\nThird, there’s no discussion in the paper of the limitations of the MCC framework. For example, the authors give the definition of M as “M represents the space of interpretable messages, that is, the Meta-Commands in the MCC framework”. However, this space is not fully described in the paper or supplementary materials. Assumably, M would need to be rewritten to apply this framework to a different MOBA game, which would require significant human effort.  \n\nFourth, related to the MCC framework, there’s no discussion of how the training data for CEN is extracted, only that “The training dataset {< o, m >} is obtained by extracting the observation o and its corresponding meta-command m from expert data.”. Some description of the methodology for extracting this dataset would be valuable. If it was done by hand this would greatly decrease the generality of this approach. \n\nFifth, there was no collection of data related to the experience of the human players, or at least there’s no reporting of said data. This seems to be a major omission and an odd choice for a human subject study design. The human player’s experience is an important aspect of any human-AI teaming. If the teaming is quantitatively effective but the humans didn’t like it, then that would negatively impact others attempting to adapt the approach. \n\nSixth and finally, the results with a single human are impressive, but from the supplementary material it’s clear there’s a major falloff when more than one human is included. This is an important result and runs counter to the author’s claims that “The experimental results show that the MCC agents can collaborate reasonably with human teammates and even generalize to collaborate with different levels and numbers of human teammates.”. I’d walk back these claims and include some acknowledgement of this falloff in the paper. \n\nThere were also some occasional issues with the language, for example “thus far simpler”-> “thus it is far simpler” and “achieving the collaboration with humans”-> “achieving effective collaboration with humans”. \n\nOverall, I’m positive on this paper, but the number of minor issues above does temper my enthusiasm.  \n\n\n1. Siu, Ho Chit, et al. \"Evaluation of human-AI teams for learned and rule-based agents in Hanabi.\" Advances in Neural Information Processing Systems 34 (2021): 16183-16195.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject