Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This submission takes on the challenge of human-agent collaboration in Honor of Kings, a MOBA (multi-player online battle arena) strategy game. It proposes a particular approach to human-agent collaboration based on an explicit communication framework (“meta-commands”) and a hierarchical approach to collaboration. The submission evaluates Meta-Command Communication, its proposed approach, in several multi-agent and human-agent experiments.
strengths_and_weaknesses: **First, an ethics flag: Did participants provide informed consent for their involvement in this research and for the use of their data toward this project?** The checklist states that the authors engaged a "process similar to IRB" and that participants signed a "risk statement" and "identity information confidentiality agreement", but it's not clear to me that they actually provided informed consent. Were they fully informed about the purpose of the research, did they confirm their approval for involvement, were they told about how to withdraw their data, could they report any misconduct or unexpected harms, etc.?

I appreciate the opportunity to review this work; it’s apparent that the authors put in a lot of effort to develop this human-agent collaboration framework and test it with human players.

I see several prominent strengths for this paper. It takes an interesting approach to human-agent collaboration, focusing on the role of explicit (bounded) strategic communication to help synchronize human and agent policies. I appreciate the use of both multi-agent and human-agent experiments to evaluate the effectiveness of the approach. And it’s terrific that the authors provide details for their experiments and approach in the supplemental information.

On balance, however, I think this work is not quite ready for acceptance. The paper is primarily weakened by insufficiencies in the experimental evaluations and description of the method. The latter may be somewhat straightforward to fix, but I imagine fixing the former may require some additional work. 

The most pressing issues I can see lie with the agent evaluation. It’s great that the experiments include several baselines against which the MCC agent is compared. However, the baselines only include three variants of the MCC agent. This is a pretty weak benchmarking approach, since it effectively tests whether ablations decrease performance. The gold standard approach would be to test against other standard agents from this domain, including some baselines that operate completely independently from the meta-command framework. As mentioned at the beginning of the paper, prior research has developed state-of-the-art agents for Honor of Kings (e.g., WuKong; Ye et al., 2020a, 2020b). How do such state-of-the-art agents---even if developed for competition against human teams---perform when placed on teams with humans?

Relatedly, for space reasons, the main text reports results for only 4 agents-1 human games. Unfortunately, these are the least informative games for evaluating the human compatibility of agents, since win rate and other team metrics can be driven entirely by agents collaborating with each other. (Again, running human-agent experiments with state-of-the-art, non-MCC agents developed for all-agent team play would help establish useful reference points for the MCC agent’s performance here.)

Finally, recent human-agent interaction work emphasizes the importance of evaluating interactive agents on both objective performance metrics and subjective metrics like satisfaction. See Du et al. (2020, NeurIPS), Strouse et al. (2021, NeurIPS), and McKee et al. (2022, AAMAS). As discussed in these papers, recent conversations around research ethics and responsible development of AI substantially increase the importance of considering human autonomy and preferences in designing interactive agents. Can we really claim that an agent is “better” (or human-compatible at all) if people dislike interacting with it?

Aside from the evaluation approach, a few scattered observations on other issues:
- With regard to the field of human-agent collaboration, the paper claims more novelty than is warranted. In particular, it ignores work demonstrating effective human-agent collaboration in Overcooked from prior years at this conference: Carroll et al., (2019, NeurIPS) and Strouse et al. (2021, NeurIPS).
- Related to the prior point on baselines, the paper claims on page 3 that state-of-the-art AI agents “can only defeat human players but cannot collaborate well due to the communication gap between agents and humans”. Can the authors supply any evidence for the claim that OpenAI-Five, WuKong, etc. would fail with human teammates? (In particular, because of the communication gap between humans and agents?)
- It is unclear to me how several parameters are chosen (e.g., $T_{mc}$).
- I don’t understand the justification for the reward formula on line 196. Why is there a trade-off parameter? Why not just sum reward without that multiplication factor?
- Overall, there are some recurring issues with the manuscript style, particularly around statements of fact/necessity and unevidenced claims.
   - On the former, the manuscript often uses language implying that a claim is a fact or a necessary statement, rather than a proposal (“We propose [...]”). I’d recommend the authors be a bit more deferential toward alternative approaches to their methods. For example, one section of the paper claims that “in MOBA games, a macro-strategy comprises three components: where to go, what to do, and how long” (p. 4). If this is a definition inherited from another paper (“X et al. (2020) introduce the concept of “macro-strategies”, which they describe as…”) or that this paper is introducing (“We propose that a _macro-strategy_ consists of [...]”), that should be stated more directly so that readers can engage with those arguments. Otherwise, I would argue that this is an insufficient definition: What about _who_ is involved in the strategy? What about conditional strategies, which are only triggered by certain conditions or have branching paths? Similar problems occur elsewhere, such as the section on page 3 that asserts “the HAC problem in MOBA games can generally be divided in the Human-to-AI (H2A) and the AI-to-Human (A2H) scenarios”. What about symmetric communication scenarios, or scenarios where there is no communication at all? Again, the manuscript is a bit too assertive in making general claims.
   - Relatedly, there are a number of unevidenced and unsubstantiated claims throughout the manuscript. I’d gently encourage the authors to consider the duty we have to our readers and any researchers who will build from our work: we should reference prior work or offer empirical evidence when we make claims. What evidence supports the following claims?
     - “However, these AI systems can only defeat human players but cannot collaborate well due to the communication gap between agents and humans” (p. 3)
     - “However, these tasks do not consider long-term macro-strategies planning and collaborating” (p. 3)
     - “The HAC problem in MOBA games can generally be divided into the Human-to-AI (H2A) and the AI-to-Human (A2H) scenarios” (p. 3)
     - “It normally takes a human [...] 300 time steps (20 seconds)” “to complete a macro-strategy in MOBA games” (p. 4)
     - “Besides, humans always want their meta-commands to be responded to as much as possible” (p. 8)
     - A final question: the introduction suggests that human-agent collaboration is harder or distinct from multi-agent collaboration. Can the authors explain why this is the case?

I’d like to thank the authors again for their contributions to these research problems. The authors claim that this research is a first for the field of human-agent collaboration: if that’s the case, I think addressing these issues would do the field a service, setting solid groundwork for others who will follow in this area.
questions: Did participants provide informed consent for participating in this research and providing their data for research purposes?

How do state-of-the art agents such as WuKong perform in human-agent teams in Honor of Kings?

Do participants prefer playing on a team with the MCC agent over state-of-the-art agents (e.g., WuKong)?

Can the authors explain why human-agent collaboration methods developed for Overcooked (Carroll et al., 2019; Strouse et al., 2021) would fail in this setting?

See also the revision recommendations in the "Strengths and Weaknesses" section.
limitations: I didn’t see any coverage of technical or conceptual limitations (e.g., how training sequentially might affect performance or whether the proposed method would work in non-MOBA collaborative domains). I would recommend that the authors discuss those areas, and reflect a bit on any potential negative societal impact. If they believe there is no risk, they should mention that.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper the authors introduce an approach to allow for human-AI collaboration (HAC) in the game Honor of Kings. This is made possible by devising a meta-command communication (MCC)-based framework with three components. First, messages (human-agent or agent-agent) are converted to this meta-command representation, then the optimal meta-command is chosen, and finally a sequence of actions are selected conditioned on the optimal meta-command. The paper introduces this framework, how it is trained, and then gives the results of agent-agent and agent-human experiments. 
strengths_and_weaknesses: In terms of strengths, the paper introduces a novel and detailed approach to human-AI collaboration/human-AI teaming that should extend to at least other MOBA games but could potentially serve as the basis for a general communication framework in domains with similar constraints. The paper itself, outside of some small language issues, is overall very well-written, and the results are fairly convincing. I have minor concerns but overall this seems to be a promising approach. 

In terms of weaknesses, as I mentioned above, I have a number of minor concerns. First, there’s a surprising lack of discussion motivating the work presented in this paper. Essentially all readers are given is “We expect this work can be a foundation for future HAC research in complex environments.”. Some indication of planned use cases/applications would be appreciated. Are the authors thinking of using these agents as friendly bots in Honor for Kings? Or is this purely theoretical and meant to develop HAC approaches?

Second, while I largely agree with the author’s summary of related work, I think they undersell the extent to which Hanabi requires a simplified version of their MCC framework [1]. Given how human and AI agents most communicate using the “hint tokens”, there’s actually a fair amount of overlap between the frameworks. Some acknowledgement of this in the paper would be beneficial. 

Third, there’s no discussion in the paper of the limitations of the MCC framework. For example, the authors give the definition of M as “M represents the space of interpretable messages, that is, the Meta-Commands in the MCC framework”. However, this space is not fully described in the paper or supplementary materials. Assumably, M would need to be rewritten to apply this framework to a different MOBA game, which would require significant human effort.  

Fourth, related to the MCC framework, there’s no discussion of how the training data for CEN is extracted, only that “The training dataset {< o, m >} is obtained by extracting the observation o and its corresponding meta-command m from expert data.”. Some description of the methodology for extracting this dataset would be valuable. If it was done by hand this would greatly decrease the generality of this approach. 

Fifth, there was no collection of data related to the experience of the human players, or at least there’s no reporting of said data. This seems to be a major omission and an odd choice for a human subject study design. The human player’s experience is an important aspect of any human-AI teaming. If the teaming is quantitatively effective but the humans didn’t like it, then that would negatively impact others attempting to adapt the approach. 

Sixth and finally, the results with a single human are impressive, but from the supplementary material it’s clear there’s a major falloff when more than one human is included. This is an important result and runs counter to the author’s claims that “The experimental results show that the MCC agents can collaborate reasonably with human teammates and even generalize to collaborate with different levels and numbers of human teammates.”. I’d walk back these claims and include some acknowledgement of this falloff in the paper. 

There were also some occasional issues with the language, for example “thus far simpler”-> “thus it is far simpler” and “achieving the collaboration with humans”-> “achieving effective collaboration with humans”. 

Overall, I’m positive on this paper, but the number of minor issues above does temper my enthusiasm.  


1. Siu, Ho Chit, et al. "Evaluation of human-AI teams for learned and rule-based agents in Hanabi." Advances in Neural Information Processing Systems 34 (2021): 16183-16195.

questions: 1. What is the authors’ motivation for this work?
2. How would the authors relate Hanabi hint tokens to their MCC framework?
3. How did the authors extract the training data for CEN?
4. Why didn’t the author’s collect data on human experience? Or if they did, why not report them?
limitations: As discussed above, there’s a number of technical limitations not discussed by the authors. Ethically, everything seems to be fine.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper introduces an approach for effective  Human-AI collaboration in the MOBA game  Honor of Kings. This is achieved by developing the interpretable Meta-Command Communication (MCC) based framework as a bridge for effective and interpretable  Human - AI communication within the game. 
strengths_and_weaknesses: The work follows the line of new AI approaches development within the setting of MOBA computer games. The particular MOBA - the Honor of Kings is used as the environment in this work, HoK along with DOTA II appeared in a few preceding AI studies.  

The paper presents the first Human-AI interaction study within a significantly more complex MOBA game environment than the preceding HAI state-of-the-art including the Hanabi, Overcooked, and Capture the Flag environments. To achieve the studied goal, the paper introduces an interpretable protocol for H2AI and AI2H communication - the meta-command communication engine. The experimental evaluation is extensive and well documented. The paper is generally well written, with many figures describing their approach, results from the experiment, and an ablation study. 

On one hand it is hard to criticise the empirical novelty of the presented work, on the other hand, I object the clear lack of reproducibility of the work : the training/evaluation environment is not available, the datasets of the games used for meta-command training are not shared, agent checkpoints are not available as well,  and the overall training procedure required a formidable amount of compute.  Moreover, the method has been tested in the restricted setting of a single environment (Honor of Kings) with a pool of handcrafted meta-commands . I can see clear advantages of the work for the HoK player community, especially if the bot is deployed online, and available for online play/ practice. However, I have a hard time noticing clear contributions of the work to the overall (multi-agent) AI community.
 
I would like the authors to think about and present their point of view on what overall benefit their research provides for the overall AI community, which goes beyond the community of HoK players in light of the apparent lack of reproducibility of the method.   

Ideally, as the environment and source code are proprietary, I would like to see the method being applied in parallel to an open environment, e.g. Dota II open API, so other researchers can test it and reproduce it in their own research. 

To conclude, I'm on the fence about accepting the paper, with the main disadvantage being the lack of reproducibility of the whole approach. If the AC confirms that it is fine to present the results obtained using proprietary code, without the possibility of reproducing I will be fine accepting the paper.
questions: * What is the benefit the presented research contributes to the overall AI community given the lack of reproducibility?
* Do you plan to apply the method in parallel to some open environment and open-source the results? Like Dota2, for example, Dota2 is presented on Fig.1 , but the method has not been evaluated in the setting of Dota2, which is a bit confusing. 
* Can the hand-crafted command extraction function be replaced with a learned function? to transfer the approach to the setting of different games.
* As I understand the evaluation of AI agents collaborating with Humans has been done based on the win rate. Are there any results evaluating the human player impression from pairing up with AI team-mates, i.e., is it easy to recognize a human team-mate from the AI team-mate?
limitations: I do not think that the limitations associated with the reproducibility of the research have been properly addressed. This includes access to the game training environment, trained checkpoints, etc..
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The paper describes a system which is able to collaborate with human players in MOBA for the communication, selection, and execution of mid-level tasks (20s). They propose a simple communication method which is used by agents and human players, which is translated to an internal representation, evaluated in comparison with other goals, and executed using 3 different systems. Few details of the training sets are provided. The system is evaluated in comparison with other weaker strategies and is only about 10% better than an agent which does not communicate (MC-Base). In tests with human players, insufficiently described, it shows a good performance when paired with strong humans.
strengths_and_weaknesses: Strengths:
The paper is ambitions and, to a certain extent, complete, in the sense of aiming to solve an entire problem which is complex and, to some extent, hard to define, develop, and evaluate adequately.

The results of the paper (the ones which are acceptable, see Weaknesses) are promising, although since there is no baseline with other human players, it is hard to evaluate the actual performance level of the system which was developed.

The most important contribution of this paper, in my view, is providing some evidence that massive use of NNs and GPUs, in supervised learning, may be able to create collaborative AI players which understand mid-level instructions in complex game scenarios.


Weaknesses:

The paper is hard to read, mostly because there is an excessive effort to include everything in the main text. It comprises significant amount of work and hardware training, but it takes a very careful examination and reading to understand it. The paper is not clear about what is in the appendix (for instance, the details of the human experiment), so the reader has to mine things as it reads along. It was a frustrating experience to read this paper.

A key problem of the paper is that it exaggerates its claims. For instance, the MCC framework is very limited as a communication strategy, since it takes care only of mid-level strategy (20s). Collaboration ranges from quick commands to complex strategies which cover the whole game. That issue is never discussed in the game, and the proposal of the MCC framework seems to have never considered how human players actually communicate and interact in a real game. MCC seems to have been designed for easy of translation.

The authors also use the term interpretable in a way that is confusing with the current use in the field. While the Machine Learning field uses the term often to convey ML models whose behavior can be understood by human beings, in this case it means commands, issues by either machines or humans, in a subset of human language. At best, it is a confusing use, at worst, it is misleading.


There are some key methodological flaws in the paper. The main one, a very important one, is related to where the data used to train the system comes from, how much data is used, and etc.

It is not clear why the authors did not compare with the WR of only human teams, which would provide some sort of baseline to the results. Also, the analysis of the Agent-Only tests is too limited and, in many ways, misleading. Figure 6 shows that the MC-Base system is almost as good as MCC, which raises questions of how effective and important communication is in the evaluation task used in the paper. In Figure 5, we see that clearly in the under-60% win rate against MC-Base. This is an important result which is ignored in the paper, misrepresenting, at some extent, how good the results are. Papers should be as clear about what did not work as about what worked, which is clearly not this case.

Moreover, there is so little information about the players used in the test which is hard to trust the results. The authors blindly accepted the game makers' classification and abilities of the players, which could be simply wrong. 

This paper also raises some ethical concerns. The authors never described where the training data comes from. Is it from real users' data? In this case, where they informed and consented to have their data used this way? Also, collaborative and communicative AI players raise concerns of cheating and of commercial exploitation of money-based platforms. They authors ignore those issues.

Finally, I found it problematic when the authors claim that they are the first to investigate the HAC problem in MOBA games, especially considering that this kind of research dates back to the 90s. In particular, the authors should look into David Chapman's pioneering work in the field:
* Vision, Instruction, and Action by David Chapman , The MIT Press 1991
* Agre, Philip E., and David Chapman. "What are plans for?." Robotics and autonomous systems 6.1-2 (1990): 17-34.


questions: 1. Where does the training data come from? How much data? Were the users informed of the use of the data for training an AI system? Did they consent?
2. Did you access the ability level of the gamers provided by the game company? 
3. Why did you not compare human-human results with human-ai results?
4. Why did MC-Base performed so well, and why this was not properly discussed in the paper?
limitations: As I mentioned in the S&W section, the authors tend to overstate their claims, especially considering that: (1) they are only considering mid-level instructions; (2) they did not compare to a human baseline; (3) they do not know if real human players would use their MCC framework in real-stake games; and (4) there is no evidence that the populations they tested are representative.
review_title: null

Meta-review: metareview: The paper proposes a mechanism for human-AI collaborative play in the game Honor of Kings. The high-level approach is sensible: communicating happens within a small space of "meta-commands", which can be converted to coherent chunks of agent behaviour. The complexity of the system is high, the experiments are done at large scale, and the empirical findings are impressive (in single-human collaboration at least). However, the reviewers also point out a large array of concerns to be resolved; of particular concern, and insufficiently addressed, are the ethical issues (informed consent) raised by multiple reviewers.

Overall, the reviewers find that the paper is not ready for publication yet, and I concur. I hope the authors will integrate the rich feedback from this reviewing process in a future (extensively rewritten) iteration. 
confidence: Certain
Decision: Reject