Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes improvements to an existing deep RL agent for the game Honor of Kings, by introducing a meta-controller trained by supervised learning to predict macro goals high-level players would like to pursue next.
main_review: The originality of the method is limited - hierarchical control systems are well established and games of similar complexity and with similar mechanics have been learnt previously - but it remains interesting to the sub-community focused on these types of applications as additional evidence of approaches that can be successful utilized in modern games.

I have some concerns about the correctness of statements in the paper that could significantly improve my rating of this paper if adequately addressed in the author's reply:

(1) Figure 6b shows win rate against human without variance. Given the very small differences in percentages on the y-axis, there may not be any significant improvement over the previous RL baseline. Can you provide stronger evidence that the proposed method significantly improves performance in comparison to the chosen baseline?

(2) The contributions on page 2 state the proposed method uses the same computational resources as the baseline, but the proposed method introduces an additional pre-training stage for the macro classifier and augments the policy state space. Both of these modifications will require additional compute - the macro classifier in its own training and the augmented policy space I assume will increase training time for the RL agent. Should this statement instead be that once the agent is trained they use the same resources at the evaluation stage? If so, the claim that this is "unlike league-based methods" does not hold.

(3) On line 40 it is claimed that "the agent should exploit [the opponent] instead of playing with a single policy" but these two statements are not mutually exclusive. A single policy can learn to exploit any opponent, either by playing a Nash equilibrium strategy or using opponent modelling to recognize online the type of opponent it is playing and condition its policy on this additional observation.

(4) Line 85 claims "there is no obvious signal ... to represent macro strategies in MOBA games" as a way to differentiate from prior work but the proposed method defines macro goals (section 3.1.1), uses them throughout training and claims they are applicable to other MOBA games (line 143). I think there are other ways to differentiate the proposed method from the past work cited, but this current statement is contradicted by the work done and so currently weakens the evidence of originality presented in the paper.

Other issues that are lowering my current rating of the quality and clarity of the paper:

(5) Line 232-233 gives statistics about the agents response time, but provides no grounding for how this compares to human players.

(6) In Figure 9 do the built-in bots have an ELO rating of 0? Or is this the no-op agent as in Figure 6?

Minor comments that could be easily addressed if accepted (no need to respond):
- Due to the complexity of the approach and scale of the study, some details are missing that would be interesting insights to the overall system. Most notably, no results are provided for the meta-controller pre-training and the focal loss is not defined (presumably because it is defined in Lin et al. 2017) - adding these to the appendix would make this a more self contained reference. 
- Line 190: "even the lineups" -> "even WHEN the lineups"
- Line 263: "the marco state" -> macro
limitations_and_societal_impact: There is limited discussions of limitations in the paper, but plenty that could be discussed to improve the paper:

(1) Elo scores are used throughout the paper to evaluate the agent, without acknowledgement of known issues with this form of evaluation when games (including MOBAs) have non-transitive interaction between policies. See for example Balduzzi et al. Re-evaluating Evaluation [NeurIPS 2018]

(2) Due to the scale of the environment, there are limited ablations and repeats. This is an existence proof of the ability for agents to play at this level, and not a rigorous study or the robustness of this approach. I think this is an acceptable form of publication, but this detail should be acknowledged.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper proposes a novel method for learning policies to play MOBA (Multi-player Online Battle Arena) games which makes use of expert supervision to extract and learn the prediction of high-level or macro goals (meta-controller) which is subsequently used to train a policy using reinforcement learning to execute each goal with a low-level micro-controller. The proposed method has an emphasis on diversity in strategies which is built-in by sampling from the meta-controller and is also evaluated with quantitative and qualitative experiments. This diversity in strategies is stated to be the reason for improvement over human performance when compared to state of the art baselines. Several qualitative demonstrations of the proposed method (Section 4.4) show how non-conventional strategies are adopted more easily than baselines. A human vs machine study is also presented for measuring the win-rate of the proposed method and baselines and how it changes with more test time i.e. how the win-rate changes when humans are able to learn and improve their own performance against the AI matchups.
main_review: Overall, the paper presents a novel method for pushing the state of the art of AI performance in MOBA games. The large-scale human study to evaluate performance against baselines and the performance exceeding the state-of-the-art baselines clearly demonstrate the effectiveness of the proposed MGG method. Specific analyses also support the diversity claims made about the learned strategies. However, the paper in its current state leaves a lot to be desired in terms of clarity, broader motivation and accessibility. Despite the thorough empirical analysis made in this paper, it remains inaccessible to the reader due to these writing issues. 
 
Below are detailed comments on the strengths and weaknesses that informed my evaluation of the paper.
 
 
## Strengths
 
1. The presented MGG method’s main selling point is the diversity in strategies that are aimed to improve performance against human matchups, since humans are capable of overcoming unimodal/repetitive strategies by AI agents given enough time and practice. The claim for diversity in the proposed method is supported by a quantitative analysis with measurements of goal-entropy and Davies-Bouldin Index of the goal embeddings, and a qualitative analysis of specific cases of non-conventional strategies in the chosen game (Honor of Kings) which typically do not align with the default reward models proposed for this game.
 
2. The large scale human-study with human vs AI matches clearly demonstrates that the proposed method retains performance (w.r.t the baseline) with increasing human test time, which is in line with the hypothesis that diverse strategies are better than unimodal strategies against human matchups.
 
3. The choice of using expert demonstrations for learning only the high-level macro-goal prediction network is intuitively appealing -- it aims to mimic broader human intentions without worrying about matching the exact execution of each intention.
 
4. For readers that are familiar with MOBA games, the two case studies in Sections 4.4.1 and 4.4.2 (and the videos linked in the paper) are impressive demonstrations of what the proposed method is capable of in practice. Such strategies seem to be very difficult to learn for AI agents but are easy or obvious for human players.
 


## Weaknesses
 

#### Lack of clarity in method description.

1. The second paragraph in Section 3 is confusing due to lack of proper explanation and use of arbitrary new notation. The difference between macro states and macro goals is not clear. The role of f i.e. the macro state extraction function is not made clear until much later in the paper. The “general reward function” is never explained -- what does this mean and what does it consist of? What does the “common strategy” mean? Why is notation introduced for human macro states when this notation is never used again in the entire paper (i.e. served no purpose later on). It could have been explained without any notation. In my opinion, the goal space should be completely defined at this stage of the paper, along with the choice of goal dimension and a description of what the dimensions of the goal specify. Also, notation for intrinsic reward R(f(S), G, A) is introduced without being accompanied by a definition (the definition is much later in the paper).

2. Auxiliary tasks are first mentioned in L135 but what auxiliary tasks are used is not mentioned until L173. Since only a single auxiliary task of auto-encoding is used, this explanation does not deserve to be pushed to such a later stage from it’s first mention.

3. L161 states that C is the number of frames between state and its goal, but it is not made clear how C is specified. Later in the paper, C is then stated to be a hyperparameter which has a fixed value. This description about C being a hyperparam of choice and its value should all be in the same first mention of C in the paper.

4. Equation 2 (and L192 below it) assume that the goal space admits a metric in the form of an L1 norm, but the goal space is not defined at this point in the paper. There is an implicit assumption that the goal space is a multi-dimensional euclidean space (R^d).
 

#### Accessibility. 

1. The paper does not introduce key MOBA-specific terminology before using it at several points in the paper. A basic explanation of the mechanics, heros and progression milestones of the game should be explained before terms like “destroying the crystal”, “over 102 heroes…” or “ban/pick capabilities” are used. However, given that this paper specifically focuses solely on MOBA games, this is a minor issue.
The paper mentions “league based” methods in the intro (L43 and L66) but does not explain what this category of prior work means until L81, where a very vague explanation is presented -- “league-based methods train multiple agents simultaneously to explore strategy space and require massive computation”.

2. The “general reward function” is mentioned several times in the paper but only introduced on L274 after citing Ye et al. [2020a] without any description. This seems to be an important fact to include in the paper well in advance.
The fact that the proposed method is an instance of an imitation learning method where expert data is used is not made clear during comparisons with baselines. It is not mentioned that the “RL-baseline” method does not use any human supervision, which is an important point to note in the comparison.
 

#### Empirical issues.

1. For the main result of AI vs human win rate obtained from the large scale human study, the Y-axis range in Figure 6 (b) seems to be extremely narrow (98.72% - 98.86%). Due to the absence of error bars, it is very difficult to judge the statistical significance of the presented results. It is also very surprising that the RL-baseline, despite lacking diversity, is able to retain such a high win rate after 26 hours of accumulated human test time. This seems to strongly oppose the premise of the paper that humans can easily overcome AI strategies if they lack diversity.

2. Lack of error bars: Figures 6, 9 and Tables 1,2,3 lack error bars, despite the authors having answered “Yes” to the checklist question about whether they have included error bars.
 

## Other minor errors

1. L259 -- A brief explanation of the Davies-Bouldin Index and its relevance should be presented instead of just citing the original paper.
2. Last line of Conclusions seems very vague, what does “non-transitivity in strategies” mean? The directions for future work can be made more clear.
3. Table 2 and Section 4.3.2 mention measuring “state entropy”. Shouldn’t this be called “macro state entropy” instead?

limitations_and_societal_impact: Yes, the authors have addressed the limitations of the paper adequately. They should also cite Ye et al. [2020a] for the detailed broader impacts that they have mentioned for MOBA game based AI.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors propose a method for using limited human data to generate macro-goals that serve as an auxiliary imitation loss and reward for deep reinforcement learning agents playing a mobile 5v5 MOBA game (Honor of Kings). The proposed approach provides 'macro' supervision for agent behavior while letting the agent learn the finer details and micro actions to reach goals or maximize game reward.
Experiments compare the proposed approach to a variety of baselines and indicate higher ELO scores and lower exploitability by human players.
main_review: # Overview

"Learning Diverse Policies in MOBA Games via Macro-Goals" presents a compelling direction for combining offline human feedback with a reinforcement learning policy, and the evaluation criteria investigating exploitability is novel and supports the paper's stated goals. Certain areas of the paper require work to help the reader reproduce the results or understand the significance of the outcomes. 

# Originality

The paper's combination of human data using imitation learning and reinforcement learning has been previously explored in the context of 'human preferences' and summarization work. The approach is novel in the context of MOBA games and demonstrates that in long-action sequences this form of supervision is applicable and can lead to demonstrably more diverse policies.

The choice of neural network, features, domain, and other aspects are not the focus of the paper, and are also not novel.

The paper measures the exploitability of the agents using human evaluations (Figure 6) and this evaluation format has not been used previously in the context of learning game AI policies. This test, while difficult to reproduce, serves as an invaluable showcase for the method's impact on realistic play. 

# Clarity

The paper's stated goals and application domain is clearly stated. Similarly, the purpose of the experiments is explained and motivated. Certain results and their applicability hinder the reading and contextualization of the work. As an example, 4.4.3 discusses capability analysis while using ELO scores. The compared approaches are stated to be 'converged', however this term is loaded and hard to transfer across architectures and setups. Similarly the results in Figure 6 and 9 do not provide statistical significance of the results, or potentially a comparison across different points during training (as a way of further confirming that the systems have indeed converged). As such, the conclusions from many of these experiments are hard to measure. A key question that should be answered is: "does using human games and an additional goal reward in this way provide better results provided with the same resources/time/energy as our reinforcement learning baseline?". Ideally a good control would allow the reinforcement learning policy to train supervised from human trajectories and then train with self-play, and account for the additional parameters and training time required by the imitation training.

Section 4.4.1 Why is achieving "resource intent strategies" important? Does the trained policy *need* to be good at this to improve at the game, or is the purpose to be as similar to human play?

Section 4.4.2 discusses human performance for monster resources: what is this human evaluation? how are they chosen? How many attempts are done by humans? Are these humans professional or amateur players?

# Significance
The proposed approach is tested on a challenging benchmark and uses significant resources to train the agents and evaluate them.
The significance of this work is limited by two key factors: 1) research in this area is important, but the proposed technique is hard to reproduce owing to the high computational cost, lack of a public interface or system to test and train Honor of Kings agents. 2) the results are provided without statistical significance numbers or selection criteria for human players. The ELO numbers and reward scores should ideally be amended in the final version of the paper with some measure to understand the size of the gains over GAIL or RL-Baseline. The human player performance is hard to asses, and perhaps an ELO score for human players, or a description of the selection process would help. (specifically the exploitability plot in Figure 6 strongly depends on the kind of players that tested either agent, and whether they were allowed to play against either system, or only one kind repeatedly, etc.)

# Areas for improvement

## Statistical significance:
Figure 6 and Figure 9 provide the most impressive results of the paper, but lack any statistical significance metrics which would help understand the gains between RL-baseline, GAIL, and MGG.

## Wording:
L249 "this test effectively proves" I would reword this as "this test suggests" or "provides evidence for" as conclusive demonstrations with human testing and a limited number of samples are unlikely, but do provide a great benchmark.
L259, L265 "marco states" -> "macro states"

## Citation style:
OpenAI Five citation should preferably use https: //openai.com/bibtex/openai2019dota.bib (e.g. OpenAI et al.)

## Related Work:
L78 "All the mentioned works except the SL model adopt a single strategy regardless of the
79 heroes and lineups, which cannot take full advantage of heroes and lineups’ characteristics" -> Slight correction: OpenAI Five has a latent vector encoding lane preferences + a representation of the heroes that affect the employed strategy, so in practice the strategy will change dramatically between games when those aspects are randomized.
limitations_and_societal_impact: # Limitations

The authors do not describe specific limitations of their work. In particular use of human data can be seen as a limitation as compared to other MOBA efforts. Furthermore the introduction of additional rewards and losses may complicate or destabilise training, and this should be addressed or discussed.

# Societal Impact

The authors properly address the societal impact of their work in the checklist and the main text.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper addresses RL agent training in MOBA games, specifically the ability to acquire a diverse set of policies when playing a variety of strategic scenarios (team compositions, aka lineups). Human example data is processes to extract future states reached from initial states, treating these future states as goals. A supervised learning model is trained to predict the future goal state from a given game state using an attention module that accounts for the strategic scenario (lineup) and game state (unit features & aggregate game stats). An existing RL framework is then augmented to condition on this goal state and trained with a reward that encourages reducing the distance to this goal in addition to the baseline reward from the environment. A wide variety of results show the method is better able to adapt to unique strategic scenarios (lineups) and exhibits a broader array of behaviors compared to the baseline agent or an imitation learning agent (GAIL) while retaining strong performance against human players.

main_review: The paper has several strengths:
- Strong empirical results.
- Variety of experimental comparisons. The experiments test a variety of features of the approach in terms of diversity of strategies, adaptability to new lineups, and overall competitive performance.

The paper's primary limitations:
- Limited novelty: Using imitation learning to bootstrap long horizon decisions from human data is not novel per se. The novelty lies in the combination of elements.
- Lacking discussion of limitations and remaining open problems. As detailed below there are some areas where there is a gap between the performance of the method and the ideal that should be discussed.


Below are comments specific to the review criteria prompts.

# Originality

Using imitation learning to bootstrap long horizon decisions from human data is not novel, nor is goal-conditioned learning. The primary technical novelties here lie in the specific modeling choices and their integration. I believe this integration is novel and valuable given the complexity of the domain and problem. It is clear this builds on prior efforts while enabling diversification of policies in a different way.

I am not sure if the attention mechanism employed for learning goals to predict is specifically new, but it seems to be.

Related work is adequately cited for the relevant techniques to compare.

# Quality

The results partially support the claims. Some specific points:
1. The new method does not seem to improve over the baseline agent against humans: Figure 6b shows a ~0.04% difference in (cumulative) win rate. The text claims this is superior performance but it does not seem obvious that is true (and there is no statistical test as the evaluation was performed once).
2. MGG does cover more of behavior space compared to the RL-baseline or players (Figure 7). But MGG also more parts of the space that players do not occupy. It is not clear whether this is a good thing or a problem.
3. There is no comparison of computational costs of the techniques. While greater efficiency is not claimed per se, this would help complete the picture.
4. Figure 8 results show a substantial gap remaining between the MGG method and player behavior.

The methods employed are appropriate for training RL agents in team games, and incorporating goal prediction into the policy training is clear incremental improvement over prior work.

The paper is not careful to address weaknesses or remaining gaps. Points (1), (2), and (4) above were not addressed. Generally limitations are not discussed. The checklist indicates assumptions were addressed in section 3, but there is no explicit text to this point.


# Clarity

The text can be difficult to read at points, but is generally understandable.

The organization is adequate and the reader is informed of relevant methods and prior work related to the techniques (like double clipping).

Some clarifications and additions that would improve the text:
- What were the instructions given to participants?
- What is the value of the in-game rewards given to participants? 
- How was consent obtained from participants?
- (describing supervised learning) "Finally, the macro-goal is predicted via a convergence model of 1 million steps." Not sure what this means. Does this indicate the model trained for 1 million steps and obtained convergence at that point?
- Table 3: How were the human values for different goals determined? For the RL agent we know the conditioned goal but it was not clear how the human goal was decided.
- Consider briefly explaining the Davies-Bouldin index for readers unfamiliar with clustering.


# Significance

The results are modestly important. Researchers looking to condition agent behavior on human examples of strategic choices could adapt the work to other domains. This will be valuable to those training RL agents starting from imitation learning in domains where strategic or other long-term choices are relevant.

The paper advances the state of the art in getting diversity (without compromising quality) for MOBA games in specific and likely for RL game AI agents in general. The comparison to GAIL shows this is superior to at least some alternatives for the same problem.
limitations_and_societal_impact: No. Potential negative societal impacts are not discussed. MOBAs as real-time control problems feel like natural models for military automation tasks (training agents to perform various team activities). The paper-specific design to allow specification of a goal that controls agent behavior is easily applied to controlling RL agents toward whatever an actor using the agent may desire, including potentially harmful actions.

No limitations of what the method can achieve is discussed. These are explained above under "Quality". For example, the Figure 8 results show a substantial gap remaining between the MGG method and player behavior - why?. In Figure 7 MGG produces large clusters of behaviors human players do not - what might this indicate? In Figure 6b the differences between MGG and RL-baseline around on the order of 0.04% - is this statistically significant? Why is it so small? None of these are major failings, but they are not framed as limitations to be considered or explained.
review_title: null

Meta-review: After reading the reviews, authors response and discussions, I suggest to accept the paper. The ethical concerns have been answered and the authors took action to conform to what was required by the ethical review. Questions and some concerns on the method have been answered by the authors during the rebuttal and one reviewer upgraded its score from weak reject to a weak accept. The authors try to tackle the problem of learning in a real-time multi-player game which is extremely difficult. Even if the method is not theoretically sound and heavily rely on human knowledge, it is still a technical prowess to achieve good performance on such games. I will also suggest the authors to change their claim concerning the analysis of Fig 6 (b) where the difference between RL and their method is extremely minor. 
Decision: Accept (Poster)