Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games

Many artificial intelligence (AI) applications often require multiple intelligent agents to work in a collaborative effort. Efficient learning for intra-agent communication and coordination is an indispensable step towards general AI. In this paper, we take StarCraft combat game as a case study, where the task is to coordinate multiple agents as a team to defeat their enemies. To maintain a scalable yet effective communication protocol, we introduce a Multiagent BidirectionallyCoordinated Network (BiCNet [’bIknet]) with a vectorised extension of actor-critic formulation. We show that BiCNet can handle different types of combats with arbitrary numbers of AI agents for both sides. Our analysis demonstrates that without any supervisions such as human demonstrations or labelled data, BiCNet could learn various types of advanced coordination strategies that have been commonly used by experienced game players. In our experiments, we evaluate our approach against multiple baselines under different scenarios; it shows state-of-the-art performance, and possesses potential values for large-scale real-world applications. Introduction The last decade has witnessed massive progresses in the field of Artificial Intelligence (AI). With supervision from labelled data, machines have, to some extent, exceeded humanlevel perception on visual recognitions and speech recognitions, while fed with feedback reward, single AI units (aka agents) defeat humans in various games including Atari video games (Mnih et al. 2015), Go game (Silver et al. 2016), and card game (Brown and Sandholm 2017). Yet, true human intelligence embraces social and collective wisdom which lays an essential foundation for reaching the grand goal of Artificial General Intelligence (AGI) (Goertzel and Pennachin 2007). As demonstrated by crowd sourcing, aggregating efforts collectively from the public would solve the problem that otherwise is unthinkable by a single person. Even social animals like a brood of well-organised ants could accomplish challenging tasks such as hunting, building a kingdom, and even waging a war, although each ant by itself is weak and limited. Interestingly, in the coming era of algorithmic economy, AI agents with a certain rudimentary level of artificial collective intelligence start to emerge from multiple domains. Typical examples include the trading robots ∗Previously as title: “Multiagent Bidirectionally-Coordinated Nets for Learning to Play StarCraft Combat Games”, Mar 2017. †The first two authors have equal contributions. Correspondence to Jun Wang, jun.wang@cs.ucl.ac.uk. gaming on the stock markets (Deboeck 1994), ad bidding agents competing with each other over online advertising exchanges (Wang, Zhang, and Yuan 2017), and e-commerce collaborative filtering recommenders predicting user interests through the wisdom of the crowd (Schafer, Konstan, and Riedl 1999). We thus believe a next grand challenge of AGI is to answer how multiple AI agents could learn human-level collaborations, or competitions, from their experiences with the environment where both of their incentives and economic constraints co-exist. As the flourishes of deep reinforcement learning (DRL) (Mnih et al. 2015; Silver et al. 2016), researchers start to shed light on tackling multiagent problems (Schmidhuber 1996) with the enhanced learning capabilities, e.g., (Sukhbaatar, Fergus, and others 2016; Mordatch and Abbeel 2017). In this paper, we leverage a real-time strategy game, StarCraft1, as the use case to explore the learning of intelligent collaborative behaviours among multiple agents. Particularly, we focus on StarCraft micromanagement tasks (Synnaeve et al. 2016), where each player controls their own units (with different functions to collaborate) to destroy the opponents army in the combats under different terrain conditions. Such game is considered as one of the most difficult games for computers with more possible states than Go game (Synnaeve et al. 2016). The learning of this large-scale multiagent system faces a major challenge that the parameters space grows exponentially with the increasing number of agents involved. As such, the behaviours of the agents can become so sophisticated that any joint learner method (Sukhbaatar, Fergus, and others 2016) would be inefficient and unable to deal with the changing number of agents in the game. We formulate multiagent learning for StarCraft combat tasks as a zero-sum Stochastic Game. Agents are communicated by our proposed bidirectionally-coordinated net (BiCNet), while the learning is done using a multiagent actor-critic framework. In addition, we also introduce parameter sharing to solve the scalability issue. We observe that BiCNet can automatically learn various optimal strategies to coordinating agents, similar to what experienced human players would adopt in playing the StarCraft game, ranging from trivial move without collision to a basic tactic hit and run to sophisticated cover attack, and focus fire without overkill. We have conducted our experiments by testing over a set of combat tasks with different levels of difficulties. Our method Trademark of Blizzard Entertainment. ar X iv :1 70 3. 10 06 9v 4 [ cs .A I] 1 4 Se p 20 17 outperforms state-of-the-art methods and shows its potential usage in a wide range of multiagent tasks in the real-world applications.