Superior Performance with Diversified Strategic Control in FPS Games Using General Reinforcement Learning

This paper offers an overall solution for first-person shooter ( FPS ) games to achieve superior performance using general reinforcement learning ( RL ) . We introduce an agent in ViZDoom that can surpass previous top agents ranked in the open ViZDoom AI Competitions by a large margin . The proposed framework consists of a number of generally applicable techniques , including hindsight experience replay ( HER ) based navigation , hindsight proximal policy optimization ( HPPO ) , rule-guided policy search ( RGPS ) , prioritized fictitious self-play ( PFSP ) , and diversified strategic control ( DSC ) . The proposed agent outperforms existing agents by taking advantage of diversified and human-like strategies , instead of larger neural networks , more accurate frag skills , or hand-craft tricks , etc . We provide comprehensive analysis and experiments to elaborate the effect of each component in affecting the agent performance , and demonstrate that the proposed and adopted techniques are important to achieve superior performance in general end-to-end FPS games . The proposed methods can contribute to other games and real-world tasks which also require spatial navigation and diversified behaviors . 1 INTRODUCTION . Games have been considered as challenging benchmarks in evaluating reinforcement learning ( RL ) algorithms . In games , agent can feel free to explore through infinite trial and error . OpenAI Gym ( Brockman et al. , 2016 ) has provided many wrapped game environments , such as Atari games ( Bellemare et al. , 2013 ) and Mujuco robotic control problems ( Todorov et al. , 2012 ) , through an unified interface . Meanwhile , many RL algorithms , such as TRPO ( Schulman et al. , 2015 ) , PPO ( Schulman et al. , 2017 ) , DQN ( Mnih et al. , 2015 ) and DDPG ( Lillicrap et al. , 2019 ) , etc. , have been demonstrated to achieve superior performance in these environments . Recent advances in solving more complex games , including GO ( Silver et al. , 2016 ) , DOTA 2 ( Berner et al. , 2019 ) , and StarCraft II ( Vinyals et al. , 2019 ) , further demonstrate that general RL method can be widely applied in solving simulated environments . First-person shooter ( FPS ) games , such as Quake ( Jaderberg et al. , 2019 ) and Doom ( Kempka et al. , 2016 ) , are also known by their complexity , while they are considered to be closer to real-world tasks because solving FPS games requires perfect navigation skills conditioning on raw screen/camera input . Normally , the partially observed screen only contains limited information , and the agent can not obtain the global state of the environment and the information of other agents . Many reallife applications , such as searching and rescuing robotics , and autonomous driving , etc. , are very similar to FPS games . Another challenge posed in these environments is that the state transition is not static . However , most existing intelligent agents are specifically trained to work well in some fixed environments , while they are unable to act diversely when the environment dynamics changes . In this paper , we take FPS games as the benchmark environment . We propose an overall solution to train intelligent agent in FPS games that can dynamically adjust its strategy according to environmental changes induced by diverse opponents . Specifically , we focus on the game of ViZDoom . We introduce an agent with diversified strategic control ( DSC ) that can surpass previous top agents ranked in the open ViZDoom AI Competitions 1,2 by a large margin . The solution framework proposed in this paper consists of a number of general techniques in RL , including hindsight experience replay ( HER ) ( Andrychowicz et al. , 2017 ) , hindsight proximal policy optimization ( HPPO ) , rule-guided policy search ( RGPS ) ( Han et al. , 2021 ) , prioritized fictitious self-play ( PFSP ) ( Vinyals et al. , 2019 ) , and diversified strategic control ( DSC ) , etc . We conduct comprehensive experimental results to show the importance of each of the introduced techniques in training intelligent agents . Our proposed methods can contribute to other games and real-world tasks which require spatial navigation and diversified behaviors that are similar to FPS games . 2 RELATED WORK . The proposed learning framework is based on the widely adopted actor-critic RL methods , in which a value function and a policy are learned simultaneously . Many algorithms , such as TRPO ( Schulman et al. , 2015 ) , PPO ( Schulman et al. , 2017 ) , DDPG ( Lillicrap et al. , 2019 ) and TD3 ( Fujimoto et al. , 2018 ) , etc. , all follow the actor-critic architecture and have achieved state-of-the-art results in various applications . In our learning framework , we will adopt PPO as a baseline RL algorithm . In ViZDoom , a most fundamental skill that the agent has to learn is navigation . Considering a navigation task , we need to define a target in advance . Therefore , the navigation problem in ViZDoom can naturally be viewed as a goal-conditioned task , which has been well-studied in the RL literature . Among many advanced algorithms , hindsight experience replay ( HER ) ( Andrychowicz et al. , 2017 ) provides a smart solution that it replaces the goal in failed experiences with any practically achieved one to pretend that the agent obtains a positive reward . By doing so , the agent has a much higher chance to see successful trajectories . There have been rich approaches demonstrating that HER and its variants are effective in solving goal-conditioned tasks . In complex tasks like ViZDoom , the problem can usually be decomposed into multiple stages . As we will show in the method section , our learning framework consists of three training stages , each of which focuses on solving a specific problem . One challenge in multi-stage training is that the agent at a later stage is easy to forget what has been learned in early stages . To avoid this embarrassing situation , policy distillation ( Rusu et al. , 2015 ) has been imported to keep the training policy staying close to the parameters trained in earlier stages . For example , in ( Vinyals et al. , 2019 ) , the training policy in RL phase is kept close to the intial policy trained using supervised learning via a policy distillation term . Our learning framework also utilize policy distillation as an important component . To obtain intelligent and superior AI agents in competitive multi-agent games , self-play ( SP ) ( Silver et al. , 2018 ) is often necessary to generate high-quality competitions . It conducts an automatic curriculum learning by letting the agent combat with itself or its own historical models . In addition to AlphaGo ( Silver et al. , 2016 ) , SP has been demonstrated effective in many other complex games , such as hide-and-seek environment ( Baker et al. , 2019 ) and DOTA 2 ( Berner et al. , 2019 ) . Variants of SP , such prioritized fictitious self-play ( PFSP ) proposed in AlphaStar ( Vinyals et al. , 2019 ) , have also been verified to be effective . We will use SP and PFSP in our learning framework to train intelligent agents . 3 BACKGROUND IN VIZDOOM . ViZDoom ( Kempka et al. , 2016 ) is a complex FPS game , and its formal competition scheme is using the Deathmatch mode , in which all agents fight against each other . In this mode , agent with the highest Frag score , which is defined as defeating counts minus the suicide counts , wins the game . Unlike the games of Go , Atari , and Starcraft II , where the player only needs to interact with the environment or compete with another opponent in one versus one zero-sum game , in the Deathmatch of ViZDoom Competitions in 2016 and 2017 , the player needs to fight against other 7 independent agents with diverse strategies . ViZDoom is a multi-agent environment with imperfect information that each agent can only observes very limited information in the environment at a time step . In addition , the agent takes raw image 1http : //vizdoom.cs.put.edu.pl/competitions/vdaic-2016-cig 2http : //vizdoom.cs.put.edu.pl/competitions/vdaic-2017-cig as input and we need to train end-to-end policy , where the state space is of large image size . These characteristics of ViZDoom pose great challenges to training the agent from scratch . There are already some works that have achieved well performance in ViZDoom . Clyde ( Ratcliffe et al. , 2017 ) imports the LSTM ( Hochreiter & Schmidhuber , 1997 ) structure to the Asynchronous Advantage Actor-Critic ( A3C ) algorithm ( Mnih et al. , 2016 ) , resulting a basic intelligent agent in ViZDoom . F1 ( Wu & Tian , 2016 ) who won the championship of ViZDoom AI Competitions 2016 is also trained with A3C algorithm . It also uses human-prior knowledge to conduct reward shaping and curriculum design to assist the agent learning . There is also another work ( Huang et al. , 2019 ) using deep recurrent Q-learning network as a high-level controller . It combines auxiliary tasks ( opponent detection and depth prediction ) to manage the combo-actions . However , these agents all have a common shortcoming : the strategy of their agents is fixed . For example , in the map ( see Figure . 1 ) of ViZDoom , one agent prefers to circle around the outer loop ( green line ) , and the other agent prefers to act in the middle magma area ( red line ) . These two agents may never meet each other in a competition game , and thus can not get Frag scores either . Therefore , how to obtain diversified strategy in ViZDoom still remains an open problem in the literature . 4 METHODOLOGY . The entire learning framework consists of three stages of training , by decomposing the complex task in the ViZDoom game into different levels of control , i.e. , navigation , frag and strategic control . For complex systems , multi-stage learning is necessary and more efficient to build strong artificial intelligence . An overview of our multi-stage learning framework is shown in Figure 2 . We briefly explain the framework here and elaborate the details in the following subsections . In stage 1 , we aim to obtain an expert navigation agent , which only takes moving actions . Specifically , we enable the agent to navigate by following a given target , and this is very useful for stage 3 when training the agent at a strategic level . Apparently , navigation to a given target is the essential problem considered by goal-conditioned RL ( Andrychowicz et al. , 2017 ) . It has been well demonstrated that RL methods using hindsight experience replay ( HER ) ( Andrychowicz et al. , 2017 ) are effective in solving goal-conditioned tasks . Therefore , we adopt HER in our navigation stage . We are not aware of any game AIs that explicitly take advantage of HER . The original HER method is implemented with the off policy method such as DDPG and TD3 . For training strong AIs in complex video games , it has been demonstrated that actor-critic algorithms , such as PPO , with discrete actions are more effective ( Schulman et al. , 2017 ) . In order to apply PPO and HER together , we adopt the same idea as the recent Hindsight Trust Region Policy Optimization ( HTRPO ) method ( Zhang et al. , 2021 ) , proposed the Hindsight Proximal Policy Optimization method , which we call HPPO . Based on an excellent navigation policy , in the second training stage , we let the agent participate in the formal battle game in ViZDoom , which is also termed as ‘ Frag ’ in ViZDoom . At this stage , we use Prioritized Fictitious Self-Play ( PFSP ) ( Vinyals et al. , 2019 ) to let the training agent play against itself and its historical versions with prioritization . To conveniently reuse the navigation policy , we create a new action head for frag in addition to the moving actions . So far , the trained agent can play formal ViZDoom games by randomly feeding a navigation target to the policy . In a formal ViZDoom match in the historical competitions , one player will play against another 7 players in the map . Therefore , in the last stage , we propose to learn a policy at a strategic level to output a specific target , so that the agent itself can decide where to navigate . Again , to smoothly reuse the pre-trained policy in the previous two stages , we create a new action head for deciding the target . In this way , the agent should be aware of its opponents ’ strategies and then decide where to go to encounter them . A detailed infrastructure of the proposed multi-stage learning system has been depicted in Figure 2 .

Summary:  The authors combined several different techniques from RL field to solve FPS game, ViZDoom. The game is splitted into three stages: navigation, frag and strategic control, respectively and the authors proposed a multi-stage learning system to tackle this problem. Navigation stage is solved by combining hindsight experience replay with PPO, which is named HPPO. Rule-guided policy search is used in frag stage and self-play is used for strategic control stage. Along the three stages, the trained model in the previous stage is always reused by adding new head to the neural network. The proposed multi-stage learning system outperforms SOTA agents(top ranking agents in open ViZDoom AI competitions) by a large margin. They also showed other advantages of the proposed system, such as learned skills are more diverse instead of fixed.