Superior Performance with Diversified Strategic Control in FPS Games Using General Reinforcement Learning

This paper offers an overall solution for first-person shooter ( FPS ) games to achieve superior performance using general reinforcement learning ( RL ) . We introduce an agent in ViZDoom that can surpass previous top agents ranked in the open ViZDoom AI Competitions by a large margin . The proposed framework consists of a number of generally applicable techniques , including hindsight experience replay ( HER ) based navigation , hindsight proximal policy optimization ( HPPO ) , rule-guided policy search ( RGPS ) , prioritized fictitious self-play ( PFSP ) , and diversified strategic control ( DSC ) . The proposed agent outperforms existing agents by taking advantage of diversified and human-like strategies , instead of larger neural networks , more accurate frag skills , or hand-craft tricks , etc . We provide comprehensive analysis and experiments to elaborate the effect of each component in affecting the agent performance , and demonstrate that the proposed and adopted techniques are important to achieve superior performance in general end-to-end FPS games . The proposed methods can contribute to other games and real-world tasks which also require spatial navigation and diversified behaviors . 1 INTRODUCTION . Games have been considered as challenging benchmarks in evaluating reinforcement learning ( RL ) algorithms . In games , agent can feel free to explore through infinite trial and error . OpenAI Gym ( Brockman et al. , 2016 ) has provided many wrapped game environments , such as Atari games ( Bellemare et al. , 2013 ) and Mujuco robotic control problems ( Todorov et al. , 2012 ) , through an unified interface . Meanwhile , many RL algorithms , such as TRPO ( Schulman et al. , 2015 ) , PPO ( Schulman et al. , 2017 ) , DQN ( Mnih et al. , 2015 ) and DDPG ( Lillicrap et al. , 2019 ) , etc. , have been demonstrated to achieve superior performance in these environments . Recent advances in solving more complex games , including GO ( Silver et al. , 2016 ) , DOTA 2 ( Berner et al. , 2019 ) , and StarCraft II ( Vinyals et al. , 2019 ) , further demonstrate that general RL method can be widely applied in solving simulated environments . First-person shooter ( FPS ) games , such as Quake ( Jaderberg et al. , 2019 ) and Doom ( Kempka et al. , 2016 ) , are also known by their complexity , while they are considered to be closer to real-world tasks because solving FPS games requires perfect navigation skills conditioning on raw screen/camera input . Normally , the partially observed screen only contains limited information , and the agent can not obtain the global state of the environment and the information of other agents . Many reallife applications , such as searching and rescuing robotics , and autonomous driving , etc. , are very similar to FPS games . Another challenge posed in these environments is that the state transition is not static . However , most existing intelligent agents are specifically trained to work well in some fixed environments , while they are unable to act diversely when the environment dynamics changes . In this paper , we take FPS games as the benchmark environment . We propose an overall solution to train intelligent agent in FPS games that can dynamically adjust its strategy according to environmental changes induced by diverse opponents . Specifically , we focus on the game of ViZDoom . We introduce an agent with diversified strategic control ( DSC ) that can surpass previous top agents ranked in the open ViZDoom AI Competitions 1,2 by a large margin . The solution framework proposed in this paper consists of a number of general techniques in RL , including hindsight experience replay ( HER ) ( Andrychowicz et al. , 2017 ) , hindsight proximal policy optimization ( HPPO ) , rule-guided policy search ( RGPS ) ( Han et al. , 2021 ) , prioritized fictitious self-play ( PFSP ) ( Vinyals et al. , 2019 ) , and diversified strategic control ( DSC ) , etc . We conduct comprehensive experimental results to show the importance of each of the introduced techniques in training intelligent agents . Our proposed methods can contribute to other games and real-world tasks which require spatial navigation and diversified behaviors that are similar to FPS games . 2 RELATED WORK . The proposed learning framework is based on the widely adopted actor-critic RL methods , in which a value function and a policy are learned simultaneously . Many algorithms , such as TRPO ( Schulman et al. , 2015 ) , PPO ( Schulman et al. , 2017 ) , DDPG ( Lillicrap et al. , 2019 ) and TD3 ( Fujimoto et al. , 2018 ) , etc. , all follow the actor-critic architecture and have achieved state-of-the-art results in various applications . In our learning framework , we will adopt PPO as a baseline RL algorithm . In ViZDoom , a most fundamental skill that the agent has to learn is navigation . Considering a navigation task , we need to define a target in advance . Therefore , the navigation problem in ViZDoom can naturally be viewed as a goal-conditioned task , which has been well-studied in the RL literature . Among many advanced algorithms , hindsight experience replay ( HER ) ( Andrychowicz et al. , 2017 ) provides a smart solution that it replaces the goal in failed experiences with any practically achieved one to pretend that the agent obtains a positive reward . By doing so , the agent has a much higher chance to see successful trajectories . There have been rich approaches demonstrating that HER and its variants are effective in solving goal-conditioned tasks . In complex tasks like ViZDoom , the problem can usually be decomposed into multiple stages . As we will show in the method section , our learning framework consists of three training stages , each of which focuses on solving a specific problem . One challenge in multi-stage training is that the agent at a later stage is easy to forget what has been learned in early stages . To avoid this embarrassing situation , policy distillation ( Rusu et al. , 2015 ) has been imported to keep the training policy staying close to the parameters trained in earlier stages . For example , in ( Vinyals et al. , 2019 ) , the training policy in RL phase is kept close to the intial policy trained using supervised learning via a policy distillation term . Our learning framework also utilize policy distillation as an important component . To obtain intelligent and superior AI agents in competitive multi-agent games , self-play ( SP ) ( Silver et al. , 2018 ) is often necessary to generate high-quality competitions . It conducts an automatic curriculum learning by letting the agent combat with itself or its own historical models . In addition to AlphaGo ( Silver et al. , 2016 ) , SP has been demonstrated effective in many other complex games , such as hide-and-seek environment ( Baker et al. , 2019 ) and DOTA 2 ( Berner et al. , 2019 ) . Variants of SP , such prioritized fictitious self-play ( PFSP ) proposed in AlphaStar ( Vinyals et al. , 2019 ) , have also been verified to be effective . We will use SP and PFSP in our learning framework to train intelligent agents . 3 BACKGROUND IN VIZDOOM . ViZDoom ( Kempka et al. , 2016 ) is a complex FPS game , and its formal competition scheme is using the Deathmatch mode , in which all agents fight against each other . In this mode , agent with the highest Frag score , which is defined as defeating counts minus the suicide counts , wins the game . Unlike the games of Go , Atari , and Starcraft II , where the player only needs to interact with the environment or compete with another opponent in one versus one zero-sum game , in the Deathmatch of ViZDoom Competitions in 2016 and 2017 , the player needs to fight against other 7 independent agents with diverse strategies . ViZDoom is a multi-agent environment with imperfect information that each agent can only observes very limited information in the environment at a time step . In addition , the agent takes raw image 1http : //vizdoom.cs.put.edu.pl/competitions/vdaic-2016-cig 2http : //vizdoom.cs.put.edu.pl/competitions/vdaic-2017-cig as input and we need to train end-to-end policy , where the state space is of large image size . These characteristics of ViZDoom pose great challenges to training the agent from scratch . There are already some works that have achieved well performance in ViZDoom . Clyde ( Ratcliffe et al. , 2017 ) imports the LSTM ( Hochreiter & Schmidhuber , 1997 ) structure to the Asynchronous Advantage Actor-Critic ( A3C ) algorithm ( Mnih et al. , 2016 ) , resulting a basic intelligent agent in ViZDoom . F1 ( Wu & Tian , 2016 ) who won the championship of ViZDoom AI Competitions 2016 is also trained with A3C algorithm . It also uses human-prior knowledge to conduct reward shaping and curriculum design to assist the agent learning . There is also another work ( Huang et al. , 2019 ) using deep recurrent Q-learning network as a high-level controller . It combines auxiliary tasks ( opponent detection and depth prediction ) to manage the combo-actions . However , these agents all have a common shortcoming : the strategy of their agents is fixed . For example , in the map ( see Figure . 1 ) of ViZDoom , one agent prefers to circle around the outer loop ( green line ) , and the other agent prefers to act in the middle magma area ( red line ) . These two agents may never meet each other in a competition game , and thus can not get Frag scores either . Therefore , how to obtain diversified strategy in ViZDoom still remains an open problem in the literature . 4 METHODOLOGY . The entire learning framework consists of three stages of training , by decomposing the complex task in the ViZDoom game into different levels of control , i.e. , navigation , frag and strategic control . For complex systems , multi-stage learning is necessary and more efficient to build strong artificial intelligence . An overview of our multi-stage learning framework is shown in Figure 2 . We briefly explain the framework here and elaborate the details in the following subsections . In stage 1 , we aim to obtain an expert navigation agent , which only takes moving actions . Specifically , we enable the agent to navigate by following a given target , and this is very useful for stage 3 when training the agent at a strategic level . Apparently , navigation to a given target is the essential problem considered by goal-conditioned RL ( Andrychowicz et al. , 2017 ) . It has been well demonstrated that RL methods using hindsight experience replay ( HER ) ( Andrychowicz et al. , 2017 ) are effective in solving goal-conditioned tasks . Therefore , we adopt HER in our navigation stage . We are not aware of any game AIs that explicitly take advantage of HER . The original HER method is implemented with the off policy method such as DDPG and TD3 . For training strong AIs in complex video games , it has been demonstrated that actor-critic algorithms , such as PPO , with discrete actions are more effective ( Schulman et al. , 2017 ) . In order to apply PPO and HER together , we adopt the same idea as the recent Hindsight Trust Region Policy Optimization ( HTRPO ) method ( Zhang et al. , 2021 ) , proposed the Hindsight Proximal Policy Optimization method , which we call HPPO . Based on an excellent navigation policy , in the second training stage , we let the agent participate in the formal battle game in ViZDoom , which is also termed as ‘ Frag ’ in ViZDoom . At this stage , we use Prioritized Fictitious Self-Play ( PFSP ) ( Vinyals et al. , 2019 ) to let the training agent play against itself and its historical versions with prioritization . To conveniently reuse the navigation policy , we create a new action head for frag in addition to the moving actions . So far , the trained agent can play formal ViZDoom games by randomly feeding a navigation target to the policy . In a formal ViZDoom match in the historical competitions , one player will play against another 7 players in the map . Therefore , in the last stage , we propose to learn a policy at a strategic level to output a specific target , so that the agent itself can decide where to navigate . Again , to smoothly reuse the pre-trained policy in the previous two stages , we create a new action head for deciding the target . In this way , the agent should be aware of its opponents ’ strategies and then decide where to go to encounter them . A detailed infrastructure of the proposed multi-stage learning system has been depicted in Figure 2 .

Summary: This paper proposes a multi-stage learning framework for training high-performance agents in FPS games. Its agent can dynamically adjust its strategy according to different opponents, which is simply controlled by selecting different target areas. The paper propose a methodology called Hindsight PPO to solve goal-conditioned RL, though it may have some concerns. The framework combines several existing techniques such as rule-guided policy search,  action wrapper, prioritized fictitious self-play, etc. Experiments show that the framework learns an agent that can surpass previous top agents.