Superior Performance with Diversified Strategic Control in FPS Games Using General Reinforcement Learning
Authors: Shuxing Li
Jiawei Xu
Chun Yuan
peng sun
Zhuobin Zheng
Zhengyou Zhang
Lei Han
Venue: ICLR 2022 Submitted (2022)
Keywords: Reinforcement Learning
Hindsight Experience Replay
FPS Games

Abstract: This paper offers an overall solution for first-person shooter (FPS) games to achieve superior performance using general reinforcement learning (RL). We introduce an agent in ViZDoom that can surpass previous top agents ranked in the open ViZDoom AI Competitions by a large margin. The proposed framework consists of a number of generally applicable techniques, including hindsight experience replay (HER) based navigation, hindsight proximal policy optimization (HPPO), rule-guided policy search (RGPS), prioritized fictitious self-play (PFSP), and diversified strategic control (DSC). The proposed agent outperforms existing agents by taking advantage of diversified and human-like strategies, instead of larger neural networks, more accurate frag skills, or hand-craft tricks, etc. We provide comprehensive analysis and experiments to elaborate the effect of each component in affecting the agent performance, and demonstrate that the proposed and adopted techniques are important to achieve superior performance in general end-to-end FPS games. The proposed methods can contribute to other games and real-world tasks which also require spatial navigation and diversified behaviors.

Under review as a conference paper at ICLR 2022
SUPERIOR PERFORMANCE WITH DIVERSIFIED STRATEGIC CONTROL IN FPS GAMES USING GEN-
ERAL REINFORCEMENT LEARNING
Anonymous authors Paper under double-bling blind review
ABSTRACT
This paper offers overall solution for first-person shooter (FPS) games achieve superior performance using general reinforcement learning (RL). We in- troduce agent in ViZDoom that can surpass previous top agents ranked in the open ViZDoom Al Competitions by a large margin. The proposed framework consists of number of generally applicable techniques, including hindsight perience replay (HER) based navigation, hindsight proximal policy optimization (HPPO), rule-guided policy search (RGPS), prioritized fictitious self-play (PFSP), and diversified strategic control (DSC). The proposed agent outperforms existing agents by taking advantage of diversified and human-like strategies, instead of larger neural networks, more accurate frag skills, or hand-craft tricks, etc. We provide comprehensive analysis and experiments elaborate the effect of each component in affecting the agent performance, and demonstrate that the proposed and adopted techniques are important to achieve superior performance in general end-to FPS games. The proposed methods contribute to other games and real-world tasks which also require spatial navigation and diversified behaviors.
INTRODUCTION
Games have been considered as challenging benchmarks evaluating reinforcement learning (RL) algorithms. In games, agent can feel free to explore through infinite trial and error. OpenAl Gym (Brockman et al. (0166) has provided many wrapped game environments, such as Atari games (Bellemare al. 2013) and Mujuco robotic control problems (Todorov et al. 2012). through an unified interface. Meanwhile, many RL algorithms, such as TRPO (Schulman et al. 2015), PPO (Schulman et al. 2017). DQN (Mnih al.) 2013) and DDPG (Lillicrap et al. 2019). etc., have been demonstrated to achieve superior performance in these environments. Recent advances in solving more complex games, including GO (Silver et al., (016)), DOTA (Berner et al. 2019),
and StarCraft II (Vinyals et al.) 2019), further demonstrate that general RL method can be widely applied in solving simulated environments.
First-person shooter (FPS) games, such as Quake (Jaderberg et al. 2019) and Doom (Kempka et al. 2016), also known by their complexity, while they are considered be closer to real-world tasks because solving FPS games requires perfect navigation skills conditioning on raw screen/camera input. Normally, the partially observed screen only contains limited information, and the agent cannot obtain the global state of the environment and the information of other agents. Many real- life applications, such as searching and rescuing robotics, and autonomous driving, etc., are very similar to FPS games. Another challenge posed in these environments is that the state transition is not static. However, most existing intelligent agents are specifically trained to work well in some fixed environments, while they unable to act diversely when the environment dynamics changes.
In this paper, we take FPS games as the benchmark environment. We propose an overall solution to train intelligent agent in FPS games that dynamically adjust its strategy according envi- ronmental changes induced by diverse opponents. Specifically, we focus on the game ViZDoom. We introduce an agent with diversified strategic control (DSC) that can surpass previous top agents

---

Under review as a conference paper at ICLR 2022
ranked in the open ViZDoom AI Competitions by large margin. The solution framework proposed in this paper consists of number of general techniques in RL, including hindsight expe- rience replay (HER) (Andrychowicz et al.] 2017), hindsight proximal policy optimization (HPPO), rule-guided policy search (RGPS) (Han et al., (021), , prioritized fictitious self-play (PFSP) (Vinyals) et al.) (019)), and diversified strategic control (DSC), etc. We conduct comprehensive experimental results to show the importance of each of the introduced techniques in training intelligent agents. Our proposed methods can contribute to other games and real-world tasks which require spatial navigation and diversified behaviors that are similar to FPS games.
RELATED WORK
The proposed learning framework is based on the widely adopted actor-critic RL. methods, in which a value function and policy are learned simultaneously. Many algorithms, such TRPO (Schulman et al. 2015), PPO (Schulman et al.) 2017). DDPG (Lillicrap et al.) 2019) and TD3 (Fujimoto et al.) 2018 etc., all follow the actor-critic architecture and have achieved state-of- results in various applications. In our learning framework, we will adopt PPO as a baseline RL algorithm.
In ViZDoom, most fundamental skill that the agent has to learn navigation. Considering a nav- igation task, we need to define target in advance. Therefore, the navigation problem in ViZDoom can naturally be viewed as coal-conditioned task, which has been well-studied in the RL literature. Among many advanced algorithms, hindsight experience replay (HER) (Andrychowicz, et al., 2017) provides a smart solution that it replaces the goal in failed experiences with any practically achieved one to pretend that the agent obtains a positive reward. By doing so, the agent has much higher chance to see successful trajectories. There have been rich approaches demonstrating that HER and its variants are effective in solving condicoond tasks.
In complex tasks like ViZDoom, the problem can usually be decomposed into multiple stages. As will show in the method section, our learning framework consists of three training stages, each of which focuses on solving a specific problem. One challenge in multi-stage training is that the agent later stage is easy forget what has been learned in early stages. To avoid this embarrassing situation, policy distillation (Rusu et al. 2015) has been imported to keep the training policy staying close to the parameters trained in earlier stages. For example, in (Vinyals al. (019). the training policy in RL phase is kept close to the intial policy trained using supervised learning via a policy distillation term. Our learning framework also utilize policy distillation as an important component.
To obtain intelligent and superior AI agents in competitive multi-agent games, self-play (SP) (Silver al. 2018) is often necessary to generate high-quality competitions. conducts an automatic curriculum learning by letting the agent combat with itself or its own historical models. In addition to AlphaGo (Silver et al., 2016)) SP has been demonstrated effective in many other complex games, such as hide-and-seek environment (Baker et al. 2019) and DOTA 2 (Berner et al.) 2019). Variants of SP, such prioritized fictitious self-play (PFSP) proposed in AlphaStar (Vinyals et al. 2011)). have also been verified to be effective. We will use SP and PFSP in our learning framework to train intelligent agents.
BACKGROUND IN VIZDOOM
ViZDoom (Kempka et al. 2016) is a complex FPS game, and its formal competition scheme is using the Deathmatch mode, in which all agents fight against each other. In this mode, agent with the highest Frag score, which defined as defeating counts minus the suicide counts, wins the game. Unlike the games of Go, Atari, and Starcraft II, where the player only needs to interact with the environment or compete with another opponent in one versus one zero-sum game, in the Deathmatch of ViZDoom Competitions in 2016 and 2017, the player needs to fight against other 7 independent agents with diverse strategies.
ViZDoom is multi-agen environment with imperfect information that each agent can only observes very limited information in the environment at a time step. In addition, the agent takes raw image
http://vizdooom.cco cs. httpp///ww..pppooooo/////mmmmmaa
wttp:n/c...c.c....eceee put.cdu. wwt.ccccu.llcoompetiioon////ddii plccmppetiions//vaii-221ieeii
2

---

Under review as a conference paper at ICLR 2022
Map
Coridors, internal and external platforms have cifferent heights. The connection between the insida and the outside staircase and three windows.
Player independent. combination.
buttoon angle
Score Defeat agents to get points
Rules
Eight players each other. Rank the number enemies defeated minus the number suidiees. Randomly reborn after
Figure 1: The ViZDoom environment.
input and need to train end-to-end policy, where the state space is of large image size. These characteristics of ViZDoom pose great challenges to training the agent from scratch.
works that have achieved well performance in ViZDoom. Clyde (Ratcliffe eet al. 2017) imports the LSTM (Hochreiter & Schmidhuber 1997) structure to the Asynchronous Advantage Actor-Critic (A3C) algorithm (Mnih et al., (016), resulting basic intelligent agent in ViZDoom. F1 (Wu& & Tian 2016) who won the championship of ViZDoom AI Competitions 2016 is also trained with A3C algorithm. also uses human- -aiiaar--rrorroo knowledge to conduct reward shaping and curriculum design to assist the agent learning. There is also another work (Huang et al. 2019) using deep recurrent Q-learning network as a high-level controller. It combines auxiliary tasks (opponent detection and depth prediction) to manage the combo-actions. However, these agents all have common shortcoming: the strategy of their agents is fixed. For example, in the map (see Figure. [ of ViZDoom, one agent prefers to circle around the outer loop (green line), and the other agent prefers to act the middle magma area (red line). These two agents may never meet each other in competition game, and thus not get Frag scores either. Therefore, how to obtain diversified strategy in ViZDoom still remains an open problem in the literature.
There already
METHODOLOGY
The entire learning framework consists three stages of training, by decomposing the complex task in the ViZDoom game into different levels of control, i.e., navigation, frag and strategic control. For complex systems, multi-stage learning is necessary and more efficient to build strong artificial intelligence. An overview of our multi-stage learning framework is shown in Figure 2
We briefly explain the framework here and claborate the details in the following subsections. In stage 1, we aim to obtain expert navigation agent, which only takes moving actions. Specifically, we enable the agent to navigate by following a given target, and this is very useful for stage 3 when training the agent considered by goal-conditioned RL (Andrychowicz et al. 2017). It has been well demonstrated that RL methods using hindsight experience replay (HER) (Andrychowicz solving goal-conditioned tasks. Therefore, we adopt HER in our navigation stage. We are not aware of any game AIs that explicitly take advantage of HER. The original HER method implemented with the off policy method such as DDPG and TD3. For training strong AIs in complex video games, it has been demonstrated that actor-critic algorithms, such as PPO, with discrete actions are more effective (Schulman et al. (017).. In order to apply PPO and HER together, we adopt the same idea as the recent Hindsight Trust Region Policy Optimization (HTRPO) method (Zhang al.) 2021), proposed the Hindsight Proximal Policy Optimization method, which we call HPPO. Based on an excellent navigation policy, in the second training stage, we let the agent participate in the formal battle game in ViZDoom. which is also termed 'Frag" in ViZDoom. At this stage, Prioritized Fictitious Self-Play (PFSP) (Vinyals et al. (019) to let the training agent play against itself and its historical versions with prioritization. To conveniently reuse the navigation policy,
strategic level. Apparently, navigation to given target is the essential problem
al.) 2017) are effective in
3

---

Under review as a conference paper at ICLR 2022
navigation control
frag control
strategic control
Move button angle button
Move button
angle button shoot button
Self 2
Stage
Stage
Stage 3
Navi training via HPPO
Attack training via PPO
strategic training via PPO & distillation
Obs
Obs
Obs
Tral
Fandon
Troj
Random
Traj
Seif
Postere
Trajectory nelsbliing
Environment
Plstilaton boo
network
suys lus
Frog policy
Frog policy
Neural Anxt
Strwiegis
Nail polloy
Pesiifaios
had policy
polley
PP0 loos
hppo
Act loncspling sci
Act
Mida button
shoot bullon
Miore
shout
verapper
x O Cr+ t-!
*t O Am-i
Figure 2: The infrastructure of our multi stage learning system. Among the stages, the dashed lines represent model reuse. In our implementation, the goals are encoded into high-dimensional one-hot vectors, which are denoted as the variable Z.
create action head for frag in addition to the moving actions. So far, the trained agent play formal ViZDoom games by randomly feeding navigation target to the policy. In a formal ViZDoom match in the historical competitions, one player will play against another players in the map. Therefore, in the last stage, we propose to learn a policy at a strategic level to output a specific target, that the agent itself can decide where navigate. Again, smoothly reuse the pre-trained policy in the previous two stages, we create a new action head for deciding the target. In this way. the agent should be aware of its opponents' strategies and then decide where go encounter them. A detailed infrastructure of the proposed multi-stage learning system has been depicted in Figure
4.1 STAGE GOAL-CCNDDT NAVIGATION
FPS games pose great challenges on the player's moving and shooting skills. When human players play FPS games, they always demonstrate diverse strategies for moving and shooting, conditioning on their opponents' strategies in one episode. For example, if a professional player knows that one opponent is used to hide at specific location in the map, he/she will navigate somewhere that can easily monitor or shoot that location. For Al agents, an explicit way to modeling such navigation strategies is define goals. Then, the problem is naturally converted to the well-studied goal- conditioned RL. There have been a rich literature one of the most effective one is using hindsight experience replay (HER). Researches based on HER often consider continuous control tasks based on DDPG, and these tasks are relatively simple environments. In ViZDoom, the action space discrete, and we choose to apply PPO and HER to solve the navigation problem. Directly training PPO using hindsight experiences is problematic, since PPO is an on- on-policy method while hindsight experiences are handeraft data which does not follow the policy any more. Therefore, we have correct the objective in PPO by re-sampling trajectories from the hindsight experiences, following the Hindsight TRPO method (Zhang et al. 2019). This method is referred to the Hindsight PPO (HPPO). Formally, HPPO maximizes the
goal-condition RL methods, among which

---

Under review as a conference paper at ICLR 2022
following objective:
Tours (ax sk.gg), g To Ta 18.97 A Aoud (8e, 0r.
Louis (0) = RE
min
Tona
(ax 8k, 9) Tool (ae 81, 0²)
t=
k==
clip
II k=
Tord
(ax
9ks, sksg")
To (ax st, 1111-,11+ A (8c, (sn,ar:)))
9')
Toor
(ak
Sk: 9) Tous (at St: (')
(1)
where g and g' denote the original and hindsight goals, respectively, and indicates the trajec- tory (so, oo, r..g....). 0 is the policy parameter and Ond is the parameter since last update. Ano (8t, (e:g is the advantage function. e is the clip range and we set = 0.2 as suggested by PPO. When g g'. the object reduces to the original PPO objective.
In ViZDoom, we discretize the map into 20 areas (shown in Figure 8.. whose center positions compose the set of goals. We simply code each goal as 20-dimensional one-hot embedding, while it is clear that the goals can be defined in other ways.
Another practically useful technique is to transfer a sequence of consecutive actions into more effi- cient action combos assisting moving. This module is referred to as an action wrapper. For example, when the action wrapper detects 4 consecutive discrete actions of turning to left (by one degree), the action wrapper will output immediate action of turning left by 20 degrees. In addition, when the policy network continues outputting more than 4 moving forward commands, the action wrapper will push a speedup moving forward command. These action combos can effectively help the agent to perform human-like behaviors.
4.2 STAGE 2: FRAG BY MAINTAINING THE EXPERTISE IN MULTI-GOAL NAVIGATION
At the second stage, we allow the agent to play formal full game in ViZDoom competitions, inher- iting the expertise in multi-goal navigation learned from stage 1. To conveniently reuse the trained parameters from stage I, we create separate action head for controlling frag, with only a few pa- rameters that are randomly initialized. The head outputs a binary value indicating whether the agent shoots or not at the current step. Note that in ViZDoom, the player is able to perform navigation and frag at the same time step. By decoupling the navigation actions and frag actions, the agent would explore better policy to coordinate navigation and frag. This is similar to control multi-agent systems with a centralized policy.
To promote data efficiency, we take advantage of the previously demonstrated effective technique in StarCraft II, named Rule-Guided Policy Search (RGPS) (Han et al., 202T). RGPS distills some straightforward domain knowledge into a pure neural network to skip unnecessary explorations. For training the frag head, we use a simple handcraft rule that an opponent shows up in a certain range of the center of the raw screen, then shoot, without executing any moving actions.
Moreover, to make sure that the policy network avoid forgetting the navigation skills learned at the first stage, we maintain a policy distillation term to let the policy trained in the second stage stay close to that trained in the first stage only for the navigation head. It's important to emphasize that, the reason why we continue train the navigation head is that the navigation head is not just useful for moving, it is also important to coordinate with the frag action for accurate movement and shooting. The overall objective function at this stage is:
Lrug
Lypo
L frag-navigation
+ frag + p L
navigation
where Lero indicates the standard PPO loss for optimizing both the navigation and frag policies, LeGPS is the RGPS term specially defined for the frag head, and Lassin is a policy distillation term preventing navigation performance decrease for the navigation head. X and are two hyper- parameters.
4.3 STAGE DIVERSIFIED STRATEGIC CONTROL
An important challenge in FPS games sharing the same competition mode in ViZDoom is that in one match, there are multiple diverse players, e.g., ViZDoom, combating with each other. In the game of GO (Silver et al. 2016), DOTA 2 (Berner et al. 2019) and StarCraft II (Vinyals al.
5

---

Under review as a conference paper at ICLR 2022
2019). the competitions are between two players, i..., versus one zero-sum games. Therefore, simultaneously combating with many opponents requires more diversified skills from strategic level.
At the final training stage, aim to train strategic policy that can dynamically control the agent's 's strategy conditioning on the observed opponents' behaviors. Specifically, we let the agent decide the current goal by itself. This is achieved by creating new action head again and reuse the previous trained parameters from stage 2. The strategic policy outputs a goal selected from the goal set, and then the goal is fed into the navigation and frag heads to perform navigation and frag actions. Such auto-regressive actions have been investigated in AlphaStar (Vinyals et al. 2019) as well. Different from stage 2, we abandon PFSP and propose a new game matching scheme to generate self-play matches. In a formal game of ViZDoom, there are 8 players in total to compete with each other. For a specific episode, we let the opponents execute fixed strategy randomly sampled from the beginning of the match. This can be easily achieved by sampling fixed goal for them. In addition to the observation taken by the navigation and frag heads, the strategic policy takes as input an additional feature, that is statistical map calculated from the cumulative historical imperfect observation in this episode. Specifically, the map records a score for each pixel, where the score is the cumulative Frag scores the agent gains at this specific pixel. At the testing phase in ViZDoom, the raw observation is the screen image, and a bird view of the entire map by accurately positioning additional auxiliary network positioning the
the agent is not available. Therefore, we train agent in the bird view map by taking the raw screen image as input. This auxiliary network is trained using supervised learning during the RL training phase. by cheating on acquiring the current global coordinates of the agent.
should be emphasized that the strategic policy should control the agent infrequently, Otherwise, the agent might change its goal to violate its current strategy very often. To void the situation, the strategic action head is activated only when the agent successfully reaches a goal or reborn. This poses a new challenge that the amount of effective data samples collected for training the strategic policy will considerably limited. For example, there are around 10,000 moves in a formal match, but the number of frames at which the strategic head is activated is only about 30-40. Therefore, to enhance the exploration efficiency at the strategic level, we employ the RGPS method (Han et al.) 2021) to incorporate some straightforward domain knowledge to guide strategic exploration. The handcraft principle is quite simple that the probability of selecting a goal is proportional to the summation of the recorded Frag scores belonging to the according area in the statistical map (see Figure The objective function at this stage is
L Stnttgiic
Lppo
strategic
ALRGSS
Stratgic
(3)
where both the PPO loss and RGPS loss are defined on the strategic policy, and the navigation and frag heads are not updated any more in this stage.
5 EXPERIMENTS
In this section, we provide comprehensive empirical studies for the proposed learning framework. The finally trained agent is referred to as the Diversified Strategic Control (DSC) Agent. In Sec- tion 5.1 visualize the representation learned by the last embedding layer shared by the policy and value networks to provide in-depth understanding of the agent. In Section we report various ablation experiments study the impact of the proposed components in affecting the agent perfor- mance. Finally, we report the overall testing performance of DSC-Agent by creating competition league, composed by the top ranking agents in previous open ViZ.Doom Al competitions, including
Marvin (Wydmuch al.) 2018), the champion in Track1 of ViZDoom AI Competition 2017, which is trained by first imitating human expert replays and then by RL.
Fl (Wu & Tian 2016), the champion in Trackl of ViZDoom AI Competition 2016, which is trained with curriculum reinforcement learning.
Axon (Wydmuch et al. 2018), the third place in Track1 of ViZDoom AI Competition 2017, which uses a similar training pipeline as Marvin.
YanShi (Wydmuch et al. 2018), the sixth place in Track1 of ViZDoom AI Competition 2017, which adopts perception module and planning module.

---

Under review as a conference paper at ICLR 2022
A
Basic Competitive Strategy
Automatically Explored Behaviors
Value Function
fring
detsupring
aalion
+ scitce wrapper
decneppiy
high
Patrolling among goals
quickly turning ground
decauping ontin acapper
decruppiing
BL
Seeking for opparent
6
Tracking
low
acton
ddeespping
Figure 3: (A) Visualization of the neuron activation for three basic competitive strategies learned by DSC-Agent. (B) Three novel high-level tactics discovered by DSC-Agent, with their statistics of occurrence per match for different configurations. (C) Visualization of the value function predicted by DSC-Agent.
5.1 VISUALIZATION OF THE LEARNED REPRESENTATION
We use t-SNE (Van der Maaten & Hinton 2008 to visualize the learned representations in the last embedding layer of the agent's network. We evaluate the DSC-Agent by performing 106 time steps in a formal match, and use the generated data for visualization by t-SNE. Figure shoows the results. In Figure A), I-SNE visualizes the activated neurons according three frequently observed be- havior patterns at the strategic level: (1) navigation to the target, (2) patrolling among targets, and (3) seeking for opponents. The visualization illustrates the learned representations memorize these strategic behaviors via different neuron regions, similar to human brains. These three strategic be- haviors are as expected, since the proposed multi-stage learning framework carefully developped to induce these behaviors. However, surprisingly. Figure B) shows that three additional behavior types are automatically discovered by the agent: (4) firing in advance, (5) quickly turning around to face the opponent, and (6) tracking observed opponent. Behaviors (4)-(6) are observed less fre- quently compared to (1)-(3), and they are scattered on the neurons as denoted in the visualized value function in Figure B C). On the right-hand side of Figure BB), the counts of observed behaviors (4)-(6) are recorded for different configurations, and we can observe that decoupling the action into multiple heads and using action wrapper helpful discover these novel tactics. Figure C) vi- sualizes the value of the corresponding behaviors, and we can observe that all the previously Tearned behaviors deserve much higher values in this neuron map.
Activation degree
Figure 4: (A) and (B) Visual representation of DSC-Agent's image processing network by using Class Activation MApping(CAMM). (C) Visual representation of ASC-Agent's strategic control pro- cessing network by using Class Activation MapPing(CAM)).

---

Under review as a conference paper at ICLR 2022
We also visualize where Age-t pays attention in the image imput by using Class Activation Mapping (CAM) (Zhou et al. 2016). Figure [[A) and (B) show the network activation for each pixel on the imput screen when observing supplies and opponents, respectively. These figures demonstrate that the agent can accurately focus on these important objects.
In Figure 4 C), also visualize the attention on the bird view map constructed by the auxiliary net- work mentioned Section 4.1 It demonstrates that the agent focuses on a few important locations from the strategic level.
5.2 EVALUATION ON STRATEGIC DIVERSITY
45
Converped strategy distributton
Goal-18
201
Boal-15
Picbabl by
Doa-13 3-
Boa--11
Frage Scone
Goal
Doal -0-
20
Beal
Geat Q4
30
G9ol 02
Goal 00
Goal 01
Goal 62
Goal- 43-
04
Gcal ds
Coal- 06
Goal 07-
Goal 58
Coal 09-
Gool 10-
coal 11-
16-
Goal 17
Goal 18
Goal 13.
Goal
Goal
Goal
Goal
Goal-
0004-00
0038-8 59
09a1:00
Gool 48
003132
6668-37
60d 39
Marvin
Axon
DSC-Agont
opponent strategy
Figure 5: Left: average Frags scores for each compared AI agent when playing against different testing opponents. The scores computed by averaging over 10 episodes, each of which lasts for 10 minutes. Right: a matrix showing the output distribution of the strategic policy in DSC-Agent (rows) against diverse opponents (columns).
As we have claimed multiple times that DSC-Agent superior compared existing state-of-the AIs by deploying a strategic policy that can decide the goal for itself. To verify this, we perform evaluation on strategic diversity in this section.
Recall that the agent trained at stage 2 can take any goal as input and act according to the goal in full ViZDoom match. Therefore, it is naturally to create a set of testing agents by selecting several representative goals from the goal set and feeding them into the trained agent at stage 2.
The left figure in Figure 5 shows the testing scores for all the oftte-of-thearr methods. Each bar indicates the average testing Frags score of a specific Al agent against 7 DSC-Agents from stage by feeding a specific goal. As we can observe, when playing against opponents with different strategies, the compared AIs also show unstable scores. It is obvious that DSC-Agent can achieve the highest scores against all the opponents. The right figure in Figure shows a matrix, in which the row indicates the output distribution of the strategic policy in DSC-Agemt over the goals and the column indicates the strategy of the opponents. By referencing the goal map in Figure R is easy that the goals selected by DSC-Agent in rows closely related to the goal indicated in the column, spatially. For example, for the opponent taking Goal-00 strategy, DSC-Agent prefers take strategies with Goal-00, Goal-10 and Goal-19, from which locations the agent can directly observe the location indicated by Goal-00.
5.3 ABLATION STUDIES
We provide comprehensive ablation studies to verify the importance of each component proposed in our learning framework. In Figure ( A), we evaluate the navigation performance in stage by comparing different configurations with PPO, HPPO and action wrapper. The results demonstrated that HPPO is much more efficient than PPO for goal-conditioned navigation, and its performance further enhanced by applying the action wrapper. Figure 6 B) reports the final performance of DSC-Agent by testing against other state-of-the-art Als in Figure 7 by varying the game matching scheme in stage 2. Similar to the results (Vinyals et al. 2019). a combination of PFSP and SP is the most effective way. Similarly, Figure shows the importance of action decoupling, RGPS in
8

---

Under review as a conference paper at ICLR 2022
stages 2 and 3, and strategic control. As observe, all the components important to allow ASC-Agent to achieve the final performance.
-PPO- Action Wiapper
HPPO
PPC+ Action Winpper
PPO
PRSP SP
20.5
SP
25
25
Mally Myne
Strategi' Control
26.6
Stage
RGPS
Action Decoupllng
PPO Free
20.0
Frage
Figure 6: (A) Comparison of HPPO, PPO and action wrapper in stage 1. (B) Evaluating the game matching scheme used in stage 2 on affecting the final testing performance of DSC-Agent. (C) Evaluation of the importance of action decoupling, RGPS in stages 2 and and strategic control. All scores are averaged over 10 episodes, each of which has 10 minutes game length, during the Deathmatch in Figure
5.4
FINAL COMPETITION RESULTS
DSC-Agent
26.6
Yanshi
17.28
Axon
20.16
Figure 7: DSC-Agent vs. previous top AIs in Deathmatch.
Finally, we report the most important results by creating a formal league competition among Marvin, F1, Axon, YanShi, and DSC-Agent. The regulation of the competition follows that in Track1 of ViZDoom AI Competition 2017. We use a known map and fixed weapons in Limited Deathmatch. All the AI agents play against each other for 12 rounds of 10 minutes Deathmatch. All the AI agents are implemented using a Tesla M40 GPU, and the Frag scores are calculated as number of frags minus the suicide penalty. The overall results shown in Figure 7 in which DSC-Agent outperforms other methods by a large margin.
CONCLUSION
We have proposed a multi-stage learning framework for solving FPS games using general RL. The learning system consists of number of advanced techniques and we have verified their importance by conducting comprehensive results in the experiments. The results demonstrate that our agent surpasses previous top ranking AIs by large margin, by taking advantage of diversified strate- gies instead of larger neural networks or hand-craft tricks. We also show that solving IPS games using general RL methods is possible without the usage of expert demonstrations and opponents' information. The proposed learning framework be easily extended other FPS games, and are interested to apply this framework to solve real-worth applications that share similarity with FPS games in future.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. arXiv preprint arXiv: 10X.:v:100..000 7077010979 2017.
Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. arXiv preprint arxi:: 1909.0758... 2019.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemystaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, al. Dota 2 with large scale deep reinforcement learning. arXiv preprint 1rXiv:1112.06690, 2019.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint 16Xiv:666.001444 2016.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. Impala: Scalable distributed deep-rl with importance weighted actor-leareer architectures, 2018.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning, pp. 1587-1596. PMLR, 2018.
Lei Han, Jiechao Xiong, Peng Sun, Xinghai Sun, Meng Fang. Qingwei Guo, Qiaobo Chen, Tengfei Shi, Hongsheng Yu, Xipeng Wu, and Zhengyou Zhang. Tstarbot-x: An open-sourced and com- prehensive study for efficient league training in starcraft full game, 2021.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Shiyu Huang, Hang Su, Jun Zhu, and Ting Chen. Combo-action: Training agent for fps game with auxiliary tasks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 954-961, 2019.
Max Jaderberg, Wojciech M Czarnecki, lain Dunning, Luke Marris, Guy Lever, Antonio Gar- cia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari Morcos, Avraham Ruderman, et al. Human-level performance in 3d multiplayer games with population-base reinforcement learn- ing. Science, 364(6443)::59 865, 2019.
Michal Kempka, Marek Wydmuch, Grzegorz Rune, Jakub Toczek, and Wojeiech Jaskowski. Viz- doom: A doom-based ai research platform for visual reinforcement learning, 2016.
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning, 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-333 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference machine learning, pp. 1928-1937. PMLR, 2016.
Dino Stephen Ratcliffe, Sam Devlin, Udo Kruschwitz, and Luca Citi. Clyde: A deep reinforcement learning doom playing agent. In Workshops at the Thirty-First AAAI Conference on Artificial Intelligence, 2017.
Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirk- patrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distil- lation. arXiv preprint 151 06699, 2015.
10

---

Under review as a conference paper at ICLR 2022
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1 1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):844448 2016.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 164 1144, 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9)11), 2008
Oriol Vinyals, Igor Babuschkin, Wojciech M Czamecki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning, Nature, 575(7782):50--35, 2019.
Yuxin Wu and Yuandong Tian. Training agent for first-person shooter game with actor-critic cur- riculum learning. 2016.
Marek Wydmuch, Michal Kempka, and Wojeiech Jaskowski. Vizdoom competitions: Playing doom from pixels. IEEE Transactions Games, 11(3)2248 2599,2299200 2018.
Hanbo Zhang, Site Bai, Xuguang Lan, David Hsu, and Nanning Zheng. Hindsight trust region policy optimization. arXiv preprint
2019.
Hanbo Zhang, Site Bai, Xuguang Lan, David Hsu, and Nanning Zheng. Hindsight trust region policy optimization, 2021.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2016.

---

Under review as a conference paper at ICLR 2022
APPENDIX
A ENVIRONMENT
ViZDoom (Kempka et al. 2016) is an Al research platform based on the FPS game Doom. We adopt the CIG 2017 competition track 1 (Wydmuch et al. 2018), where 8 Al players join in maze (shown in Figure ( and play against each other. After period of 10 minutes (in-game time), the players are ranked by the FRAG, which is defined as defeating count minus suicides (due to own rocket splash).
In experiment, the observation is RGB image, which is the first uust-person-vicee view screen pixels as what a human player sees. The action discrete in the size 10, represent- ing "TURN "EFTT, "TURN RIGHT", "MOVE
"MOVT.LEFT". MMOEE.OORWAR) "MOVE_BAS BARKWARD". "SPEED", "TURNI80", "TURN LEFT RIGHT DELTA", "ATTACK". should be emphasized that, these buttons be the same time, combine the joint actions, to facilitate decoupling action and auxiliary action.
7
5
19
10
16
2
13
8
17
3
6
11
14
18
4
15
12
Figure 8: Map of ViZDoon track 1: we discretize the map into 20 areas, each of them also used to represent opponent with fixed strategy style.
B TRAINING ARCHITECTURE
We used a multi-agent deep reinforcement learning framework designed for improve the efficiency of large-scale training, which achieves high throughput and reasonable scale-up when performing distributed training. Our architecture based on an actor-learner structure (Espeholt et al.) (2018):
large collection of over 1000 actor processes play ViZDoom deathmatch with prioritized fictitious self-play at the same time. There are 35 raw frames for one in-game, in our experiments we use frame-skip skine--sip---- 2, henceforth the in game fps 17.5fps. We use 1,152 CPU cores and 32 GPUs in toal, and the GPU type is M40 where only TCP connection is used for Horovod allreduce. After every 200 player time steps, the trajectory of experience from each player's 's point of view (observations, actions, rewards, infos) is sent to the leamer.
C DSC-AGENT IN STAGE AS OPPONENTS
The DSC-Agent in stage 2, which are trained on stage one and two, but without learning the diversity strategic control. The scope of their preferred activity distribution of 20 parts of the map as shown in Figure 8 It should be emphasized that, There several noteworthy places, including No. on the upper Teft side the map, where the view is broad, and the terrain is flat; No. 3 is on the left side of the corridor; No. 6 is in the inner of the map: No. 8 is the connecting area between the inner
12

---

Under review as a conference paper at ICLR 2022
and outer parts of the map; No. 12 Located the lower side of the map, it is relatively hidden; No. 17 on the right side the corridor; No. 19 is on the upper platform of the map.
D SUPPLEMENTAL EXPERIMENT
Figure show our result of experiment, in which we evaluate our DSC-Agent and other Al agents (Marvin, FI, Axon, and YanShi) fighting 7 opponents (DSC-Agent in stage 2) who all have fixed strategic strategies in one round. Figure 100sso our DSC-Agent' averaged distribution of strategy when fighting 7 opponents (DSC-Agent stage 2).
FV/0000
26/2 00
20/6 06
20/0 08
2d:-3
11/6-12
Manat
infarahi
Figure 9: Average frags/deaths per against opponents of fixed strategy style (10 episodes of 10 minutes each).
Average distribution
0.30
0.25
0.20
0.15
0.10
0.05
0.00
2
8
10 11
16 17 18 19
Z
Figure 10: Averaged distribution of strategy: carry out 10 rounds each type (the total is 20) of bots respectively, and calculated the average strategy distribution of DSC-Agent.
E NETWORK ARCHITECTURE
We employ a neural network shown in Figure II which consists of 3 blocks of convolution layer followed by blocks of max pooling layer and LSTM block. The inputs are the original observa- tion and the enlarged observation in the center area. And the input z is from its own strategic head.
13

---

Under review as a conference paper at ICLR 2022
mask; St
origin X
Conv
Conv
Max pool
Conv
Max pool
LSTM
Max pool
Max pool
center Xt-
Conv
Conv
Conv
fc
Z
fc
concat
navigation
navi action
head
strategic head
strategic action
frag head
frag action
Figure 11: Overview of the network structure of DSC Agent.
14


Reviews:
[{"review_id": "pUH--VhQLCP", "reviewer": "Reviewer_9yoB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The overall performance seems to be better than the state-of-the-art.\n\n2. The motivation of each component is intuitive and straightforward. \n\n3. The presentation of this paper is clear.\n\n\nWeaknesses:\n\n1. The baselines compared are from 2016 & 2018. How're latest competitions or related work after them? It seems like this part of the related work is missing, while the following work can be easily found:\n- Song, Shihong, et al. \"Playing FPS Games With Environment-Aware Hierarchical Reinforcement Learning.\" IJCAI. 2019.\n- Xu, Zhiwei, et al. \"HAVEN: Hierarchical Cooperative Multi-Agent Reinforcement Learning with Dual Coordination Mechanism.\" arXiv preprint arXiv:2110.07246 (2021).\n\n2. The ablation study in this paper is not convincing enough to show the usefulness of different components and different stages. For example, there is no clear evidence showing that the whole process has to be done in a three-stage way. How about merging two of them, or learning them in one step?\n\n3. The source code of this paper is unavailable, which makes it hard to reproduce and understand the details in the training process. ", "summary_of_the_paper": "This paper proposed a multi-stage learning framework for solving FPS games, with hindsight experience replay, goal conditioned reinforcement learning, and prioritized self-play. This whole work includes an overall solution, with combinations of existing work.", "summary_of_the_review": "With the above strengths and weaknesses, I tend to give \"marginally below the acceptance threshold\". But I'm open change my rating after seeing the rebuttal from the authors to clarify my concerns.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "j89yKWr6_PH", "reviewer": "Reviewer_jKT2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nExperiments are well-designed, and ablation studies are also presented. Visualizations make the paper much more interesting and understandable. This paper shows that instead of using larger NN, assembling existing techniques are also able to achieve SOTA performance. \nAlthough the network is trained end-to-end,  they do use some hand-crafted domain specific rules in stage 2 and stage 3. Also, in this multi-stage learning system, each stage is manually selected/formed.  \n\nReproducibility: \n\nThe neural network architecture is provided, but the used code and settings of hyper-parameters are not presented. 1152 CPUs and 32 GPUs are used, so I guess this work could be really hard to reproduce.  \n\n\nRelation to prior work: \n\nAuthors briefly explained the used related techniques and also baselines they used for comparison. Since the proposed learning framework is so different, there are not too much comparisons and relations depicted here. The proposed learning system is a multi-stage system, maybe it is better to cite some papers which are also using multi-stage learning system to solve tasks/games. \n\n \nCould be improved: \n\nAuthors proposed a multi-stage learning system, and it would be interesting to compare with a flat learning system with the same/similar architecture; \n\nNot only report means of different runs in Fig.6, but also report the standard error; \n\nTo make the loss terms more detailed, in Formular 2 and 3, losses for RGPS and policy distil are not specified; \n \nThe implementation work is impressive, but the description of the contribution and the method is not clear enough.", "summary_of_the_paper": "\nThe authors combined several different techniques from RL field to solve FPS game, ViZDoom. The game is splitted into three stages: navigation, frag and strategic control, respectively and the authors proposed a multi-stage learning system to tackle this problem. Navigation stage is solved by combining hindsight experience replay with PPO, which is named HPPO. Rule-guided policy search is used in frag stage and self-play is used for strategic control stage. Along the three stages, the trained model in the previous stage is always reused by adding new head to the neural network. The proposed multi-stage learning system outperforms SOTA agents(top ranking agents in open ViZDoom AI competitions) by a large margin. They also showed other advantages of the proposed system, such as learned skills are more diverse instead of fixed. \n\n ", "summary_of_the_review": "\nStrengths: \n\nExperiments are well-designed, and ablation studies are also presented. Visualizations make the paper much more interesting and understandable. This paper shows that instead of using larger NN, assembling existing techniques are also able to achieve SOTA performance. \n\n \n\nWeaknesses: \n\nAlthough the network is trained end-to-end, but they do use some hand-crafted domain specific rules in stage 2 and stage 3. Also, in this multi-stage learning system, each stage is manually selected/formed.  \nOnly a single game is used for empirical validation, the proposed method is quite complex, consisting of the combination of a number of existing methods,  the clarity of the contributed method is not sufficient, and no explanation is provided to explain how the method works and why the performance is better.\nOverall the work is somewhat interesting, but not significant enough to warrant publication. Reproducibility is problematic.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VYTWgKFko-9", "reviewer": "Reviewer_73Nb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n+ The paper is well-written and the logic is clear.\n+ The visualization in the experiment clearly illustrates that the agent learns how to navigate, shoot, and other strategies.\n+ The paper provides a careful design for the FPS game and achieves superior performance. \n\nWeaknesses:\n- To use off-policy data in hindsight experience replay, the paper adds an importance-sampling term to the PPO loss function, following the Hindsight TRPO method. However, there are no more techniques to reduce the high variance brought by cumulative multiplication terms. And the paper does not compare the HPPO with other goal-conditioned approaches in the experiment. \n- I'm wondering why choosing different regions can represent different strategy styles. In my opinion, there are many policy styles in FPS games, such as aggressive/defensive/wandering, etc. These policy styles cannot be simply concluded by where the agent goes. For example, an agent can go to a same region for many reasons. It may go there to attack, to defend, or to detect the situation......\n- In Stage 3, the strategic policy takes the statistical map recording the Frag scores as an input. I'm wondering how the agent cumulates Frag scores in unseen areas when testing, and whether the auxiliary network‘s prediction is accurate when facing unknown opponents.\n- The experimental setup in evaluating strategy diversity is unfair. The evaluation opponent(Goal-00 to Goal-19) is used when training the DSC-Agent in stage 3. It is normal that DSC-Agent can get higher scores than other benchmark Agents. \n- The paper should conduct more experiments to evaluate the strategic diversity of the agent which is the main contribution of the paper. How the agent dynamically changes its strategy as the game progresses, gathering more information about opponents. How does the agent develop strategies when fighting against agents not seen in training? ", "summary_of_the_paper": "This paper proposes a multi-stage learning framework for training high-performance agents in FPS games. Its agent can dynamically adjust its strategy according to different opponents, which is simply controlled by selecting different target areas. The paper propose a methodology called Hindsight PPO to solve goal-conditioned RL, though it may have some concerns. The framework combines several existing techniques such as rule-guided policy search,  action wrapper, prioritized fictitious self-play, etc. Experiments show that the framework learns an agent that can surpass previous top agents. ", "summary_of_the_review": "I think this paper deserves credit for training a superior performance agent by designing a multi-stage learning framework combining several techniques. However, this paper has some flaws:\n- The proposed HPPO algorithm may suffer from high variance, and it is better to compare it with other methods.\n- The relationship between strategy style and the target area is unclear. \n- Some methods in Stage 3 are unclear.\n- The experimental setup in evaluating strategy diversity is unfair.\n- The paper should have more experiments to evaluate the strategic diversity of the agent.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-pKvzWAHsYf", "reviewer": "Reviewer_ahya", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Pros\n- This work takes a problem-driven approach to improve the performance of a reinforcement learning agent on VizDoom. Although most of the RL literature starts from conceptual ideas to experimental results, I consider the present work to be valuable as it assembles and confronts some well-known techniques in RL to concrete problems.\n- The impact of the action wrapper and the rule-guided policy search is a great example that sometimes less is more. Using simple tricks on top of generic methods can sometimes be much more effective than complicated approaches.\n\n### Cons\n- The quality and clarity of the writing is problematic. Besides the numerous spelling mistakes, the imprecise language often confuses the reader. For instance:\n\t* \"...the state transition is not static\", we generally talk about stationary or non-stationary environments. I'm not sure what static means in this case?\n\t* \"..., while they are enabled to act diversely when the environment dynamics changes.\" The term `diverse` in the context of reinforcement learning is most often associated with Quality-Diversity that aims to generate a large collection of diverse solutions/policies. It is unclear whether the authors want to highlight the ability of their agent to adapt to changing dynamics (few-shot learning for instance) or if they want their agent to learn a policy performing well against opponents displaying a diverse set of strategies. For this reason, the method name, Diversified Strategic Control, is confusing as well.\n\t* The title mentions \"General Reinforcement Learning\". I'm not sure what the authors try to communicate? Besides the fact that general reinforcement learning feels undefined, the described approach uses tricks specific to FPS, like the statistical map provided as extra features to the agent's network or the rules defined in the 2 RGPS losses.\n- As highlighted by the authors, the proposed approach consist of a combination of existing techniques. Although replicating and using these techniques require significant engineering effort, these can't be considered novel contributions. Other works that took a similar work setting, often made novel and original contributions, e.g. AlphaStar and their league system.\n- The work is certainly complex from an engineering point of view: handcrafted curriculum learning, hindsight experience replay, distributed training strategy. As such, particular attention to providing all the required information to replicate the work has to be initiated. This aspect of the work is deeply lacking.\n\t- The values of all the hyper-parameters are not provided e.g. optimizers, losses coefficients, training parameters, etc.\n\t- Many terms and aspects are not defined. The authors don't explain: how they applied Policy Distillation, how the prioritized self-play has been implemented and put in action, what's the training dynamic (how long each stage lasts), where do the baselines come from (were they re-implemented or available as part of the competition?), etc.\n\tThese missing bits of information would make the replication of the results nearly impossible to achieve and don't help the reader understand how the approach fully works and how it was implemented. I would encourage the authors to make a few steps in this direction as the lack of transparency on their method is puzzling.\n\n### Questions\n- Figure 5 only provides the score for a few goals. Why is that, and could the performance of the agents for the other goals be provided in the appendix?\n- Regarding the training at stage 3, the authors mention that the selection of goals should be infrequent to avoid goal switching. Have the authors tried to constrain the switching of goals by restricting the goals to be adjacent to the current agent's goal, or by introducing a penalty in the reward?\n- How are the end-of-life events processed? When the agent dies, is it considered the end of the episode? What are the implications for the hidden state of the LSTM, does it persist over the 10 minutes of the Deathmatch?\n- What are the motivations behind having chosen PPO? The work requires applying Hindsight experience replay and importance sampling which increases the variance of the gradient. Have the authors tried using an off-policy algorithm that suits more easily HER?\n- I would like additional clarifications on the goal selection. If I understand correctly the third head can decide the goal. If, for instance, the agent detects an enemy in an area that is not its current goal. How can the agent decide to change its strategy given that the third head is only activated once the agent reaches its current goal?\n\n### Minor Comments\n- Hindsight Trust Region Policy Optimization appears twice in the bibliography.\n- Figure 2 is really hard to understand. It would benefit from a more extensive caption describing in greater detail each component and the transitions from one stage to the next.\n", "summary_of_the_paper": "This paper proposes an approach to improve the performance of reinforcement learning agents on the ViZDoom FPS game (Deathmatch mode as used in the AI competition 2016, 2017). The method consists of a combination of existing techniques, e.g. PPO, Hindishght Experience Replay, Rule-Guided Policy Search, (Prioritized) Self-Play. The resulting algorithm is compared to the top-ranked agents from previous ViZDom AI Competitions (2016, 2017). The approach is shown to outperform these baselines and an ablation study is provided to highlight the impact on the performance due to the integration of each system component.", "summary_of_the_review": "The paper improves the performance over the previously top-ranked solutions of the ViZDoom competition. Their approach consists of a combination of known technics well-engineered together. As such, the work lacks originality and novel contributions. In addition, many pieces of information that would enable a complete understanding of the method and the reproducibility of the results are missing. Consequently, I consider that the current version of the work does not meet the required standard for acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a method for training agents in FPS games, and achieve good results in a VizDoom setting. The method combines a number of different components and ideas, and it is not clear which of these are crucial to the success. In particular, ablations of the method are missing, as well as more runs to test variability and diversity. In addition, the paper is not all that easy to read. Reviewers had a number of partly overlapping concerns, of which I've tried to distil the main ones above. While the empirical results are promising, it is clear that much more work is needed to distil this method into generalizable knowledge.
Decision: Reject