Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposed a multi-stage learning framework for solving FPS games, with hindsight experience replay, goal conditioned reinforcement learning, and prioritized self-play. This whole work includes an overall solution, with combinations of existing work.
main_review: Strengths:
1. The overall performance seems to be better than the state-of-the-art.

2. The motivation of each component is intuitive and straightforward. 

3. The presentation of this paper is clear.


Weaknesses:

1. The baselines compared are from 2016 & 2018. How're latest competitions or related work after them? It seems like this part of the related work is missing, while the following work can be easily found:
- Song, Shihong, et al. "Playing FPS Games With Environment-Aware Hierarchical Reinforcement Learning." IJCAI. 2019.
- Xu, Zhiwei, et al. "HAVEN: Hierarchical Cooperative Multi-Agent Reinforcement Learning with Dual Coordination Mechanism." arXiv preprint arXiv:2110.07246 (2021).

2. The ablation study in this paper is not convincing enough to show the usefulness of different components and different stages. For example, there is no clear evidence showing that the whole process has to be done in a three-stage way. How about merging two of them, or learning them in one step?

3. The source code of this paper is unavailable, which makes it hard to reproduce and understand the details in the training process. 
summary_of_the_review: With the above strengths and weaknesses, I tend to give "marginally below the acceptance threshold". But I'm open change my rating after seeing the rebuttal from the authors to clarify my concerns.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a multi-stage learning framework for training high-performance agents in FPS games. Its agent can dynamically adjust its strategy according to different opponents, which is simply controlled by selecting different target areas. The paper propose a methodology called Hindsight PPO to solve goal-conditioned RL, though it may have some concerns. The framework combines several existing techniques such as rule-guided policy search,  action wrapper, prioritized fictitious self-play, etc. Experiments show that the framework learns an agent that can surpass previous top agents. 
main_review: Strengths: 
+ The paper is well-written and the logic is clear.
+ The visualization in the experiment clearly illustrates that the agent learns how to navigate, shoot, and other strategies.
+ The paper provides a careful design for the FPS game and achieves superior performance. 

Weaknesses:
- To use off-policy data in hindsight experience replay, the paper adds an importance-sampling term to the PPO loss function, following the Hindsight TRPO method. However, there are no more techniques to reduce the high variance brought by cumulative multiplication terms. And the paper does not compare the HPPO with other goal-conditioned approaches in the experiment. 
- I'm wondering why choosing different regions can represent different strategy styles. In my opinion, there are many policy styles in FPS games, such as aggressive/defensive/wandering, etc. These policy styles cannot be simply concluded by where the agent goes. For example, an agent can go to a same region for many reasons. It may go there to attack, to defend, or to detect the situation......
- In Stage 3, the strategic policy takes the statistical map recording the Frag scores as an input. I'm wondering how the agent cumulates Frag scores in unseen areas when testing, and whether the auxiliary network‘s prediction is accurate when facing unknown opponents.
- The experimental setup in evaluating strategy diversity is unfair. The evaluation opponent(Goal-00 to Goal-19) is used when training the DSC-Agent in stage 3. It is normal that DSC-Agent can get higher scores than other benchmark Agents. 
- The paper should conduct more experiments to evaluate the strategic diversity of the agent which is the main contribution of the paper. How the agent dynamically changes its strategy as the game progresses, gathering more information about opponents. How does the agent develop strategies when fighting against agents not seen in training? 
summary_of_the_review: I think this paper deserves credit for training a superior performance agent by designing a multi-stage learning framework combining several techniques. However, this paper has some flaws:
- The proposed HPPO algorithm may suffer from high variance, and it is better to compare it with other methods.
- The relationship between strategy style and the target area is unclear. 
- Some methods in Stage 3 are unclear.
- The experimental setup in evaluating strategy diversity is unfair.
- The paper should have more experiments to evaluate the strategic diversity of the agent.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes an approach to improve the performance of reinforcement learning agents on the ViZDoom FPS game (Deathmatch mode as used in the AI competition 2016, 2017). The method consists of a combination of existing techniques, e.g. PPO, Hindishght Experience Replay, Rule-Guided Policy Search, (Prioritized) Self-Play. The resulting algorithm is compared to the top-ranked agents from previous ViZDom AI Competitions (2016, 2017). The approach is shown to outperform these baselines and an ablation study is provided to highlight the impact on the performance due to the integration of each system component.
main_review: ### Pros
- This work takes a problem-driven approach to improve the performance of a reinforcement learning agent on VizDoom. Although most of the RL literature starts from conceptual ideas to experimental results, I consider the present work to be valuable as it assembles and confronts some well-known techniques in RL to concrete problems.
- The impact of the action wrapper and the rule-guided policy search is a great example that sometimes less is more. Using simple tricks on top of generic methods can sometimes be much more effective than complicated approaches.

### Cons
- The quality and clarity of the writing is problematic. Besides the numerous spelling mistakes, the imprecise language often confuses the reader. For instance:
	* "...the state transition is not static", we generally talk about stationary or non-stationary environments. I'm not sure what static means in this case?
	* "..., while they are enabled to act diversely when the environment dynamics changes." The term `diverse` in the context of reinforcement learning is most often associated with Quality-Diversity that aims to generate a large collection of diverse solutions/policies. It is unclear whether the authors want to highlight the ability of their agent to adapt to changing dynamics (few-shot learning for instance) or if they want their agent to learn a policy performing well against opponents displaying a diverse set of strategies. For this reason, the method name, Diversified Strategic Control, is confusing as well.
	* The title mentions "General Reinforcement Learning". I'm not sure what the authors try to communicate? Besides the fact that general reinforcement learning feels undefined, the described approach uses tricks specific to FPS, like the statistical map provided as extra features to the agent's network or the rules defined in the 2 RGPS losses.
- As highlighted by the authors, the proposed approach consist of a combination of existing techniques. Although replicating and using these techniques require significant engineering effort, these can't be considered novel contributions. Other works that took a similar work setting, often made novel and original contributions, e.g. AlphaStar and their league system.
- The work is certainly complex from an engineering point of view: handcrafted curriculum learning, hindsight experience replay, distributed training strategy. As such, particular attention to providing all the required information to replicate the work has to be initiated. This aspect of the work is deeply lacking.
	- The values of all the hyper-parameters are not provided e.g. optimizers, losses coefficients, training parameters, etc.
	- Many terms and aspects are not defined. The authors don't explain: how they applied Policy Distillation, how the prioritized self-play has been implemented and put in action, what's the training dynamic (how long each stage lasts), where do the baselines come from (were they re-implemented or available as part of the competition?), etc.
	These missing bits of information would make the replication of the results nearly impossible to achieve and don't help the reader understand how the approach fully works and how it was implemented. I would encourage the authors to make a few steps in this direction as the lack of transparency on their method is puzzling.

### Questions
- Figure 5 only provides the score for a few goals. Why is that, and could the performance of the agents for the other goals be provided in the appendix?
- Regarding the training at stage 3, the authors mention that the selection of goals should be infrequent to avoid goal switching. Have the authors tried to constrain the switching of goals by restricting the goals to be adjacent to the current agent's goal, or by introducing a penalty in the reward?
- How are the end-of-life events processed? When the agent dies, is it considered the end of the episode? What are the implications for the hidden state of the LSTM, does it persist over the 10 minutes of the Deathmatch?
- What are the motivations behind having chosen PPO? The work requires applying Hindsight experience replay and importance sampling which increases the variance of the gradient. Have the authors tried using an off-policy algorithm that suits more easily HER?
- I would like additional clarifications on the goal selection. If I understand correctly the third head can decide the goal. If, for instance, the agent detects an enemy in an area that is not its current goal. How can the agent decide to change its strategy given that the third head is only activated once the agent reaches its current goal?

### Minor Comments
- Hindsight Trust Region Policy Optimization appears twice in the bibliography.
- Figure 2 is really hard to understand. It would benefit from a more extensive caption describing in greater detail each component and the transitions from one stage to the next.

summary_of_the_review: The paper improves the performance over the previously top-ranked solutions of the ViZDoom competition. Their approach consists of a combination of known technics well-engineered together. As such, the work lacks originality and novel contributions. In addition, many pieces of information that would enable a complete understanding of the method and the reproducibility of the results are missing. Consequently, I consider that the current version of the work does not meet the required standard for acceptance.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: 
The authors combined several different techniques from RL field to solve FPS game, ViZDoom. The game is splitted into three stages: navigation, frag and strategic control, respectively and the authors proposed a multi-stage learning system to tackle this problem. Navigation stage is solved by combining hindsight experience replay with PPO, which is named HPPO. Rule-guided policy search is used in frag stage and self-play is used for strategic control stage. Along the three stages, the trained model in the previous stage is always reused by adding new head to the neural network. The proposed multi-stage learning system outperforms SOTA agents(top ranking agents in open ViZDoom AI competitions) by a large margin. They also showed other advantages of the proposed system, such as learned skills are more diverse instead of fixed. 

 
main_review: 
Experiments are well-designed, and ablation studies are also presented. Visualizations make the paper much more interesting and understandable. This paper shows that instead of using larger NN, assembling existing techniques are also able to achieve SOTA performance. 
Although the network is trained end-to-end,  they do use some hand-crafted domain specific rules in stage 2 and stage 3. Also, in this multi-stage learning system, each stage is manually selected/formed.  

Reproducibility: 

The neural network architecture is provided, but the used code and settings of hyper-parameters are not presented. 1152 CPUs and 32 GPUs are used, so I guess this work could be really hard to reproduce.  


Relation to prior work: 

Authors briefly explained the used related techniques and also baselines they used for comparison. Since the proposed learning framework is so different, there are not too much comparisons and relations depicted here. The proposed learning system is a multi-stage system, maybe it is better to cite some papers which are also using multi-stage learning system to solve tasks/games. 

 
Could be improved: 

Authors proposed a multi-stage learning system, and it would be interesting to compare with a flat learning system with the same/similar architecture; 

Not only report means of different runs in Fig.6, but also report the standard error; 

To make the loss terms more detailed, in Formular 2 and 3, losses for RGPS and policy distil are not specified; 
 
The implementation work is impressive, but the description of the contribution and the method is not clear enough.
summary_of_the_review: 
Strengths: 

Experiments are well-designed, and ablation studies are also presented. Visualizations make the paper much more interesting and understandable. This paper shows that instead of using larger NN, assembling existing techniques are also able to achieve SOTA performance. 

 

Weaknesses: 

Although the network is trained end-to-end, but they do use some hand-crafted domain specific rules in stage 2 and stage 3. Also, in this multi-stage learning system, each stage is manually selected/formed.  
Only a single game is used for empirical validation, the proposed method is quite complex, consisting of the combination of a number of existing methods,  the clarity of the contributed method is not sufficient, and no explanation is provided to explain how the method works and why the performance is better.
Overall the work is somewhat interesting, but not significant enough to warrant publication. Reproducibility is problematic.
review_title: null

Meta-review: The authors propose a method for training agents in FPS games, and achieve good results in a VizDoom setting. The method combines a number of different components and ideas, and it is not clear which of these are crucial to the success. In particular, ablations of the method are missing, as well as more runs to test variability and diversity. In addition, the paper is not all that easy to read. Reviewers had a number of partly overlapping concerns, of which I've tried to distil the main ones above. While the empirical results are promising, it is clear that much more work is needed to distil this method into generalizable knowledge.
Decision: Reject