Title: Training Agent for First-Person Shooter Game with Actor-Critic Curriculum Learning
Authors: Yuxin Wu
Yuandong Tian
Venue: ICLR 2017
Keywords: Reinforcement Learning
Applications
Games

Abstract: In this paper, we propose a novel framework for training vision-based agent for First-Person Shooter (FPS) Game, in particular Doom.
Our framework combines the state-of-the-art reinforcement learning approach (Asynchronous Advantage Actor-Critic (A3C) model) with curriculum learning. Our model is simple in design and only uses game states from the AI side, rather than using opponents' information. On a known map, our agent won 10 out of the 11 attended games and the champion of Track1 in ViZDoom AI Competition 2016 by a large margin, 35\% higher score than the second place.

TL;DR: We propose a novel framework for training vision-based agent for First-Person Shooter (FPS) Game, Doom, using actor-critic model and curriculum training. 

Review discussion:
Are the graphs in Fig 5 constructed using a single hyper-parameter sweep? I think the authors should report results with many random initialization to make the comparisons more robust
Thanks for the comments!
We are current working on more experiments. Will give you an update of the paper before rebuttal ends.
I did not find where you explicitly describe what "model 0", "model 1", ... are? I presume they are snapshots from some learning curve? Please be explicit.
Another aspect that would improve the value of the paper would be to discuss your tuning efforts, and how robust the results are compared to it -- from my perspective, this is a single run on highly tuned hyper-parameters and reward-shaping settings?
Thanks reviewer for the comments!
The 7 different models (model 0 to model 6) are the trained models after specific curriculum defined in Table 3.
For example, Model 0 is the model trained after class 0. We used an old table to train these models, which has only 7 curricula instead of 8.
Therefore you see Model 0 – Model 6 and Class 0 – Class 7.  From Fig. 4, you will see for each class, model x is in general better than model y for x > y, showing curricula improve decent performance.
We are working on more extensive experiments for the next version of the paper. For different part of experiment sessions, we have multiple runs (but Fig. 5 is indeed a single run. We will do more on this part).
"In this work, we use Huber loss instead of L2 loss in Eqn 2." Why? Seems like this was not done in the original A3C paper. Can you justify your choice here (citations would be nice, maybe the DQN paper?).. do you have experiments with L2 loss to compare? How critical is this? Does it help with stability / hyperparameter robustness? Should we all switch to Huber loss for the value network? At the moment, this choice appears to be somewhat arbitrary.
Thanks for your question!
When error is large, Huber loss grows linearly (instead of quadratically), therefore its gradient stays within ±1. Using Huber loss is equivalent to clipping the gradient, which is commonly used to stabilize training process. The original A3C paper mentions "gradient norm clipping" -- it's unclear what was exactly used, but the effect should be similar. The DQN paper explicitly describes where and how the clipping is applied, and our use of Huber loss follows their settings.
In this paper we mainly focus on introducing a system that trains strong Doom AI bots. And we will try our best to explain various choices in designing the system. In this case the aforementioned reason provides justification why this choice is not arbitrary, and we'll also provide some experiments in the next version to further justify this choice.
This is a solid paper that applies A3C to Doom, enhancing it with a collection of tricks so as to win one of the VizDoom competitions. I think it is fair to expect the competition aspect to overshadow the more scientific approach of justifying every design decision in isolation, but in fact the authors do a decent job at the latter.
Two of my concerns have remained unanswered (see AnonReviewer2, below).
In addition, the citation list is rather thin, for example reward shaping has a rich literature, as do incrementally more difficult task setups, dating back at least to Mark Ring’s work in the 1990s. There has also been a lot of complementary work on other FPS games. I’m not asking that the authors do any direct comparisons, but to give the reader a sense of context in which to place this.
This paper basically applies A3C to 3D spatial navigation tasks.
- This is not the first time A3C has been applied to 3D navigation.
In fact the original paper reported these experiments.
Although the experimental results are great, I am not sure if this paper has any additional insights to warrant itself as a conference paper. It might make more sense as a workshop paper
-  Are the graphs in Fig 5 constructed using a single hyper-parameter sweep? I think the authors should report results with many random initializations to make the comparisons more robust
- Overall the two main ideas in this paper -- A3C and curriculums -- are not really novel but the authors do make use of them in a real system.
The paper describes approaches taken to train learning agents for the 3D game Doom. The authors propose a number of performance enhancements (curriculum learning, attention (zoomed-in centered) frames, reward shaping, game variables, post-training rules) inspired by domain knowledge.
The enhancements together lead to a clear win as demonstrated by the competition results. From Fig 4, the curriculum learning clearly helps with learning over increasingly difficult settings. A nice result is that there is no overfitting to the harder classes once they have learned (probably because the curriculum is health and speed). The authors conclude from Fig 5 that the adaptive curriculum is better and more stable that pure A3C;
however, this is a bit of a stretch given that graph. They go on to say that Pure A3C doesn't learn at all in the harder map but then show no result/graph to back this claim. Tbl 5 shows a clear benefit of the post-training rules.
If the goal is to solve problems like these (3D shooters), then this paper makes a significant contribution in that it shows which techniques are practical for solving the problem and ultimately improving performance in these kinds of tasks.
Still, I am just not excited about this paper, mainly because it relies so heavily of many sources of domain knowledge, it is quite far from the pure reinforcement learning problem. The results are relatively unsurprising. Maybe they are novel for this problem, though.
I'm not sure we can realistically draw any conclusions about Figure 6 in the paper's current form. I recommend the authors increase the resolution or run some actual metrics to determine the fuzziness/clarity of each row/image: something more concrete than an arrow of already low-resolution images.
--- Added after rebuttal:
I still do not see any high-res images for Figure 6 or any link to them, but I trust that the authors will add them if accepted.
We just made Fig 6 high-resolutional. Thanks for your understanding!
We thank the reviewers for their insightful comments!
All reviewers agree that this paper makes a solid contribution with good experimental results. It is not uncommon to see application-oriented papers using a combination of multiple techniques to achieve strong performance. This category covers many seminar works, e.g., deep reinforcement learning for Atari games (applying deep models to traditional Q-learning), or even AlphaGo (supervised learning, policy gradient, value function, Monte-Carlo Tree Search, self-play). It may be a bit shortsighted to judge such strong performing papers with a single criterion.
Confusion about the domain:
Reviewer3 mentions that the paper "basically applies A3C to 3D spatial navigation tasks.", which is not true. In the deathmatch game of Doom, multiple players explore the maze and fight against each other to get a higher score, which is defined as #kills - #suicide. In this task, part of the goal is to learn anti-enemy tactics (e.g., dodging the rocket shot from the enemy, e.g., video:  http://yuandong-tian.com/dodge.mp4).
Therefore, Doom is a very different environment compared to pure navigation tasks proposed in previous works (e.g, https://arxiv.org/abs/1611.03673, https://arxiv.org/abs/1609.05143). Both Reviewer1 and Reviewer2 clearly mentioned that it is a gaming scenario (3D shooter game), rather than a navigation task.
One major issue brought about by the reviewers is its novelty.
In terms of applicational contribution, as pointed out by Reviewer1, this paper made "a significant contribution in that it shows which techniques are practical for solving the problem and ultimately improving performance in these kinds of tasks". Previous research in 3d game environments using a similar framework (a3c + curriculum training), or using more complicated approaches (e.g, memory-equipped approach like Arnold in https://arxiv.org/abs/1609.05521) does not achieve this level of performance.
Therefore, our method could serve as a baseline (we will release the code and pre-trained models) for follow-up research.
In terms of technical contributions, in this paper we use a modified version of A3C that stores a single model in the server side to collect game experience from multiple clients, which is different from original A3C that stores models on the client side and perform asynchronous updates.
As a result, the proposed approach uses less memory.
A related concern from the reviewers is whether papers focusing on a specific game like Doom will lead to better general solutions to the reinforcement learning problem. We believe the answer is yes.
For example, many papers on Atari games focus on Montezuma's Revenge, a particular game with very sparse reward and requires strong exploration techniques. Go, as a game with fully-observable but enormous states, is another example that could lead to a better understanding of general RL techniques. In our paper, having different levels of enemies is critical for our curriculum training to work. This could motivate following works that use parametrized factors in the environment to bootstrap the training, or apply similar approaches on changing/online environments.
Other issues:
We will run more experiments regarding to Fig. 5 and add more figures on curriculum learning regarding the full map.
We have more videos showing that the agent indeed learns tactics e.g., dodging. See the following link: http://yuandong-tian.com/dodge.mp4. The low-resolution image in Fig. 6 will be replaced by high-resolutional images.
We will add more discussions about the method of the paper and compare conceptually with others, which will naturally bring about more related works.

Meta-review:
role: PC
data: comment: This paper provides a number of performance enhancements inspired by domain knowledge. Taken together, these produce a compelling system that has shown itself to be the best-in-class as per the related competition.
 Experts agree that the authors do a good job at justifying the majority of the design decisions.
 
 pros:
 - insights into the SOTA Doom player
 
 cons:
 - lack of pure technical novelty: the various elements have existed previously
 
 This paper comes down to a matter of taste in terms of appreciation of SOTA systems or technical novelty.
 With the code being released, I believe that this work will have impact as a benchmark, and as a guidebook
 as to how features can be combined for SOTA performance on FPS-style scenarios.

Decision: Accept (Poster)