Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: Hi, quick question: can you please clarify how you perform n-step Q-Learning on auxiliary tasks (unless you use n=1?), since it has to be off-policy? Thanks!
review_title: n-step Q-Learning for auxiliary tasks

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper is about improving feature learning in deep reinforcement learning, by augmenting the main policy's optimization problem with terms corresponding to (domain-independent) auxiliary tasks. These tasks are about control (learning other policies that attempt to maximally modify the state space, i.e. here the pixels), immediate reward prediction, and value function replay. Except for the latter, these auxiliary tasks are only used to help shape the features (by sharing the CNN+LSTM feature extraction network). Experiments show the benefits of this approach on Atari and Labyrinth problems, with in particular much better data efficiency than A3C.

The paper is well written, ideas are sound, and results pretty convincing, so to me this is a clear acceptance. At high level I only have few things to say, none being of major concern:
- I believe you should say something about the extra computational cost of optimizing these auxiliary tasks. How much do you lose in terms of training speed? Which are the most costly components?
- If possible, please try to make it clearer in the abstract / intro that the agent is learning different policies for each task. When I read in the abstract that the agent "also maximises many other pseudo-reward functions simultaneously by reinforcement learning", my first understanding was that it learned a single policy to optimize all rewards together, and I realized my mistake only when reaching eq. 1.
- The "feature control" idea is not validated empirically (the preliminary experiment in Fig. 5 is far from convincing as it only seems to help slightly initially). I like that idea but I am worried by the fact the task is changing during learning, since the extracted features are being modified. There might be stability / convergence issues at play here.
- Since as you mentioned, "the performance of our agents is still steadily improving", why not keep them going to see how far they go? (at least the best ones)
- Why aren't the auxiliary tasks weight parameters (the lambda_*) hyperparameters to optimize? Were there any experiments to validate that using 1 was a good choice?
- Please mention the fact that auxiliary tasks are not trained with "true" Q-Learning since they are trained off-policy with more than one step of empirical rewards (as discussed in the OpenReview comments)

Minor stuff:
- "Policy gradient algorithms adjust the policy to maximise the expected reward, L_pi = -..." => that's actually a loss to be minimized
- In eq. 1 lambda_c should be within the sum
- Just below eq. 1 r_t^(c) should be r_t+k^(c)
- Figure 2 does not seem to be referenced in the text, also Figure 1(d) should be referenced in 3.3
- "the features discovered in this manner is shared" => are shared
- The text around eq. 2 refers to the loss L_PC but that term is not defined and is not (explicitly) in eq. 2
- Please explain what "Clip" means for dueling networks in the legend of Figure 3
- I would have liked to see more ablated versions on Atari, to see in particular if the same patterns of individual contribution as on Labyrinth were observed
- In the legend of Figure 3 the % mentioned are for Labyrinth, which is not clear from the text.
- In 4.1.2: "Figure 3 (right) shows..." => it is actually the top left plot of the figure. Also later "This is shown in Figure 3 Top" should be Figure 3 Top Right.
- "Figure 5 shows the learning curves for the top 5 hyperparameter settings on three Labyrinth navigation levels" => I think it is referring to the left and middle plots of the figure, so only on two levels (the text above might also need fixing)
- In 4.2: "The left side shows the average performance curves of the top 5 agents for all three methods the right half shows..." => missing a comma or something after "methods"
- Appendix: "Further details are included in the supplementary materials." => where are they?
- What is the value of lambda_PC? (=1 I guess?)

[Edit] I know some of my questions were already answered in Comments, no need to re-answer them
review_title: A solid submission

final_score: rating: null
review_content: - "Labyrinth also supports continuous motion unlike the Minecraft platform of (Oh et al., 2016), which is a 3D grid world." Not true anymore, the latest Minecraft platform (Malmo) allows getting continuous motion. (although it is true that this specific paper uses the 3D grid)
- 10x less steps to train a model is great, but how much longer does each step take? Since you train on many things simultaneously, is there still a wall-time speedup?
- It seems like feature control is almost the same thing as pixel control, but from the "viewpoint" of the "hidden state". Did you try having stacked recurrent layers and doing this feature control at each layer? If not, do you think it would be beneficial? 
- In the total absence of external rewards, the base policy would not learn anything meaningful. Did you attempt to do on-policy learning of some variants of pixel or feature control? Maybe as some kind of pretraining phase?
- You used n=20-step return, what \gamma value accompanies this?

Thanks!
review_title: A few questions

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a way of adding unsupervised auxiliary tasks to a deep RL agent like A3C. Authors propose a bunch of auxiliary control tasks and auxiliary reward tasks and evaluate the agent in Labyrinth and Atari. Proposed UNREAL agent performs significantly better than A3C and also learns faster. This is definitely a good contribution to the conference. However, this is not a surprising result since adding additional auxiliary tasks that are relevant to the goal should always help in better and faster feature shaping. This paper is a proof of concept for this idea.

The paper is well written and easy to follow by any reader with deep RL expertise.

Can authors comment about the computational resources needed to train the UNREAL agent?

The overall architecture is quite complicated. Are the authors willing to release the source code for their model?

--------------------------------------------------------
After rebuttal:
No change in the review.
review_title: Review

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This work proposes to train RL agents to also perform auxiliary tasks, positing that doing so will help models learn stronger features.
They propose two pseudo-control tasks, control the change in pixel intensity, and control the activation of latent features. They also propose a supervised regression task, predict immediate reward following a sequence of events. The latter is learned offline via a skewed sampling of an experience replay buffer in order to balance seeing reward or not to 1/2 chance.
Such agents perform significantly well on discrete-action-continuous-space RL tasks, and reach baseline performance in 10x less iterations. 

This work contrasts with traditional "passive" unsupervised or model-based learning. Instead of forcing the model to learn a potentially useless representation of the input, or to learn the possibly impossible (due to partial observability) task-modelling objective, learning to control local and internal features of the environment complements learning the optimal control policy.

To me the approach is novel and proposes a very interesting alternative to unsupervised learning that takes advantage of the "possibility" of control that an agent has over the environment.
The proposed tasks are explained at a rather high level, which is convenient to understand intuition, but I think some lower level of detail might be useful. For example L_PC should be explicitly mentioned, before reaching the appendix. Otherwise this work is clear and easily understandable by readers familiar with Deep RL.
The methodology is sound, on one hand hand the distribution of best hyperparameters might be different for A3C and UNREAL, but also measuring top-3 ensures that, presuming that the both best hyperparameters for A3C and for UNREAL are within the explored intervals, the per-method best hyperparameters are found.
I think one weakness of the paper (or rather, considering the number of things that can fit in a paper, crucially needed future work) is that there is very little experimental analysis of the effect of the auxiliary tasks appart from their (very strong) effect on performance. In the same vein, pixel/feature control seems to have the most impact, in Labyrinth just A3C+PC beats anything else (except UNREAL), I think it would have been worth looking at this, either in isolation or in more depth, measuring more than just performance on RL tasks.


review_title: Review

Meta-review: There was broad consensus by the reviewers that this paper should be accepted. There was also a good deal of discussion about detailed aspects of the paper. I think the directions in which this paper points are going to be of interest to many in the community. As one reviewer put is, the idea here seems to involve taking advantage of the "possibility" of control that an agent may have over the environment. This is formulated in terms of auxiliary control and auxiliary prediction tasks, which share an underlying CNN and LSTM representation.
 
 The revision posted by the author's addresses a number of the questions, suggestions and concerns of reviewers.
Decision: Accept (Oral)