Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: Depth is an interesting auxiliary task as it helps the agent localize itself, can you comment on whether other computer vision tasks such as texture or lighting prediction would improve the agent representation, and at what point this might become too hand-engineered?

The paper claims that the auxiliary loss helps bootstrap the learning, what are the contributions of the different losses in the gradient at various stages of training?
review_title: other auxiliary tasks and gradient contribution

final_score: rating: null
review_content: Why is the learning of depth perception unsupervised? Appendix B.2 states that the Z-buffer is used. Isn't this a supervised signal?
review_title: Unsupervised depth perception

final_score: rating: 5: Marginally below acceptance threshold
review_content: I do like the demonstration that including learning of auxiliary tasks does not interfere with the RL tasks but even helps. This is also not so surprising with deep networks. The deep structure of the model allows the model to learn first a good representation of the world on which it can base its solutions for specific goals. While even early representations do of course depend on the task performance itself, it is clear that there are common first stages in sensory representations like the need for edge detection etc. Thus, training by additional tasks will at least increase the effective training size. It is of course unclear how to adjust for this to make a fair comparison, but the paper could have included some more insights such as the change in representation with and without auxiliary training. 

I still strongly disagree with the implied definition of supervised or even self-supervised learning. The definition of unsupervised is learning without external labels. It does not matter if this comes from a human or for example from an expensive machine that is used to train a network so that a task can be solved later without this expensive machine. I would call EM a self-supervised method where labels are predicted from the model itself and used to bootstrap parameter learning. In this case you are using externally supplied labels, which is clearly a supervised learning task!

review_title: Depth is supervise learning

final_score: rating: null
review_content: - in section 2.2, is the label l_t 1 if p_t is close to *any* p_t'? Or do you train for specific t'-s?
- Figure 7 is quite interesting. I suppose each trajectory is for a single episode? Did you try plotting more than 1 trajectory per goal to see if the memory cells would activate similarly across trajectories with the same goal?
- Nitpick maybe: You are saying depth prediction is an unsupervised task, but you clearly have a supervision signal to your depth predictor, i.e. the depth! Or am I missing something? I understand there is no real labeling, since depth is a part of your "input", but I feel like this clashes with common use of the word 'unsupervised'.
- Did you try using other auxiliary tasks proposed in Jaderberg et al? I'm wondering since I'm curious to know if there is a point at which having too many auxiliary objectives hinders learning, or if they contribute positively to one another (by just making the representations better).

Thanks!


review_title: A few questions

final_score: rating: 7: Good paper, accept
review_content: This relatively novel work proposes to augment current RL models by adding self-supervised tasks encouraging better internal representations. 
The proposed tasks are depth prediction and loop closure detection. While these tasks assume a 3D environment as well some position information, such priors are well suited to a large variety of tasks pertaining to navigation and robotics.

Extensive experiments suggest to incorporating such auxiliary tasks increase performance and to a large extent learning speed.
Additional analysis of value functions and internal representations suggest that some structure is being discovered by the model, which would not be without the auxiliary tasks.


While specific to 3D-environment tasks, this work provides additional proof that using input data in addition to sparse external reward signals helps to boost learning speed as well as learning better internal representation. It is original, clearly presented, and strongly supported by empirical evidence.

One small downside of the experimental method (or maybe just the results shown) is that by picking top-5 runs, it is hard to judge whether such a model is better suited to the particular hyperparameter range that was chosen, or is simply more robust to these hyperparameter settings. Maybe an analysis of performance as a function of hyperparameters would help confirm the superiority of the approach to the baselines. My own suspicion is that adding auxiliary tasks would make the model robust to bad hyperparameters.

Another downside is that the authors dismiss navigation literature as "not RL". I sympathize with the limit on the number of things that can fit in a paper, but some experimental comparison with such literature may have proven insightful, if just in measuring the quality of the learned representations.

review_title: Review

final_score: rating: 7: Good paper, accept
review_content: This paper shows that a deep RL approach augmented with auxiliary tasks improves performance on navigation in complex environments. Specifically, A3C is used for the RL problem, and the agent is simultaneously trained on an unsupervised depth prediction task and a self-supervised loop closure classification task. While the use of auxiliary tasks to improve training of models including RL agents is not new, the main contribution here is the use of tasks that encourage learning an intrinsic representation of space and movement that enables significant improvements on maze navigation tasks.

The paper is well written, experiments are convincing, and the value of the auxiliary tasks for the problem are clear. However, the contribution is relatively incremental given previous work on RL for navigation and on auxiliary tasks. The work could become of greater interest provided broader analysis and insights on either optimal combinations of tasks for visual navigation (e.g. the value of other visual / geometry-based tasks), or on auxiliary tasks with RL in general.  As it is, it is a useful demonstration of the benefit of geometry-based auxiliary tasks for navigation, but of relatively narrow interest.
review_title: well presented, convincing, but of limited novelty

Meta-review: The paper proposes an approach to navigating in complex environments using RL agents that have auxiliary tasks besides just the successful navigation itself (for instance, the task of predicting depth from images). The idea is a nice one, and the demonstration is fairly compelling. The one aspect that seems a bit unsatisfying is that the the approach does seem a bit ad-hoc, and could be made more formal, but presenting these results on a challenging task like this navigation problem is certainly sufficient for the paper to be worth accepting. The pros and cons are as follows:
 
 
 Pros:
 + Idea of formulating auxiliary tasks is a nice one and the precise form in which it is done here appears novel
 + Good results on a challenge task of maze navigation from visual data
 
 Cons:
 - Methodology does seem a bit ad-hoc, it would be nice to see if some of the auxiliary task mechanisms could be formalized beyond simple "this is what worked for this domain"
Decision: Accept (Poster)