Title: Learning to Navigate in Complex Environments
Authors: Piotr Mirowski
Razvan Pascanu
Fabio Viola
Hubert Soyer
Andy Ballard
Andrea Banino
Misha Denil
Ross Goroshin
Laurent Sifre
Koray Kavukcuoglu
Dharshan Kumaran
Raia Hadsell
Venue: ICLR 2017
Keywords: Deep learning
Reinforcement Learning

Abstract: Learning to navigate in complex environments with dynamic elements is an important milestone in developing AI agents. In this work we formulate the navigation question as a reinforcement learning problem and show that data efficiency and task performance can be dramatically improved by relying on additional auxiliary tasks to bootstrap learning. In particular we consider jointly learning the goal-driven reinforcement learning problem with an unsupervised depth prediction task and a self-supervised loop closure classification task. Using this approach we can learn to navigate from raw sensory input in complicated 3D mazes, approaching human-level performance even under conditions where the goal location changes frequently. We provide detailed analysis of the agent behaviour, its ability to localise, and its network activity dynamics, that show that the agent implicitly learns key navigation abilities, with only sparse rewards and without direct supervision.

TL;DR: We proposed a deep RL method, augmented with memory and auxiliary learning targets, for training agents to navigate within large and visually rich environments that include frequently changing start and goal locations

Review discussion:
We have just submitted an updated version of the paper, including additional references as well as new results on those agents that are enhanced with auxiliary tasks.
Specifically, we investigated:
1) a new way of performing depth prediction, by formulating it as a classification task (over the quantized depth image) and compared it to depth regression.
2) the use of depth prediction as an auxiliary task for the policy LSTM, instead of the convnet.
3) the effect, during actor critic training, of reward clipping on the performance of the agent as well as on the stability of RL learning.
Why is the learning of depth perception unsupervised? Appendix B.2 states that the Z-buffer is used. Isn't this a supervised signal?
Thank you for your comment.
We believe that there may be a clash of nomenclature and that the definitions of unsupervised or supervised learning are not without ambiguity. From our perspective, supervised learning implies that the labels are annotations made by humans. These labels are usually expensive to collect - hence the idea of using unsupervised learning to help supervised learning.
Moreover, our reasoning is that depth is part of the inputs; as we are reconstructing a subset of the input from another, we call it unsupervised learning. This is one accepted definition of unsupervised learning (i.e. modeling p(x) in some form). We can see however how one could see depth prediction as supervised learning.
For this reason, we will replace “unsupervised loss” by “self-supervised loss” in the revised version of the paper.
- in section 2.2, is the label l_t 1 if p_t is close to *any* p_t'? Or do you train for specific t'-s?
- Figure 7 is quite interesting. I suppose each trajectory is for a single episode? Did you try plotting more than 1 trajectory per goal to see if the memory cells would activate similarly across trajectories with the same goal?
- Nitpick maybe: You are saying depth prediction is an unsupervised task, but you clearly have a supervision signal to your depth predictor, i.e. the depth! Or am I missing something? I understand there is no real labeling, since depth is a part of your "input", but I feel like this clashes with common use of the word 'unsupervised'.
- Did you try using other auxiliary tasks proposed in Jaderberg et al? I'm wondering since I'm curious to know if there is a point at which having too many auxiliary objectives hinders learning, or if they contribute positively to one another (by just making the representations better).
Thanks!
Thank you very much for your comments.
In section 2.2, l_t is equal to 1 if p_t is close to any p_t’. We only consider time points t’ that follow the last respawn, such that (t - t’) < 400 steps.
In other words, l_t = 1 if the agents “walks back in its recent footsteps” since the last time it spawned in the maze, otherwise it is l_t = 0.
Alternatively, we could say that the agent “paints” the squares of the maze that it has visited (but every time that the agent respawns during an episode, the “painting” goes away).
Thank you very much for your suggestion for Figure 7. We will include this cross-episode analysis per goal location in the revision.
Please see our reply to AnonReviewer3 regarding the supervised nature of the depth prediction task.
Unifying our work with the work of Jaderberg, Mnih, Czarnecki et al. is something we will look at next. We think that the questions posed are definitely important and we do not have a definitive answer yet.
But in general we notice that composing multiple auxiliary losses on distinct parts of the network tends to help, though there may be diminishing returns when composing losses on the same modules.
For instance, we noticed in our early experiments that combining depth prediction from the convnet and loop closure from the LSTM improved upon merely doing depth prediction from the convnet.
However, adding loop closure on top of depth prediction from the LSTM did slightly hurt the performance. We will run an additional experiment with depth prediction from the convnet and from the LSTM, without loop closure prediction, to further investigate the combination of losses. We hypothesize that putting multiple auxiliary loss together might sometimes hurt or not work as well because of the overlap in the features they help building, because of the lack of capacity in the LSTM, or because it can make the optimization problem harder - as we are merely doing a weighted sum of the gradients from the different losses.
Depth is an interesting auxiliary task as it helps the agent localize itself, can you comment on whether other computer vision tasks such as texture or lighting prediction would improve the agent representation, and at what point this might become too hand-engineered?
The paper claims that the auxiliary loss helps bootstrap the learning, what are the contributions of the different losses in the gradient at various stages of training?
Thank you so much for your comments.
Regarding hand-engineering of the losses:
We have not tried texture or lightning prediction, but our guess is that both of them would help. We believe that the nature of the auxiliary loss is not that crucial, and that any meaningfully picked task can work, as long as it can be solved quickly by the neural network and has some relationship to the observations of the agent. Examples of alternatively tasks are presented in https://arxiv.org/abs/1611.05397. If the system were over-engineered, this would mean that it is heavily depending on a particular choice of the task or of the environment and might not generalise well. This might be the case with textures or lighting conditions, as opposed to a colour- and illumination-invariant depth perception.
Regarding the different stages of learning:
Do you mean making the comparison in terms of the norm? We are worried that comparing gradients is at best non-meaningful, at worst deceitful. E.g. the fact that the gradient of a particular loss is always smaller than the gradient of the other does not mean that this gradient has a smaller contribution in learning. This smaller mass could be concentrated on particular weights, or affect particular weights that the other loss does not (i.e. the two gradients are orthogonal to each other which is to be expected in high dimensions). The non-stationarity of the gradient distribution is also hard to capture, which makes the comparison hard.
For what is worth, we believe that the model converges quickly on the auxiliary loss at the beginning of the training, then spends a lot of the time trying to solve the RL task. If anything, we believe that this is why phrasing the prediction task as a classification rather than regression works better (as learning a classification task is much faster). It is unclear what kind of representation this auxiliary loss builds, but, at a high level, being able to predict depth means that one can learn something about the environment (e.g., understand to some extend its geometry) which we believe to be useful. We do want to insist that we believe that many variations of the auxiliary tasks would work, which points towards a more generic mechanism rather than a specific correlation between depth prediction and the kind of task we try to solve.
This relatively novel work proposes to augment current RL models by adding self-supervised tasks encouraging better internal representations.
The proposed tasks are depth prediction and loop closure detection.
While these tasks assume a 3D environment as well some position information, such priors are well suited to a large variety of tasks pertaining to navigation and robotics.
Extensive experiments suggest to incorporating such auxiliary tasks increase performance and to a large extent learning speed.
Additional analysis of value functions and internal representations suggest that some structure is being discovered by the model, which would not be without the auxiliary tasks.
While specific to 3D-environment tasks, this work provides additional proof that using input data in addition to sparse external reward signals helps to boost learning speed as well as learning better internal representation. It is original, clearly presented, and strongly supported by empirical evidence.
One small downside of the experimental method (or maybe just the results shown) is that by picking top-5 runs, it is hard to judge whether such a model is better suited to the particular hyperparameter range that was chosen, or is simply more robust to these hyperparameter settings. Maybe an analysis of performance as a function of hyperparameters would help confirm the superiority of the approach to the baselines. My own suspicion is that adding auxiliary tasks would make the model robust to bad hyperparameters.
Another downside is that the authors dismiss navigation literature as "not RL". I sympathize with the limit on the number of things that can fit in a paper, but some experimental comparison with such literature may have proven insightful, if just in measuring the quality of the learned representations.
Thank you for your comments.
We provided additional analysis, in Appendix section C.4, to address your comments. For each of the experiments in this paper, 64 replicas were run with hyperparameters (learning rate, entropy cost) sampled from the same interval. Figure 12 shows that the Nav architectures with auxiliary tasks achieve higher results for a comparatively larger number of replicas, suggesting that auxiliary tasks make learning more robust to the choice of hyperparameters - in line with the reviewer’s intuition. This observation is particularly strong for the small static maze (more than a third of the replicas for FF A3C and LSTM A3C baselines do not even reach the goal, whereas less than 10 Nav agents out of 64 replicas suffer from this).
In this paper we focused on the potential benefits of auxiliary tasks in enhancing the navigational capacities of agents that use deep RL to map pixels directly to actions - rather than designing a new state-of-the-art navigation system. Our discussion of the literature reflected this focus, but was not intended to be dismissive of other navigation approaches such as SLAM.
I do like the demonstration that including learning of auxiliary tasks does not interfere with the RL tasks but even helps. This is also not so surprising with deep networks. The deep structure of the model allows the model to learn first a good representation of the world on which it can base its solutions for specific goals.
While even early representations do of course depend on the task performance itself, it is clear that there are common first stages in sensory representations like the need for edge detection etc.
Thus, training by additional tasks will at least increase the effective training size. It is of course unclear how to adjust for this to make a fair comparison, but the paper could have included some more insights such as the change in representation with and without auxiliary training.
I still strongly disagree with the implied definition of supervised or even self-supervised learning. The definition of unsupervised is learning without external labels. It does not matter if this comes from a human or for example from an expensive machine that is used to train a network so that a task can be solved later without this expensive machine. I would call EM a self-supervised method where labels are predicted from the model itself and used to bootstrap parameter learning. In this case you are using externally supplied labels, which is clearly a supervised learning task!
Regarding the reviewer’s comment about supervised vs. unsupervised terminology, we agree that the nomenclature can be confusing.
While we do not agree with the reviewer's views on supervised or unsupervised learning, we have removed the qualifier from the paper, and simply use “auxiliary task”. We emphasise that the fundamental approach and results are independent from the nomenclature used.
In order to evaluate the reviewer’s suggestion that auxiliary tasks accelerate learning through increased training, leading to faster representation learning, we did asymptotic performance analysis on the existing agents, documented in Appendix section C.5 “Asymptotic performance of the agents”. Note that the asymptotic performance (see Table 3) still shows a significant advantage, indicating that the contribution of these tasks does more than simply accelerate learning. As Table 3 shows, the performance and the position accuracy of the baseline agent significantly increase after twice the number of training steps (going from 57 points to 90 points, and from 33.4% to 66.5%), but still do not reach the performance and accuracy of the Nav A3C+D2 agent.
We also provide new evidence for the specific benefit of the depth prediction task: i.e. that in our experiments, it provides a greater performance benefit than the reward prediction task introduced in [Jaderberg et al., 2016 ICLR submission].
As Yann LeCun mentions in his NIPS Plenary Talk, Predictive Learning is a better term if the notion of unsupervised vs supervised is confusing. Predicting rewards or inverse-dynamics are also in a sense supervised objectives in an MLE sense.
The judgement should not be based on notions that predicting an image like next state is true unsupervised learning and other auxiliary predictions are supervised. The idea is to learn better representations by bootstrapping more signals, which can come from normal model-free RL objectives as well as depth / reward / next-state predictions. The idea of depth prediction is novel and should be seen as an important contribution for Predictive Learning in RL, trying to move away from predicting single scalar values like value-functions. Combining other forms of Predictive Learning with RL is a hot topic for the future, and this paper being one of the only 2 papers addressing this with extensive experiments, should be accepted.
Rating: 8: Top 50% of accepted papers, clear accept
Confidence: 5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature
This paper shows that a deep RL approach augmented with auxiliary tasks improves performance on navigation in complex environments.
Specifically, A3C is used for the RL problem, and the agent is simultaneously trained on an unsupervised depth prediction task and a self-supervised loop closure classification task.
While the use of auxiliary tasks to improve training of models including RL agents is not new, the main contribution here is the use of tasks that encourage learning an intrinsic representation of space and movement that enables significant improvements on maze navigation tasks.
The paper is well written, experiments are convincing, and the value of the auxiliary tasks for the problem are clear.
However, the contribution is relatively incremental given previous work on RL for navigation and on auxiliary tasks. The work could become of greater interest provided broader analysis and insights on either optimal combinations of tasks for visual navigation (e.g. the value of other visual / geometry-based tasks), or on auxiliary tasks with RL in general.  As it is, it is a useful demonstration of the benefit of geometry-based auxiliary tasks for navigation, but of relatively narrow interest.
I am not an author of this paper.
However, out of curiosity, I would like to know from AnonReviewer1 a small clarification about a couple of sentences:
i) "While the use of auxiliary tasks to improve training of models including RL agents is not new", the main contribution here is the use of tasks that encourage learning an intrinsic representation of space and movement that enables significant improvements on maze navigation tasks.
ii) "However, the contribution is relatively incremental given previous work on RL for navigation and on auxiliary tasks." The work could become of greater interest provided broader analysis and insights on either optimal combinations of tasks for visual navigation (e.g. the value of other visual / geometry-based tasks), or on auxiliary tasks with RL in general.
I would like to point out that the only other Deep RL paper with auxiliary tasks is from Jaderberg et al, and a paper from DeepMind itself, and was a "parallel" submission to ICLR 17 (the same conference).
So, I do not think the novelty of one paper should be judged based on the existence of the other since they were both submissions made at the same time to the same conference.
I would be interested to know if there has been some other work in Deep RL with auxiliary tasks, preceding both these works, so that the comment is still applicable.
I do not have any comments on the appropriateness of the rating. I am just curious to know if there have been previous works.
Yes, there have been previous work. Lample and Chaplot (https://arxiv.org/abs/1609.05521) use prediction of enemy as an auxiliary task for playing Doom with DQN.
Our approach uses depth prediction as an auxiliary task, which is more general than the enemy-detection task employed by Lample and colleagues. In additional experiments, we provide evidence that speaks to this point: depth-prediction provides benefits not only in navigational scenarios that require shortest path planning, but also scenarios with minimal navigational demands (e.g. collecting rewarded objects and avoiding negatively rewarding objects).
Further, our experiments incorporate an important control lacking by this previous work: specifically, we show that auxiliary depth prediction is much better than using RGB+depth inputs.
We thank the reviewer for suggesting ways in which our paper could be of broader interest: specifically, if it could afford more insights into i) optimal combinations of auxiliary tasks for navigation and ii) the role of auxiliary tasks in non-navigational RL tasks. We address these points through additional results which have been added to the paper. These suggest that depth-prediction from the policy LSTM yields increases performance to a greater extent than a different auxiliary task (i.e. reward prediction: from Jaderberg et al., 2016 ICLR submission), with no additional benefit gained from combining both auxiliary tasks together.
Further, we demonstrate a more general benefit of the depth prediction task in enhancing performance in scenarios that involve minimal navigational demands (i.e. no requirement for shortest path planning). We describe these findings in more detail in section 5.3, table 2, appendix C.4 and figure 10 of the revised paper.
I agree with Piotr. Enemy Prediction is a very task-specific (to Doom) auxiliary task. It is as specific as saying we could have an auxiliary task of ball tracking in games like Pong and Breakout. The goal in Lample et al was to get DQN work for Doom, and it doesn't work stand-alone due to which they introduced more objectives like enemy prediction.
Depth Map Prediction is more generic, and applicable to 3D Deep Reinforcement Learning problems in general (in worlds like Minecraft, Labyrinth, etc).
Therefore, I believe the criticism on the basis of novelty of auxiliary tasks is not appropriate for this paper. The only other auxiliary task paper is a parallel submission to the same conference.
We have addressed the points/suggestions raised through several additional experiments which have been added to the paper in the latest revision.
1) Exploring the optimal combinations of auxiliary tasks: we now include an additional auxiliary task, reward prediction (shown to be effective across a range of tasks in Jaderberg et al. 2016, ICLR submission). Results show that depth prediction is superior to reward prediction in the navigational settings examined, with the combination of the two being no more effective (section 5.3, table 2).
2) The focus of our paper on navigation and the depth prediction auxiliary task: we now present results showing that auxiliary depth prediction is beneficial in scenarios that have minimal navigational demands, suggesting its general effectiveness (appendix C.3; figure 10).
3) Whether auxiliary tasks increase robustness to hyperparameters: we provide a more systematic analysis to show that this does seem to be the case (appendix C.4; figure 11).
4) Whether auxiliary tasks simply accelerate training: we provide evidence that they do more that this through analyses of asymptotic performance (appendix C.5; table 3), effects on the representations learnt (i.e. indexed by position decoding), demonstration that depth prediction shows benefits compared to the reward prediction auxiliary task in the navigation domain.

Meta-review:
role: PC
data: comment: The paper proposes an approach to navigating in complex environments using RL agents that have auxiliary tasks besides just the successful navigation itself (for instance, the task of predicting depth from images). The idea is a nice one, and the demonstration is fairly compelling. The one aspect that seems a bit unsatisfying is that the the approach does seem a bit ad-hoc, and could be made more formal, but presenting these results on a challenging task like this navigation problem is certainly sufficient for the paper to be worth accepting. The pros and cons are as follows:
 
 
 Pros:
 + Idea of formulating auxiliary tasks is a nice one and the precise form in which it is done here appears novel
 + Good results on a challenge task of maze navigation from visual data
 
 Cons:
 - Methodology does seem a bit ad-hoc, it would be nice to see if some of the auxiliary task mechanisms could be formalized beyond simple "this is what worked for this domain"

Decision: Accept (Poster)