Title: LEARNING TO NAVIGATE IN COMPLEX ENVIRONMENTS

Abstract: Learning to navigate in complex environments with dynamic elements is an important milestone in developing AI agents. In this work we formulate the navigation question as a reinforcement learning problem and show that data efficiency and task performance can be dramatically improved by relying on additional auxiliary tasks leveraging multimodal sensory inputs. In particular we consider jointly learning the goal-driven reinforcement learning problem with auxiliary depth prediction and loop closure classification tasks. This approach can learn to navigate from raw sensory input in complicated 3D mazes, approaching human-level performance even under conditions where the goal location changes frequently. We provide detailed analysis of the agent behaviour 1 , its ability to localise, and its network activity dynamics, showing that the agent implicitly learns key navigation abilities.

1. INTRODUCTION:

The ability to navigate efficiently within an environment is fundamental to intelligent behavior. Whilst conventional robotics methods, such as Simultaneous Localisation and Mapping (SLAM), tackle navigation through an explicit focus on position inference and mapping (Dissanayake et al., 2001) , here we follow recent work in deep reinforcement learning (Mnih et al., 2015; 2016) and propose that navigational abilities could emerge as the by-product of an agent learning a policy that maximizes reward. One advantage of an intrinsic, end-to-end approach is that actions are not divorced from representation, but rather learnt together, thus ensuring that task-relevant features are present in the representation. Learning to navigate from reinforcement learning in partially observable environments, however, poses several challenges. First, rewards are often sparsely distributed in the environment, where there may be only one goal location. Second, environments often comprise dynamic elements, requiring the agent to use memory at different timescales: rapid one-shot memory for the goal location, together with short term memory subserving temporal integration of velocity signals and visual observations, and longer term memory for constant aspects of the environment (e.g. boundaries, cues).

To improve statistical efficiency we bootstrap the reinforcement learning procedure by augmenting our loss with auxiliary tasks that provide denser training signals that support navigation-relevant representation learning. We consider two additional losses: the first one involves reconstruction of a low-dimensional depth map at each time step by predicting one input modality (the depth channel) from others (the colour channels). This auxiliary task concerns the 3D geometry of the environment, and is aimed to encourage the learning of representations that aid obstacle avoidance and short-term trajectory planning. The second task directly invokes loop closure from SLAM: the agent is trained to predict if the current location has been previously visited within a local trajectory.

Figure 1 : Views from a small 5 × 10 maze, a large 9 × 15 maze and an I-maze, with corresponding maze layouts and sample agent trajectories. The mazes, which will be made public, have different textures and visual cues as well as exploration rewards and goals (shown right).

To address the memory requirements of the task we rely on a stacked LSTM architecture (Graves et al., 2013; Pascanu et al., 2013) . We evaluate our approach using five 3D maze environments and demonstrate the accelerated learning and increased performance of the proposed agent architecture. These environments feature complex geometry, random start position and orientation, dynamic goal locations, and long episodes that require thousands of agent steps (see Figure 1 ). We also provide detailed analysis of the trained agent to show that critical navigation skills are acquired. This is important as neither position inference nor mapping are directly part of the loss; therefore, raw performance on the goal finding task is not necessarily a good indication that these skills are acquired. In particular, we show that the proposed agent resolves ambiguous observations and quickly localizes itself in a complex maze, and that this localization capability is correlated with higher task reward.

2. APPROACH:

We rely on a end-to-end learning framework that incorporates multiple objectives. Firstly it tries to maximize cumulative reward using an actor-critic approach. Secondly it minimizes an auxiliary loss of inferring the depth map from the RGB observation. Finally, the agent is trained to detect loop closures as an additional auxiliary task that encourages implicit velocity integration.

The reinforcement learning problem is addressed with the Asynchronous Advantage Actor-Critic (A3C) algorithm (Mnih et al., 2016) that relies on learning both a policy π(a t |s t ; θ) and value function V (s t ; θ V ) given a state observation s t . Both the policy and value function share all intermediate representations, both being computed using a separate linear layer from the topmost layer of the model. The agent setup closely follows the work of (Mnih et al., 2016) and we refer to this work for the details (e.g. the use of a convolutional encoder followed by either an MLP or an LSTM, the use of action repetition, entropy regularization to prevent the policy saturation, etc.). These details can also be found in the Appendix B.

The baseline that we consider in this work is an A3C agent (Mnih et al., 2016) that receives only RGB input from the environment, using either a recurrent or a purely feed-forward model (see Figure 2a, b ). The encoder for the RGB input (used in all other considered architectures) is a 3 layer convolutional network. To support the navigation capability of our approach, we also rely on the Nav A3C agent (Figure 2c ) which employs a two-layer stacked LSTM after the convolutional encoder. We expand the observations of the agents to include agent-relative velocity, the action sampled from the stochastic policy and the immediate reward, from the previous time step. We opt to feed the velocity and previously selected action directly to the second recurrent layer, with the first layer only receiving the reward. We postulate that the first layer might be able to make associations between reward and visual observations that are provided as context to the second layer from which the policy is computed. Thus, the observation s t may include an image x t ∈ R 3×W ×H (where W and H are the width and height of the image), the agent-relative lateral and rotational velocity v t ∈ R 6 , the previous action a t-1 ∈ R N A , and the previous reward r t-1 ∈ R.

x t r t-1 {v t , a t-1 } enc x t enc enc Loop (L) Depth (D 1 ) a. FF A3C c. Nav A3C d. Nav A3C +D 1 D 2 L x t r t-1 {v t , a t-1 } enc x t b. LSTM A3C Depth (D 2 )

Figure 2d shows the augmentation of the Nav A3C with the different possible auxiliary losses. In particular we consider predicting depth from the convolutional layer (we will refer to this choice as D 1 ), or from the top LSTM layer (D 2 ) or predicting loop closure (L). The auxiliary losses are computed on the current frame via a single layer MLP. The agent is trained by applying a weighted sum of the gradients coming from A3C, the gradients from depth prediction (multiplied with β d1 , β d2 ) and the gradients from the loop closure (scaled by β l ). More details of the online learning algorithm are given in Appendix B.

2.1. DEPTH PREDICTION:

The primary input to the agent is in the form of RGB images. However, depth information, covering the central field of view of the agent, might supply valuable information about the 3D structure of the environment. While depth could be directly used as an input, we argue that if presented as an additional loss it is actually more valuable to the learning process. In particular if the prediction loss shares representation with the policy, it could help build useful features for RL much faster, bootstrapping learning. Since we know from (Eigen et al., 2014 ) that a single frame can be enough to predict depth, we know this auxiliary task can be learnt. A comparison between having depth as input versus as an additional loss is given in Appendix C, which shows significant gain for depth as a loss.

Since the role of the auxiliary loss is just to build up the representation of the model, we do not necessarily care about the specific performance obtained or nature of the prediction. We do care about the data efficiency aspect of the problem and also computational complexity. If the loss is to be useful for the main task, we should converge faster on it compared to solving the RL problem (using less data samples), and the additional computational cost should be minimal. To achieve this we use a low resolution variant of the depth map, reducing the screen resolution to 4x16 pixels2 .

We explore two different variants for the loss. The first choice is to phrase it as a regression task, the most natural choice. While this formulation, combined with a higher depth resolution, extracts the most information, mean square error imposes a unimodal distribution (van den Oord et al., 2016) .

To address this possible issue, we also consider a classification loss, where depth at each position is discretised into 8 different bands. The bands are non-uniformally distributed such that we pay more attention to far-away objects (details in Appendix B). The motivation for the classification formulation is that while it greatly reduces the resolution of depth, it is more flexible from a learning perspective and can result in faster convergence (hence faster bootstrapping).

2.2. LOOP CLOSURE PREDICTION:

Loop closure, like depth, is valuable for a navigating agent, since can be used for efficient exploration and spatial reasoning. To produce the training targets, we detect loop closures based on the similarity of local position information during an episode, which is obtained by integrating 2D velocity over time. Specifically, in a trajectory noted {p 0 , p 1 , . . . , p T }, where p t is the position of the agent at time t, we define a loop closure label l t that is equal to 1 if the position p t of the agent is close to the position p t at an earlier time t . In order to avoid trivial loop closures on consecutive points of the trajectory, we add an extra condition on an intermediary position p t being far from p t . Thresholds η 1 and η 2 provide these two limits. Learning to predict the binary loop label is done by minimizing the Bernoulli loss L l between l t and the output of a single-layer output from the hidden representation h t of the last hidden layer of the model, followed by a sigmoid activation.

3. RELATED WORK:

There is a rich literature on navigation, primarily in the robotics literature. However, here we focus on related work in deep RL. Deep Q-networks (DQN) have had breakthroughs in extremely challenging domains such as Atari (Mnih et al., 2015) . Recent work has developed on-policy RL methods such as advantage actor-critic that use asynchronous training of multiple agents in parallel (Mnih et al., 2016) . Recurrent networks have also been successfully incorporated to enable state disambiguation in partially observable environments (Koutnik et al., 2013; Hausknecht & Stone, 2015; Mnih et al., 2016; Narasimhan et al., 2015) . 2016) also used the Minecraft domain to show the benefit of combining feedforward deep-Q networks with the learning of resuable skill modules (cf options: (Sutton et al., 1999) ) to transfer between navigation tasks. Tai & Liu (2016) trained a convnet DQN-based agent using depth channel inputs for obstacle avoidance in 3D environments. Barron et al. (2016) investigated how well a convnet can predict the depth channel from RGB in the Minecraft environment, but did not use depth for training the agent.

Auxiliary tasks have often been used to facilitate representation learning (Suddarth & Kergosien, 1990) . Recently, the incorporation of additional objectives, designed to augment representation learning through auxiliary reconstructive decoding pathways (Zhang et al., 2016; Rasmus et al., 2015; Zhao et al., 2015; Mirowski et al., 2010) , has yielded benefits in large scale classification tasks. In deep RL settings, however, only two previous papers have examined the benefit of auxiliary tasks. Specifically, Li et al. (2016) consider a supervised loss for fitting a recurrent model on the hidden representations to predict the next observed state, in the context of imitation learning of sequences provided by experts, and Lample & Chaplot (2016) show that the performance of a DQN agent in a first-person shooter game in the VizDoom environment can be substantially enhanced by the addition of a supervised auxiliary task, whereby the convolutional network was trained on an enemy-detection task, with information about the presence of enemies, weapons, etc., provided by the game engine.

In contrast, our contribution addresses fundamental questions of how to learn an intrinsic representation of space, geometry, and movement while simultaneously maximising rewards through reinforcement learning. Our method is validated in challenging maze domains with random start and goal locations.

4. EXPERIMENTS:

We consider a set of first-person 3D mazes from the DeepMind Lab environment (Beattie et al., 2016) (see Fig. 1 ) that are visually rich, with additional observations available to the agent such as inertial information and local depth information. 3 The action space is discrete, yet allows finegrained control, comprising 8 actions: the agent can rotate in small increments, accelerate forward or backward or sideways, or induce rotational acceleration while moving. Reward is achieved in these environments by reaching a goal from a random start location and orientation. If the goal is reached, the agent is respawned to a new start location and must return to the goal. The episode terminates when a fixed amount of time expires, affording the agent enough time to find the goal several times. There are sparse 'fruit' rewards which serve to encourage exploration. Apples are worth 1 point, strawberries 2 points and goals are 10 points. Videos of the agent solving the maze are linked in Appendix A.

In the static variant of the maze, the goal and fruit locations are fixed and only the agent's start location changes. In the dynamic (Random Goal) variant, the goal and fruits are randomly placed on every episode. Within an episode, the goal and apple locations stay fixed until the episode ends. This encourages an explore-exploit strategy, where the agent should initially explore the maze, then retain the goal location and quickly refind it after each respawn. For both variants (static and random goal) we consider a small and large map. The small mazes are 5 × 10 and episodes last for 3600 timesteps, and the large mazes are 9 × 15 with 10800 steps (see Figure 1 ). The RGB observation is 84 × 84.

The I-Maze environment (see Figure 1 , right) is inspired by the classic T-maze used to investigate navigation in rodents (Olton et al., 1979) : the layout remains fixed throughout, the agent spawns in the central corridor where there are apple rewards and has to locate the goal which is placed in the alcove of one of the four arms. Because the goal is hidden in the alcove, the optimal agent behaviour must rely on memory of the goal location in order to return to the goal using the most direct route. Goal location is constant within an episode but varies randomly across episodes.

The different agent architectures described in Section 2 are evaluated by training on the five mazes. Figure 3 shows learning curves (averaged over the 5 top performing agents). The agents are a feedforward model (FF A3C), a recurrent model (LSTM A3C), the stacked LSTM version with velocity, previous action and reward as input (Nav A3C), and Nav A3C with depth prediction from the convolution layer (Nav A3C+D 1 ), Nav A3C with depth prediction from the last LSTM layer (Nav A3C+D 2 ), Nav A3C with loop closure prediction (Nav A3C+L) as well as the Nav A3C with all auxiliary losses considered together (Nav A3C+D 1 D 2 L). In each case we ran 64 experiments with randomly sampled hyper-parameters (for ranges and details please see the appendix). The mean over the top 5 runs as well as the top 5 curves are plotted. Expert human scores, established by a professional game player, are compared to these results. The Nav A3C+D 2 agents reach human-level performance on Static 1 and 2, and attain about 91% and 59% of human scores on Random Goal 1 and 2.

In Mnih et al. (2015) reward clipping is used to stabilize learning, technique which we employed in this work as well. Unfortunately, for these particular tasks, this yields slightly suboptimal policies because the agent does not distinguish apples (1 point) from goals (10 points). Removing the reward clipping results in unstable behaviour for the base A3C agent (see Appendix C). However it seems that the auxiliary signal from depth prediction mediates this problem to some extent, resulting in stable learning dynamics (e.g. Figure 3f , Nav A3C+D 1 vs Nav A3C*+D 1 ). We clearly indicate whether reward clipping is used by adding an asterisk to the agent name.

Figure 3f also explores the difference between the two formulations of depth prediction, as a regression task or a classification task. We can see that the regression agent (Nav A3C*+D 1 [MSE] ) performs worse than one that does classification (Nav A3C*+D 1 ). This result extends to other maps, and we therefore only use the classification formulation in all our other results4 . Also we see that predicting depth from the last LSTM layer (hence providing structure to the recurrent layer, not just the convolutional ones) performs better.

We note some particular results from these learning curves. In Figure 3 (a and b), consider the feedforward A3C model (red curve) versus the LSTM version (pink curve). Even though navigation seems to intrinsically require memory, as single observations could often be ambiguous, the feedforward model achieves competitive performance on static mazes. This suggest that there might be good strategies that do not involve temporal memory and give good results, namely a reactive policy held by the weights of the encoder, or learning a wall-following strategy. This motivates the dynamic environments that encourage the use of memory and more general navigation strategies.

Figure 3 also shows the advantage of adding velocity, reward and action as an input, as well as the impact of using a two layer LSTM (orange curve vs red and pink). Though this agent (Nav A3C) is better than the simple architectures, it is still relatively slow to train on all of the mazes. We believe that this is mainly due to the slower, data inefficient learning that is generally seen in pure RL approaches. Supporting this we see that adding the auxiliary prediction targets of depth and loop closure (Nav A3C+D 1 D 2 L, black curve) speeds up learning dramatically on most of the mazes (see Table 1 : AUC metric). It has the strongest effect on the static mazes because of the accelerated learning, but also gives a substantial and lasting performance increase on the random goal mazes.

Although we place more value on the task performance than on the auxiliary losses, we report the results from the loop closure prediction task. Over 100 test episodes of 2250 steps each, within a large maze (random goal 2), the Nav A3C*+D 1 L agent demonstrated very successful loop detection, reaching an F-1 score of 0.83. A sample trajectory can be seen in Figure 4 (right). Evaluation of a single best performing agent is done through analysis on 100 test episodes. Goals gives the number of episodes where the goal was reached one more more times. Position Accuracy is the classification accuracy of the position decoder. Latency 1:>1 is the average time to the first goal acquisition vs. the average time to all subsequent goal acquisitions. Score is the mean score over the 100 test episodes.

5. ANALYSIS:



5.1. POSITION DECODING:

In order to evaluate the internal representation of location within the agent (either in the hidden units h t of the last LSTM, or, in the case of the FF A3C agent, in the features f t on the last layer of the conv-net), we train a position decoder that takes that representation as input, consisting of a linear classifier with multinomial probability distribution over the discretized maze locations. Small mazes (5 × 10) have 50 locations, large mazes (9 × 15) have 135 locations, and the I-maze has 77 locations. Note that we do not backpropagate the gradients from the position decoder through the rest of the network. The position decoder can only see the representation exposed by the model, not change it.

An example of position decoding by the Nav A3C+D 2 agent is shown in Figure 6 , where the initial uncertainty in position is improved to near perfect position prediction as more observations are acquired by the agent. We observe that position entropy spikes after a respawn, then decreases once the agent acquires certainty about its location. Additionally, videos of the agent's position decoding are linked in Appendix A. In these complex mazes, where localization is important for the purpose of reaching the goal, it seems that position accuracy and final score are correlated, as shown in Table 1 . A pure feed-forward architecture still achieves 64.3% accuracy in a static maze with static goal, suggesting that the encoder memorizes the position in the weights and that this small maze is solvable by all the agents, with sufficient training time. In Random Goal 1, it is Nav A3C+D 2 that achieves the best position decoding performance (85.5% accuracy), whereas the FF A3C and the LSTM A3C architectures are at approximately 50%.

In the I-maze, the opposite branches of the maze are nearly identical, with the exception of very sparse visual cues. We observe that once the goal is first found, the Nav A3C*+D 1 L agent is capable of directly returning to the correct branch in order to achieve the maximal score. However, the linear position decoder for this agent is only 68.5% accurate, whereas it is 87.8% in the plain LSTM A3C agent. We hypothesize that the symmetry of the I-maze will induce a symmetric policy that need not be sensitive to the exact position of the agent (see analysis below). A desired property of navigation agents in our Random Goal tasks is to be able to first find the goal, and reliably return to the goal via an efficient route after subsequent re-spawns. The latency column in Table 1 shows that the Nav A3C+D 2 agents achieve the lowest latency to goal once the goal has been discovered (the first number shows the time in seconds to find the goal the first time, and the second number is the average time for subsequent finds). Figure 5 shows clearly how the agent finds the goal, and directly returns to that goal for the rest of the episode. For Random Goal 2, none of the agents achieve lower latency after initial goal acquisition; this is presumably due to the larger, more challenging environment.

5.2. STACKED LSTM GOAL ANALYSIS:

Figure 7 (a) shows shows the trajectories traversed by an agent for each of the four goal locations.

After an initial exploratory phase to find the goal, the agent consistently returns to the goal location. We visualize the agent's policy by applying tSNE dimension reduction (Maaten & Hinton, 2008) to the cell activations at each step of the agent for each of the four goal locations. Whilst clusters corresponding to each of the four goal locations are clearly distinct in the LSTM A3C agent, there are 2 main clusters in the Nav A3C agent -with trajectories to diagonally opposite arms of the maze represented similarly. Given that the action sequence to opposite arms is equivalent (e.g. straight, turn left twice for top left and bottom right goal locations), this suggests that the Nav A3C policy-dictating LSTM maintains an efficient representation of 2 sub-policies (i.e. rather than 4 independent policies) -with critical information about the currently relevant goal provided by the additional LSTM.

5.3. INVESTIGATING DIFFERENT COMBINATIONS OF AUXILIARY TASKS:

Our results suggest that depth prediction from the policy LSTM yields optimal results. However, several other auxiliary tasks have been concurrently introduced in (Jaderberg et al., 2017) , and thus we provide a comparison of reward prediction against depth prediction. Following that paper, we implemented two additional agent architectures, one performing reward prediction from the convnet using a replay buffer, called Nav A3C*+R, and one combining reward prediction from the convnet and depth prediction from the LSTM (Nav A3C+RD 2 ). (Jaderberg et al., 2017) . We report the AUC (Area under learning curve), averaged over the best 5 hyperparameters.

6. CONCLUSION:

We proposed a deep RL method, augmented with memory and auxiliary learning targets, for training agents to navigate within large and visually rich environments that include frequently changing start and goal locations. Our results and analysis highlight the utility of un/self-supervised auxiliary objectives, namely depth prediction and loop closure, in providing richer training signals that bootstrap learning and enhance data efficiency. Further, we examine the behavior of trained agents, their ability to localise, and their network activity dynamics, in order to analyse their navigational abilities.

Our approach of augmenting deep RL with auxiliary objectives allows end-end learning and may encourage the development of more general navigation strategies. Notably, our work with auxiliary losses is related to (Jaderberg et al., 2017) which independently looks at data efficiency when exploiting auxiliary losses. One difference between the two works is that our auxiliary losses are online (for the current frame) and do not rely on any form of replay. Also the explored losses are very different in nature. Finally our focus is on the navigation domain and understanding if navigation emerges as a bi-product of solving an RL problem, while Jaderberg et al. ( 2017) is concerned with data efficiency for any RL-task.

Whilst our best performing agents are relatively successful at navigation, their abilities would be stretched if larger demands were placed on rapid memory (e.g. in procedurally generated mazes), due to the limited capacity of the stacked LSTM in this regard. It will be important in the future to combine visually complex environments with architectures that make use of external memory (Graves et al., 2016; Weston et al., 2014; Olton et al., 1979) to enhance the navigational abilities of agents. Further, whilst this work has focused on investigating the benefits of auxiliary tasks for developing the ability to navigate through end-to-end deep reinforcement learning, it would be interesting for future work to compare these techniques with SLAM-based approaches. C ADDITIONAL RESULTS

ACKNOWLEDGEMENTS:



C.1 REWARD CLIPPING:

Figure 9 shows additional learning curves. In particular in the left plot we show that the baselines (A3C FF and A3C LSTM) as well as Nav A3C agent without auxiliary losses, perform worse without reward clipping than with reward clipping. It seems that removing reward clipping makes learning unstable absence of auxiliary tasks. For this particular reason we chose to show the baselines with reward clipping in our main results.

C.2 DEPTH PREDICTION AS REGRESSION OR CLASSIFICATION TASKS:

The right subplot of Figure 9 compares having depth as an input versus as a target. Note that using RGBD inputs to the Nav A3C agent performs even worse than predicting depth as a regression task, and in general is worse than predicting depth as a classification task.

C.3 NON-NAVIGATION TASKS IN 3D MAZE ENVIRONMENTS:

We have evaluated the behaviour of the agents introduced in this paper, as well as agents with reward prediction, introduced in (Jaderberg et al., 2017) (Nav A3C*+R) and with a combination of reward prediction from the convnet and depth prediction from the policy LSTM (Nav A3C+RD 2 ), on different 3D maze environments with non-navigation specific tasks. In the first environment, Seek-Avoid Arena, there are apples (yielding 1 point) and lemons (yielding -1 point) disposed in an arena, and the agents needs to pick all the apples before respawning; episodes last 20 seconds.

The second environment, Stairway to Melon, is a thin square corridor; in one direction, there is a lemon followed by a stairway to a melon (10 points, resets the level) and in the other direction are 7 apples and a dead end, with the melon visible but not reachable. The agent spawns between the lemon and the apples with a random orientation. Both environments have been released in DeepMind Lab (Beattie et al., 2016) . These environments do not require navigation skills such as shortest path planning, but a simple reward identification (lemon vs. apple or melon) and persistent exploration. As Figure 10 shows, there is no major difference between auxiliary tasks related to depth prediction or reward prediction. Depth prediction boosts the performance of the agent beyond that of the stacked LSTM architecture, hinting at a more general applicability of depth prediction beyond navigation tasks.

C.4 SENSITIVITY TOWARDS HYPER-PARAMETER SAMPLING:

For each of the experiments in this paper, 64 replicas were run with hyperparameters (learning rate, entropy cost) sampled from the same interval. Figure 11 shows that the Nav architectures with auxiliary tasks achieve higher results for a comparatively larger number of replicas, hinting at the fact that auxiliary tasks make learning more robust to the choice of hyperparameters.

C.5 ASYMPTOTIC PERFORMANCE OF THE AGENTS:

Finally, we compared the asymptotic performance of the agents, both in terms of navigation (final rewards obtained at the end of the episode) and in terms of their representation in the policy LSTM. Rather than visualising the convolutional filters, we quantify the change in representation, with and

:

The image is cropped before being subsampled to lessen the floor and ceiling which have little relevant depth information.

The environments used in this paper are publicly available at https://github.com/deepmind/lab.

An exception is the Nav A3C*+D1L agent on the I-maze (Figure3c), which uses depth regression and reward clipping. While it does worse, we include it because some analysis is based on this agent.

Video of the Nav A3C*+D1L agent on the I-maze: https://youtu.be/PS4iJ7Hk_BU

Video of the Nav A3C*+D1L agent on static maze 1: https://youtu.be/-HsjQoIou_c

Video of the Nav A3C*+D1L agent on static maze 2: https://youtu.be/kH1AvRAYkbI

Video of the Nav A3C*+D1L agent on random goal maze 1: https://youtu.be/5IBT2UADJY0

Video of the Nav A3C*+D1L agent on random goal maze 2: https://youtu.be/e10mXgBG9yo

Figures and Tables:

Figure 2: Different architectures: (a) is a convolutional encoder followed by a feedforward layer and policy (π) and value function outputs; (b) has an LSTM layer; (c) uses additional inputs (agent-relative velocity, reward, and action), as well as a stacked LSTM; and (d) has additional outputs to predict depth and loop closures.

Deep RL has recently been used in the navigation domain.Kulkarni et al. (2016) used a feedforward architecture to learn deep successor representations that enabled behavioral flexibility to reward changes in the MazeBase gridworld, and provided a means to detect bottlenecks in 3D VizDoom.Zhu et al. (2016) used a feedforward siamese actor-critic architecture incorporating a pretrained ResNet to support navigation to a target in a discretised 3D environment.Oh et al. (2016) investigated the performance of a variety of networks with external memory(Weston et al., 2014) on simple navigation tasks in the Minecraft 3D block world environment.Tessler et al. (

Figure 3: Rewards achieved by the agents on 5 different tasks: two static mazes (small and large) with fixed goals, two static mazes with comparable layout but with dynamic goals and the I-maze. Results are averaged over the top 5 random hyperparameters for each agent-task configuration. Star in the label indicates the use of reward clipping. Please see text for more details.

Figure 4: left: Example of depth predictions (pairs of ground truth and predicted depths), sampled every 40 steps. right: Example of loop closure prediction. The agent starts at the gray square and the trajectory is plotted in gray. Blue dots correspond to true positive outputs of the loop closure detector; red cross correspond to false positives and green cross to false negatives. Note the false positives that occur when the agent is actually a few squares away from actual loop closure.

Figure 5: Trajectories of the Nav A3C*+D1L agent in the I-maze (left) and of the Nav A3C+D2 random goal maze 1 (right) over the course of one episode. At the beginning of the episode (gray curve on the map), the agent explores the environment until it finds the goal at some unknown location (red box). During subsequent respawns (blue path), the agent consistently returns to the goal. The value function, plotted for each episode, rises as the agent approaches the goal. Goals are plotted as vertical red lines.

Figure 6: Trajectory of the Nav A3C+D2 agent in the random goal maze 1, overlaid with the position probability predictions predicted by a decoder trained on LSTM hidden activations, taken at 4 steps during an episode. Initial uncertainty gives way to accurate position prediction as the agent navigates.

Figure 7: LSTM cell activations of LSTM A3C and Nav A3C*+D1L agents from the I-Maze collected over multiple episodes and reduced to 2 dimensions using tSNE, then coloured to represent the goal location. Policy-dictating LSTM of Nav A3C agent shown.

Figure 9: Results are averaged over the top 5 random hyperparameters for each agent-task configuration. Star in the label indicates the use of reward clipping. Please see text for more details.

Figure 10: Comparison of agent architectures over non-navigation maze configurations, Seek-Avoid Arena and Stairway to Melon, described in details in(Beattie et al., 2016). Image credits for (c) and (d):(Jaderberg et al., 2017).

Comparison of four agent architectures over five maze configurations, including random and static goals. AUC (Area under learning curve), Score, and % Human are averaged over the best 5 hyperparameters.

Table content: <table><tr><td/><td/><td colspan="3">Mean over top 5 agents</td><td/><td colspan="2">Highest reward agent</td><td/></tr><tr><td>Maze</td><td>Agent</td><td>AUC</td><td>Score</td><td>% Human</td><td>Goals</td><td>Position Acc</td><td>Latency 1:&gt;1</td><td>Score</td></tr><tr><td>I-Maze</td><td>FF A3C*</td><td>75.5</td><td>98</td><td>-</td><td>94/100</td><td>42.2</td><td>9.3s:9.0s</td><td>102</td></tr><tr><td/><td>LSTM A3C*</td><td>112.4</td><td>244</td><td>-</td><td>100/100</td><td>87.8</td><td>15.3s:3.2s</td><td>203</td></tr><tr><td/><td>Nav A3C*+D1L</td><td>169.7</td><td>266</td><td>-</td><td>100/100</td><td>68.5</td><td>10.7s:2.7s</td><td>252</td></tr><tr><td/><td>Nav A3C+D2</td><td>203.5</td><td>268</td><td>-</td><td>100/100</td><td>62.3</td><td>8.8s:2.5s</td><td>269</td></tr><tr><td/><td>Nav A3C+D1D2L</td><td>199.9</td><td>258</td><td>-</td><td>100/100</td><td>61.0</td><td>9.9s:2.5s</td><td>251</td></tr><tr><td>Static 1</td><td>FF A3C*</td><td>41.3</td><td>79</td><td>83</td><td>100/100</td><td>64.3</td><td>8.8s:8.7s</td><td>84</td></tr><tr><td/><td>LSTM A3C*</td><td>44.3</td><td>98</td><td>103</td><td>100/100</td><td>88.6</td><td>6.1s:5.9s</td><td>110</td></tr><tr><td/><td>Nav A3C+D2</td><td>104.3</td><td>119</td><td>125</td><td>100/100</td><td>95.4</td><td>5.9s:5.4s</td><td>122</td></tr><tr><td/><td>Nav A3C+D1D2L</td><td>102.3</td><td>116</td><td>122</td><td>100/100</td><td>94.5</td><td>5.9s:5.4s</td><td>123</td></tr><tr><td>Static 2</td><td>FF A3C*</td><td>35.8</td><td>81</td><td>47</td><td>100/100</td><td>55.6</td><td>24.2s:22.9s</td><td>111</td></tr><tr><td/><td>LSTM A3C*</td><td>46.0</td><td>153</td><td>91</td><td>100/100</td><td>80.4</td><td>15.5s:14.9s</td><td>155</td></tr><tr><td/><td>Nav A3C+D2</td><td>157.6</td><td>200</td><td>116</td><td>100/100</td><td>94.0</td><td>10.9s:11.0s</td><td>202</td></tr><tr><td/><td>Nav A3C+D1D2L</td><td>156.1</td><td>192</td><td>112</td><td>100/100</td><td>92.6</td><td>11.1s:12.0s</td><td>192</td></tr><tr><td>Random Goal 1</td><td>FF A3C*</td><td>37.5</td><td>61</td><td>57.5</td><td>88/100</td><td>51.8</td><td>11.0:9.9s</td><td>64</td></tr><tr><td/><td>LSTM A3C*</td><td>46.6</td><td>65</td><td>61.3</td><td>85/100</td><td>51.1</td><td>11.1s:9.2s</td><td>66</td></tr><tr><td/><td>Nav A3C+D2</td><td>71.1</td><td>96</td><td>91</td><td>100/100</td><td>85.5</td><td>14.0s:7.1s</td><td>91</td></tr><tr><td/><td>Nav A3C+D1D2L</td><td>64.2</td><td>81</td><td>76</td><td>81/100</td><td>83.7</td><td>11.5s:7.2s</td><td>74.6</td></tr><tr><td>Random Goal 2</td><td>FF A3C*</td><td>50.0</td><td>69</td><td>40.1</td><td>93/100</td><td>30.0</td><td>27.3s:28.2s</td><td>77</td></tr><tr><td/><td>LSTM A3C*</td><td>37.5</td><td>57</td><td>32.6</td><td>74/100</td><td>33.4</td><td>21.5s:29.7s</td><td>51.3</td></tr><tr><td/><td>Nav A3C*+D1L</td><td>62.5</td><td>90</td><td>52.3</td><td>90/100</td><td>51.0</td><td>17.9s:18.4s</td><td>106</td></tr><tr><td/><td>Nav A3C+D2</td><td>82.1</td><td>103</td><td>59</td><td>79/100</td><td>72.4</td><td>15.4s:15.0s</td><td>109</td></tr><tr><td/><td>Nav A3C+D1D2L</td><td>78.5</td><td>91</td><td>53</td><td>74/100</td><td>81.5</td><td>15.9s:16.0s</td><td>102</td></tr></table>

Table2suggests that reward prediction (Nav A3C*+R) improves upon the plain stacked LSTM architecture (Nav A3C*) but not as much as depth prediction from the policy LSTM (Nav A3C+D 2 ). Combining reward prediction and depth prediction (Nav A3C+RD 2 ) yields comparable results to depth prediction alone (Nav A3C+D 2 ); normalised average AUC values are respectively 0.995 vs. 0.981. Future work will explore other auxiliary tasks.

Table content: <table/>

Comparison of five navigation agent architectures over five maze configurations with random and static goals, including agents performing reward prediction Nav A3C*+R and Nav A3C+RD2, where reward prediction is implemented following

Table content: <table><tr><td>Navigation agent architecture</td></tr></table>

Peer review (iclr2017):
I do like the demonstration that including learning of auxiliary tasks does not interfere with the RL tasks but even helps. This is also not so surprising with deep networks. The deep structure of the model allows the model to learn first a good representation of the world on which it can base its solutions for specific goals. While even early representations do of course depend on the task performance itself, it is clear that there are common first stages in sensory representations like the need for edge detection etc. Thus, training by additional tasks will at least increase the effective training size. It is of course unclear how to adjust for this to make a fair comparison, but the paper could have included some more insights such as the change in representation with and without auxiliary training. I still strongly disagree with the implied definition of supervised or even self-supervised learning. The definition of unsupervised is learning without external labels. It does not matter if this comes from a human or for example from an expensive machine that is used to train a network so that a task can be solved later without this expensive machine. I would call EM a self-supervised method where labels are predicted from the model itself and used to bootstrap parameter learning. In this case you are using externally supplied labels, which is clearly a supervised learning task!