Model-based Navigation in Environments with Novel Layouts Using Abstract $2$-D Maps

Efficiently training agents with planning capabilities has long been one of the major challenges in decision-making . In this work , we focus on zero-shot navigation ability on a given abstract 2-D occupancy map , like human navigation by reading a paper map , by treating it as an image . To learn this ability , we need to efficiently train an agent on environments with a small proportion of training maps and share knowledge effectively across the environments . We hypothesize that model-based navigation can better adapt agent ’ s behaviors to a task , since it disentangles the variations in map layout and goal location and enables longer-term planning ability on novel locations compared to reactive policies . We propose to learn a hypermodel that can understand patterns from a limited number of abstract maps and goal locations , to maximize alignment between the hypermodel predictions and real trajectories to extract information from multi-task off-policy experiences , and to construct denser feedback for planners by n-step goal relabelling . We train our approach on DeepMind Lab environments with layouts from different maps , and demonstrate superior performance on zero-shot transfer to novel maps and goals . 1 INTRODUCTION . If we provide a rough solution of a problem to an agent , can the agent learn to follow the solution effectively ? In this paper , we study this question within the context of maze navigation , where an agent is situated within a maze whose layout has never been seen before , and the agent is expected to navigate to a goal without first training on or even exploring this novel maze . This task may appear impossible without further guidance , but we will provide the agent with additional information : an abstract 2-D occupancy map illustrating the rough layout of the environment , as well as indicators of its start and goal locations ( “ task context ” in Figure 1 ) . This is akin to a tourist attempting to find a landmark in a new city : without any further help , this would be very challenging ; but when equipped with a 2-D map with a “ you are here ” symbol and an indicator of the landmark , the tourist can easily plan a path to reach the landmark without needing to explore or train excessively . Navigation is a fundamental capability of all embodied agents , both artificial and natural , and therefore has been studied under many settings . In our case , we are most concerned with zero-shot navigation in novel environments , where the agent can not perform further training or even exploration of the new environment ; all that is needed to accomplish the task is technically provided by the abstract 2-D map . This differs from the vast set of approaches based on simultaneous localization and mapping ( SLAM ) typically used in robot navigation ( Thrun et al. , 2005 ) , where the agent can explore and build an accurate occupancy map of the environment prior to navigation . Recently , navigation approaches based on deep reinforcement learning ( RL ) approaches have also emerged , although they often require extensive training in the same environment ( Mirowski et al. , 2017 ; 2018 ) . Some deep RL approaches are even capable of navigating novel environments with new layouts without further training ; however , these approaches typically learn the strategy of efficiently exploring the new environment to understand the layout and find the goal , then exploiting that knowledge for the remainder of the episode to repeatedly reach that goal quickly ( Jaderberg et al. , 2017 ) . In contrast , since the solution is essentially provided to the agent via the abstract 2-D map , we require a more stringent version of zero-shot navigation , where it should not explore the new environment ; instead , we expect the agent to produce a near-optimal path in its first ( and only ) approach to the goal . Although the solution is technically accessible via the abstract 2-D map , some challenges remain to use it effectively . First , although we assume that the layout in the 2-D map is accurate , the map does not correspond to the state space of the agent in the environment , so the agent must learn the correspondence between its state and locations in the 2-D map . Second , actions in the 2-D map also can not be directly mapped into the agent ’ s action space ; moving betweend adjacent “ cells ” in the 2-D map requires a sequence of many actions , specified in terms of the agent ’ s translational and rotational velocities . Hence , one can not simply perform graph search on the 2-D map , then execute the abstract solution directly on the agent . Instead , we propose approaches that learn to use the provided abstract 2-D map via end-to-end learning . Concretely , we propose two approaches for navigation using abstract 2-D maps : • MMN ( Map-conditioned Multi-task Navigator ) : A model-based approach that learns a hypermodel ( Ha et al. , 2016 ) , which uses the provided 2-D map to produce a parameterized latent-space transition function fφ for that map . This transition function fφ is jointly trained with Monte-Carlo tree search ( MCTS ) to plan ( in latent space ) to reach the specified goal Schrittwieser et al . ( 2019 ) . • MAH ( Map-conditioned Ape-X HER DQN ) : A model-free approach based on Ape-X Deep QNetworks ( DQN ) ( Horgan et al. , 2018 ) , a high-performing distributed variant of DQN , that takes in the provided 2-D map as additional input . Furthermore , we supplement it with our proposed n-step modification of hindsight experience replay ( HER ) ( Andrychowicz et al. , 2017 ) . In experiments performed in DeepMind Lab ( Beattie et al. , 2016 ) , a 3-D maze simulation environments shown in Figure 1 , we show that both approaches achieve effective zero-shot navigation in novel environment layouts , and the model-based MMN is better at long-distance navigation . 2 BACKGROUND . We consider a distribution of navigation tasks ρ ( T ) . Each task is different in two aspects : map layout and goal location . ( 1 ) Abstract map . The layout of each navigation task is specified by an abstract map . Specifically , an abstract map m ∈ RN×N is a 2-D occupancy grid , where cell with 1s ( black ) indicate walls and 0s ( white ) indicate nagivable spaces . A cell does not correspond to the agent ’ s world , so the agent needs to learn to localize itself given an abstract 2-D map . We generate a set of maps and guarantee that any valid positions are reachable , i.e. , there is only one connected component in a map . ( 2 ) Goal position . Given a map , we can then specify a pair of start and goal position . Both start and goal are represented as a “ one-hot ” occupancy grid g ∈ R2×N×N provided to the agent . For simplicity , we use g to refer to both start and goal , and we denote the provided map and start-goal positions c = ( m , g ) as the task context . We formulate each navigation task as a goal-reaching Markov decision process ( MDP ) , consisting of a tuple 〈S , A , P , RG , ρ0 , γ〉 , where S is the state space , A is the action space , P is the transition function P : S ×A → ∆ ( S ) , ρ0 = ρ ( s0 ) is the initial state distribution , and γ ∈ ( 0 , 1 ] is the discount factor . We assume transitions are deterministic . For each task , the objective is to reach a subset of state space SG ⊂ S indicated by a reward function RG : S × A → R. We denote a task as T = 〈P , RG , ρ0〉 , since a map and goal specify the dynamics and reward function of a MDP , respectively . In the episodic goal-reaching setting , the objective is typically not discounted ( γ = 1 ) and the reward is −1 for all non-goal states , i.e. , RG ( s , a ) = −I [ s 6= g ] , g ∈ SG . 3 MAP-CONDITIONED PLANNING GIVEN ABSTRACT 2-D MAPS . To build a map-based navigation agent efficient in both training and transfer , there are several technical challenges . ( 1 ) A local change in map may introduce entirely different task structure , so we need the model and planner to adapt to the task context in a different way than conditioning on state , and not directly condition on the entire task context . ( 2 ) During training , we can only rely on a very small proportion of training tasks ( e.g. , 20 of 13×13 maps ) . This requires the agent to be efficient in terms of understanding each task ’ s structure , i.e. , the layout of training maps and goal locations . ( 3 ) Since reward is sparse and model learning and exploration are done simultaneously , we need to fully utilize the knowledge in the environment to train the model and planner , such as transition tuples and failure experiences . Corresponding to the challenges , we introduce the task-conditioned hypermodel first , followed by the forward pass of the planning computation by using the hypermodel in inference . We then detail the backward pass on the training target and optimization process . 3.1 TASK-CONDITIONED HYPERMODEL We aim to build a model adaptive to given abstract 2-D maps for the navigation planner . In a single-task training schema , a naive solution is to separately learn a parameterized transition function fi ( s , a ) for different maps . However , we want to share knowledge between tasks in navigation domain , in which maps have some common computational patterns . For example , in Figure 2 , moving right on center of the box on the left map shares some computation with the right one . This also applies to the larger scale of map area and also the reward prediction . When the agent is able to capture this type of computational pattern , it can better predict what will happen when transferring to a new task . We propose to build a meta network hψ , or hypermodel , to learn the “ computation ” of the transition model fψ si- multaneously for all maps with abstract 2-D maps as input . The transition model for task T ( mapgoal pair ) is a function fi that maps current ( latent ) state and action to a next ( latent ) state . The set { fi } represents transition functions of all tasks belonging to a navigation schema ( e.g. , a certain size of map ) , and these tasks have similar structure . We parameterize a transition function fi as a neural network with its parameter vector φi . We assume the set of transition networks have similar structure that can be characterized by a set of context variables c = ( m , g ) , i.e. , the abstract 2-D map and goal.1 This implies that parameter vectors φi live in a low-dimensional manifold . Thus , we define a mapping h : C → Φ that maps the context of a task to the parameter vector φi of its transition function fi . We parameterize h also as a network with parameter ψ:2 hψ : c 7→ φ , fφ : s , a 7→ s′ . ( 1 ) This can be viewed as soft weight sharing between multiple tasks . It efficiently maps lowdimensional structure in the MDP , specified by the map , to computation of the transition model . It may also be viewed as a learned “ dot-product ” between task context cT and state and action st , at to predict the next state . The idea of predicting the weights of a main network using another meta-network is also known as HyperNetworks ( Ha et al. , 2016 ; von Oswald et al. , 2019 ) . 1Concretely , a task context c ∈ R4×N×N has four components : downsampled global occupancy map , cropped local occupancy map , and ” one-hot ” goal and start occupancy maps , where N is downsampled size . 2We only predict weights of the transition model fφ : S × A → S which operates on a latent state space . The mapping from environment observations to latent states e : O → S is not predicted by a meta network . Since the latent space is low-dimensional , it is feasible to predict weight matrices of a transition network for it .

Summary: This submission tackles the Point Goal navigation task given access to the agent’s starting location, current state (position and velocity), the goal location and a top-down map. The submission presents two approaches for tackling this task. First is MMN (Map-conditioned Multi-task Navigator) which is model-based approach which learns to hyper-network to convert map input to a transition function. Second approach is MAH (Map-conditioned Ape-X HER DQN) which is model-free approach using Ape-X DQN with Hindsight experience replay.