Map Induction: Compositional spatial submap learning for efficient exploration in novel environments

1 INTRODUCTION . Humans efficiently use spatial reasoning to navigate through and forage in new environments . We easily find our way around new buildings , and quickly ( without exhaustive search ) infer promising foraging locations . Which neurally-informed computational cognitive mechanisms make this possible , and could they be leveraged for better exploration in AI ? In this work we empirically study human exploration in novel spaces , and propose a formal computational model that predicts human behaviour by leveraging Bayesian inference about the structure of unobserved space ( which we term map induction ) . We also show that map induction can improve performance of a Partially Observable Markov Decision Process ( POMDP ) planner during exploration . We begin by reviewing the literature on human spatial cognition involved in exploration , and parallel developments of exploration algorithms , followed by introducing our computational modelling , and experimental results . Distributed non-metric representations . Human exploration relies on approximate spatial representations ( Wang & Brockmole , 2003 ; Warren et al. , 2017 ; Wiener & Mallot , 2003 ) that support near-optimal planning in daily life ( Bongiorno et al. , 2021 ) while sacrificing global metric accuracy ( Vuong et al. , 2019 ; Newcombe et al. , 1999 ; Zhu & Levinson , 2015 ) . These representations are acquired by combining redundant observations of local landmarks , such as views from multiple perspectives ( Gillner & Mallot , 1998 ; Foo et al. , 2005 ) . The use of distributed spatial representations for cost-effective navigation also has a long history in AI . Examples include a computational theory of wayfinding based on multiple locally unique but globally similar landmarks ( Prescott , 1996 ) , which prioritized error recovery over accuracy . The Spatial Semantic Hierarchy model ( Kuipers & Levitt , 1988 ) used distributed spatial graphs as maps for navigation ( Kuipers , 2000 ; Kuipers et al. , 2004 ) . Simultaneous Localization and Mapping ( SLAM ) algorithms circumvent scale limitations by dividing the world into manageable locally metric submaps with topological relationships between submaps ( Bosse et al. , 2003 ; Fairfield et al. , 2010 ) . By contrast , recent recurrent neural network models of navigation have focused more closely on the metric structure of explored spaces and global map formation , showing how networks trained on path integration can take shortcuts ( Banino et al. , 2018 ) . Hierarchical organization . Humans minimize planning costs by representing spaces hierarchically – based on visible spatial boundaries ( Kosslyn et al. , 1974 ) , geography ( Stevens & Coupe , 1978 ) , or otherwise interpreting spaces as composed of sub-regions ( Hirtle & Jonides , 1985 ) . Hierarchical spatial organisation is evident in a tendency to plan paths between regions , before planning subpaths within each region ( Bailenson et al. , 2000 ; Newcombe et al. , 1999 ; Wiener & Mallot , 2003 ; Wang & Brockmole , 2003 ; Balaguer et al. , 2016 ) , and in increased reaction times when switching between hierarchy levels during plan execution ( Balaguer et al. , 2016 ; Kryven et al. , 2021 ) . Hierarchical state-spaces in non-spatial planning domains include drawing ( Tian et al. , 2020 ) inverse inference ( Shu et al. , 2020 ) , and reasoning about topology ( Tomov et al. , 2020 ) . Hierarchical representations in AI are often expressed as options – a hierarchical grouping of actions frequently performed together ( Sutton et al. , 1999 ) . Assuming that a novel Markov Decision Processes ( MDP ) is sampled from a family of similar MDPs , its rewards can be learned as hierarchically derived from a parent distribution shared by the MDP family ( Wilson et al. , 2012 ) . Singh et al . ( 2012 ) proposed a computational framework for learning efficient state-space representations by recognizing which sequences of actions lead to identical observations – for example , grouping together paths that lead to observing the same landmark . A principle of grouping game-board states based on rotation and reflection symmetries has been used to optimize state-space representations in the game of Go ( Silver et al. , 2017 ) . Shared reference frames . From young children to hunter-gatherers , humans spontaneously organize sensory precepts into patterns ( Pitt et al. , 2021 ) , and use them to generalize between tasks ( Tian et al. , 2020 ; Lake & Piantadosi , 2020 ; Schulz et al. , 2018 ) . Generalization and transfer learning in spatial domains include phenomena such as mirror-invariant neural scene representations ( Dilks et al. , 2011 ) , and reusing of reference frames across similar environments ( Marchette et al. , 2014 ) . Shared reference frames may occur in other mammals as well , as suggested by evidence of rodents reusing grid-cell maps between similar parts of environments ( Derdikman et al. , 2009 ; Carpenter et al. , 2015 ) . However , these studies have neither studied the process of discovering structural patterns during spatial exploration , nor considered formal quantitative models of how structured map knowledge may be built and reused . Map induction . We propose a map induction hypothesis – that humans optimize exploration of new spaces by representing maps as composed of reusable reference frames – which can be modeled by using program induction to represent unseen space as composed of regions encountered in the observed areas . Program induction generally refers to inferring a program that likely produced a given sequence of observations ( Stuhlmuller et al. , 2010 ; Lake et al. , 2015 ) . For example , given a sequence of numbers , the inferred program may be an arithmetic progression ; given a natural texture the program may be a reaction-diffusion equation ( Camazine et al. , 2020 ) . Although Bayesian models of concept-learning by probabilistic program induction have been developed for many domains ( Stuhlmuller et al. , 2010 ; Lake et al. , 2015 ; Tian et al. , 2020 ) , the use of program induction to understand spatial structures and environments is unexplored . In this work map induction refers to inferring program ( s ) that could have generated environment , given past observations . Given one of such programs , an agent can anticipate the structure of an unseen environment to optimize exploration . For example , we can plan a foraging trip by anticipating that a landscape of hills will continue in a previously observed pattern , and that the distribution of berries in the unseen space will be similar to the distribution observed so far ( see Figure 1 ) . In this work we adopt a scientific and engineering goal : ( 1 ) to empirically study how humans learn spatial representations , and ( 2 ) describe a computational model that formalizes human-like map induction and can potentially be leveraged to optimize exploration in AI . We show that a Partially Observed Markov Decision Process ( POMDP ) planner augmented with a Hierarchical Bayesian map induction predicts human exploration better than a naive , non-inductive , POMDP ( Experiment 1 ) . We also show that human exploration relies on probabilistic distributions over likely mapsgenerating programs , as opposed to using only the most likely map ( Experiment 2 ) . In the next section , we introduce a novel Map Induction Task , which we use to empirically study exploration . We then give a detailed description of computational models in Section 3 . In Section 4 , we describe two human experiments , and compare behavioral metrics with our models ’ predictions . 2 MAP INDUCTION TASK . The Map Induction Task ( MIT ) is designed to present subjects with novel environments and provide a context for learning novel spatial representations . In this work we are especially interested in environments including repeating regions , such that certain parts of the environment can be predicted from incomplete observations . For example , upon entering a unit in an apartment building a human explorer will likely categorize it as a new region , but after exploring several units , one should be able to anticipate the remaining floor-plan , or even consider several possibilities consistent with prior observations – such as , that the right and the left wing of the building could mirror each other , or two types of units could occur in alteration , etc . In nature predictable structures occur in patterns of hills and valleys , branching riverbeds , and plants favoring certain features of the environment ( see Figure 1 ) , which in theory can be probabilistic , rather than exactly structured repetitions . The subjects ’ task is to forage for rewards in novel partially observable 3-dimensional environments , which contain multiple rewards . To keep experiment duration under 45 minutes , in the current version of the MIT task each environment consisted of repeated units – structural blocks repeated throughout the environment with rewards ( diamonds ) hidden in specific parts of each unit ( see Figure 2B ) . However , in theory the environment structure can also be a probabilistic mixture of predictable and unpredictable layouts . The subjects have no advance knowledge of environment structure and are instructed to collect all diamonds ( see Figure 2 C ) . Subjects use a keyboard and mouse ( or a trackpad ) to navigate , which are standard navigation controls in first-person games . The distribution of rewards is initially unknown to the subjects , but can be predicted from the layout of the environment – for example , the rewards may be placed in the same part of a unit each time a given unit occurs . While the MIT task can be completed without map induction , or even without an explicit map representation ( e.g . by following a wall ) such a naive strategy would take longer than selectively exploring only the parts likely to contain diamonds . Each trial continues either until all diamonds are collected or until a timeout is reached , whichever comes first . 3 COMPUTATIONAL MODELS . In this section we formalize the map induction hypothesis in order to quantitatively and computationally test its predictions . We first consider computational steps needed to optimize performance in the MIT task : ( 1 ) inferring that the environment is composed of repeating units ( 2 ) inferring the most likely structure ( s ) of the environment ( 3 ) inferring the distribution of rewards within the environment , and ( 4 ) planning the shortest route that collects all rewards . Here ( 1 ) and ( 2 ) refer to map induction , and ( 3 ) may involve further program induction over the distribution of rewards ( e.g . the rewards could be placed in the top left corner of every unit , in every odd unit , etc . ) The reward-generating program could potentially be arbitrarily complex ; however , in the current work we consider short programs for simplicity . Because of uncertainty in the environment , this compu- tational pipeline is executed in the context of a replanning loop . Finally , step ( 4 ) involves solving a Partially Observed Markov Decision Process ( POMDP ) that uses distribution of hypothetical environments to plan . While it is possible to do ( 4 ) without performing the first three steps , this would entail assuming an uninformed prior that any unobserved cell could equally likely be a wall , an empty space , or a reward – which would result in exploring exhaustively . Formally , we can model this process by a Bayesian generative framework consisting of four computational modules ( see Figure 3A ) corresponding to the four steps described above . Each module can be independent of the others and can express different modeling assumptions . In the remainder of this section , we discuss the function of each module in detail and define three specific combinations of module implementations , to express three distinct computational hypotheses formalizing the principle of map induction in different ways .

Summary: This work is motivated by the hypothesis that humans maintain a hierarchical spatial representation when exploring new environments, such that shared patterns (due to the hierarchy) between spatial regions can be used to predict how still unvisited places would look. A simple discrete 2D computational model based on probabilistic program induction is proposed for predicting the map at yet unseen places. The model assumes a discrete set of possible transformations (flips, rotations, concatenations, etc.) of small extracted regions (submaps of previous observations), based on which a distribution over possible completions of empty map regions can be formulated.  A control task is then considered, in which an agent needs to explore a completely new environment and collect rewards (tokens) in the process. Under strong structural assumptions for the test environments (e.g. it is ensured that patterns actually repeat in a very uniform way, rooms look the same, rewards are placed consistently, etc.), it is shown that model-based planning under the map-induction model results in more efficient exploration & reward-collection behavior. Furthermore, a study with actual human subjects is conducted where they need to solve the same task. Results suggest that real-life exploration behavior is consistent with the most expressive version of the proposed computational model, where a full distribution over possible map completions is maintained (as opposed to a MAP estimate or an uninformed model).