Map Induction: Compositional spatial submap learning for efficient exploration in novel environments

1 INTRODUCTION . Humans efficiently use spatial reasoning to navigate through and forage in new environments . We easily find our way around new buildings , and quickly ( without exhaustive search ) infer promising foraging locations . Which neurally-informed computational cognitive mechanisms make this possible , and could they be leveraged for better exploration in AI ? In this work we empirically study human exploration in novel spaces , and propose a formal computational model that predicts human behaviour by leveraging Bayesian inference about the structure of unobserved space ( which we term map induction ) . We also show that map induction can improve performance of a Partially Observable Markov Decision Process ( POMDP ) planner during exploration . We begin by reviewing the literature on human spatial cognition involved in exploration , and parallel developments of exploration algorithms , followed by introducing our computational modelling , and experimental results . Distributed non-metric representations . Human exploration relies on approximate spatial representations ( Wang & Brockmole , 2003 ; Warren et al. , 2017 ; Wiener & Mallot , 2003 ) that support near-optimal planning in daily life ( Bongiorno et al. , 2021 ) while sacrificing global metric accuracy ( Vuong et al. , 2019 ; Newcombe et al. , 1999 ; Zhu & Levinson , 2015 ) . These representations are acquired by combining redundant observations of local landmarks , such as views from multiple perspectives ( Gillner & Mallot , 1998 ; Foo et al. , 2005 ) . The use of distributed spatial representations for cost-effective navigation also has a long history in AI . Examples include a computational theory of wayfinding based on multiple locally unique but globally similar landmarks ( Prescott , 1996 ) , which prioritized error recovery over accuracy . The Spatial Semantic Hierarchy model ( Kuipers & Levitt , 1988 ) used distributed spatial graphs as maps for navigation ( Kuipers , 2000 ; Kuipers et al. , 2004 ) . Simultaneous Localization and Mapping ( SLAM ) algorithms circumvent scale limitations by dividing the world into manageable locally metric submaps with topological relationships between submaps ( Bosse et al. , 2003 ; Fairfield et al. , 2010 ) . By contrast , recent recurrent neural network models of navigation have focused more closely on the metric structure of explored spaces and global map formation , showing how networks trained on path integration can take shortcuts ( Banino et al. , 2018 ) . Hierarchical organization . Humans minimize planning costs by representing spaces hierarchically – based on visible spatial boundaries ( Kosslyn et al. , 1974 ) , geography ( Stevens & Coupe , 1978 ) , or otherwise interpreting spaces as composed of sub-regions ( Hirtle & Jonides , 1985 ) . Hierarchical spatial organisation is evident in a tendency to plan paths between regions , before planning subpaths within each region ( Bailenson et al. , 2000 ; Newcombe et al. , 1999 ; Wiener & Mallot , 2003 ; Wang & Brockmole , 2003 ; Balaguer et al. , 2016 ) , and in increased reaction times when switching between hierarchy levels during plan execution ( Balaguer et al. , 2016 ; Kryven et al. , 2021 ) . Hierarchical state-spaces in non-spatial planning domains include drawing ( Tian et al. , 2020 ) inverse inference ( Shu et al. , 2020 ) , and reasoning about topology ( Tomov et al. , 2020 ) . Hierarchical representations in AI are often expressed as options – a hierarchical grouping of actions frequently performed together ( Sutton et al. , 1999 ) . Assuming that a novel Markov Decision Processes ( MDP ) is sampled from a family of similar MDPs , its rewards can be learned as hierarchically derived from a parent distribution shared by the MDP family ( Wilson et al. , 2012 ) . Singh et al . ( 2012 ) proposed a computational framework for learning efficient state-space representations by recognizing which sequences of actions lead to identical observations – for example , grouping together paths that lead to observing the same landmark . A principle of grouping game-board states based on rotation and reflection symmetries has been used to optimize state-space representations in the game of Go ( Silver et al. , 2017 ) . Shared reference frames . From young children to hunter-gatherers , humans spontaneously organize sensory precepts into patterns ( Pitt et al. , 2021 ) , and use them to generalize between tasks ( Tian et al. , 2020 ; Lake & Piantadosi , 2020 ; Schulz et al. , 2018 ) . Generalization and transfer learning in spatial domains include phenomena such as mirror-invariant neural scene representations ( Dilks et al. , 2011 ) , and reusing of reference frames across similar environments ( Marchette et al. , 2014 ) . Shared reference frames may occur in other mammals as well , as suggested by evidence of rodents reusing grid-cell maps between similar parts of environments ( Derdikman et al. , 2009 ; Carpenter et al. , 2015 ) . However , these studies have neither studied the process of discovering structural patterns during spatial exploration , nor considered formal quantitative models of how structured map knowledge may be built and reused . Map induction . We propose a map induction hypothesis – that humans optimize exploration of new spaces by representing maps as composed of reusable reference frames – which can be modeled by using program induction to represent unseen space as composed of regions encountered in the observed areas . Program induction generally refers to inferring a program that likely produced a given sequence of observations ( Stuhlmuller et al. , 2010 ; Lake et al. , 2015 ) . For example , given a sequence of numbers , the inferred program may be an arithmetic progression ; given a natural texture the program may be a reaction-diffusion equation ( Camazine et al. , 2020 ) . Although Bayesian models of concept-learning by probabilistic program induction have been developed for many domains ( Stuhlmuller et al. , 2010 ; Lake et al. , 2015 ; Tian et al. , 2020 ) , the use of program induction to understand spatial structures and environments is unexplored . In this work map induction refers to inferring program ( s ) that could have generated environment , given past observations . Given one of such programs , an agent can anticipate the structure of an unseen environment to optimize exploration . For example , we can plan a foraging trip by anticipating that a landscape of hills will continue in a previously observed pattern , and that the distribution of berries in the unseen space will be similar to the distribution observed so far ( see Figure 1 ) . In this work we adopt a scientific and engineering goal : ( 1 ) to empirically study how humans learn spatial representations , and ( 2 ) describe a computational model that formalizes human-like map induction and can potentially be leveraged to optimize exploration in AI . We show that a Partially Observed Markov Decision Process ( POMDP ) planner augmented with a Hierarchical Bayesian map induction predicts human exploration better than a naive , non-inductive , POMDP ( Experiment 1 ) . We also show that human exploration relies on probabilistic distributions over likely mapsgenerating programs , as opposed to using only the most likely map ( Experiment 2 ) . In the next section , we introduce a novel Map Induction Task , which we use to empirically study exploration . We then give a detailed description of computational models in Section 3 . In Section 4 , we describe two human experiments , and compare behavioral metrics with our models ’ predictions . 2 MAP INDUCTION TASK . The Map Induction Task ( MIT ) is designed to present subjects with novel environments and provide a context for learning novel spatial representations . In this work we are especially interested in environments including repeating regions , such that certain parts of the environment can be predicted from incomplete observations . For example , upon entering a unit in an apartment building a human explorer will likely categorize it as a new region , but after exploring several units , one should be able to anticipate the remaining floor-plan , or even consider several possibilities consistent with prior observations – such as , that the right and the left wing of the building could mirror each other , or two types of units could occur in alteration , etc . In nature predictable structures occur in patterns of hills and valleys , branching riverbeds , and plants favoring certain features of the environment ( see Figure 1 ) , which in theory can be probabilistic , rather than exactly structured repetitions . The subjects ’ task is to forage for rewards in novel partially observable 3-dimensional environments , which contain multiple rewards . To keep experiment duration under 45 minutes , in the current version of the MIT task each environment consisted of repeated units – structural blocks repeated throughout the environment with rewards ( diamonds ) hidden in specific parts of each unit ( see Figure 2B ) . However , in theory the environment structure can also be a probabilistic mixture of predictable and unpredictable layouts . The subjects have no advance knowledge of environment structure and are instructed to collect all diamonds ( see Figure 2 C ) . Subjects use a keyboard and mouse ( or a trackpad ) to navigate , which are standard navigation controls in first-person games . The distribution of rewards is initially unknown to the subjects , but can be predicted from the layout of the environment – for example , the rewards may be placed in the same part of a unit each time a given unit occurs . While the MIT task can be completed without map induction , or even without an explicit map representation ( e.g . by following a wall ) such a naive strategy would take longer than selectively exploring only the parts likely to contain diamonds . Each trial continues either until all diamonds are collected or until a timeout is reached , whichever comes first . 3 COMPUTATIONAL MODELS . In this section we formalize the map induction hypothesis in order to quantitatively and computationally test its predictions . We first consider computational steps needed to optimize performance in the MIT task : ( 1 ) inferring that the environment is composed of repeating units ( 2 ) inferring the most likely structure ( s ) of the environment ( 3 ) inferring the distribution of rewards within the environment , and ( 4 ) planning the shortest route that collects all rewards . Here ( 1 ) and ( 2 ) refer to map induction , and ( 3 ) may involve further program induction over the distribution of rewards ( e.g . the rewards could be placed in the top left corner of every unit , in every odd unit , etc . ) The reward-generating program could potentially be arbitrarily complex ; however , in the current work we consider short programs for simplicity . Because of uncertainty in the environment , this compu- tational pipeline is executed in the context of a replanning loop . Finally , step ( 4 ) involves solving a Partially Observed Markov Decision Process ( POMDP ) that uses distribution of hypothetical environments to plan . While it is possible to do ( 4 ) without performing the first three steps , this would entail assuming an uninformed prior that any unobserved cell could equally likely be a wall , an empty space , or a reward – which would result in exploring exhaustively . Formally , we can model this process by a Bayesian generative framework consisting of four computational modules ( see Figure 3A ) corresponding to the four steps described above . Each module can be independent of the others and can express different modeling assumptions . In the remainder of this section , we discuss the function of each module in detail and define three specific combinations of module implementations , to express three distinct computational hypotheses formalizing the principle of map induction in different ways .

Summary: This paper tests a hypothesis of how humans choose the path to explore and maximize reward in unvisited space based on their past experience. The central hypothesis is that human uses program induction to generate prediction of possible spatial map. Based on human behavior in two experiments, the paper further compared different models of map prediction, and demonstrated that humans actually consider the distribution of possible maps instead of only consider the most likely map.