Grid Cell-Inspired Fragmentation and Recall for Efficient Map Building
Authors: Jaedong Hwang
Zhang-Wei Hong
Eric R Chen
Akhilan Boopathy
Pulkit Agrawal
Ila R Fiete
Venue: Accepted by TMLR ()

Abstract: Animals and robots navigate through environments by building and refining maps of space. These maps enable functions including navigation back to home, planning, search and foraging. Here, we use observations from neuroscience, specifically the observed fragmentation of grid cell map in compartmentalized spaces, to propose and apply the concept of Fragmentation-and-Recall (FARMap) in the mapping of large spaces. Agents solve the mapping problem by building local maps via a surprisal-based clustering of space, which they use to set subgoals for spatial exploration. Agents build and use a local map to predict their observations; high surprisal leads to a "fragmentation event" that truncates the local map. At these events, the recent local map is placed into long-term memory (LTM) and a different local map is initialized. If observations at a fracture point match observations in one of the stored local maps, that map is recalled (and thus reused) from LTM. The fragmentation points induce a natural online clustering of the larger space, forming a set of intrinsic potential subgoals that are stored in LTM as a topological graph. Agents choose their next subgoal from the set of near and far potential subgoals from within the current local map or LTM, respectively. Thus, local maps guide exploration locally, while LTM promotes global exploration. We demonstrate that FARMap replicates the fragmentation points observed in animal studies. We evaluate FARMap on complex procedurally-generated spatial environments and realistic simulations to demonstrate that this mapping strategy much more rapidly covers the environment (number of agent steps and wall clock time) and is more efficient in active memory usage, without loss of performance.

Published in Transactions on Machine Learning Research (07/2024)
Grid Cell-Inspired Fragmentation and Recall for Efficient Map Building
Jaedong Hwang
jdhwwnggmmit.cmm edu
Massachusetts Institute of Technology
Zhang-Wa Wei Hong
zuhong@hitt. edu
Massachusetts Institute of Technology
Eric Chen
ericro@mit. edu
Massachusetts Institute of Technology
Akhilan Boopathy
aktilan edu
Massachusetls Institute of Technology
Pulkit Agrawal
pulkitag@mit. edu
Massachusetts Institute of Technology
Ila Fiete
fiete edu
Massachusetts Institute Technology
Reviewed on OpenReview: /ttps://openeeee wwtp:/vee..eeeeee.oomoe net/ forum? c7800J6Q67
Abstract
Animals and robots navigate through environments by building and refining maps of space. These maps enable functions including navigation back to home, planning, search and foraging. Here, we use observations from neuroscience, specifically the observed fragmentation of grid cell map in compartmentalized spaces, to propose and apply the concept of Fragmentation- (FARMap) in the mapping of large spaces. Agents solve the mapping problem by building local maps via surprisal-based clustering of space, which they use to set subgoals for spatial exploration. Agents build and use a local map to predict their observations: high surprisal leads to a "fragmentation event" that truncates the local map. At these events, the recent local map is placed into long-term memory (LTM) and a different local map is initialized. If observations a fracture point match observations one of the stored local maps, that map recalled (and thus reused) from L.M. The fragmentation points induce natural online clustering of the larger space, forming set of intrinsic potential subgoals that are stored in LTM as a topological graph. Agents choose their next subgoal from the set of near and far potential subgoals from within the current local map or LTM, respectively. Thus, local maps guide exploration locally, while LTM promotes global exploration. We demonstrate that FARMap replicates the fragmentation points observed in animal studies. We evaluate FARMap on complex procedurally generatte spatial environments and realistic simulations to demonstrate that this mapping strategy much of agent steps and wall clock time) and is more efficient in active memory usage, without loss of performance!
rapidly covers the environment (number
https://jd770. git.0b..tthuuu..... hotiib.iioorooece///Attgg

---

Published in Transactions on Machine Learning Research (07/2024)
1 Introduction
Human episodic memory breaks our continuous experience of the world into episodes or fragments that are divided by event boundaries corresponding to large changes of place, context, affordances, and perceptual
inputs (Baldassano et al. 2017 Ezzyat & Davachi 2011 Newtson & Engquist, 1976; Richmond & Zacks,
2017: Swallow el al. 2009 Zacks Swallow] 2007). The episodic nature of memory a core component. of how we construct models of the world. It has been conjectured that episodic memory makes it easier to perform memory retrieval, and to use the retrieved memorics in chunks that are relevant to the current context. These observations suggest certain locality fragmented nature in how model the world.
Chunking of experience has been shown to play a key role in perception, planning, learning and cognition in humans and animals (De Groot 1946 Egan & Schwartz 1979 Gobet & Simon 1998; Gobet et al. 2001
Simon 1974.. In the hippocampus, place cells appear to chunk spatial information by defining separate maps when there has been a sufficiently large change context or in other non-spatial or spatial variables, through a process called remapping; see Coolgin et al. (2008); Fyhn et al. (2007). Grid and place cells in the hippocampal formation have also been shown to fragment their representations when the external world their own behaviors have changed only gradually rather than discontinuously in the same environment (Carpenter et al.
2015 Derdikman et al. 2009 Low et al. 2021 (Figure [
Inspired by the concept of online fragmentation and recall (remapping to the eristing fragment) proposed for grid cells Klukas et al. (2021), we propose a new framework for map-building, FARMap, schematized in Figure Dp. This model combines three idcas: 1) when faced with a complex world, can be more efficient to build and combine multiple (and implicitly simpler) local models than to build a single global (and implicitly complex) model, 2) boundaries between local models should occur when local model ceases to be predictive, and 3) the local model boundaries define natural subgoals, which can guide more efficient hierarchical exploration.
As an agent explores, it predicts its next observation. Based on a measure of surprisal between its observation and prediction, there can be a fragmentation event, at which point the agent writes the current model into long-term memory (LTM) and initiates a new local model. While exploring the space, the agent consults its LTM, and recalls existing model if it returns to the corresponding space. The agent uses its current. local model to act locally, and its LTM to act more globally. We apply this concept to solve the spatial map building problem.
We first simulate animal studies (Derdikman et al. 2009 Carpenter et al. 2015) using FARMap showing its ability to fragment environments at the same locations as observed in animals. This confirms that FARMap accurately replicates the fragmentation points noted in animal research. We then evaluate the proposed framework on procedurally-geeeratte spatial environments. Experimental results support the effectiveness of the proposed framework; FARMap explores the spatial environment with much less memory and computation time than its baseline by large margins the agent only refers to the local model and uses both memories for setting subgoals.
The contribution of this paper is three-fold:
We propose a new framework for mapping based on Tragmeentaiion-An-RReeee or FARMap, that exploits grid cell-like map fragmentation via surprisal combined with a long-term memory to perform efficient online map building.
We contribute srreedurally-generatte environments for spatial exploration, with parametrically controllable complex shapes that include multiple rooms and pathways.
We demonstrate the efficacy of our framework in spatial map-building tasks. Our experiments show that FARMap reduces wall-clock time and the number of steps (actions) taken to map large spaces, and requires smaller online memory size relative to baselines.
2

---

Published in Transactions on Machine Learning Research (07 (0244)
(a)
(b)
R
Observation
* Fragmentation
AA Long-Term mode fropmene
Environment
Agent
Memory
Figure 1: (a) Firing fields of grid cells in various environments from Derdikman al. (2009) (top) and Carpenter et al. 2015 (bottom). The firing pattern changes at the boundary between two regions (frag- mentation). (b) Overview of our approach. Given an observation from the environment, the FARMap agent decides whether to fragment the space based on how well it can predict the observation. If fragmentation occurs, the current map (or model) fragment is stored in long-term memory (LTM)); the agent then initializes a new map (or model) fragment. Conversely, the current observation closcly matches the observations stored LTM, the agent loads an existing map (or model) fragment from there (recall). Based on the current. fragment, the agent selects an action to explore the environment.
2 Related Work
2.1 Fragmentation of Grid Cell Maps
Mammalian entorhinal grid cells generate highly regular periodic spatial reprosentations that tile open enviromments (Hafting et al. 2005). This periodic response is hypothesized to be a general allothetic spatial coordinate system that represents displacements. The spatial response independent of the speed and direction of movement and is believed to be formed through integration self-velocity estimates. However, the regular periodic firing pattern of grid cells becomes fragmented in more complex spatial layouts, such when an environment contains multiple subdivisions (Carpenter et al.] 2015 Derdikman al. 2009 Fyhn et al. 2007). For instance, there is fracture in the periodic response at sharp turns of narrow corridor and in doorways, where the grid phase appears to be remapped jumps discretely to distinct value. A recent manuscript [Kuuaas et al.) 2021) builds a model to predict when such discrete remapping events might occur even though the agent explores the environment in a continuous trajectory. They formulated map fragmentation as a clustering computation, and showed how online clustering based on observational surprisal results in fragmentations that match the neuroscientific observations grid cells and also match normative clustering algorithms like DBSCAN (Ester et al. 1996). However, that work did not extensively explore the functional utility of grid cell-like map fragmentation. Here, show that surprisal-baset fragmentation, which fits the biological Iragmentation dala, is biologically plausible principle that enables agents to efficiently build maps of various environments online without getting stuck in local loops.
2.2 Grid Cell-Inspired SLAM
Grid cells have received attention in roboties due their potential to produce more robust spatial navigation. Milford et al. (2004) propose model based on continuous attractor dynamics (Samsonovich & McNaughton, 1997h and more recently with grid cells (Ball et al.
2013 Milford et al. 2010), to achieve correct loop
closure during noisy odometry. Similarly, Zhang et al. (2021) employ growing self-rgaaizing maps inspired
by the hippocampus for the same purpose. Yu et. al. (2019) exlend OaSSRALSLAM (Ball el. al. 2013) to 3D environments via conjunctive pose cell model that employs 3D grid cell. These methods focus on the orror-correctin propertics of grid cell dynamics. They do not consider fragmented grid cell maps and the possibility that these map fragments might represent the construction of subgoals which could be used for further spatial exploration.
3

---

Published in Transactions on Machine Learning Research (07 (224))
Short-Term Memory (STM)
0
Local Map
subgoal
Planner
surprisal
Observation ( Ott)
Surprisal distribution
Choose subgnal.
Actions
Fregmentation
Initiaiize a locsl map.
Previous Action [ (4g-1)
Recall stored local map.
Store the current local
LTM subgoal
de
m
Navigation
agent's current Incation
Racal
Maps
Connectivity Graph
5TM subgoal :LMM subgeal
Traymentation
Long-Term Memory (LTM)
fracture points
Figure 2: Illustration of the FARMap framework. Navigation (black arrow): Given the current observation which is egocentric top-down view with restricted field of view and previous action, the agent updates its short-term memory (STM) and selects subgoal from the current local map in STM or the local map connectivity graph stored in LTM. The planner generates sequence of actions for the shortest path to the subgoal. Recall (dashed arrow): If the agent arrives at a fracture point (circle in the map), a corresponding local map is recalled from LTM and the current local map stored in LTM is updated. Fragmentation (gray arrow): If the current surprisal is higher than a threshold, the current local map stored LTM and a new local map initialized. of is a spatially transformed observation with the same size the current local map to update the map.
2.3 Frontier-based SLAM
Active SLAM (simultaneous localization and mapping) agents must efficiently explore spaces to build maps. A standard approach is define the frontier between observed and unobserved regions 2D environment, and then select exploratory goal locations from the of frontier states (Yamauchi) 1997). ). Frontier-based exploration has been extended to 3D environments (Dai et al. 2020 Dornhege & Kleiner 2011) and used building block of more sophisticated exploration strategies (Stachniss et al. [004]). Although conceptually simple, fronticr- based exploration can be quite effective compared to more sophisticated
ceission-thcoetti
exploration (Holz et al.) 2010). A cost of frontier-based exploration is the use of global maps and global frontiers, making the process memory expensive and search intensive. In contrast to frontier-base based exploration, our approach learns the surprising parts an environment as intrinsic subgoals, selecting among those as the exploratory goals.
2.4 Submap-Bass Based SLAM
Submap-Basee SLAM algorithms involve mapping a space by breaking it into local submaps that are connected to one another via a topological graph. Such Submap-Baser STAM methods are usually designed to avoid path integration errors when building maps of large spaces (e.g. Fairfield et al. (2010)) and reduce the computational cost of planning paths between start and target position (Fairfield et al. 2010 Maffei et al. 2013). Maffei et al. (2013) add DP-SLAM (Eliazar & Parr 2003) to SegSLAM to reduce the search space, generating segments periodically at fixed time-intervals. CCoset & Nagatani (2001) generate new landmarks in an environment to build a lopological graph of the landmarks and navigates based on the graph. FARMap is closely related to these methods in that we build multiple submaps. However, FARMap divides space based on propertics of the space (how predictable the space based on the local map or modcl), and does so in an online manner using surprisal. As we show below, this fragmentation strategy can lead improvements in performance compared to random periodic fragmentation.
4

---

Published in Transactions on Machine Learning Research (07/2024)
3 Fragmentaion and Recall based Spatial Mapping (FARMap)
3.1 Motivation and Overview
Animals explore spaces efficiently even in large environments by using grid cells' remapping that divides environment into multiple subregions. This remapping can be modeled as surprisal-based online frag- mentation (Klukas et al. 2021). Here, we propose a fragmeentation-an--reell based spatial map-building strategy (FARMap) inspired by remapping of grid cells. FARMap tackles the problem of SLAM algorithms: the memory cost and search cost, of finding subgoals grow rapidly with environment size; for agents exploring a large space, the computational costs could explode.
While exploring an environment, an agent builds a local model (map) and uses it in short-term memory (STM) to compute a surprisal signal that depends the current observation and the agent's 's local model-based prediction. When the surprisal exceeds some threshold, this corresponds to fragmentation event. At this event, the local model is wrillen to long-term memory (LTM) which builds connectivity graph that relates model fragments to each other so that it can share information across local models without direct access to the stored models in LTM. Then, the agent initializes an entirely new local model. Conversely, if the agent revisits the fracture point, the agent recalls the corresponding model fragment (local model). Hence, the agent can preserve and reuse previously acquired information. Figure 2 shows how an agent decides its next subgoal given the observation and the previous action with fragmentation and recall. LTM (except the connectivity graph portion) can be regarded as external memory, while STM is modeled as working memory. This external memory is accessed or updated only during fragmentation or recall processes. Consequently, this can be beneficial for machines with limited memory access (see Appendix D. We also discuss LTM retrieval overhead in Appendix C
3.2 Overall Procedure of Spatial Navigation
[ presents the overall procedure of FARMap at time I. On top of the Frontier algorithm YYamauchi 1997), we have colored the FARMap algorithm blue. Given the previous action a;-1, current observation Oz, local predictive map M at time 1, we first update the map following Eq. 1 and caleulate the surprisal s, following Eq. 2
Algorithm
If the agent is located at the fracture point where fragmentation happened between the current local map, Mo and another local map stored in LTM (Line 6). we store M and da in LTM, and the stored map fragment is recalled to STM. On the other hand, if the z-scored surprisal 2, calculated using the running mean and standard deviation of surprisal within the current local map is greater than threshold, P (Line 9). we store M."",, and 9e in LTM, and initialize a new map in STM. During this process, the current locations in both Mę and the new map are marked as fracture points (Section 3.4 After checking recall and fragmentation, we find the desirable local map fragments that are less explored than other fragments (Section 3.6). If the current map is not the desirable map, we set the subgoal as the fracture point between the current map and the desirable map. Otherwise, we first find frontier-edges and calculate the weight of each frontier-edge F; using weighted sampling with weight w; following Eq. 3 (w, is 1/d4 in the case of the Frontier model). The subgoal is defined as the nearest frontier from the centroid of the sampled frontier-edge. Finally, a planner generates a sequence of actions to navigate to the subgoal (Section 3.7). Note that while the agent moves based on the sequence, it keeps updating the map and checking fragmentation and recall.
3.3 Local Map
The STM has local predictive spatial map, M [CC+1)xHX where height H and width W grow as the agent extends its observations in the local region by adding newly discovered regions. The first C' channels of M denote color and the last channel denotes the agent's confidence in each spatial cell. In this paper, we will focus only on the update of the confidence channel (the C-th channel). The local predictive map is simply a temporally decaying trace of recent sensory observations like a natural agent (Zhang el al. 2005):
M 10 7 M + (1 7) o'c.
(1)
5

---

Published in Transactions on Machine Learning Research (07/2024)
Algorithm 1 FARMap Procedure at time t. FARMap algorithm is colored in blue top of Frontier
algorithm (Yamauchi 1997).
Require: a spatial map M previous action d;-1, current observation or, short-term memory STM, long-term memory LTM, position at time t, pos,, decay factor fragmentation threhsold and hyperpa- rameter 6.
Ensure: Updated map, M and a sequence of actions {a}
procedure STEP
Sec. 3.3
M 7 M + (1- - of
Update the current local map
Calculate 9, = ct following Eq.
(st Mr)/0z
5:
-Ntroutier Nknown
6:
if pos, = fracture point then
b Recall Store M change M.
7:
LTM + Store(pos,, 9cr M.
Sec. 3.4
STM Recall LTM)
else if 24 p then
Fragmentation
10:
LTM f- Store(pos,, Ges Mşurr)
11:
Initialize a new map M in STM.
12:
end if
Sec. 3.5
< 13:
Update running mean Me+1 and standard deviation 0++1 of surprisal. 4: g = arg max; dicte if # then
14:
D Eq.
15:
> Subgoal based connectivity between fragments. subgoal + the fracture point between the current fragment e and fragment g
16:
17:
else
18:
Find frontier-edges (Fi} and their centroids {centroid,).
Scc. 3.6
19:
d, = pos: centroid;||a ||1-
20:
w; 1/d:- [Fi (F, is not located spatially behind the agent) > 1(-) is if the condition is true else 0.
21:
22:
Select frontier-edge Fg based on the weighted sampling with {w;).
23:
subgoal +- the nearest frontier Fg from its centroid.
24:
end if
Sec. 3.7
25:
sequence of actions, {a} +- Planner(subgodd M.
> Dijkstra's algorithm
26: end procedure
where y is a decay factor and o E RIC-I)xxxx is the egocentric view input observation in the environment at time sized as (h, w). The last channel of the observation indicates visibility caused by occlusion or restricted field of view (FOV); visible (1) or invisible (0) in each cell. The red region visible and others are invisible in Figure of E (C++))x denotes spatially transformed observation to M to update the current observation to the local map in the correct position using rotation and zero-padding, Figure 3 shows a toy illustration ion of how to transform the current observation to update the local map and how the map size grows. We first rotate the observation based on the head direction of the agent in the map and then zero-pad it so that it has the same size as the local map considering the agent's current location in the map. II the observation does not fit. within the map due the agent's 's location, we add zero-padding (gray in the figure) to both the transformed observation and the local map. Then, we update the local map by adding the transformed observation. For example, once the exploration has started, the memory size is h w (H h. W w) and if the agent moves step upward, the size changes to (h + 1) (H = h W w).
3.4 Fragmentation and Recall
Fragmentation Fragmentation occurs if the z-scored current surprisal ((s: - pe)/os) exceeds a threshold, P, where &, denotes surprisal time t, and μt and o, represent running mcan and standard deviation. Initially, for each new map, the agent collects surprisal statistics and is not permitted to further fragment space until the number of samples is greater than 25 (to ensure large enough sample conditions for statistics).

---

Published in Transactions on Machine Learning Research (07 (07/202) /222)
117
h
Agem's location at
dec
Agen's
W
W
Known cell
W
H
H
H
[-1,0
M
[-10
M
Figure 3: Schematic illustrations of how the local map is updated. In this figure, only consider the visibility of each cell ignoring occupancy and color for simplification. (a) We first rotate the current observation o, based on the head dircction of the agent in the local map. Then, the observation is zero-padded to match the same size as the local map. Finally, the local map is updated by adding the transformed observation d, (b) If the current observation does not fit within the local map due to the agent's location, we add zero- pooodpaddnngg (gray) both the obsorvation and the local map. Hence, the size of the local map incrcases (H changes).
We also store the ratio de of the number of frontier cells (Nirontier) to the number of known cells (Nknown) and the distance between each fracture point in the current local map M.",, as further explained in this section. The ratio is used for guiding agents on whether not to move to other local maps. When M is stored in LTM, it is connected with adjacent map fragments that share the same fracture point in the connectivity graph. In other words, the node of the graph is model fragment, and a connection denotes that both fragments share a fracture point. In our implementation, we designed a fracture border to prevent unnecessary overlaps and over-fragmentation. The fracture border extends from a fracture point to the left and right. based on the agent's 's head direction until the border reaches either the frontier or an occupied cell. When the agent crosses this border, the recall process is triggered. This mechanism ensures that the agent does not need be exactly on fracture point to recall existing maps, thus enhancing flexibility and preventing vverffragmentation. The fracture points themselves used subgoals to switch to adjacent map fragments ment ioned in Section 3.6. Note that agent does not need to fragment an environment if it is a single open space, such as a large rectangular or circular arena.
Recall Each local map records the fracture points. At these points, there overlaps with other map fragments. When the agent moves to fracture point in the current local map, the corresponding local map is recalled from LTM and the current one is stored in LTM.
3.5 Surprisal
The surprisal serves as a criterion for fragmentation, which can be any uncertainty estimate of the future, such as negative confidence or future prediction error. We employ the local predictive map for measuring surprisal. The scalar surprisal signal s; = 1 - ot is generated using the local map in STM and the current obscrvation, where 6; quantifies the average similarity of the visible part of the observation the local predictive map M before update:
M M &.
4:
(2)
The agent is assumed to maintain running estimate of the mean 14 and standard deviation oa of past surprisals, stored part of the current map.

---

Published in Transactions on Machine Learning Research (07/ (224))
3.6 Subgoal
Subgoals are decided by using either the current. local map in STM or the connectivity graph in LTM. The former enlarges the current local map while the latter helps find the next local map to explore. An agent explores the local region in the environment unless the current surprisal is too low (e.g., 2-score is smaller than -1) and there is a less explored local map nearby.
Subgoals made with the current local map are based on frontier-basee subgoals (Yamauchi 1997] for exploring the local region. Each cell in the region categorized known and unknown based on whether was previously observed not, and occupied and unoccupied (empty) based on its occupancy. In the current local map, we first. find all frontiers which are unknown cells adjacent to the known unoccupied cells. A group of consecutive frontiers is called a 'frontier-edgee and Yamauchi (1997) uses the nearest centroid of the frontier-edge as a subgoal. Unlike standard SLAM methods that employ the entire map, our map in STM only covers a subregion of the environment. After fragmentation, the region where the agent came from has several frontiers (border of two local modcls) forming a frontier-cdge. It leads the agent to go back to the previous area and recall the correspouding map fragment. This would lead to the agent moving between two map fragments for long time. Therefore, prioritize the frontier-edge that is not located spatially behind the agent. The subgoal is sampled with the following weight w, for each frontier-edge Fy:
Fi 1(F is not located spatially behind the agent)
W0
33)
where de the distance between the current position and the centroid of F, and 1()) is the indicator function that is 1 if the condition true otherwise 0.
Once the agent finishes mapping the local region, it should move to different subregions. However, subgoals from the current local map can misguide the agent to the already explored region since the agent does not have information beyond the map. Hence, we employ the connectivity graph of local maps stored in LTM. We leverage the discovery ratio (the ratio of the number of frontier cells to the number of known cells) q mentioned above to find the most desirable subregions to explore. We also utilize the Manhattan distance between the current agent location and the fracture point between the current (c-th) local map and the connected i-th local map, di.. where de,e and dj,- xs if the j-th local map disconnected to the current. map. Then, the desirable local map is selected as
4i
g arg max
(4)
di,c + é
where e denotes the preference of staying in the current local map; a smaller value encourages staying in the current local map. If is not equal to c, the fracture point between the current local map and the g-th local map is set as the subgoal. Once the agent arrives at the fracture point, the corresponding local map is recalled and the agent recursively checks Eq. 4 until g is the arrived subregion. Note that the distances between fracture points stored in the recalled local map are precomputed since they fixed.
3.7 Planner
The planner takes subgoal and the current spatial map in STM and finds the shortest path within the map from the current agent location to the subgoal. We use Dijkstra's algorithm for planning path to the next subgoal. However, the planner can be any path planning method such as the A algorithm (Hart et al. 1968)
or RRT Vallee 1998).
4 Procedurally-Generatee Environment
We build a procedurally-generaae environment for the map-building experiments. Figure 9 and Algorithm 2 in Appendix show the procedure of map generation. We first generate prid-patterned square rooms and randomly connect and merge them. Then, we flip boundary cells (empty or occupied) multiple times for diversity. Formally, given the length of square S, the interval between square rooms, L, and the size of the grid, (N,M), M), first generate the binary square grid map M E {0(empty); (occupied) (NN LXx
Let
8

---

Published in Transactions on Machine Learning Research (07 (07/2024) 2224)
(a) Observation
(b) Small (size: 3249)
(c) Medium (size: 13689)
(d) Large (size: 23568)
Figure 4: Environments. Empty cells (that can be occupied by the agent) are black; walls are randomly colored. (a) Top-down visualization of the agent's local field of view (FOV) (agent: red triangle; shaded region: observation) within an enviromment (b). The agent has only a locally restricted egocentric view. The right side is occluded by a wall. (b) Top-down view of one environment. The red box marks the region shown in (a). (c), (d) Examples of medium and large environments.
8; be the i-th square in row-major order in M. For each of the adjacent square pairs, we connect two squares with probability Peonnect as a width w unif(1, 1n.f(1,2,..... S 1 or merge (a special case of connecting with width S) them with probability Pmorge- Then, we flip all boundarics between occupied and empty cells K times with probability Pfip- After flipping the boundaries, there are several isolated (i.e., not connected other submaps) submaps in M. We only use the submaps where the sizes are greater than a threshold (352 in our implementation). After creating maps, we randomly colorize each occupied cell and scale them up by factor of 3. Note that the proposed environment has much more complex maps compared to Minigrid (Chevalier- (Chevalie-Boiivert et al. 2018). Please refer to Appendices D and F.1 for more details.
Figure 4 shows examples of enviromments and observation. The walls in the environment are randomly colored and are composed of various narrow and wide pathways. For each trial, the agent is randomly placed before it begins to explore the environment. Figure 4a illustrates an example of the agent's view in the small environment shown in Figure 4b The agent is presented a red triangle and the obscrved cells are shaded. The agent has restricted field of view with occlusion (130%).
5 Experiments
In this section, we conduct experiments for FARMap comparing with its baselines on the proposed procedurally generated map environments and robot simulations. We conduct an ablation sludy, and sensitivity analysis of respectively. To quantify the difficulty of the proposed environments for the RL exploration algorithm, we mcasure the performance of RND (Burda al.) 2019) in the environments in Appendix J
hyperparameters in Appendices
H and [
We measure the map coverage, memory usage, and wall-clock time for each environment at each time step our evaluation critcria and calculate the mean and standard deviation over all runs. The memory usage in each enviromment is calculated as ratio of the local map size (memory size, H x W) to the environment size. Note that the local map size is the asymptotically dominant factor in the memory. We compare FARMap with standard muntier-based exploration (Frontier) (Yamauch) 1997p. Please refer to Appendix F for the experimental settings.
5.1 Comparison with Grid Cell Remapping
We conduct experiments within simulated environments that replicate existing rat studies Derdikman et al, 2009: Carpenter et al.) 2015) to validate that the fracture points generated by FARMap correspond with the actual remapping locations where activation patterns are changed in these experiments. Since there is no established metric for quantifying remapping patterns of grid cells in neuroscience, we qualitatively compare the fracture points with the remapping locations. Figure 3 illustrates that the fracture points align closely with the actual remapping locations of rats' grid cells observed in the experiments. This alignment can be attributed to our agent's 's egocentric view and its limited field of view, similar to that of the rats in the
9

---

Published in Transactions on Machine Learning Research (07/2024)
cell neural activity
FARMap Simulation
grid neural activity
FARMap Simulation
rmmapenng
fuctaarpeent
Viart louazon
(a) Derdikman al. (2009)
Carpenter al. (2015)
Figure 5: Comparison between remapping locations of grid cells in neuroscience experiments (Derdikman ct al. 2009 Carpenter et al. 2015) and fracture points of FARMap in simulation. The red rectangles and emerald circles denote the actual remapping locations and fracture points, and the red triangle is the start location of the simulation. The fracture points are well aligned with the actual remapping locations.
experiment. When the agent passes through narrow pathway (Figure 5 turns corner (Figure 5p), it encounters new obscrvations that were previously occluded. These new observations significantly increasc the surprisal. When the surprisal drops again below threshold from above, it triggers a fragmentation event, as in Klukas et al. (2021). The difference in surprisal before and after these events due to the sudden exposure to new, unpredicted information, which is more pronounced after the agent has turned or moved past the occlusion.
5.2 FARMap in SeoeedurallyGGnnerae Environments
We conduct experiments on 1,500 different environments to show that FARMap can explore new environments without prior training. Note that FARMap is not a learning algorithm; rather, it operates similarly to Frontier and other SLAM methods by efficiently exploring new environments without learning. Figure 0 summarizes the course of exploration
the performance small (size < 5,000), medium (5,000 < size < 15,000), and large (size 2 15,000). The lines in the plots are the average of all experiments or a group of experiments and the shaded areas are standard errors of the mean which are not visible due to a large number of trials. FARMap clearly outperforms the baseline on every step, which means that explores the environment more efficiently. On the other hand, FARMap generally uses a stable amount memory on average (40 %) over all experiments while Frontier requires much more memory as map coverage increases. The average memory usage of FARMap is almost consistent in any group of environments the agent explores environments while the usage of Frontier keeps increasing.
1,500 environments with three groups based on their sizes;
Figure 7 and Table D analyze memory size and wall-clock-time changes depending the environment size. 'Random Exploration' denotes an agent moving randomly at every step. Although its runtime is fast, it cannot explore as many areas when the environment becomes larger or more complex. The memory usage of FARMap in each environment is measured by the biggest memory size during exploration since the size is dynamically changed by fragmentation and recall. FARMap clearly outperforms the baseline with a much less wall-clock time while planning. This is because our agent only refers the subregion of the environment, not using the entire map. Especially in large environments, it is approximately four times faster than the baseline. Moreover, FARMap requires less memory than the baseline, intervals are caused by aggregating results from multiple high-variance environments (see Appendix G). We also measure the ratios of memory usage and map coverage and of wall-clock time and map coverage in Table 2 The result shows that FARMap has a smaller ratio in all criteria, which means that it requires fewer time and memory resources to explore 1% of an environment.
mentioned above. The high conficlence
5.3 Dynamic Environment
Inspired by Random Disco Maze (Bdia et al. 2020 we build medium-sized 345 dynamic environments where the wall colors change every time step that contributes an increase in surprisal due to mismatched predictions. Table B shows that all methods work well in the environments, and FARMap retains its efficiency compared to Frontier in terms of memory and wall-clock time.
10

---

Published in Transactions on Machine Learning Research (07/2024)
Envrooments
Envronmenns
Medium Enviormerts
02-
10-
12-
42
48-
$2.0
0200
5000
1050
2002
3202
03-
93
00-
63-
01-
03-
48-
2200
2006
2000
4303
$200
1000
2006
3000
4000
5000
1000
2000
3003
4200
2300 3000
5000
kkgs
wtgs
Figure 6: Growth in agent-explored map region a function of the number of steps in the environment matches the performance of an augmented Frontier-based baseline with less memory use. Mean spatial map coverage performance (top) and mean memory usage (bottom) as a function of the mumber steps taken in various sizes of environment sets. FARMap achieves better or comparable exploration than a Frontier-based exploration bascline (Fronticr) (Yamauchi] 1997). while using only about half the memory on average. The memory benefit increases in a larger environment.
Pearson corr. coeff. (p-value)
Pearson coeff. (p-value) Frantier 0,82 (0:00)
20000
Frontier 0.07 11.05e-190)
3000
FARMap (3.28881))
FARMap 0.37 (8.26e-99)
Memary Sire
15000
2000
Wall- Coll-Coock
10000
Frontier FARMap
1000
5000
0-
5000
10000
15000
20000 25000 30000
5000 10000 15000
25000 30000
Environment Size (x 1000)
Enviranment Size (x 1000)
Figure 7: Relative memory and wall-clock time advantage of FARMap to Frontier grow with environment size. Comparison of memory cost (left) and wall-clock time (right) as function of environment size (circles: experimental results; line: linear regression fit). FARMap requires substantially less memory and is much faster than other methods.
Table 1: Comparison of average map coverage (%), memory use (%), and wall-clock time (s) for small, medium, and large environments. The memory usage advantage of FARMap relative to counterpart grows with environment size. The numbers in parentheses are 95 % confidence intervals generated by bootstrap with million samples. f: Random Exploration does not need memory for exploration.
Model
(size 5.000)
Modiam 150000 dae 15,000)
Large 15,0003
Meemoy
Time
Tiee
Menooy
Explestion
17.5
30.6
18.
50.8 (8.5, 117.9)
Vevecter
Yararrh
97.9 (76.0 100
Wh. (61 8, 68
200. (154, 773)
763 (15.6, 99.5)
73.5 (131)) R03)
871 9 1900, 2000)
41.4 (6-1, 643)
10. (3.8, 543)
1901.0 (217, 3189)
A1p
99.0 (90.3 100.0)
(81.4, 88.0) 278.2 (189, 638)
96.4 (15.6, 100.0)
02.9 (12.5, 90.2)
(191, 528)
(6.1, 97.7)
31.4 (S.8, 86.3) 382.5 (202, 683)
5.4 FARMap in Robot Operation Simulation
We simulate FARMap in four continuous environments with TurtleBot3 (Burger) via Robot Operation System (ROS) (Macenski et. al.] 2022) with Gazebo simulator. ROS is one of the standard libraries for conducting robotic experiments, and it allows for straightforward deployment to real robots at no additional cost. Unlike experiments performed Scction 5.2. the observation here involves a 360-dcgroc first-person view via the default laser scan. We utilize the default global planner in the 'move_ base' package. Frontier and FARMap tested in four continuous 3D environments with fixed starting location (Figure g for
11

---

Published in Transactions on Machine Learning Research (07/2024)
Table 2: Comparison of the ratios of memory usage and map coverage, and of wall-clock time and map coverage. Smaller value denotes the model more efficient than others. FARMap has the smallest ratios in all comparisons.
Small
Medium
Largo
Model
Memory / Coverage Time /Coverage
Memory /Coverage
Time / Coverage
Memory /Coverage
Time Coverage
Frontier
0.83
3.71
0.96
11.43
1.07
30.46
FARMap 0.80
3.52
0.73
3.72
0.55
6.22
Table 3: All methods have stable performance in dynamic environments. We measure average map coverage (%), memory use (%), and wall-clock time (s) for dynamic environments with 95% confidence intervals computed by bootstrap with one million samples.
Method
Coverage (%6
Memory (%)
Time (s)
Frontier
95.0 (72.2, 100.0)
86.6 (64.5, 90.0)
742.2 (385.6, 1361.7)
FARMap
95.5 (72.5, 100.0)
67.9 (37.0, 89.6)
386.0 (154.5, 521.7)
Apent [turtisbot]
Environment
AWS Office
Environment
Observation
Environment
American
Figure Robot simulation environments. The turtlebot agent moves around with a 360-degree laser scan sensor to map the entire space.
2500 steps using five different random seeds. The laser scan operates at a frequency of 2.5Hz, meaning that the agent updates the local map every 0.4 seconds.
Table presents comparison between FARMap and Frontier in terms of map coverage and memory usage measurements without any normalization. We do not wall-clock time for the comparison as it is now related to the agent step. In most environments, FARMap has better exploration performance with less memory, Although FARMap consumes more memory than Frontier in the AWS Office, its memory-to-ooverage ratio better than Frontier's (1.16 compared to 1.26, respectively).
5.5 FARMap with Neural SLAM
We conducted experiments on both PARMap and Frontier integrated with the pre-trained Neural SLAM (Chap lot et al. 2020) obtained from the official repository for the Gibson (Shen et al. 2021] exploration task with the Habitat simulator (Szot et al.) 2021). We use 'American" used in Section 5.4 as example. For the fair comparison with FARMap and Neural SLAM, replaced the global policy in FARMap or Frontier to establish the 'long-term goal', following Chaplot et al. (2020). This essentially means that we employ a Neural SLAM module to convert. RGB observations to a 2D map and a Local Policy to generate discrete actions based on the given global goal. Table demonstrates that Neural SLAM, when substituting FARMap for global policy, attains superior exploration performance. In contrast, incorporating Frontier led a decrement in performance. These experimental outcomes also hint at the potential advantages of applying our Fragmentationnand-ree concept to exploration methods that leverage maps.
12

---

Published in Transactions on Machine Learning Research (07/2024)
Table 4: FARMap has better performance with less memory and time in 3D robot simulation environment while it has similar performance with more time in environment The number parenthesis denotes 95%
confidence interval.
Environment 1
Enviromment 2
AWS Office
EErdogan 2019
American Shen et al. 2021
Model
Coverage (k)
Memory (k)
Coverage (k)
Memory (k)
Coverage (k)
Memorr (K)
Coverage (K)
Memory (K)
Frontier
7.0 1.4] 20.5 (+ 1.0)
8.3 (+ 0.6)
32.8 (4 34.4)
38.2
30.0)
48.1 (+ 20.8)
13.8
3.11
11.0 2.1)
FARMap
7.7
(= 1.0)
20.1 (= 2.4)
8.3 (= 0.1)
23.0 (= 8.6)
57.0 (# 4.7)
66.0 (# 14.3)
15.8
(4 4.2)
10.6 (# 3.7)
Table 5: Comparison of Neural SLAM and its adaptations with Frontier and FARMap on the Gibson American environment.
Model
% Cov.
Cov. (m²*
Neural SLAAM
Chaplot
2022
64.795
Neural SLAM w/o global policy
Frontier
0.733
58.103
Neural SLAM w/o global policy FARMap
66.012
6 Discussion
We have proposed new framework for exploration based on local models and fragmentation, inspired by how natural agents explore space efficiently through grid cells' remapping. Our framework dynamically fragments the exploration space based on the current surprisal in real time and stores the current. model fragment. in long-term memory (LTM). Stored fragments are recalled when the agent returns to the state where the fragmentation happened so that the agent can reuse the local information. Accordingly, the agent can refer to longer-term local information. This method shows potential for broad application in tasks involving streaming observations or data that are recurrent or reused (Hwang et al. 2023). Specifically, we have applied this to the setting of spatial exploration. The surprisal is generated by short-term memory (STM) using a local map in FARMap. FARMap closely replicates the fragmentation behavior observed in animal studies. This alignment with biological systems underscores the potential our framework for capturing essential aspects of natural exploration processes.
FARMap outperforms the baseline method (Yamauchi, 1997) in terms of reduced wall-clock time, memory requirements, and action count while enhancing map- building performance in both static and dynamic discrete environments as well as in continual robot simulations. Considering that Yamauchi (1997) is still a
core algorithm in the recent state-of-the-art SLAM methods (Bonetto et al. 2022 Hess et al.) 2016 Placed
et al. 2022) (e-g-, Figure 12), FARMap is complementary in that. it addresses a key inefficiency of global map- based approaches, which is that they can be memory-intensive and computationally demanding.
Our paper aims to be a proof-of-concept for fragmentation and recall in spatial map-building inspired by biological principles using frontier-based exploration and Neural SLAM (Chaplot et al. 2020). We believe that this concept be applicable to other exploration paradigms and various applications (see Appendix D This concept can make large-scale exploration, which typically requires a huge memory size and long-ranged memory span, significantly more efficient.
Broader Impact Statement
Our main focus of this work is to connect neuroscience and spatial exploration so that two different research communities interact more actively with each other. Our method can be exploited for military purposes like other spatial exploration methods or SLAM.
Acknowledgement
We appreciate Jim Neidhoefer helping implement the robot simulation experiment. This rescarch was supported by ARO MURI W911NF2310277 and Ila Fiete is supported by the Office of Naval Research, the Howard Hughes Medical Institute (HHMT), and NIH (NTMFF-MH119046))
13

---

Published in Transactions on Machine Learning Research (07 (224))
References
Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martín Arjovsky, Alexander Pritzel, and Andew Bolt. Never give up: Learning directed exploration strategies. In JCLR, 2020.
Christopher Baldassano, Janice Chen, Asich Zadbood, Jonathan W Pillow, Uri Hasson, and Kenneth A Norman. Discovering event structure in continuous narrative perception and memory. Newron, 95(:): 709 721, 2017.
David Ball, Scott Heath, Janet Wiles, Gordon Wyeth, Peter Corke, and Michael Milford. Openratslam: open souree brain-based slam system. Autonomous Robots, 34:149-176, 2013.
Elia Bonetto, Pascal Goldschmid, Michael Pabst, Michael Black, and Aamir Ahmad. irotate: Active visual slam for omnidiroctional robots. Robotics and Autonomous Systems, 154:104102, 2022.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In ICLR, 2019.
Francis Carpenter, Daniel Manson, Kate Jeffery, Neil Burgess, and Caswell Barry. Grid cells form global representation of connected enviromments. Current Biology, 25(9):1176-1182 2015.
Devendra Singh Chaplot, Dhiraj Gandhi, Saurabh Gupta, Abhinav Gupta, and Ruslan Salakhutdinov. Learning to explore using active neural slam. In ICLR, 2020.
Maxime Chevalier- Lucas Willems, and Suman Pal. Minimalistic gridworld environment for gymna- sium. htt.:/thhh..../aaaaaaaaoooaaaaaa
2018.
Howie Choset and Keiji Nagatani. Topological simultaneous localization and mapping (slam): toward exact localization without explicit localization. TRA, 17(2):125-137, 2001.
Laura Lee Colgin, Edvard I Moser, and May- May-Britt Moser. Understanding memory through hippocampal remapping. Trends in neurosciences, (6)))469-47) 2008.
Anna Dai, Sotiris Papatheodorou, Nils Funk, Dimos Tzoumanikas, and Stefan Leutenegger. Fast frontier-based information-drrven autonomous exploration with an mav. ICRA, 2020.
Adrianus Dingeman De Groot. Het denken van den schaker, een experimennteellsschooogie studie. Noord- Hollandsche Uitgevers Maatschappij, 1946.
Dori Derdikman, Jonathan R Whitlock, Albert Tsao, Marianne Fyhn, Torkel Hafting, May-Britt Moser, and Edvard Moser. Fragmentation of grid cell maps in a multicompartment environment. Nalure Neuroscience, 12(10))13255 1332, 2009.
Christian Dornhege and Alexander Kleiner. A vrontier-void-basell approach for autonomous exploration in 3d. In SSRR, pp. 351-356, 2011.
Dennis E Egan and Barry J Schwartz. Chunking in recall of symbolic drawings. Memory & cognition, 7(2): 149-158, 1979.
Austin Eliazar and Ronald Parr. Dp-slam: Fast, robust simultaneous localization and mapping without predetermined landmarks. In IJCAI, 2003.
Melih Erdogan. DateesrrooCaacoooooootlltddddddd -wweoaaaaoaeeooooooeeeeoeeaedeee and-Maps
GitHub, 2019. URL https://github.... com/
Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. In KDD, 1996.
Youssef Ezzyat and Lila Davachi. What constitutes an episode episodic memory? Psychological science, 22 (2):243 252, 2011.
14

---

Published in Transactions on Machine Learning Research (07/2024)
Nathaniel Fairfield, David Wettergreen, and George Kantor. Segmented slam in throe-dimensionall environ- ments. Journal of Field Robolics, 27(1):85-100 103, 2010.
Meire Fortunato, Melissa Tan, Ryan Faulkner, Steven Hansen, Adrià Puigdomênech Badia, Gavin Buttimore, Charles Deck, Joel Z Leibo, and Charles Blundell. Generalization of reinforcement learners with working and episodic memory. In NeurIPS, 2019.
Marianne Fyhn, Torkel Hafting, Alessandro Treves, May-Britt Moser, and Edvard I Moser. Hippocampal remapping and grid realignment in entorhinal cortex. Nature, 446((1332)190-114 2007.
Fernand Gobet and Herbert A Simon. Expert chess memory: Revisiting the chunking hypothesis. Memory, (3):225 255, 1998.
Fernand Gobet, Peter CR Lane, Steve Croker, Peter CH Cheng, Gary Jones, lain Oliver, and Julian M Pine. Chunking mechanisms in human learning. Trends in cognitive sciences, 5(6):236 243, 2001.
Giorgio Grisetti, Rainer Kümmerle, Cyrill Stachniss, and Wolfram Burgard. A tutorial on graph-based slam. IEEE Intelligent Transportation Systems Magazine, 2(4):31-43, 2010.
Torkel Hafting, Marianne Fyhn, Sturla Molden, May-Britt Moser, and Edvard Moser. Microstructure of a spatial map in the entorhinal cortex. Nalure, 436(7052):801 806, 2005.
Peter E Hart, Nils J Nilsson, and Bertram Raphael. A formal basis for the heuristic determination of minimum cost paths. IEEE Transactions on Systems Science and Cybernelics, 4(2):100-101) 107,000-17,, 1968.
Wolfgang Hess, Damon Kohler, Holger Rapp, and Daniel Andor. Real-time loop closure in 2d lidar slam. In ICRA, pp. 1271-1278, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 98):11335))780, 1997.
Dirk Holz, Nicola Basilico, Francesco Amigoni, and Sven Behnke. Evaluating the efficiency of frontier-based exploration strategies. In ISR, pp. 1-8, 2010.
Chia-Chun Hung, Timothy Lillicrap, Josh Abramson, Yan Wu, Mehdi Mirza, Federico Carnevale, Arun Ahuja, and Greg Wayne. Optimizing agent behavior over long time scales by transporting value, Nature communications, 10(1):5223, 2019.
Jaedong Hwang, Zhang- Wei Hong, Eric Chen, Akhilan Boopathy, Pulkit Agrawal, and ll Fiete. Neuro- inspired fragmentation and recall to overcome catastrophic forgetting in curiosity. In NewrIPSW 2023 (Iutrinsicallyy Molivaled Open-ended Learning), 2023.
Mirko Klukas, Sugandha Sharma, YiLun Du, Tomas Lozano-Perez, Leslie Kaelbling, and Ila Fiete. Fragmented spatial maps from surprisal: state abstraction and efficient. planning. bioRain, 2021.
Mihir Kulkarni, Mihir Dharnadhikari, Marco Tranzatio, Samuel Zimmermann, Victor Reijgwart, Paolo De Petris, Huan Nguyen, Nikhil Khedekar, Christos Papachristos, Lionel Ott, et al. Autonomous teamed exploration of subterrancan environments using logged and acrial robots. In ICRA, 2022.
Andrew Lampinen, Stephanie Chan, Andrea Banino, and Felix Hill. Towards mental time travel: hierarchical memory for reinforcement learning agents. NeurIPS, 2021.
Steven M Valle.. Rapidly-exploring random trees: A new tool for path planning. Technical report, Computer Science Department, lowa State University, 1998.
Rachel S Lee, Yotam Sagiv, Ben Engelhard, llana В Witten, and Nathaniel D Daw. A feature-specific prodiction error model explains dopaminergic heterogencity. bioRaie, 2022.
Isabel IC Low, Alex H Williams, Malcolm G Campbell, Scott W Linderman, and Lisa M Giocomo. Dynamic and reversible remapping of network representations in an unchanging environment. Newron, 109(18): 1967-2980 2021.
15

---

Published in Transactions on Machine Learning Research (07/2024)
Steven Macenski, Tully Foote, Brian Gerkey, Chris Lalancette, and William Woodall. Robot operating system 2: Design, architecture, and uses in the wild. Science Robolies, 7(66), 2022.
Renan Maffei, Vitor Jorge, Mariana Kolberg, and Edson Prestes. Segmented dp-slam. In JROS, 2013.
Michael J Milford, Gordon F Wyeth, and David Prasser. Ratslam: a hippocampal model for simultaneous localization and mapping. In ICRA, 2004.
Michael J Milford, Janet Wiles, and Gordon F Wyeth. Solving navigational uncertainty using grid cells on robots. PLoS computational biology, 6(11), 2010.
James KochizulkiFreeman Sahaj Singh Maini, and Zoran Tiganj. Evidence accumulation in deep rl agents powered by a cognitive model. In CearIPSW-memany 2022.
Darren Newtson and Gretchen Engquist. The perceptual organization of ongoing behavior. Journal of Experimental Social Psychology, 12(5))436 450, 1976.
Edward Н Nieh, Manuel Schottdorf, Nicolas W Freeman, Ryan J Low, Sam Lewallen, Sue Ann Koay, Lucas Pinto, Jeffrey L Gauthier, Carlos D Brody, and David W Tank. Geometry abstract learned knowledge the hippocampus. Nature, 505(7865):80 84, 2021.
Yael Niv. Reinforcement learning in the brain. Journal of Mathematical Psychology, 53(3):139 154, 2009.
Dabal Pedamonti, Samia Mohinta, Martin V Dimitrov, Hugo Malagon- Stephane Ciocchi, and Rui Ponte Costa. Hippocampal networks support reinforcement learning in partially observable environments. bioftriv, 2023.
Lucas Pinto, Kanaka Rajan, Brian DePasquale, Stephan Y Thiberge, David W Tank, and Carlos D Brody. desendep changes in the large-scale dynamics and necessity of cortical regions. Neuron, 104(4): 810-824, 2019.
Julio A Placed, Juan J Gómez Rodríguez, Juan D Tardós, and José A Castellanos. Explorb-slam: Active visual slam exploiting the pose-graph topology. In Pberian Robolics conference, pp. 199 210, 2022.
Alexander Pritzel, Benigno Uria, Sriram Srinivasan, Adria Puigdomenech Badia, Oriol Vinyals, Demis Hassabis, Daan Wierstra, and Charles Blundell. Neural episodic control. In ICML, 2017.
Stefano Recanatesi, Matthew Farrell, Guillaume Lajoie, Sophie Deneve, Mattia Rigotti, and Eric Shea- Brown. Prodictive learning a network mechanism for extracting low-dimensional latent space representations. Nature communications, 12(1):1417, 2021.
Lauren Richmond and Jeffrey M Zacks. Constructing experience: Event models from perception to action. Trends in cognitive sciences, 21(12):9622995) 980, 2017.
Samuel Ritter, Jane Wang, Zeb Kurth-Nelson, Siddhant Jayakumar, Charles Blundell, Razvan Pascamu, and Matthew Botvinick. Been there, done that: Meta-learning with episodic recall. In ICML, 2018a.
Samuel Ritter, Jane X Wang, Zeb Kurth-Nelson, and Matthew Botvinick. Episodic control as meta- reinforcement learning. BioRwie, pp. 360537, 2018b.
Alexei Samsonovich and Bruce L McNaughton. Path integration and cognitive mapping in continuous attractor neural network model. Journal of Neuroscience, 17(15):5900-590 5920,, 1997.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. Xiv preprint (.X.x:1177.06../ 2017.
Bokui Shen, Fei Xia, Chengshu Li, Roberto Martin-Martin, Linxi Fan, Guanzhi Wang, Claudia Pérez- D' D'Arino, Shyamal Buch, Sanjana Srivastava, Lyne Tchapmi, et al. igibson 1.0: A simulation environment for interactive tasks in large realistic scenes. Tn IROS, 2021.
16

---

Published in Transactions on Machine Learning Research (07/2024)
Herbert A Simon. How big is chunk? by combining data from several experiments, basic human memory unit can be identified and measured. Science, 183(4124):482-488 1974.
C. Stachniss, D. Hahnel, and W. Burgard. Exploration with active loop-closing for fastslam. In IROS, 2004.
Khena M Swallow, Jeffrey M Zacks, and Richard A Abrams. Event boundaries in perception affect memory encoding and updating. Journal of Raperimental Psychology: General, 138(2):236, 2009.
Andrew Szot, Alex Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir Vondrus, Sameer Dharur, Franziska Meier, Wojciech Galuba, Angel Chang, Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, and Dhruv Batra. Habitat 2.0: Training home assistants to rearrange their habitat. In NewrIPS, 2021.
Edward C Tolman. Cognitive maps in rais and men. Psychological revieu, 55(4):189, 1948.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 2017.
Charles Vorhees and Michael T Williams. Morris water maze: procedures for assessing spatial and related forms of learning and memory, Nature protocols, 1(2):848 858, 2006.
Brian Yamauchi. A frontier- approach for autonomous exploration. In CIRA, 1997.
Fan Yang, Dung-Han Han Lee, John Keller, and Sebastian Scherer. Graph-based topological exploration planning in large-scale 3d environments. In ICRA, 2021.
Fangwen Yu, Jianga Shang, Youjian Hu, and Michael Milford. Neuroslam: a brain-inspired slam system for 3d environments. Biological cybernetics, 113:515-545, 2019.
Jeffrey M Zacks and Khena M Swallow. Event segmentation. Current directions in psychological science, 16 (2):80-84, 2007.
Shaowu Zhang, Fiola Bock, Aung Si, Juergen Tautz, and Mandyam V Srinivasan. Visual working memory in decision making by honey bees. PNAS, 102(14):5250 5255, 2005.
Yukun Zhang, Mengyuan Chen, Dehong Tian, and Lingmei Ding. Biomimetic slam algorithm based on growing self-organizing map. IEEE Access, 9:134660 134671,, 2021.
Appendix
A Bridging between Neuroscience and Machine Learning
Our contributions hold significant relevance for the machine learning (ML.) community, beyond robotic ST.AM. They also help to connect back to neuroscience, by generating functional hypotheses that can be tested in the brain.
Our interdisciplinary approach, which leverages biological principles towards making broader AT advancements, has the potential to develop more robust and efficient AT systems and inspire novel ML algorithms. The fraggentaaiion-nnd-Beee
framework is not only effective with traditional SLAM (Yamauchi) 1997) but also improves the exploration performance of SLAM based neural networks (Chaplot et al. 2020). This shows the potential of FARMap when it combines with a neural network-based spatial exploration approach.
Further, FARMap is designed as a fundamental algorithm that should be relevant beyond spatial tasks like robotic navigation. For instance, it can be leveraged for any model- based learning, from motor control to reinforcement learning. The agent builds local models (where "local" may be in terms of spatial location for navigating agent or in state space for motor control, for instance). When the local model fails to predict the next states well, the agent may select different local model or build new model. Thus, the concept of
17

---

Published in Transactions on Machine Learning Research (07/2024)
surprisal-based clustering and memory-effiiient mapping can be extended to other areas of machine learning, such spatial exploration, memory optimization, reinforcement learning, and autonomous decision-making, and goal-directed behavior.
Conversely, in neuroscience, where the phenomenon of spatial fragmentation was observed, its implications for function have not yet been studied or appreciated. FARMap supplies hypotheses for the functional roles of the observed map fragmentation, motivating experiments to test these hypotheses. In addition, FARMap provides unifying algorithm for episodic memory, spatial navigation, and perhaps also segmented motor control. It predicts that universal "surprisal" signal might exist in the brain and play central role in signaling and inducing model or map fragmentations across behavioral domains. Thus, FARMap motivates new experiments in neuroscience: to search for signals that trigger the building of entirely new models or maps, across domains of spatial mapping, cognitive modeling, and motor control.
B Additional Related Works
B.1 Graph-Base Based SLAM
Graph-based SLAM (Grisetti et al., 2010: Yang al. 2021 Kulkarni et al. 2022) constructs topological graph for efficient exploration by reducing the dimensionality of the planning problem. Once this graph is established, a planner utilizes it to navigate toward subgoals. GBPlanner Yangg et al. 2021 Kulkarni et al. 2022) creates a random graph in the local region and uses it for path planning. This reduces computational cost for local path planning by reusing sparse graph nodes although it still uses a frontier. In contrast, FARMap aims for efficient exploration in terms of memory, time, and the number of steps by dividing the environment (i.e., fragmentation) and the topological graph is used for moving one subregion another. We believe that there is potential synergy between sraph-based SLAM and FARMap. Such synergy can be achieved by substituting fronticr-based based exploration with graph-based approach, pairing global fragmentation from FARMap with relatively local planning from graph-based moth
B.2 Reinforcement Learning in Neuroscience
Many animal experiments involving goal-directed learning combine rewards (or punishments) with lasks, such as evidence accumulation Ftochizuki-Freemaat et al. 2022 Nieh et al. 2021 Lee et al. 2022 Pinto et al.)
2019) and simple visual cues (Vorhecs & Williams 2006). This has led computational neuroscientists to use reinforcement learning (RL) to model brain functions, resulting in models with good explanatory power for both neural and behavioral data (Nvv) 2009; Pedamonti et al., 2023 Recanatesi et al. 2021: Vorhees & Williams 2006). However, spatial learning (Tolman] (9488) served as a powerful rebuttal reward-based learning and the behaviorist school in Psychology: Animals typically acquire and represent spatial information rewards. Grid and place cells form spatial representations, and
even in the absence of spatially-condiioonn grid cell remapping is observed in free-moving animals with no operant or reinforcement-bese training. Thus, goal-directed learning is not required for spatial representations and remapping. However, spatial representations and remapping could very well play an important role in spatial reinforcement learning. framework into goal-conditioedd RL is an intriguing direction for
Integrating the fragmentation-anddreee future research, beyond the scope of the present paper.
B.3 Memory-Based Reinforcement Learning
Although the reinforcement learning (RL) algorithm is beyond the scope of this paper, FARMap is similar to based RL in the sense that it uses memories. Hung et al. (2019) combine LSTM (Hochreiter) & Schmidhuber 1997) with external memory, along with encoder and decoder for the memory. Ritter ct al. (2018a b) use DND (Pritzel et al. 2017) to store the states of LSTM with its inputs and retrieve old states to update the state of LTM in meta-reinforcemeer learning tasks. Similarly, Fortunato et al. (2019) use working memory and an episodic memory structure but employ an output of the episodic memory as an input for the working memory. On the other hand, Lampinen et al. (2021) utilize a hicrarchical LTM with chunks and attention for long-term recall inspired by Transformers (Vaswani et al. 2017)) however, their chunks formed periodically rather than based on content and are not used as intrinsic options for
18

---

Published in Transactions on Machine Learning Research (07/2024)
exploration. Our spatial map-building framework is similar to memory-based RL methods in terms of having two memory architectures inspired by the brain. However, FARMap fragments an environment (or space) in an online manner and recalls stored memories inspired by grid cells, while memory-based RL stores previous states. Morcover, we use the connectivity graph of STMs to find the next subgoal for efficient map building. We would like to emphasize that FARMap not a rein reinmmncement learning method. On the other hand, we believe that our proposed concept, frugmentation aaamenttaiion-addrrealll can be applicable to memory-based reinforcement learning by reducing search space in the memory.
C LTM Retrieval Overhead
FARMap needs to consider the retrieval time of LTM since it is not located in the main memory. If the memory (RAM) is larger than the environment so that we can even use LTM on RAM, retrieval time is not a concern, and FARMap is useful in boosting speed, although it might use more memory. In our original scenarios, LTM is an external memory (non-volatile memory). Usually, SSD's speed (including bandwidth and read/write) is around 300-600 MB/s while RAM (DDRA) operates 5-25 GBps. In this case, SSD read/ /write can be bottleneck. However, the flash memory speed is around 5 GBps, and the retrieval time for the map will be negligible compared to the planning time. It is generally not recommended to use hard disk drive (HDD), whose data transfer rate is around 100 MB/s.
Discussion about the Proposed Environment
D.1 Wall Color and Exploration
When the environment is static, the color of the wall does not affect FARMap; the model can still detect surprisal events effectively even if the wall color remains constant. On the other hand, in dynamic environments, where the color of walls changes at each time step (as discussed in Section 5.3), the color changes cout ribure an increase in surprisal due to mismatched predictions. Despite this increased surprisal, FARMap remains effective in detecting and managing fragmentation events, demonstrating its robustness in both static and dynamic scenarios.
D.2 Comparison with MiniGrid
MiniGrid (Chevalier- Chevaeier-Boisverrt et al. 2018) is specifically designed for Reinforcement Learning (RL) experiments and involves tasks that require an agent to understand and navigate environments by avoiding hazards, picking up colored keys, opening doors, and recognizing goals. This setup necessitates an RL policy, whereas FAR.ap is not based on reinforcement learning algorithms. Additionally, the MiniGrid environment composed of simple rectangular rooms and corridors, which do not provide the level of complexity we require for evaluating our method. To thoroughly test FARMap's capabilities, we use complex maze environments that better challenge the agent's 's ability to manage fragmentation and recall in a variety of spatial configurations.
E Potential Applications
In this section, we introduco several potential applications where FARMap can be helpful by reducing memory and time costs.
E.1 Mars Exploration
Mars exploration rovers such Opportunity and Curiosity have limited resources. For example, the Curiosity rover has 256 MB of RAM and 2GB of flash memory [ However, the mission range on Mars may be much larger than the RAM. Therefore, elficient mapping is required and we believe that FARMap could be helpful in Mars exploration.
hhttps://maat /ttp http://twwaaaaae./////////////ta
19

---

Published in Transactions on Machine Learning Research (07/2024)
Table The statistics of the size of environments in the dataset.
Statistics
All
Small
Medium
Large
The number of environments
1500
1015
345
140
Average size
5697.8
2466.7
8532.4
22138.7
Standard deviation of size
6265.8
1253.7
2828.5
4872.9
E.2 2D/3D Mapping with LiDAR
As mentioned in Section 5.4 FARMap capable of utilizing observations from LiDAR for map-building in continuous environments. The resolution of the sensor be set to cell unit. Considering the properties of the Robot Operating System (ROS) (Macenski et al. 2022), we believe that FARMap can be easily deployed to real robot. Additionally, it is feasible to extend it to 3D by using 3D voxel mapping instead of 2D pixel mapping. This approach can prove beneficial in large-scale environments such as buildings, airports, and houses.
F Experimental Details
Our models are implemented on PyTorch and the experiments are conducted on an Intel(R) Xoon(R) CPU E5-2650 v4 2.20GHz for spatial exploration experiments and on an NVIDIA Titan V for RND and Neural SLAM.
F.1 FARMap Environment Generation
To generate the environment, we run map generation (Algorithm 2 200 times and then use the 300 largest- sized maps. All maps scaled up by factor of after colorization for the task. On every trial, sample S and N from {3, 4, 5, 6, 7} and set M = N. K, L N are sampled from [0, 10] and [1,3], respectively. We set Peonuect: Pmerge and Pilip to 0.25, 0.25, and 0.05, respectively. Figure iillustrates the procedure of environment generation described in Algorithm Table shows the statistics of the size of the generated environments.
F.2 FARMap
We run the agent on 1,500 different environments: 300 different maps with five random seeds and the starting position and the color of the map are changed on each random seed. We set y, P, and to 0.9, 2, and 5, respectively. The observation size (h, w) (15,15). If the frontier-based exploring agent is surrounded by a large frontier-edge in an open space, the centroid of the fronticr can fall into the interior of the explored space, leading to new discovery. This causes the agent to become stuck. We improve the agent by selecting the nearest unoccupied cell from the nearest frontier state to the centroid.
F.3 RND
We train RND Burda al. 2019) for 1 million steps without extrinsic reward for each environment. RND is based on recurrent PPO Schulman et al. 2017). Table shows the architecture of RND used for the experiments. The learning rate is 0.0001, the reward discount factor is 0.99 and the number of epochs is 4. For other parameters, we use the same values mentioned in PPO and RND: we set the GAF parameter À as 0.95, value loss coofficient. as 1.0, entropy loss coofficient as 0.001, and clip ratio (e in Eq. 7 in Schulman et al. (0177) as 0.1.
G Wide Confidence Intervals
In Table D 95% confidence intervals for each measurement are generated by bootstrapping with one million samples. The confidence intervals very wide because metrics (map coverage, memory usage, and
20

---

Published in Transactions on Machine Learning Research (07/2024)
Algorithm Spatial Exploration Environment Generation
Require: N, M, L, S, K. Poonnect: Pmerge Ptip
Ensure: A set of maps, {M}.
1: procedure MAPGENERATION
JLO.SHHHNHHDDDXMMSHHAAA 2: Initialize M E {0, 13
(N, grid with interval L and each square sized
S)
Figure (1).
for 8;) E 81)); and 8, are adjacent, i j} do ((1, (conuect))
Get adjacent grid square pairs.
Connect adjacent squares with probability Peonnect-
5: if = then
6:
w ~ S - 1}
7:
Connect 8; and 8, with width w. end if
Figure (2).
8:
I. ~ Pmergg)
Merge adjacent squares with probability Pmerge-
10:
if then
11:
Merge 8; and 8y by removing the interval.
Fig Figure (3).
12:
end if end for
13:
14:
for k - 1 to K do
15:
for CE {clc E Me e xor d' 1, E Adj(c)} do
> Get boundary cells in the map.
16:
I x~ B(1, Paip)
Flip the cell with probability Piip-
17:
c= xor x
Figure (4)-(6).
18:
end for
19:
end for
20:
Divide M into set of isolated maps {m,)
Figure
21:
Filter out map in {m:}, where the size is smaller than 352.
22:
Randomly colorize the occupied cell in each map.
Figure
23:
Scale up each map in }m;} by factor of 3.
24:
end procedure
5:4
N L=1
Connect
Merge
Flip
K-9
-025
[3)
(2)
(3)
(4)
Remove small maps
Flip
Colorization
(6)
(8)
(1) We first set square grid where black and white denote empty and occupied, respectively. (2) We randomly connect and (3) merge adjacent grid. (4)-(6) We also randomly flip the boundaries of empty and occupied cells recursively. (7) Then, we remove small isolated subregions and (8) randomly colorize occupied colls. Finally, we increase the size of the map.
Figure 9: Procedure of map generation.
wall-clock time) depend on the size and the complexity of the environment, and each method is evaluated on many varied environments as shown in Table 0 and the code repository.
We also present results with much smaller groups in Figure 10 We first sort the environments based on their sizes, and then we partition the environments into 150 groups, cach of size 10, and calculato the average with bootstrapping to get a 95 % confidence interval for each group. The 95% confidence intervals measured by bootstrapping also smaller than the reported range in Table In particular, FARMap has relatively
21

---

Published in Transactions on Machine Learning Research (07/2024)
Table The architecture RND agent. The networks are divided into the policy module and RND module.
Policy module
RND module
Conv2d (8x8, 16)
Conv2d (8x8, 32)
Conv2d (4x4, 32)
Conv2d (4x4, 64)
FC (3200x512)
Conv2d (3x3, 64)
LSTM (512, 512)
FC (3136x512)
FC (512x5) x 2
FC (512x512)
FC (512x) x 2
FC (512x512)
Frenteer
FARMap
3500
2500
Coverage
Wall-Clock
Memory
10-
1500
10-
500
Figure 10: Map coverage, memory usage, and wall-clock time advantage of FARMap to Frontier grow with environment size. Comparison of these metrics as function of environment size. The mean (line) and 95% confidence interval (shade) are calculated by bootstrapping with one million samples each from 150 groups (10 environments each) ordered by size.
steady wall-clock lime across the entire environments while Frontier requires more time with high variance depending on the environments. Although the gaps between FARMap and Frontier in all metrics are small in small environments, they bocome larger as the environment size grows. In other words, FARMap is better than Frontier in all environments in terms of map coverage, memory usage, and wall-clock time.
Ablation Study
Table g illustrates the ablation study of components of FARMap. Each component contributes to improving the performance while it increases memory and time which are negligible compared to the baseline performance (44.4, 1261.0, respectively). We also evaluated FARMap, random fragmentation, and uniform fragmentation methods top of FARMap. This is demonstrate that surprisal produces effective fragmentations that maintain the exploration performance with low memory and fast wall-clock time. Random and Uniform models only change the fragmentation criteria and other parts (c.g., LTM, subgoal sclection, and planning) remain the same. Table 9 shows that there is a trade-off between frequency of fragmentation and memory usage and wall-clock time. FARMap achieves better exploration performance than random and uniform fragmentation models. Table 10 shows Frontier and FARMap with RRT planner (La Valle) 1998) large environment. Although the performance gaps between the two models are reduced, FARMap still outperforms Frontier.
22

---

Published in Transactions on Machine Learning Research (07/2024)
Table 8: Ablation study for each component large environments. We chose the best-performed fragmentation threshold without the z-score variant in 0.91.
2-score
LTM subgoal
Coverage
Memory
Time
51.7 (22.1, 90.5)
20.8 (10.0, 49.6)
177.3 (121.7, 254.8)
52.8 (21.9, 85.6)
30.4 (15.3, 57.7)
302.3 (167.9, 519.2)
56.6 (6.1, 97.7)
31.4 (3.8, 54.3)
352.5 (202.0, 633.0)
Table 9: Comparison of vanilla FARMap and FARMap with random, and uniform fragmentation Large Environments. The random model decides to fragment with probability on every time step and the uniform model makes a fragmentation on every Interval step (L).
Model
Coverage
Memory
Time
FARMap
56.6 (6.1, 97.7)
31.4 (3.8, 54.3)
352.5 (202, 633)
Random (P = 0.1)
45.2 (15.4, 82.3)
7.8 (4.4, 13.2)
111.3 (70.9, 141.0)
Random (P 0.05)
47.5 (18.0, 87.4)
12.1 (6.7, 20.0)
136.5 (179.8)
Random (P 0.01)
49.0 (18.6, 87.6)
24.5 (12.9, 43.7)
290.7 (148.1, 499.6)
Random (P = 0.005)
49.1
(20.5, 89.6)
30.5 (16.1, 60.0)
378.1 (201.7, 637.7)
Random (P 0.001)
54.1
(23.5, 92.4)
46.1 (20.9, 81.4)
683.4 (292.3, 1484.4)
Uniform (L 25)
49.1
(15.8, 82.5)
7.5 (4.7, 11.8)
110.8 (84.6, 143.7)
Uniform (L 50)
48.8
(17.3, 89.5)
12.6 (6.8, 20.9)
147.3 (112.9, 200.8)
Uniform [ 100)
48.3
(18.7, 88.8)
19.3 (10.7, 32.1)
216.5 (150.3, 330.2)
Uniform L 200)
48.8
(22.0, 90.0)
27.5 (14.0, 45.5)
322.2 (209.1, 612.0)
Uniform L 500)
52.2
(22.0, 90.0)
38.2 (16.5, 82.5)
484.2 (292.9, 840.6)
Uniform (L 1000)
53.4
(22.1, 91.9)
46.5 (23.4, 87.9)
712.3 (380.6, 1425.0)
Table 10: Comparison of average map coverage (%), memory (%), and wall-clock time (s) in large environments. Both Frontier and FARMap use RRT La Valle 1998 planner.
Model
Coverage
Memory
Time
Frontier (Yamauchi/ 1997
46.9 (20.7, 94.0)
49.7 (22.6, 90.3)
880.9 (395.6, 1673.1) 318.1 (174.6, 500.6)
FARMap
50.9 (17.3, 91.3)
30.4 (13.8, 55.7)
Table 11: Sensitivity analysis about fragmentation threshold, p in FARMap. The numbers parentheses are the standard deviation.
Small (size < 5,000)
Medium (5,000- < size < 15,000)
Large (size > 15,000)
P
Coverage
Memory
Time
Coverage
Memory
Time
Coverage
Memory
Time
1.0
99.1
71.5
117.9
87.1
39.6
146.2
60.9
17.9
148.4
1.5
99.1
75.7
158.0
87.6
50.2
180.1
59.7
23.3
188.9
2.0 (ours)
99.0
79.1
278.2
86.4
62.9
321.4
56.6
31.4
352.5
2.5
98.8
80.7
207.1
89.0
79.7
557.3
58.4
56.9
770.5
3.0
98.8
81.5
296.1
91.0
85.0
698.2
60.9
67.9
1068.0
Sensitivity Analysis for Hyperparameters in FARMap
We test FARMap with various hyperparameters; fragmentation threshold (p), decaying factor (v), and All experiments are conducted in the same environments. While comparing hyperparameter, fix the
23

---

Published in Transactions on Machine Learning Research (07/2024)
Table 12: Sensitivity analysis about decaying factor, y in Eq, The numbers in parentheses are the standard deviation.
Small (size < 5,000)
Medium (5,000 & size < 15,000)
Large (size 2 15,000)
Y
Coverage
Memory
Time
Coverage
Memory
Time
Coverage
Memory
Time
0.8
98.8
79.4
210.5
85.3
64.5
304.0
55.6
32.8
304.8
0.9 (ours)
99.0
79.1
278.2
86.4
62.9
321.4
56.6
31.4
352.5
0.95
99.1
79.0
178.3
87.3
61.3
507.5
59.2
31.9
284.7
0.99
99.1
80.8
262.3
89.3
76.5
453.8
60.4
46.7
541.5
Table 13: Sensitivity analysis about in Eq.
The numbers in parentheses the standard deviation.
Small (size 5,000)
Medium (5,000 S size
15,000)
Large (size 2 15,000)
Coverage
Memory
Time
Coverage
Memory
Time
Coverage
Memory
Time
99.0
79.1
198.0
86.8
63.0
275.3
56.6
31.5
294.8
3
99.0
79.1
198.1
86.7
63.0
271.3
56.5
31.5
294.5
(ours)
99.0
79.1
278.2
86.4
62.9
321.4
56.6
31.4
352.5
10
99.0
79.1
197.1
86.6
62.9
272.5
56.3
31.4
294.9
15
99.0
79.1
198.5
86.6
63.0
288.7
55.9
31.1
295.7
Table 14: Average map coverage (%), memory use (%), and wall-clock time (s) of RND in small, medium, large, and dynamic environments. The numbers in parentheses are 95 % confidence intervals generated by bootstrapping with one million samples across various environments. The memory usage is calculated by the ratio between the number of parameters (7.7M) and each environment size.
Environment
Coverage (%)
Memory (%)
Time (s)
Small
77.0 (31.3, 100.0)
421.7k (157.5k, 1012.5k)
31.6 (23.9, 40.4)
Medium
37.1 (11.0, 77.0)
99.7k (53.7k, 151.9k)
31.2 (23.7, 39.6)
Large
14.9 (3.4, 33.5)
36.2k (24.4k, 49.7k)
30.9 (25.0, 39.6)
Dynamic
37.2 (23.9, 35.6)
99.7k (53. 7k, 151.9k)
29.1 (10.5, 75.8)
remaining parameters as = 2.0, 6.9,6= =. Table 11 presents the performance of FARMap with different. fragmentation thresholds, p. The smaller value makes more prone fragment the space, which means it can use less memory but overly fragment the space. On the other hand, bigger threshold uses more memory with less fragmentation. Hence, we choose 2 as the threshold value (95% confidence interval if the distribution follows Gaussian). On the other hand, FARMap is robust to the decaying factor and shown in Tables 12 and 13 respectively.
J Reinforceemet Learning Method in the Proposed Environments
We run RND (Burda et al. 2019 based on PPO-LSTM [Schulman et al. 2017 to give an example of reinforcement learning exploration method in the proposed rrocedurally-generaaeed environments. Table [4] shows the performance RND in static and dynamic environments. To quantify RND's memory usage based on this measurement, we divided the number of parameters (7.7M) by the environment size. Note that is difficult compare with FARMap or Frontier directly since the RL agent is trained on each environment before testing it while FARMap and Frontier have training. However, in both sets of environments, RND has much lower coverage than FARMap but it is much faster since it does not need to update the local map and planning. We also demonstrate the average map coverage across the number of steps in Figure
24

---

Published in Transactions on Machine Learning Research (07 (07/202)) /204)
All Trammmeth
Medium Trukerments
Lang Envronment
303
393
193
100
350
120
2000
1903
820
1000
1006
2001
1301
1883
2007
3100
Figure 11: Growth in agent-explored map region as function of the number of steps from the first step in the environment matches the performance of RND. Mean spatial map coverage performance function of the number of steps taken in various sizes of environment sets. The shade denotes the standard error.
Kitliig
LIN saggoll
Graph Erclttes Maker
Lake
Read
(a) ExpIORB-SLLL SLMORR-SLAM Placed et al.
2022
(b) ExpIORB-SLAM + FARMap
Figure 12: (a) The schematie diagram of ExPIORB-STAM from Placed (2022). (b) An example of FARMap integrated EXPIORB-SLAM. FARMap encapsulates 'Frontier Detector and LTM subgoal can substitute 'Navigation Goal' from 'Graph Decision Maker.
25


Reviews:
[{"review_id": "fTqISMDWcI", "reviewer": "Reviewer_r3yZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. Biological brains are always source of inspiration for artificial intelligence development, therefore this work is well motivated and interesting. I think the study in this work is beneficial for the community.\n\n2. The experiments are thoroughly conducted from multiple perspectives. The proposed method significantly outperform the baseline (frontier).\n\nweakness:\n1. It seems that only one baseline (frontier) is adopted for comparison. Since this method was proposed in 1997, please discuss whether this represents the current best performance and whether there are newer baselines.\n\n2. Although the inspiration from biological brains is good, it would be better to discuss more on the relationship between the proposed method and the other state-of-the-arts methods. E.g. the inefficiency of the existing methods, and why the proposed method can resolve the issues.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a novel approach to build map for an environment, which is inspired by the working mechanism of the grid-cells in brains. The work focuses on the fragmentation of grid cell map in compartmentalized spaces, based on which the concept of 'fragmentation and recall' is proposed for mapping large spaces.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please discuss the two weakness points above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MYTk60k7Ex", "reviewer": "Reviewer_9Lry", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n\n1. It's interesting to see the paper shows similar results as in neuroscience experimental findings. This can potentially give explanations for the underlying mechanism of grid cells.\n\n2. The authors show the proposed method achieves better exploration rates on large simulated environments compared to the Frontier-based method by using much less memory and computation time.\n\nWeakness\n\n1. The review has a concern for the targeted audience of the paper. The paper may gain interest from neuroscience and traditional mobile robot researchers. However, it lacks connections with general machine-learning research. Maybe more computational neuroscience or robotics-oriented journals will be a better venue for the work.\n\n2. The proposed method is fully rule-based and heuristic-based. There can be many problems with such a system for robotics. First, the system will lack flexibility and adaptability as the method is confined to predefined rules. For example, it's questionable if the method can work in open environments without well-defined borders and with other types of sensors. Second, the complexity of managing and scaling the system increases significantly with the complexity of the environment. Additionally, they require substantial expert knowledge, are inefficient in unstructured environments, and have difficulty handling noisy or incomplete data.\n\n3. Therefore, it is questionable if the proposed method can be applied to real-world robots and environments. Since the experiments performed are either too abstract (2D grid environment) or too simple (simple 3D environment with structured borders), it's hard to make the connection with real-world deployments.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed an efficient active map building method for mobile agents inspired by the remapping/fragmentation of grid cells. The proposed method builds new local maps based on surprisal signals and performs exploration in existing local maps based on the explored rate. The method is rule-based and heuristic-based. Since only the local map and fracture points of the local maps are stored in memory, the method is more memory-efficient than existing active mapping methods. The authors performed experiments in simulated environments, including randomly generated 2D grid environments and existing 3D room environments. The results show the proposed method can achieve a better mapping rate by using less memory and computation time.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Recommend to move Section 3.7 to the beginning of the Method section. It will help the reader better understand the proposed method.\n\n2. If we read the paper from the computational neuroscience perspective, the readers would like to learn insights into experimental neuroscience observations. Are there any new insights from the experiments and results of this paper that can better explain the underlying mechanism of neuroscience observations? For example, the authors can perform additional ablation studies and examine if the simulated result can show a similar effect as biological ablation studies.\n\n3. The agent only recalls existing maps if it's on a fracture point. How to make sure the agent will always move on top of a fracture point when a change of a map is needed? Will the fracture points be very dense and cost a lot of memory space?\n\n4. The computation of the surprisal signal is fully based on a simple 2D pattern matching algorithm. Will this computation method generalize to real-world environments with more complex patterns and noises? If not, how will the authors deal with this problem?\n\n5. The results shown in Figure 6 and Figure 7 seem to contradict each other. Figure 6 shows the memory costs of FARMap are significantly lower than Frontier. However, in Figure 7, the memory cost differences are much less. For smaller environments, it seems the differences are even statistically insignificant.\n\n6. A similar contradiction is also observed in Table 1-3, as the memory reduction is not as significant as in Figure 6 (For the small environment).\n\n7. Since the fragmentation threshold is the lower the better in the ablation study, why not use 1.0 in the main result for all the experiments?\n\n8. What will happen if decreasing decay factor to less than 0.8?\n\n9. The authors only compare with the Frontier method. However, the mapping method in robotics always works with SLAM. How does the method compare with state-of-the-art Active SLAM methods?", "broader_impact_concerns": "There is no broader impact concerns for this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3A9DgnvXci", "reviewer": "Reviewer_VhYd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe algorithm introduced outperforms existing mapping algorithms while being more efficient.\n\nComparison with neuroscience observations.\n\nSeveral aspects of the same principle are explored and there is a good effort in explaining the algorithm.\n\n\nWeaknesses:\n\n1. The results need better quantification. For example, in Figure. 4, its not clear how well is the model able to reconstruct the original environment across the different sizes. This could be easily calculated by comparing reconstruction with original in a final panel e. And maybe compare here with a random agent and other existing methods?\n\n2. Similar to above, the same problem occurs in figure 5. Can you quantify this somehow? Maybe using a metric that has been used in the original neuroscience papers. This figure is a bit unclear. Are you simply saying that your algorithm can identify the fragment points? Is that what is claimed in neuroscience? Or rather that the grid representations change after these points? If the later, then its unclear how the model is related to experiments. Accordingly, the 5.1 section is super short, needs more explanation. This needs more work as its one of the key points made in the abstract.\n\n3. Related to the point above. More discussion related to the neuroscience literature would be helpful. For example, recent work has shown that RL agents trained to navigate partially observable environments better explain both neural and behavioural data in the hippocampus with good generalisation properties (Pedamonti et al. 2023 bioRxiv: https://doi.org/10.1101/2023.11.09.565503). Given that you also rely on partial observability, it would be important/interesting to discuss the relationship with this other work. See also more relevant papers within that paper.\n\n4. In figure 6, these results should be shown with error bars/shaded areas, to quantify the variability in the models across different initial conditions. This would help the reader see how significant the results are. Similarly in Fig. 7, can you provide the linear correlation coefficients and respective p-values?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors introduce a neuroscience-inspired algorithm to build maps locally by identifying points of fragment. New map formation occurs when high surprisal events occur. The algorithms seems to achieve good performance when compared to classical algorithms. Moreover, the authors state that their agents capture animal behaviour. The model seems to achieve promising results, but there are a few unclear points that should be addressed.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above in weaknesses a number of major points that should ideally be addressed.\n\nMinor ones:\n1. Why multi-coloured grid maps is utilised? Are the colours necessary, what happens if walls are kept with the same colour, can the model still detect surprisal events?\n2. Why use custom maze and not an existing environment such as Minigrid gym (Chevalier-Boisvert, CoRR, 2023)?\n3. How well does it generalise? Try in a new environment with similar condition (observation) but different layout.\n4. “However, FARMap divides space based on properties of the space (how predictable the space is based on the local map or model), and does so in an online manner using surprisal.” What happens if the environment is much more simple and uniform.\n5. Fig 5 unclear as why fracture points are in certain positions (right after the turn and not before as both events look similar in term of ‘surprisal’). Please elaborate on this more.\n6. The definition of memory usage as given by local map size/ environment size is not clear. Considering local map is always W x H, shouldn’t the memory remain fixed? But the figures shows some variation as number of steps increases. As described here, it seems more like it is the observation size; how does it related to short-term memory (STM) and longe-term memory (LTM) and their respective sizes?", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I recommend the paper be accepted, given that it meets the criteria. I am also recommending this article for the featured certification because it not only shows benefits in the robotics environments, but also reproduces some neuroscientific findings of where grid cell remapping occurs in the environment; thus, it highlights potentially interesting connections that could be of benefit to both fields — in addition to clearly presenting the method and results, and connecting well with the relevant literature (especially in the revised version).
Decision: Accept (poster)