MemoNav: Working Memory Model for Visual Navigation
Authors: Hongxin Li
Xu Yang
Zeyu Wang
yuran Yang
Shuqi Mei
Zhaoxiang Zhang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Image-Goal Navigation
Memory mechanism
Embodied visual navigation
Embodied AI

Abstract: We present MemoNav, a novel memory model for image-goal navigation, which utilizes a working memory-inspired pipeline to improve navigation performance. Specifically, the node features on the topological map are stored in the short-term memory (STM), as these features are dynamically updated. The MemoNav retains the informative fraction of the STM via a forgetting module to improve navigation efficiency. To learn a global representation of 3D scenes, we introduce long-term memory (LTM) that continuously aggregates the STM. Afterward, a graph attention module encodes the retained STM and the LTM to generate working memory (WM). After encoding, the WM contains the informative features in the retained STM and the scene-level feature in the LTM and is finally used to generate actions. Consequently, the synergy of these three types of memory increases navigation performance by selectively retaining goal-relevant information and learning a high-level scene feature. When evaluated on multi-goal tasks, the MemoNav outperforms the SoTA methods at all difficulty levels in both Gibson and Matterport3D scenes. The MemoNav also achieves consistent improvements on traditional 1-goal tasks. Moreover, the qualitative results show that our model is less likely to be trapped in a deadlock.

Under review as a conference paper at ICLR 2023
MEMONAV: WORKING MEMORY MODEL FOR VISUAL NAVIGATION
Anonymous authors Paper under double-blind review
ABSTRACT
We present MemoNav, novel memory model for image-goal navigation, which utilizes a working memory-inspired pipeline to improve navigation performance. Specifically, the node features on the topological map are stored the short-term memory (STM), as these features are dynamically updated. The MemoNav retains the informative fraction of the STM via a forgetting module to improve navigation efficiency. To learn global representation of 3D scenes, we introduce long-term memory (LTM) that continuously aggregates the STM. Afterward, graph attention module encodes the retained STM and the LTM to generate working memory (WM). After encoding, the WM contains the informative features in the retained STM and the scene-level feature in the LTM and is finally used to generate actions. Consequently, the synergy of these three types of memory increases navigation performance by selectively retaining goal-relevant information and learning high- level scene feature. When evaluated on multi goal tasks, the MemoNav outperforms the SoTA methods at all difficulty levels in both Gibson and Matterport3D scenes. The MemoNav also achieves consistent improvements on traditional 1 I-goal tasks. Moreover, the qualitative results show that our model is less likely to be trapped in a deadlock.
INTRODUCTION
This paper studies the image- goal navigation (ImageNav) problem, which aims to steer an agent towards a destination with the goal image in unseen environments. ImageNav has recently received much attention due to its promising applications robots navigating in the open world.
Scene memory is essential for imageNav as it provides indispensable historical information for eccision-maaing in unseen environments (Savinov et al. 2018). During navigation, this memory typically stores both scene features and the agent's navigation history (Kwon et al. 2021 These two types of information in turn help the agent generate more reasonable navigation actions by lessening the negative impact of partial observability (Parisotto & Salakhutdinov 2018). In literature, various memory mechanisms have been introduced for ImageNav, which can be classified into three categories according memory structure: (a) metric map-based methods (Chaplot et al. 2020a Chen ct al. 2019) that reconstruct local top-down maps and aggregate them into a global map. (b) stacked memory-based methods (Pashevich et al. 2021 Mezghani et al. [202]: Fang et al. 2019) that stack the past observations in chronological order, and (c) topological map-based methods (K (wonn& et al. Chaplot et al. 2020b; [eeching et al. 2020 Savinov et al. 2018) that store sparse landmark features in graph nodes. The topological map- based methods benefit from the memory sparsity of topological maps and have achieved impressive performance ImageNav.
2021 2021
However, existing topological map-based methods still suffer from two major limitations: (a) Un- awareness of useful nodes. They generally use node features for generating actions without considering the contribution of each node, thus being casily misled by redundant nodes that are uninformative of the goal. (b) Local representation. Each node feature only represents a small local area in large scene, which restricts the agent's ability to learn higher-level semantic and geometric representation of the scene.
To overcome the above two limitations, present novel ImageNav method named MemoNav. which is motivated by the classical concept of working memory in cognitive neuroscience (Cowan 2008) and in loose analogy with the working memory model in human navigation (Blacker et al.

---

Under review as a conference paper at ICLR 2023
2017). The MemoNav learns three types of scene representations: (a) Short-term memory (STM) represents the local and transient features of the nodes in a topologiaal map. (b) Long-term memory (LTM) is global node that learns scene-level representation by continuously aggregating STM. (c) Working memory (WM) learns goal-relevant features about 3D scenes and is used by a policy network to generate actions. The WM is formed by encoding the informative fraction of the STM and the LTM.
Based on the above three representations, the MemoNav navigation pipeline contains five steps: (1) STM generation. The map update module stores landmark features on the map as the STM. (2) Selective forgetting. To incorporate goal-relevant STM into the WM, a forgetting module temporarily removes nodes whose attention scores assigned by memory decoder rank below a predefined percentage. After forgetting. the navigation pipeline will not compute the forgotten node features at subsequent time steps. (3) LTTM generation. To assist the ST'M, we add glohal node to the map as the LTM. The global node links to all map nodes and continuously aggregates the features of these nodes at each time step. (4) WM generation. A graph attention module encodes the retained STM and the LTM to generate the WM. The WM utilizes the goal-relevant information in the retained STM and the scene-level feature in the LTM, thus enabling the agent to utilize informative scene representations to improve navigation performance. (5) Action generation. Two Transformer decoders use the embeddings of the goal image and the current observation to decode the WM. Then, the decoded features are used to generate navigation actions.
Consequently, with the synergy of the three representations, the MemoNav noticeably outperforms the SoTA method (Kwon et al. 2021) in the Gibson scenes (Xia et al. 2018), increasing the navigation success rate by approximately 2.9%, 1.4%, 2.4%, and 1.7% on 2, 3. and 4-goal test datasets, respectively. The comparison in the Matterport3D scenes (Chang et al. 2017) shows that the
MemoNav exhibits better transferability.
The main contributions of this paper are as follows:
We propose the MemoNav, which learns three types of scene representations (STM, LTM, and WM) improve navigation performance in the ImageNav task.
We use forgetting module to retain the STM, thereby reducing redundancy in the map and improving navigation efficiency. We also introduce a global node as the LTM. The LTM connects all nodes in the STM and learns scene-level representation that provides the agent with a global view.
We adopt a graph attention module to generate WM from the retained STM and the LTM. This module flexibly adjusts weights used for aggregating node features, which helps the agent use adaptive WM to improve performance.
The experimental results demonstrate that our model outperforms the SoTA baseline on both 1-goal and multi-goal tasks in two popular scene datasets.
2
RELATED WORK
ImageNav methods. Since an early attempt (Zhu et al.
2017 to train agents in a simulator for
ImageNav, rapid progress has been made on this task (Beeching et al. 2020
Chen et al. 2021
Wasserman et al. 2022 Al-Halah et al.] 2022). ). Several methods have utilized topological scene
representations for visual navigation, which SPTM (Savinov al. 2018 is an early work. NTS
(Chpplot et al. 20206 and VGM (won et al. 2021 incrementally build a topological map during navigation and generalize to unseen environments without exploring the scenes in advance. These methods utilize all features in the map, while the MemoNav flexibly utilizes the informative fraction of these features. Another line of work (Yadav et al. 2022 Majumdar et al. 2022 has introduced
self-supervised learning to enhance the scene representations, achieving promising navigation success rate. In contrast, we enhance the scene representations using a global node that aggregates the agent's local observation features.
Memory mechanisms for reinforcement learning. Several studies (Ritter et al., 2021 Lampinen
et al. 202 Sukhbaatar et al. 2021 Loynd et al. 2020) draw inspiration from memory mechanisms of the human brain and design reinforcement learning models for reasoning over long time horizons. Ritter et al. (Ritter et al. 2021) proposed an episodic memory storing state transitions for navigation
2

---

Under review as a conference paper at ICLR 2023
Map node
edge
Starting
Figure 1: Example episodes for 1-goal and three-goal tasks. the 1 1-goal task (left), the agent searches for the goal during exploring the scene. In the three-goal task (right), the agent continues to navigate to two more successive goals.
tasks. Lampinen al. (Lampinen al. 2021) presented hierarchical attention memory as form of "mental time-travel" [Tulving] 1985), which means recovering goal-oriented information from past experiences. Unlike this method, our model retains such information via novel forgetting module. Expire-span (Sukhbaatar et al. 2021) predicts life spans for each memory fragment and permanently deletes expired ones. Our model is different from in that we restore forgotten memory if the agent returns to visited places. The most similar work is Working Memory Graph (Loynd et al.] (020) which is also inspired by working memory in cognitive neuroscience. However, its memory capacity is fixed. In contrast, our model retains certain proportion of short-term memory in the working memory and adjusts the memory capacity when the navigation episode contains more goals.
3 BACKGROUND
3.1 TASK DEFINITION
The objective of ImageNav (Figure T is to learn policy æ to reach a target, given an image harget that contains view of the target and series of observations {lc} captured during the navigation. At the beginning of navigation, the agent receives an RGB image karget of the target. At each time step, the agent captures RGB-D panoramic image I, of the current location and generates navigational action. Following (Kwon et al., 2021). any additional sensory data (e.g. GPS and IMU) are not available.
3.2 BRIEF REVIEW OF VISUAL. GRAPH MEMORY
The MemoNav is mainly built upon VGM (Kwon al. 2021 which is briefly introduced below.
VGM incrementally builds topological map G = (V,E) from the agent's past observations where V and & denote nodes and edges, respectively. The node features (short-term memory) V E dBd are generated from observations by pretrained encoder Fine where d denotes the dimension of feature and N, the number of nodes at time l.
VGM uses graph convolutional network (GCN) to encode the topological map, obtaining the encoded memory M GCN(V). Before encoding, VGM obtains the target embedding étorget Fenc(larget) and fuses each node feature with this embedding using a linear layer.
The encoded memory is then decoded by two Transformer (Vaswani et al. 2017) decoders, Dour and Dtorget. Deur takes the current observation embedding Ecur = FFne (Teur) ) as the query and the feature vectors of the encoded memory M as the keys and values, generating a feature vector four- Similarly, Dzarget takes the target embedding étargt as the query and generates ftarget- Lastly, an LSTM-based policy network takes as input the concatenation of feur, ftarget and Cear output action distribution.
4 METHOD
The MemoNav comprises three key components: the forgetting module, the global node, and the GATv2 memory encoder. We show the pipeline of the MemoNay in Figure and describe these components in the remainder of this section.
3

---

Under review as a conference paper at ICLR 2023
16
11
Fter
Fly
Map updane
Reshiet encoder
STM
Informacion
Selective forgetting
Short tarm Memory with
Long tarm memory
Forgotten
Starpet
(e) GATu2 ancoder
Short t-rmm memory edge
Aprrett
odee M!
odase at?
aeoe M
Long-term -erm--eemm memory edge
Forgotton edge
decoders
[g) Policy network
das
Figure 2: Overview of MemoNav. (a) The memory update module builds a topological map using er, the embedding the current image I. (b) The node features in the map form the STM while a global node that links each node acts the LTM. (c) The forgetting module temporarily excludes fraction of STM whose attention scores rank below threshold (d) The retained STM and the LTM then encoded by (e) graph attention module generate the WM M,,- (f) The WM is decoded by two Transformer decoders (details the Appendix). (g) Lastly, the output of the decoding process input to policy network generate navigation actions.
4.1 SELECTIVE FORGETTING MODULE
We devise forgetting module that instructs the agent to forget uninformative experiences. Here, "forgetting" means that nodes with attention scores lower than a threshold are temporarily excluded from the navigation pipeline. This means of forgetting via attention is evidenced by research (Fukuda & Vogel 2009) revealing that the optimal performance of working memory relies on humans' ability focus attention task-relevant information. In Figure 3 visualize the attention scores for the short-term memory (STM) calculated in Dtarget of VGM. The figure shows that the agent assigns high scores to nodes close targets while paying little attention remote ones. This phenomenon indicates that not all node features in the STM are helpful and that it is more efficient to use a small number of them for navigation.
The forgetting module retains fraction of STM according the attention scores [a in D Dearget- These scores denote to what extent the goal embedding étarget attends to each node feature in the STM. After Dtarget finishes decoding. the agent temporarily "forgets" a fraction of nodes whose scores rank below a predefined percentage In other words, these nodes will be disconnected from their neighbors, and not be processed by the navigation pipeline at the subsequent time steps. the agent returns to forgotten node, this node will be added to the map and processed by the pipeline again. multi-goal task, once the agent has reached target, all forgotten nodes will be restored, as these nodes are probably useful for finding the next target.
The forgetting module used in a plug-and-play manner, which means that this module is not activated during training and switched on when evaluating and deploying. p is default to 20%. With this module, the agent can selectively retain the informative STM, while avoiding misleading experiences.
4.2 LONG-TERM MEMORY GENERATION
In addition to STM, the information in the long-term memory (LTM) also forms part of working memory (WM) (Eriesson & Kintsch 1995). Inspired by ETC (Ainslie et al. 2020) and LongFormer
4

---

Under review as a conference paper at ICLR 2023
Map sntertionsoorsss
Map-edge
Goals
Goal
Goal
Gool
Agent posiiio
Figure 3: VGM calculates redundant node features. The figure visualizes the attention scores in
D Diarget for an multi-goal episode of VGM. We can see that only small number of nodes useful for finding a single goal.
(Beltagy et al. 2020). we add a zero-initialized trainable global node 12 giobbal E IR' to the topological map as the LTM (the orange nodes in Figure RecBERT (Hong et al. 2021) adopts a recurrent state token similar to the proposed LTM, but their functions are different. The token in RecBERT aggregates visual-linguistic clues while the LTM in model summarizes the agent's past observations by continuously fusing the STM via memory encoding (the encoder is described in the next subsection).
The LTM possesses two advantages: it learns a scene-level feature and facilitates feature fusing. recent study (Ramakrishnan et al. 2022) ) suggests that embodied agents need to learn higher-level representations of the 3D environment overcome the partial observability caused by limited field-of- view sensors. From this viewpoint, the LTM stores a high-level scene representation by aggregating local node features. Moreover, another merit of the LTM facilitating feature fusing. The topological map is divided into several sub-graphs when forgotten nodes are removed. Consequently, direct message passing between the nodes separated in different sub-graphs no longer exists. The L.TM links every node in the map, acting bypath that facilitates feature fusing between these isolated sub-graphs.
4.3 WORKING MEMORY GENERATION
The third type of scene representation WM learns goal-relevant features that used generate actions. To learn adaptive WM, we utilize graph attention module GATv2 (Brody et al. 2022) 2) to encode the retained STM and the LTM since GATv2 is powerful when different nodes have different rankings of neighbors. GATv2 derives adaptive weights from node features for neighboring nodes, instead of from the Laplacian matrix. This design suitable for generating WM, especially in multi-goal tasks since the STM features that contain information about the target a path leading to target should obtain high weights while those of irrelevant places should receive lower weights. In addition, the node features in the STM are dynamically updated during navigation, as the agent replaces these features with new ones when revisiting these nodes. Therefore, it is more suitable to adaptively change the weights used to aggregate the STM features into the WM.
The WM generation is formulated as: M = {V', n globall } = GATv2((V m)jobat)) where M., represents the generated working memory, and V the encoded STM. {-.-} denotes that the LTM (a vector) is prepended the retained STM (a sequence of vectors). Note that the time step superscript of Niglobal means that the L.TM is recurrent through time.
After GATv2 encoding, the WM aggregates the goal-relevant information the retained STM as well as the scene-level representation in the LTM. Lastly, the decoders Dour and Dtarget take M,. as keys and values, generating feur and targets which are further used to generate actions.
EXPERIMENTS
DATASETS
All experiments conducted in the Habitat (Savva al. 2019) simulator with the Gibson (ia
ct al. 2018) or Matterpot3D (Chang et al. 2017) scene dataset. We adopt the same action space as in
VGM KK won al. 2021).
1-goal dataset. In the Gibson scenes. all models are trained with 72 scenes and evaluated on a public 1 --goal dataset (Mezghani al. 2021) that comprises 14 unseen scenes. Following the setting in
The 1-goal difficulty level here denotes the hard level in the public test dataset
5

---

Under review as a conference paper at ICLR 2023
VGM (Kwon et al. 2021). 1007 out of 1400 episodes in this public dataset are used for the evaluation on I-goal tasks, but the 1400 episodes are still used for ablation studies.
Multi-goal dataset. Multi-goal evaluation is more suitable for evaluating memory mechanisms used by navigation methods. By letting the agent return to visited places we can test whether memory mechanisms help the agent plan short path. If not, the agent will probably waste its time re-exploring the scene or traveling randomly. However, recent ImageNav methods seldom conduct multi-goal -oal evaluations. To further investigate how memory mechanisms assist the agent in navigation, we follow MultiON (Wani et al. 2020) to collect 700-episode multi-goal test datasets in the Gibson scenes (an example in Figure
mulli--gall task, the agent navigates an ordered sequence of goals.
We follow five rules to set sequential goals for each sample: (I) No obstacles appear inside a circle with radius of 0.75 meters centered at each goal. (2) The distance between two successive goals is no more than 10 meters. (3) All goals are placed on the same layer without altitude differences. (4) All goals are reachable from each other. (5) The final goal placed near certain previous one with the distance between them smaller than 1.5 meters. Please refer to A.4.4 for more details.
In the Matterpot3D scenes, we sample 1008 episodes for each difficulty level from the multi-goal test datasets used in Multi-ON (Wani et al., 2020). The difficulty of an episode is indicated by the number of goals. All compared methods and our model are trained on the Gibson 1 I-goal dataset and tested other difficulty levels.
Evaluation Metrics, In I-goal tasks, the success rate (SR) and success weighted by path length (SPL) (Anderson et al., 2018) are used. In multi-goal task, two metrics are borrowed from (Wani et al. 2020): The progress (PR) is the fraction of goals successfully reached, equal to the SR for 1 1-goal tasks; Progress weighted by path length (PPL) indicates navigation efficiency and is defined as
E Progress; 4
PPL
max (p;,4) where E is the total number of test episodes, 4 and pt
the shortest path distance to the final goal via midway ones, and the actual path length taken by the agent, respectively.
5.2 COMPARED METHODS AND TRAINING DETAILS
We compare the MemoNav with previous methods utilizing various types of memory. Random an agent that samples actions from a uniform distribution and navigates with oracle stopping. CNNLSTM (Zhu et al. (017) uses no maps but a hidden vector as implicit memory. ANS (Chaplot metric map-based model adapted for ImageNav in the experiments of VGM (wonn et al. 2021). NTS (Chaplot et al.) (020b) incrementally builds topological map without pre-exploring and adopts a hierarchical navigation strategy. VGM (K won et al.. 202T) (see Section
al.
2020a
3.2) is the baseline the MemoNav is built on. ZER (Al-Halah et al. 2022) is used to handle multiple navigation subtasks SLING (Wasserman et al. 2022) uses mode switch to finish last-mile navigation. OVRL YYadav et al. 2022) introduces offline visual representation learning before training agents.
Following VGM Kwon et al. 202I), for all methods except ANS and NTS, the agent is first trained using imitation learning. minimizing the negative log-likelihood of the ground-truth actions. Next, the agent is finetuned with proximal policy optimization (PPO) (Schulman et al., 2017 to improve the exploratory ability. The reward setting and auxiliary losses remain the same as in VGM. Each model is trained for five runs on a TITAN RTX GPU. The evaluation results for ANS and NTS are borrowed from the VGM paper Kwon al. 2021
5.3 QUANTITATIVE RESULTS
Comparison on Gibson. Table shows that the MemoNav outperforms all the baselines in SR on the four difficulty levels. CNNLSTM exhibits the poorest performance its limited memory provides insufficient information about the goal area. The MemoNav outperforms the metric map-based method ANS which requires pre-b maps, enjoying a large improvement in SR and SPI..
ANS is designed for exploration while NTS is not opensourced, so it is not straightforward to reproduce them for multi-goal tasks.

---

Under review as a conference paper at ICLR 2023
Table 1: Evaluation results on the Gibson (G) and Matterport3D (M) test datasets. We report the averages and standard deviations (in parentheses) by repeating experiments five times. (G*; evaluation on 4200 validation trajectories, SR: success rate (%), SPL: success weighted by path length (%)), PR: progress (%), PPL: progress weighted by path length (%))
Methods
I-goal
2-goal
3-goal
SR
SPI.
PR
PPL.
PR
PPL.
PR
PPL.
Random
17.4 17.4(0.7)
5.0
4.9(0.8)
(.8(o.
3.1 30.3)
0.4(0.3) 2.8(0.2)
1.8(0.2)
0.2
CNNLSIM
21.6(1.2)
14.6(1.1)
14.0(1.7)
9.0(..9)
9.0(1.1)
6.4(0.9)
1.6(0.1)
ANS
30.0/-)
[1.0[-
43.0,-)
26.0(-)
VGM
71.8(1.7)
67.9(2.2)
45.6(2.9)
18.7(1.))
33 33.4(2.8)
8.3(0.8)
25 25.8(1.5)
5.0ro.4)
MeusoNav (ours)
(4.7(1.5)
(8.2(1.4)
47.0(2.1)
18.3..))
(0.8(2.9)
(.0
27.0(1.0)
(.2
ZER (ViewAug)
22.0
18.8
OVRL
41.3
26.9
SLINGHOVRLLGD
37.4
MemoNay (ouns)
Random
14.5(0.5)
5.5(o.3)
8.0
0.P(o.0)
G.3
(.5
CNNT
(6.2(1.8)
9.8
10.8(0.9)
2.6(0.2)
7.7(0.8)
1.4(0.2)
M
24.4(1.1)
16.3(a.7)
(5
4.7(0.3)
12.0
2.8
MemoNav (nur))
(1.0
(6.1
18.2(0.7)
5.2(0.2)
13.3(0.2)
2.8(0.1)
Table 2: Network component ablation results. Row 1 the baseline model VGM (Kwon et al. 202T), and row 7 is our full model. The averages over five runs reported in this table while the standard deviations are placed in (G: the GATv2-based memory encoder, L: the long-term memory, F: forgetting nodes whose attention scores rank below 20%)
Components
I-goal
2-goal
3-geal
4-goal
G
L
SR
SPL
PR
PPL
PR
PPL
PR
PPL
55.8 55.8
17 17.1
45. 15.6(2.9)
18.7(1.5)
33.4(2.5)
8.3 8:
25 25.8
5.0
58.9(1.8)
48.4(1.2)
46.3
18.3(1.0)
35.3/2.1)
8.3
26.5(1.6)
4.7(0.5)
56.8 a)
17.0
45.8(2.9)
18.0
33.7(a.0)
8.0 0:
25.9(a.3)
5.1(0.5)
v
59.4(1.m
48.7(0.5)
46.3
18.8(o.9)
35.4(1.))
8.3
26.8(1.9)
4.7
60.7(1.9)
49.0(1 03
47.3(2.4)
18.9 18.9(1.)
35.8(1.4)
8.6(0.4)
27.5;1 0)
5.1
Compared with the VGM, our model exhibits a noticeable performance gain, increasing the SR by 2.9%, 1.4%. 2.4%, and 1.7% on the 3, and 4-goal tasks respectively while using 20% fewer node features. This result indicates that the MemoNav benefits from the informative scene memory and the high-level scene representation contained in the WM, obtaining a higher success rate.
The MemoNav is also tested using 90-degree FoV sensors instead of panoramas (see G* in Ta- ble ! The evaluation results on 4200 trajectories (G*) used by ZER also show that the MemoNav outperforms ZER, OVRL, and SLING.
Comparison on Matterport3D. We conduct an evaluation on the Matterport3D scenes to the models' ability generalize to other scene types. Table that our method achieves consistent performance improvements on this unseen scene dataset. Compared with VGM, the MemoNav witnesses performance gains in the success rate at the three difficulty levels, improving the SR/PR by 1.6 % 2.4%, and 1.3 % on the 1, 2, and 3-goal tasks, respectively. These results show that our method is more capable of generalizing to new scene styles.
5.4 ABLATION STUDIES AND ANALYSIS
We conduct ablation studies in the Gibson scenes to analyze the impact of each proposed component.
Performance gain of each proposed component. We ablate the three key components described in Section 4 and show the results in Table 2 Comparing row with row 1, we can see that the LTM brings a noticeable improvement in SR/PR over the baseline. Comparing row with row I, applying the forgetting module achieves slight improvements in the SR/PR. However, its cooperation with the LTM witnesses larger increases (row 4 vs. row 1). More importantly, compared with the baseline (row 1), the synergy of the three components (row 5) increases the SR/PR by 0.049 (8.8%) 0.014 (3.1 %)
7

---

Under review as a conference paper at ICLR 2023
Table 3: Ablation study of LTM. Row is our default model. Row shows the impact replacing the LTM with a random feature in the STM. Row 3 is variant that uses LTM to aggregates STM but does not incorporate the LTM into the WM. We report the averages and standard deviations (in parentheses) of five runs.
Methods
2-goal
3-goal
SR
SPL.
PR
PPL.
PR
PPL.
PR
PPI.
MemoNav
(0.7(2.1)
49.0
47.0(2.4)
18.3(..1)
35.8(a.4)
8. 8.6(0.4)
27. 27.5(s.0)
5.1
w/ reglace
39.3
47.1
45.9(2.3)
17.3(1.3)
35.0(1.3)
6.7(0.4)
(6.5(
(.3()
w/o decoding LTM
59.8 6)
48.4
46.0(2.4)
17.5
35.6 35.9
7.9(0.3)
27.0(1.0)
4.8(o.4)
Diferemoess in SPLPAL
3-gaal
P-grei
goal
Differen Bereerea ces in SRIPR (15)
Diforemoes ang. stops
20
15
2egna
3-goa
1:00
1.5
10
1.0
0.5
0.0
3
2gg0
'goa
-3.5
3ggal
-1.0
-10
Forgetting threshold
Forgetting
Forgetting threshold
(a) SPL/PPL vs.
(b) SR/PR vs. p
(c) Average steps vs. p
Figure 4: Navigation performance versus forgetting threshold (Averaged over five runs)
0.024 (7.2%). 0.017 (6.6%) at the 1, 2, 3 and 4-goal levels respectively. These results demonstrate that the three components help to solve long-ting navigation tasks with multiple sequential goals.
Importance of LTM. Table presents the results of ablation experiments for the LTM (described in Section 4.2). The first ablation experiment (row 2) replaces the LTM feature with a randomly selected STM feature each time the GATv2 encoding is finished, so that the MemoNav cannot use the scene-level representation in the LTM. Row shows that the navigation performance becomes worse at all difficulty levels compared with the full model (row 1). Furthermore, when the LTM feature is not incorporated into the WM (row 3), the performance also deteriorates. In summary, the LTM stores scene-level feature essential for improving the success rate.
Correlation between navigation performance and forgetting threshold. We evaluate our model with different forgetting thresholds p (see Section 4.1). The results shown in Figure D For clarity, the figure shows the performance differences between our full model (with the forgetting module, the LTM, and the GATv2 encoder) and the variants. The four levels exhibit different trends. When p increases (i.c. a larger fraction of STM is not incorporated into the WM), our model witnesses first increases and then drops in the SR/PR and SPL/PPL the 3 and 4-goal levels while enjoying slight gains in these metrics at the 1 and 2-goal levels. At the 1-goal level, our model obtains the highest SR and SPL when p = 80%, which means that a large fraction of node features in the map are useless when the navigation task is easy. A similar trend can also be seen at the 2-goal level. In contrast, our model exhibits an increase in the PR and PPL at the 4-goal level when p rises from 0% to 40%. However, if p rises to 80%, the metrics 20 steps to complete the tasks. These results suggest that excluding redundant STM from the WM improves navigation performance in multi-goal -oal tasks. However, excluding excessive fraction forces the agent not to utilize what it has explored, thus undermining the navigation performance.
precipitous decline, and the agent takes more than
5.5 VISUALIIZATION RESULTS
To observe how the MemoNav improves the navigation performance, we show example episodes of CNNLSTM, VGM and our model in the Gibson scenes in Fig (See Appendix for more). We can that CNNLSTM takes numerous steps explore and tends to fail in complex scenes. VGM tends to spend a large proportion of time going in circles in narrow pathways. In contrast, the trajectories of the MemoNav are shorter and smoother. For instance, the 2nd and 3rd columns in Figure show
8

---

Under review as a conference paper at ICLR 2023
243 steps
72 stops
309 steps (failed)
500 steps (faree)
VGM
89 steps
500 sleps (lelld)
450 sleps
347 steps
Trajectory
MemoNav
Time
Goals
stops
steps
steps
steps
1-goal
2-goel
3- goal
ggaal
Figure 5: Visualization of example episodes. compare selected episodes of CNNLSTM, VGM, and MemoNav at four difficulty levels in the Gibson scenes and visualize the top-down views. The number of navigation steps (the upper limit is 500) are shown at the bottom of each top-down view. Best viewed in color.
Lggoal
2:goal
égooo
0.35
6.173
0.6
Herohar
0:10
HamcHus
0.150
30.150
2.3775
Narchan
0.325
g0.20
0.100
0.3
0.15
0.675
0.055
0.10
0.080
5.000
811
0.05
0:625
0.22
Nommtzed path length
Normalzeed path length
Nermwiieed path length
Norma Lond patth length
Figure 6: Histograms of normalized path lengths of successful episodes for VGM (Kwon et al. 2021) and our model (averaged 5 runs). Our model has more short trajectories while VGM has a larger proportion of long episodes.
that the VGM agent is trapped in a bottleneck connecting two rooms, while our agent plans efficient routes. This comparison shows that the MemoNav is more capable of escaping from deadlock.
next draw a histogram the normalized path lengths of successful episodes for VGM and our model to further investigate to what degree our model escapes from deadlock. The normalized path length is calculated as horr 4// max(p,, 4) where p; and 4 are defined in Section 5.1 horm indicates how many extra steps the agent takes compared the shortest path. A larger hoorm represents a less efficient navigation episode. The histogram shown in Figure The figure shows that the histogram of our model exhibits less heavier-tailes' distribution, especially at the 4-goal level. This distribution means that our model has fewer episodes with an extremely large number of steps and agrees with the qualitative comparison.
CONCLUSION AND FUTURE WORK
This paper proposes MemoNav, novel memory model for ImageNav. This model flexibly retains the informative fraction of the short-term navigation memory via forgetting module. We also introduce an extra global node as long-term memory to learn scene-level representation. The retained short-term memory and the long-term memory are encoded by graph attention module to generate the working memory that used for generating action. The experimental results show that the MemoNav outperforms the baselines in multi-gos tasks, exhibits better transferability, and is more capable of escaping from deadlock. For future work, we will try to design trainable forgetting modules better assess which fraction of the agent's memory informative.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
We provide our source code and model configuration files in the supplementary material. We also add the python script used for generating our multi-goal Gibson datasets to the supplementary material. Any detail about data processing can be found in this script. For example, the rules used generate multi-goal samples (described in Section 5.1 are coded in the script.
LICENSES FOR REFERENCED DATASETS
Gibson:
ICS. pdf
REFERENCES
Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. Etc: Encoding long and structured inputs in transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 284, 2020.
Ziad Al-Halah, Santhosh Kumar Ramakrishnan, and Kristen Grauman. Zero experience required: Plug & play modular transfer learning for semantic visual navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp- 17031-17041, 2022.
Peter Anderson, Angel Chang. Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen
Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, et al. On evaluation of embodied navigation agents. arXiv preprint 1rX 2018.
Alan Baddeley. Working memory: theories, models, and controversies. Annual review of psychology, 63:1-29, 2012.
Edward Beeching, Jilles Dibangoye, Olivier Simonin, and Christian Wolf. Learning to plan with uncertain topological maps. In ECCV 262h-16th European Conference Computer Vision, pp. 1-24, 2020.
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint 00X0:20004005500, 2020.
Kara J Blacker, Steven M Weisberg, Nora S Newcombe, and Susan M Courtney. Keeping track of where we are: Spatial working memory navigation. Visual Cognition, 25(7-8):691-702, 2017.
Shaked Brody, Uri Alon, and Eran Yahav. How attentive are graph attention networks? In Interna- tional Conference on Learning Representations, 2022. URL
net/
FForumidd=F2xxmmsxx
Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niebner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. 20/7 International Conference on 3D Vision (3DV), pp. 667-676. IEFE, 2017.
Devendra Singh Chaplot, Dhiraj Gandhi, Saurabh Gupta, Abhinav Gupta, and Ruslan Salakhutdi- nov. Learning to explore using active neural slam. In International Conference on Learning Representations (ICLR), 2020a.
Devendra Singh Chaplot, Ruslan Salakhutdinov, Abhinav Gupta, and Saurabh Gupta. Neural topological slam for visual navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12875-12884, 2020b.
Kevin Chen, Junshen K Chen, Jo Chuang, Marynel Vázquez, and Silvio Savarese. Topological planning with transformers for vidion-and-languagee navigation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 11276-11286, 2021.
Tao Chen, Saurabh Gupta, and Abhinav Gupta. Learning exploration policies for navigation. In International Conference on Learning Representations, 2019. URL het:/pdffi:=SyMnnnl
10

---

Under review as a conference paper at ICLR 2023
Nelson Cowan. What the differences between long-term, short-term, and working memory? Progress in brain research, 169:323-333 -388, 2008.
K Anders Ericsson and Walter Kintsch. Long-tern working memory. Psychological review, 102(2): 211. 1995.
Kuan Fang, Alexander Toshev, Li Fei-Fei, and Silvio Savarese. Scene memory transformer for embodied agents in ong-horizon tasks. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pp. 538-547, 2019.
Keisuke Fukuda and Edward K Vogel. Human variation in overriding attentional capture. Journal of Neuroscience, 29(27):8726-87: 2009.
Yicong Hong. Qi Wu, Yuankai Qi, Cristian Rodriguez-Opazzo and Stephen Gould. VIn bert: A recurrent vision-and-language bert for navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1643-1653, 2021.
Obin Kwon, Nuri Kim, Yunho Choi, Hwiyeon Yoo, Jeongho Park, and Songhwai Oh. Visual graph memory with unsupervised representation for visual navigation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 15890-15899, 2021.
Andrew Lampinen, Stephanie Chan, Andrea Banino, and Felix Hill. Towards mental time travel: a hierarchical memory for reinforcement learning agents. Advances Neural Information Processing Systems, 34, 2021.
Ricky Loynd, Roland Fernandez, Asli Celikyilmaz, Adith Swaminathan, and Matthew Hausknecht. Working memory graphs. In International Conference on Machine Learning, pp. 6404-6414. PMLR, 2020.
Arjun Majumdar, Gunnar A Sigurdsson, Robinson Piramuthu, Jesse Thomason, Dhruv Batra, and Gaurav S Sukhatme. Ssl enables learning from sparse rewards in image-goal navigation. In International Conference Machine Learning. pp. 14774-14785. PMLR, 2022.
Lina Mezghani, Sainbayar Sukhbaatar, Thibaut Lavril, Oleksandr Maksymets, Dhruv Batra, Piotr Bojanowski, and Karteek Alahari. eeemory-augmente reinforcement learning for image-goal navigation. arXiv preprint arXiv:2101.05181, 2021.
Emilio Parisotto and Ruslan Salakhutdinov. Neural map: Structured memory for deep reinforcement learning. International Conference on Learning Representations, 2018.
Alexander Pashevich, Cordelia Schmid, and Chen Sun. Episodic transformer for vision-and aaiion-an--anggage navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15942-15952, 2021.
Santhosh Kumar Ramakrishnan, Tushar Nagarajan, Ziad Al-Halah, and Kristen Grauman. En- viromment predictive coding for visual navigation. In International Conference Learning Representations, 2022. URL
Samuel Ritter, Ryan Faulkner, Laurent Sartran, Adam Santoro, Matthew Botvinick, and David Raposo. Rapid task-solving in novel environments. International Conference on Learning Representations, 2021. URL httph::://// http///pppeeeeeeeee////ee/ee wwt. wtwpeeerrevieeeeeeeeeeeeeeeee
Nikolay Savinov, Alexey Dosovitskiy, and Vladlen Koltun. Semi-parametric topological memory for navigation. In International Conference Learning Representations, 2018.
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9347, 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (ooo::00700774 2017.

---

Under review as a conference paper at ICLR 2023
Sainbayar Sukhbaatar, Da Ju, Spencer Poff, Stephen Roller, Arthur Szlam, Jason Weston, and Angela Fan. Not all memories are created equal: Learning to forget by expiring. In International Conference Machine Learning, pp.
PMLR, 2021.
Endel Tulving. Memory and consciousness. Canadian Psychology/Pxyycoooogie canadienne, 26(1):1, 1985.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Saim Wani, Shivansh Patel, Unnat Jain, Angel Chang, and Manolis Savva. Multion: Benchmarking semantic map memory using multi-object navigation, Advances in Neural Information Processing Systems, 33:9700-977 97100 2020.
Justin Wasserman, Kanmesh Yadav, Girish Chowdhary, Abhinav Gupta, and Unnat Jain. Last-mile embodied visual navigation. 6th Annual Conference on Robot Learning, 2022.
Fei Xia, Amir R Zamir, Zhiyang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: Real-world perception for embodied agents. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 9068-9079, 2018.
Karmesh Yadav, Ram Ramrakhya, Arjun Majumdar, Vincent-Pierre Berges, Sachit Kuhar, Dhruv Batra, Alexei Baevski, and Oleksandr Maksymets. Offline visual representation learning for embodied navigation. arXiv preprint 1xXXi:2204.13226 2022.
Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In 2017 IEEE international conference on robotics and automation (ICRA). pp. 3357-3364. IEEE, 2017.
A
APPENDIX
A.1 LIMITATIONS
While the MemoNav witnesses a large improvement the navigation success rate multi-goal navigation tasks, still encounters limitations. The proposed forgetting module is post-processing method, as it obtains the attention scores of the decoder before deciding which nodes are to be forgotten. Future work can explore trainable forgetting modules. The second limitation that our forgetting module does not reduce memory footprint, since the features of the forgotten nodes still exist in the map for localization. Moreover, the forgetting threshold in our experiments fixed. Future work can merge our idea with Expire-span (Sukhbaatar et al. 2021 to learn an adaptive forgetting threshold.
A.2 POTENTIAL IMPACT
The notable potential of negative societal impact from this work: model is trained on 3D scans of the Gibson scenes which only contain western styles. This inadequacy of diverse scene styles may render our model biased and incompatible with indoor environments in unseen styles. As result, our model may be only available in a small fraction of real-life scenes. If our model is transferred to out-of-distribution scenes, the agent may take more steps and even bump on walls frequently.
A.3 REPRESENTATIVE MODELS OF WORKING MEMORY HUMAN BRAIN
proposed typical model describing the relationships among long- term memory (LTM), short-term memory (STM), and working memory (WM) of the human brain. According to their definitions, LTM is a large knowledge base and a record of prior experience; STM reflects faculties of the human mind that hold a limited amount of information in a very accessible state temporarily; WM includes part of STM and other processing mechanisms that help to utilize STM. Cowan et al. designed a framework depicting how WM formed from STM and LTM (shown in Figure 7). This framework demonstrates that STM is derived from temporarily activated subset
Cowan et al. (Cowan 2008)
12

---

Under review as a conference paper at ICLR 2023
Contral executive (controlied attention) processes
LTM
STM: Activated portion memory
(decay and feature interference)
WM
(chunk capacity limits)
Figure 7: The memory model by Cowan al. (Cowan 2008 This figure is borrowed and adapted from its original paper.
of LTM. This activated subset may decay function of time unless it is refreshed. The useful fraction of STM incorporated into WM via an attention mechanism avoid misleading distractions. Subsequent work by Baddeley et al. (Baddeley 2012) suggests that the central executive manipulates memory by incorporating not only part of STM but also part of LTM to assist in making a decision.
We draw inspiration from the work by Cowan et al. (Cowan 2008 and Baddeley al. (Baddeley) 2012) and reformulate the agent's navigation experience as the three types of memory defined above.
The parallel between the MemoNav and the two relevant models above is shown in the following list:
The map node features are termed "STM", since they are local and transient.
The topological map of the MemoNav maintains 100-node queue to store map nodes. This design simulates STM that holds limited amount of information in very accessible state temporarily in the human brain.
The MemoNav introduces global node aggregating prior observation features stored in the topological map, thereby simulating LTM which acts as large knowledge base.
The MemoNav utilizes a forgetting mechanism remove fraction of STM with attention scores lower than a threshold. This mechanism acts as a simple way of decaying STM.
The forgetting mechanism helps WM include part of STM.
The MemoNav incorporates the retained STM and the LTM into WM, which is subsequently used to generate navigation actions. This design simulates the working memory model by (Baddeley] 2012).
A.4 IMPLEMENTATION DETAILS
A.4.1 IMPLEMENTATION OF MEMONAV
The structure of the memory decoding module (Figure f) in the main paper) in the MemoNav remains the same as in the VGM (Kwon et al. 202 and shown in Figure [ We maintain the module hyper-parameter specified in the supplementary of the VGM paper. The forgetting module on the MemoNav requires the attention scores generated in the decoder Dtarget- Therefore, our model needs to calculate the whole navigation pipeline before deciding which fraction of the STM should be retained. This lag means that the retained STM is incorporated into the WM at the next time step. The pseudo-code of the MemoNav is shown in Algorithm
13

---

Under review as a conference paper at ICLR 2023
Attention scores
M
Keys
values
Turget
To the policy
Diarget
Query
network
faw
Deur
Ceur
Query
(f) Memory decoders
Figure 8: The structure of the memory decoding module displayed in Figure 2 (f) of the main paper. M denotes the working memory. The attention scores a generated in Dtarget are used in the forgetting module to retain informative STM.
Algorithm 1: The implementation of the MemoNav
Data: Empty topological map G {V. 8) target image harget. current time step f. forgetting percentage p. trainable observation encoder F Fencs GATv2-based encoder GATv2, Transformer decoders Dtarget and Dear, LSTM-based policy network L.TT
Result: Navigation action
Long-term memory Teglobal + E Rd:
Attention scores for graph nodes V: a + 0E
while not AgentCallStop () do
Step 1: Update the topological map I + ManCuurrenPPaoraamae
G. Uppaaeeaap(());
// Step 2: Retain the informative fraction of the STM Forgotten number n Floor (p-|V));
Sorted indices i +- Argsort(a)) (a:);
Forgotten indices forgotenn < i []:
G. ReRnoveNoees(A forgooten))
// Step 3: Memory encoding and decoding
V E R Vixd t- G. GedNodeeeatuuess ();
10
11
Working memory M + GATvv((V, Nigibal)
(V,,ihha 1);
Cour f- (I),, Etarget f- Feno (harget))
13
Seur + Dour (Eeur, Mw), ) Starget +- Drarget (Etarges Mw);
x Dtarget. GetAttScores()
// Step 4: Act ion generation
t LSTM(FC([F frarget ecurl));
(a x) (FC(()))
16
a: Prnssrrrrorsss
x));
end
14

---

Under review as a conference paper at ICLR 2023
MemaNav
8-
VGM
7-
Reward
5-
4-
3-
2
10
20
Frames [million]
Figure 9: Reward curves of the MemoNav and VGM during the RL training phase.
A.4.2 REPRODUCTION OF CNNLSTM
We reproduce CNNLSTM (Zhu et al. 2017) following the description in its original paper, but we also make some modifications to keep the comparison fair. We replace the ResNet-50 in CNNLSTM with the pretrained RGB-D encoder of VGM (Kwon et al. 2021). We also add positional embeddings to the encoded RGB-D observations to contain temporal information. Moreover, we concatenate the encoded RGB-D observations with the goal image embedding and project the concatenated feature (1024D) to a 512D feature, so that CNNLSTM can utilize the information of the goal image. The projected features of four consecutive frames are further condensed and then input as in (Zhu et al., 2017). To use the two auxiliary tasks proposed in VGM (Kwon et al.) 2021). we also introduce the linear projection layers (Linear-Red Leun--ReL--Linna used in VGM to process the embedded goal image and embedded current observation. With these modifications, the comparison between CNNLSTM and MemoNav (ours) is fair.
policy network
A.4.3 TRAINING DETAILS
We follow the two-step training routine and maintain the training hyper-parameters in the VGM paper (Kwon et al. 2021). Firstly, CNNLSTM, VGM, and MemoNav (ours) are all trained using imitation learning for 20k steps. Afterward, we finetune these models using PPO (Schulman et al. 2017) for 10M steps. Due to the performance fluctuation intrinsic reinforcement learning, the model the 10M-th step is probably not the best. Therefore, we evaluate all checkpoints in the step range [9M, 10.4M] and select the best one.
A recent study (adav et al. 2022 has pointed out that training via PPO does not converge till 500M frames and results at 1OM are highly sensitive to initialization. For better comparing the MemoNav and VGM, these two methods are trained for 10M more steps. The reward curves during the training phase shown in Figure S This figure shows that the reward gain obtained by MemoNav becomes larger when the number of training steps increases from 1OM to 20M.
A.4.4 EXTRA INFORMATION ABOUT THE MULTI-GOAL DATASETS
We follow the format of the public 1 1-goal dataset in (Mezghani al. 2021 and create 2-goal, 3goal, and 4- 4-goal test datasets on the Gibson (Xia et al. 2018) scenes. We generate 50 samples for each of the 14 test scenes. In each sample, we randomly choose target positions while still following five rules: (1) no obstacles appear inside circle with a radius of 0.75 meters centered at each target; (2) the distance between two successive targets is no more than 10 meters; (3) all targets are placed on the same layer without altitude differences. (4) all targets
reachable from each other. (5) The final target is placed near a certain previous one with the distance between them smaller than 1.5 meters. The distributions of the total geodesic distances for the three difficulty levels shown in Figure 10
15

---

Under review as a conference paper at ICLR 2023
Asg gaodenc distanee 14.30m
Aug geocenc ditance: 04m
Av receesic ostance 32 30m
3-gow
4-gowl
Runter of traectattss
100
Rumser of Exaectonio
13
25.
5/
755
10.0
1255
15.0
37.5
20. 253
10
15
20
25
10
20
30
40
50
Figure 10: Histograms of geodesic distances for the multi-goal test datasets.
A.4.5 COMPUTE REQUIREMENTS
We utilize an RTX TITAN GPU for training and evaluating our models. The imitation learning phase takes 1.5 days train while the reinforcement learning takes days.
The computation in the GATv2- proportion of the run-time of the MemoNav. The computation complexity of the encoder and the decoders are O(V + [E|d) and O d), respectively. Using the forgetting module with percentage threshold p. the computation complexity of the MemoNav can be flexibly decreased by reducing the number of nodes to p|V/.
encoder and the two Transformer decoders occupy the largest
A.5 EXTRA ABLATION STUDIES
We conduct additional ablation experiments the forgetting module in the MemoNav further investigate how its design affects the agent's ' navigation performance. The results are shown in Table
Origin of attention scores used by forgetting module. The forgetting module in the MemoNav removes the uninformative STM according the attention scores generated in D Dearget, as described in Section 1. We change these scores to those generated in Deur- The result (row 2) shows that using the attention scores generated in Deur leads slightly worse performance.
We also remove STM according to the average of the scores generated in Deur and those in D Dtarget- The result (row 3) demonstrates that this variant obtains lower SR and SPL in the I-goal setting but larger PR and PPL in harder tasks (i.e., 3 and 4-goal tasks). We hypothesize that this phenomenon occurs because averaging scores can include context information close to the agent, and not just the goal. Easy tasks require less memory, including too many scene features around the agent leads to frequent changes in the output actions, thereby undermining the performance in and 2-goal settings. In contrast, harder tasks require the agent to utilize its context information to plan short path to visited areas, so retaining memory informative for both the current position and the goal benefits navigation performance.
Effectiveness of forgetting module. This ablation investigates whether it is effective to exclude the STM with attention scores ranking below the predefined percentage p. "Effectiveness" here means that the forgetting module does retain useful node features. In this experiment, the forgetting module excludes a random fraction of the STM. We test this ablation model over five random seeds and report the average metrics. The result (row 3) shows that incorporating random fraction of the STM into the WM leads to decreases in all metrics, which validates the effectiveness of our design for retaining informative STM.
To better understand how the forgetting module works, we conduct an additional statistical experiment. In this experiment, five distance metrics are calculated: (a) distance from node to the agent, (b) distance from a node to the goal, (c) distance from node to the oracle shortest path, (d) distance from node to the shortest path segments closer to the agent, and (e) distance from a node to the
shortest path segments closer to the current goal. Then the histograms of these five metrics are drawn according to the metrics records for each forgotten/retained node at each time step that how far away these nodes are from the agent, the goal, and the shortest path. Please see Figure to better understand the definitions of distance metrics (c)(d)(e).
16

---

Under review as a conference paper at ICLR 2023
fut Fath segment dose the agent
by Fath segment dese the
Map node
Map edge
Goel
Starting location
Shortest path
Agent position
Distance
Distance from node to the shortest path
Distance
Figure 11: The visualization of distance metrics (c), (d), and (e).
Table
Extra ablation studies of the forgetting module.
I-gual
2-gual
3-goul
4-goal
Methods
SPL
PPL
PPL
PR
PPL
MemoNav
60.7 1)
49.0(1.9)
47.0(2.4)
18.3(1.1)
35.8(1.4)
8.6(0.4)
27.5(1.0)
Wr Peur sil. scorss
60.4(1.4)
48. ((1.4)
46 (1.8)
18. (8.0)
35. T(1.5)
8.0.0)
28.4 4(1.4)
5.2(0.2)
averaged
60.1 6)
48.4c0.6)
46.9(1.2)
18.4(0.5)
36.7(1.2)
8.8(0.4)
28.4(1.5)
5.4(0.2)
w. Random STM
59.7(2.0)
44.6
44.9(1.3)
15.3
33. 33.1
(.1
25.7
4.8(0.2)
We evaluate the MemoNav on the 3-goal Gibson task and draw the histograms on per-goal basis with the average results of five different seeds, as shown in Figure The figure provides two interesting findings:
The distance distribution patterns for forgotten nodes (green) and retained ones (orange) vary across goals. The distributions of the distances from forgotten nodes to goals (column 2) and to shortest path segments near goal (column 5) become uniform when the agent navigates to the third goal. In contrast, these two histograms for the retained nodes become sharper and the peaks shift to smaller distance values. This phenomenon occurs because the forgetting module selectively retains map nodes closer to and informative for the current goal.
The forgetting module has a larger impact on the distance metrics when the navigation task becomes more difficult. Specifically, when the current goal index is 1 (i.e. the task is easy), the averages of the distance metrics for forgotten nodes and retained nodes are close. When the goal index rises to 3 (i.e. the task becomes harder), larger proportion of the retained nodes are close to the goal, the shortest path, and the shortest path segments near goal while a larger proportion of the forgotten nodes are close to the agent, and the shortest path segments near agent. This difference suggests that MemoNav improves navigation performance by focusing more on the region along the shortest path and around the goal area.
These results empirically validate that the information useful for path planning are not totally lost by the forgetting module.
Extra Validation of LTM. In order observe how the learned scene-level representation in the LTM benefits the MemoNav, we replace the LTM feature with the average of all STM features. After replac- ing, the LTM no longer contains scene-level representation, but it still facilitates message passing. The results in Table shows that the MemoNav without the scene-level representation in the LTM witnesses drops in all metrics. This experiment suggests that the learned scene-level representation in the LTTM contains important information that helps to improve navigation performance.
A.6 EXTRA VISUALIZATION RESULTS
More examples of multi-goal episodes are displayed in Figure [3(a] The agent efficiently explores the scenes and finds sequential goals using the informative fraction of the node features in the map.
17

---

Under review as a conference paper at ICLR 2023
0.9
6.00
Properition
6.2
630
6:00
12
to apert
Dac goal
shortest path
to shortent agert
Dat: storten path near gost
031
Prgperton
Projectien
Proponttin
Proportion
615
0:10-
600
4.4 6 U
2.4.5 6 2011
Goal 1
Des to ageet
Diet to gool
Diet. to thorter path
Dist to thottent pith newr sgent
Det. shorteet path neer
3.35
0.29
Proertton
Plopetdon
Prosittion
Propertion
0.3
0:2
0:30
8:0
0.5
1 12
1811
Dst. to agget
Dst. lo gool
Out. to shartest path
Dit. to shotttt puth near agent
Ditt 105.11 shertest path near goal
3.5
0.9-
Propertion
9.3
0.3
3.00
68s-
6:35
4
Distance
Distance
Goal 2
Dist to agent
Didt to gool
Dict to shortest path
06: to shortest path near agent
Ditt shartest path near nearges!
0.55
035
0.5
Poppationn
Propettion
Propurtinn
Propertion
8.15
6:39
6:00
6 t
1 12234:1 13
12344677 10l
Disiance (m)
[m]
Distance [m]
Distance
Dislance
Osi. agent
Did to goal
Ost. shortest path
09c shortss Hear agent
D6A startesi pafi goll
Ioppotion
8:00
Phiportto EAA
8.70
Proportt om
0.8
Proporttion
0.39
0:30
410
8.10
222
0.33
6:30
600
800
0.0
3
Distacee (m)
Distance (m)
Dstance [m]
Distance (m)
Distance (m)
Goal 3
Figure 12: Histograms of the five distance metrics defined in Section A.5 The data of these metrics collected by evaluating the MemoNav on the 3-goal task in the Gibson scenes and averaged over five runs. The upper row (green) and lower row (orange) belong to the forgotten nodes and retained ones, respectively.
18

---

Under review as a conference paper at ICLR 2023
Table 5: Extra ablation studies of the L.T.
Methods
gggall
2-goal
3-goal
4-goal
PPI.
MemoNav
(2.7(2.1) 58.1(1.0)
19.0
(2.44
((.1)
(1.4)
8.6
((L.)))))
(.1
averaging STM LIM
47.2(1.)
45.1(1.0)
17.9(0.8)
34.4(1.3)
8.1(3.3)
26. 26.4(0.8)
4.9(0.2)
These examples show that the MemoNav agent focuses high attention only a small fraction of nodes and excludes nodes that are away from the current goal. For example, in the 3-goal example, the agent forgets the topmost node when navigating the 1st goal since this node is the farthest from the goal; the agent forgets the nodes at the bottom left corner when navigating to the 2nd and 3rd goals since these nodes remote and uninformative. The comparison with the baseline for these examples is recorded in the supplementary videos.
Failure analysis. We present examples of failed episodes in Figure 13(b) and record the proportions of various failure modes at all difficulty levels. The failure modes can mainly be categorized into four types: Stopping mistakenly, Missing the goal, Not close enough, and Over- exploring. The mode Stopping mistakenly means that the agent implements stop at the wrong place. The mode Missing the goal means that the agent has observed the goal but passes The mode Nor close enough means that the agent attempts to reach the goal it sees but implements stop outside the successful range. The mode Over-exploring means that the agent spends too much time exploring open areas without any goals. The largest probability lies in Over-exploring cases, most of which occur when the agent explores a large proportion of the scene but still fails to get close to the target area in limited time.
Visualization comparison between MemoNav with and without forgetting. We compare the trajectories of the MemoNav with and without the forgetting module to observe how this module affects the characteristics (c.g. smoothness and length) of the trajectories. Figure [4]shows that the trajectories of MemoNav using the forgetting module are smoother and shorter. Without the forgetting module, the trajectories tends to possess many tiny sharp turns and are longer. We hypothesize that a fraction of the STM is uninformative for navigation. This fraction may acts as noise and make the policy network frequently change its output actions. The forgetting module adaptively excludes this disturbing fraction of STM so that the policy network is able helpful navigation memory for decision-maln making....
19

---

Under review as a conference paper at ICLR 2023
Map nodes with attantion scores
Map edge
Forgotten nodes
Goals
Agent position
Trajectory
Goal
Goal
t-0
T+24
t-65
1=109
Tw 34
Goal 2
Goal
to 1.9
2
212
249
Current Observation
Goal
Goal
I+O
c-+41
t-103
1-767
I- 1-199
2-215
Stopping mistakenly
Missing the goal
Not close enough
Over-eclorrng
Current Observation
Current Observation
Current Observation
Gowl Observation
Goal Observation
Gool Observation
(b)
Figure 13: (a) Examples of multi-goal navigation episodes. Each example shows both the topological map and the trajectory. The graph nodes are incrementally added to the map by the agent and selectively retained by the forgetting module in the MemoNav. The yellow downward arrow denotes the current localized node of the agent. (b) Examples of failed episodes. The agent encounters four major failure mode: Stopping mistakenly, Missing the goal, Not close enough, and Over-exploring.
20

---

Under review as a conference paper at ICLR 2023
MemoNow
torgetiig
237 steps
178
274 steps
Memohaa
Trajectory
Goals Starting location
steps
steps
steps
steps
1-geal
2 goal
3ogl
4 goal
Figure 14: Visualization comparing the MemoNav with and without the forgetting module. We compare selected episodes at four difficulty levels in the Gibson scenes and visualize the top-down views. The number of navigation steps (the upper limit is 500) are shown at the bottom of each top-down view. Best viewed in color.
21


Reviews:
[{"review_id": "qihEoEg9aR", "reviewer": "Reviewer_EedW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper builds upon the VGM (Kwon et al., 2021) for image navigation. By forgetting some less informative nodes in the topological map through the attention mechanism, the short-term memory (STM) can be updated dynamically. The selected STMs are fused to obtain the scene-level feature with Long-term memory (LTM), and which are encoded through graph attention to generate the Working Memory (WM). The WM is finally used to generate the actions for navigation. The experiments of both 1-goal and multi-goal tasks are conducted in two public datasets Gibson and Matterpot3D in Habitat simulator.", "summary_of_the_review": "I think this is a practical work, the motivation and idea are interesting, and the experimental results are also sufficient. However, I worry about the that novelty of the technical contribution over baseline (VGM) seems limited. And compared to the results of VGM, the improvement of the proposed method seems marginal in some parts. ", "strength_and_weaknesses": "Strengths:\n1.\tThe paper is generally well-written and easy to follow.\n2.\tThe experiments and analysis are extensive with the ablation experiments and baselines.\n3.\tThe selective forgetting module seems capable of retaining the informative nodes in the topological map and forget the redundant nodes, which may be helpful for improving the SR/PR for 1-goal and multi-goal tasks.\n\nWeaknesses:\n1.\tThe MemoNav seems to be trained in the 1-goal setting, while used in both 1-goal and multi-goal tasks, is it suitable to use this model to evaluate the multi-goal task? It is worth discussing training different models (under multi-goal settings) to validate for multi-goal tasks.\n2.\tThe proposed MemoNav follows the VGM method of constructing the topological maps of the environments. Considering the topological maps can somehow memorize the observations during navigation as well, the advantages of proposed memory based modules are not that effective compared to the baseline, also the close SPL/PPL results are obtained compared the baseline VGM under 1-goal and multi-goal tasks. \n3.\tIt seems the regularity and relevance are not that significant in Figure 4. The 2-goal task shows a sharp drop in PP at p=0.4, and a dramatic increase at p=0.6, what are the reasons for this phenomenon?\n \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "pcEr8QG6Ca", "reviewer": "Reviewer_aALd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an image-goal navigation (ImageNav) method on top of VGM (Kwon et al. 2021) which builds a topological graph. The proposed method, MemoNav considers VGM as the short-term memory (STM) and removes some nodes based on thresholding on attention score (selective forgetting). The modified STMs are fed into the GATv2 encoder followed by a decoder. The final output of the decoder is used as an input of the policy network. The authors tested MemoNav in Gibson and Matterport3D environments and showed that the proposed method improves performance.\n", "summary_of_the_review": "As mentioned in above, the idea and achievement are good but the further analysis of experimental results is needed and the novelty is somewhat limited.\n\n- Post Rebuttal - \nAlthough I appreciate the authors dealt with my concerns, I agree with other reviewers' concerns and G+L+F is not significantly different from G+L. Hence, I maintain the original rating.", "strength_and_weaknesses": "**[Strength]**\n\n1. The paper is well-written and easy to understand.\n\n2. The authors include the failure trajectories in Figure 10 for a better understanding of the behaviors of the proposed method. \n\n\n**[Weakness]**\n\n1. Basically, the proposed MemoNav is a combination of existing methods, VGM (topological mapping, decoder) + forgetting module + GATv2 (across time). Although I do not degrade the contribution of merging together and achieving good performance, the novelty is somewhat limited. The authors argue that the forgetting module is novel in Section 2. However, attention-based dropout already exists [Ref 1].\n\n2. According to Table 2 (1-3, 2-4), the forgetting module is not that helpful and even degrades the performance in terms of SPL/PPL. The authors should specify the reason why the performance s decreased. Moreover, considering Table 3, I wonder effectiveness of using the forgetting module. I read Section A.5 but it compares with random forgetting, not the effectiveness of forgetting itself. One possible explanation might be to reduce the computation by removing the node. The authors briefly mentioned that the proposed method employs 20% fewer node features. However, further explanation is needed.\n\n3. In Figure 4, the performance is consistently decreasing in 3-goal and 4-goal settings as p grows. The authors argue that “excluding an excessive fraction forces the agent not to utilize what it has explored.” This statement should be also applicable to the smaller number of goals tasks. However, the performance is getting better in 1-goal and 2-goal tasks. The authors should analyze this trend more clearly.\n\n\n**[minor]**\n\n1. In Section 5.3, please unify the notation. The second paragraph uses absolute difference while the third paragraph uses relational difference.\n\n2. There is one paper [Ref 2] that achieves better performance than the proposed method but I will not count on this since it is arxiv paper.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "K3LVYFEWcjS", "reviewer": "Reviewer_LEke", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel graph-memory network for visual navigation. Inspired by working memory from cog-sci literature, the proposed method consists of short-term, long-term and working memories to perform the ImageNav task. A forgetting module is proposed to prune the goal-irrelevant parts of the short-term memory. Results are demonstrated on Gibson and Matterport3D datasets.", "summary_of_the_review": "The idea is interesting and novel, the paper is clearly written, and the experiments are well designed. I have concerns about the design of the forgetting module, insufficient accounting for statistical significance, unfair comparison to VGM and missing state-of-the-art baselines for ImageNav. I also have some minor concerns which I hope that authors should be able to address easily.", "strength_and_weaknesses": "\n# Strengths\n* The idea is interesting and novel to the best of my knowledge. \n* The clarity of the paper is good and the paper is easy to understand.\n* The experiments have been well designed, especially the ablations and subsequent analyses. \n\n\n# Major weaknesses\n## The design of selective forgetting module is suboptimal\n* The module deletes nodes which are not scored highly by the target decoder. But the graph also contains other valuable information which are not directly related to the goal: (1) the context around the agent's current position in the environment, and (2) the graph connectivity information from agent's current position to a node close to the target --- useful for path planning. This information is lost when selective forgetting module is used.\n* The design of the LTM module appears to be focused on undoing some of the information loss from selective forgetting, but since it is just a single node, it may not completely reverse the loss. \n* Is it necessary to perform hard removal of nodes instead of soft attention?\n* The results in Tab. 2 indicate that L+F (row 4) is better than row 1 primarily for the 1-goal task. For the multi-goal tasks, the results are slightly better, but possibly within the margin of error. Can the forgetting module be better designed to avoid losing the above information? It can lead to more significant gains across tasks.\n\n## Experiments do not sufficiently account for statistical significance\n* In Table 1, bolding does not account for statistical significance b/w VGM and MemoNav. Are the bolded numbers statistically better than the next-best values?\n    * In Gibson: 1-goal SPL, 2-goal PR / PPL, 3-goal PPL, 4-goal PPL\n    * In MP3D: 1-goal SPL, 2-goal PPL\n* In Table 2, {2,3,4}-goal results are fairly close based on standard deviation even though the averages show progression. Need to account for statistical significance of the increase in performance across rows.\n* Figure 4 - no standard deviation markings to show significance of the differences.\n\n## Unfair comparison to VGM \n* VGM uses the inferior GCN encoder instead of the GAT-v2 used for Memonav. The particular architecture of encoder used is orthogonal to the contributions of Memonav. Does Memonav outperform VGM with GCN replaced by GAT-v2?  \n\n## Missing state-of-the-art baselines --- ZER, OVRL, Last-Mile, SSL-sparse\n* VGM is no longer the state-of-the-art for ImageNav. There are a lot more recent works which show significant improvements on ImageNav and need to be compared with [R1,R2,R3,R4].\n\n\n# Minor weaknesses\n* Tab. 3 -- need average STM features baseline instead of randomly sampled STM feature\n* Figure 4 --- unclear how to pick a good forgetting threshold without having complete test results --- no single number is good across settings.\n* Tables 1 and 2 - why limited or no gains on SPL / PPL?\n* Clarification needed in paragraph below Figure 3 - is forgetting module not used during training?\n* Sec 4.1 1st paragraph - incorrect reference to Anderson et al., 2018 ? \n* Missing broader related work comparison to navigation and representation learning literature\n* Working memory graphs has been cited as a closely related work, but it has not been experimentally compared with.\n\n# References\n[R1] Al-Halah, Ziad, Santhosh Kumar Ramakrishnan, and Kristen Grauman. \"Zero experience required: Plug & play modular transfer learning for semantic visual navigation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022. \n\n[R2] Yadav, Karmesh, et al. \"Offline Visual Representation Learning for Embodied Navigation.\" arXiv preprint arXiv:2204.13226 (2022). \n\n[R3] Wasserman, Justin, et al. \"Last-Mile Embodied Visual Navigation.\" 6th Annual Conference on Robot Learning. 2022. \n\n[R4] Majumdar, Arjun, et al. \"SSL Enables Learning from Sparse Rewards in Image-Goal Navigation.\" International Conference on Machine Learning. PMLR, 2022. \n\n\n# Post-rebuttal update\nI thank the authors for their extensive experiments and detailed responses. However, my concerns with the forgetting module remain, and the experiments are incomplete since the models are not trained to convergence and results are statistically similar to baselines. It is hard to overlook these glaring concerns. This work is promising, but incomplete. I maintain my rating.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject