Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper builds upon the VGM (Kwon et al., 2021) for image navigation. By forgetting some less informative nodes in the topological map through the attention mechanism, the short-term memory (STM) can be updated dynamically. The selected STMs are fused to obtain the scene-level feature with Long-term memory (LTM), and which are encoded through graph attention to generate the Working Memory (WM). The WM is finally used to generate the actions for navigation. The experiments of both 1-goal and multi-goal tasks are conducted in two public datasets Gibson and Matterpot3D in Habitat simulator.
strength_and_weaknesses: Strengths:
1.	The paper is generally well-written and easy to follow.
2.	The experiments and analysis are extensive with the ablation experiments and baselines.
3.	The selective forgetting module seems capable of retaining the informative nodes in the topological map and forget the redundant nodes, which may be helpful for improving the SR/PR for 1-goal and multi-goal tasks.

Weaknesses:
1.	The MemoNav seems to be trained in the 1-goal setting, while used in both 1-goal and multi-goal tasks, is it suitable to use this model to evaluate the multi-goal task? It is worth discussing training different models (under multi-goal settings) to validate for multi-goal tasks.
2.	The proposed MemoNav follows the VGM method of constructing the topological maps of the environments. Considering the topological maps can somehow memorize the observations during navigation as well, the advantages of proposed memory based modules are not that effective compared to the baseline, also the close SPL/PPL results are obtained compared the baseline VGM under 1-goal and multi-goal tasks. 
3.	It seems the regularity and relevance are not that significant in Figure 4. The 2-goal task shows a sharp drop in PP at p=0.4, and a dramatic increase at p=0.6, what are the reasons for this phenomenon?
 

clarity,_quality,_novelty_and_reproducibility: The paper is well-written, and the contributions of the paper are easy to follow. Compared with the current methods, this paper contributes to node selection and encoding, but less in constructing topological maps. The code is submitted, the paper should be reproducible. 
summary_of_the_review: I think this is a practical work, the motivation and idea are interesting, and the experimental results are also sufficient. However, I worry about the that novelty of the technical contribution over baseline (VGM) seems limited. And compared to the results of VGM, the improvement of the proposed method seems marginal in some parts.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel graph-memory network for visual navigation. Inspired by working memory from cog-sci literature, the proposed method consists of short-term, long-term and working memories to perform the ImageNav task. A forgetting module is proposed to prune the goal-irrelevant parts of the short-term memory. Results are demonstrated on Gibson and Matterport3D datasets.
strength_and_weaknesses: 
# Strengths
* The idea is interesting and novel to the best of my knowledge. 
* The clarity of the paper is good and the paper is easy to understand.
* The experiments have been well designed, especially the ablations and subsequent analyses. 


# Major weaknesses
## The design of selective forgetting module is suboptimal
* The module deletes nodes which are not scored highly by the target decoder. But the graph also contains other valuable information which are not directly related to the goal: (1) the context around the agent's current position in the environment, and (2) the graph connectivity information from agent's current position to a node close to the target --- useful for path planning. This information is lost when selective forgetting module is used.
* The design of the LTM module appears to be focused on undoing some of the information loss from selective forgetting, but since it is just a single node, it may not completely reverse the loss. 
* Is it necessary to perform hard removal of nodes instead of soft attention?
* The results in Tab. 2 indicate that L+F (row 4) is better than row 1 primarily for the 1-goal task. For the multi-goal tasks, the results are slightly better, but possibly within the margin of error. Can the forgetting module be better designed to avoid losing the above information? It can lead to more significant gains across tasks.

## Experiments do not sufficiently account for statistical significance
* In Table 1, bolding does not account for statistical significance b/w VGM and MemoNav. Are the bolded numbers statistically better than the next-best values?
    * In Gibson: 1-goal SPL, 2-goal PR / PPL, 3-goal PPL, 4-goal PPL
    * In MP3D: 1-goal SPL, 2-goal PPL
* In Table 2, {2,3,4}-goal results are fairly close based on standard deviation even though the averages show progression. Need to account for statistical significance of the increase in performance across rows.
* Figure 4 - no standard deviation markings to show significance of the differences.

## Unfair comparison to VGM 
* VGM uses the inferior GCN encoder instead of the GAT-v2 used for Memonav. The particular architecture of encoder used is orthogonal to the contributions of Memonav. Does Memonav outperform VGM with GCN replaced by GAT-v2?  

## Missing state-of-the-art baselines --- ZER, OVRL, Last-Mile, SSL-sparse
* VGM is no longer the state-of-the-art for ImageNav. There are a lot more recent works which show significant improvements on ImageNav and need to be compared with [R1,R2,R3,R4].


# Minor weaknesses
* Tab. 3 -- need average STM features baseline instead of randomly sampled STM feature
* Figure 4 --- unclear how to pick a good forgetting threshold without having complete test results --- no single number is good across settings.
* Tables 1 and 2 - why limited or no gains on SPL / PPL?
* Clarification needed in paragraph below Figure 3 - is forgetting module not used during training?
* Sec 4.1 1st paragraph - incorrect reference to Anderson et al., 2018 ? 
* Missing broader related work comparison to navigation and representation learning literature
* Working memory graphs has been cited as a closely related work, but it has not been experimentally compared with.

# References
[R1] Al-Halah, Ziad, Santhosh Kumar Ramakrishnan, and Kristen Grauman. "Zero experience required: Plug & play modular transfer learning for semantic visual navigation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022. 

[R2] Yadav, Karmesh, et al. "Offline Visual Representation Learning for Embodied Navigation." arXiv preprint arXiv:2204.13226 (2022). 

[R3] Wasserman, Justin, et al. "Last-Mile Embodied Visual Navigation." 6th Annual Conference on Robot Learning. 2022. 

[R4] Majumdar, Arjun, et al. "SSL Enables Learning from Sparse Rewards in Image-Goal Navigation." International Conference on Machine Learning. PMLR, 2022. 


# Post-rebuttal update
I thank the authors for their extensive experiments and detailed responses. However, my concerns with the forgetting module remain, and the experiments are incomplete since the models are not trained to convergence and results are statistically similar to baselines. It is hard to overlook these glaring concerns. This work is promising, but incomplete. I maintain my rating.

clarity,_quality,_novelty_and_reproducibility: The paper clarity is good. The idea is novel. The code has been provided along with instructions for reproducibility.
summary_of_the_review: The idea is interesting and novel, the paper is clearly written, and the experiments are well designed. I have concerns about the design of the forgetting module, insufficient accounting for statistical significance, unfair comparison to VGM and missing state-of-the-art baselines for ImageNav. I also have some minor concerns which I hope that authors should be able to address easily.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an image-goal navigation (ImageNav) method on top of VGM (Kwon et al. 2021) which builds a topological graph. The proposed method, MemoNav considers VGM as the short-term memory (STM) and removes some nodes based on thresholding on attention score (selective forgetting). The modified STMs are fed into the GATv2 encoder followed by a decoder. The final output of the decoder is used as an input of the policy network. The authors tested MemoNav in Gibson and Matterport3D environments and showed that the proposed method improves performance.

strength_and_weaknesses: **[Strength]**

1. The paper is well-written and easy to understand.

2. The authors include the failure trajectories in Figure 10 for a better understanding of the behaviors of the proposed method. 


**[Weakness]**

1. Basically, the proposed MemoNav is a combination of existing methods, VGM (topological mapping, decoder) + forgetting module + GATv2 (across time). Although I do not degrade the contribution of merging together and achieving good performance, the novelty is somewhat limited. The authors argue that the forgetting module is novel in Section 2. However, attention-based dropout already exists [Ref 1].

2. According to Table 2 (1-3, 2-4), the forgetting module is not that helpful and even degrades the performance in terms of SPL/PPL. The authors should specify the reason why the performance s decreased. Moreover, considering Table 3, I wonder effectiveness of using the forgetting module. I read Section A.5 but it compares with random forgetting, not the effectiveness of forgetting itself. One possible explanation might be to reduce the computation by removing the node. The authors briefly mentioned that the proposed method employs 20% fewer node features. However, further explanation is needed.

3. In Figure 4, the performance is consistently decreasing in 3-goal and 4-goal settings as p grows. The authors argue that “excluding an excessive fraction forces the agent not to utilize what it has explored.” This statement should be also applicable to the smaller number of goals tasks. However, the performance is getting better in 1-goal and 2-goal tasks. The authors should analyze this trend more clearly.


**[minor]**

1. In Section 5.3, please unify the notation. The second paragraph uses absolute difference while the third paragraph uses relational difference.

2. There is one paper [Ref 2] that achieves better performance than the proposed method but I will not count on this since it is arxiv paper.


clarity,_quality,_novelty_and_reproducibility: As mentioned in **[Strength]**, the paper is well written and includes many ablation studies and visualization. However, some analysis regarding results or explanation (e.g., A4.2. what is special linear projection layers) are missing. 

summary_of_the_review: As mentioned in above, the idea and achievement are good but the further analysis of experimental results is needed and the novelty is somewhat limited.

- Post Rebuttal - 
Although I appreciate the authors dealt with my concerns, I agree with other reviewers' concerns and G+L+F is not significantly different from G+L. Hence, I maintain the original rating.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a memory architecture for visual navigation, which includes a short-term memory (that is dynamically updated), a long-term memory (which continuously aggregates information from short-term memory), and a working memory that is generated by attention over short-term and long-term memory. 

The proposed idea had some novel aspects (however reviewers EedW and aALd have concerns about the similarity with previous work). The paper is also generally well-written.

The reviewers brought up several weaknesses, some of which were addressed by the authors during the rebuttal period. For example, concerns regarding forgetting important information were addressed by additional experiments. 

justification_for_why_not_higher_score: The paper has issues regarding the statistical significance of the results and also lack of enough training. So, the conclusions and the claims about performance improvements and comparisons with other methods are not entirely valid.
Decision: Reject