SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Comparing more mapping schemes. In the previous sections, we have shown the priority of our proposed fine-grained goal prompting that mapping the intermediate high-resolution activation maps into the affine factors. To further verify the necessity of fine-grained and high-resolution mapping in the mid fusion mechanism, we provide detailed ablation on the semantic mapping methods, where we shift the source of the semantic goal prompt from the average pooled feature of each activation map to a high-dimension feature in the last layer. The poor performance of these two variants in Table 4 further indicates the importance of fine-grained and high-resolution mapping.
Comparing FiLM with self-attention. We conduct an experiment that replaces the FiLM module with a self-attention module. Specifically, we project the flattened feature map from the first layer of the goal encoder into the query and the correspondent sequence from the observation encoder
Layer	SPL	SR
1	50.4%	77.3%
2	44.4%	69.2%
3	45.9%	67.3%
4	37.1%	52.5%
Table 3: Choice of fusion layer. Early layers contain informative clues for prompting the observation encoder.
Methods	SPL	SR
Self-attention	12.2%	13.9%
Ours	50.4%	77.3%
Table 5: How to perform mid-fusion? Self-attention performs significantly worse than FiLM layers.
Mapping Method	SPL	SR
FG/HR	50.4%	77.3%
Semantic (each layer)	24.0%	32.0%
Semantic (last layer)	24.4%	32.3%
Table 4: How to perform semantic mapping? Neither mapping the global mean of each activation layer or semantic-level feature works.
Setting	SPL	SR
Ours w/o background	45.2%	64.4%
Ours w/ background	50.4%	77.3%
into key and value. Then, we utilize the self-attention operation to merge the goal and observation features. Experiment results are shown in Table 5.
Importance of environment context in goal image. We leverage the semantic annotation from 145 scenes in HM3D v2 dataset and set the pixel of background (e.g., uncountable object such as wall and floor) to zero according to the ground truth segmentation map. Numbers are reported in Table 6. From the experimental results, our method suffered from a slight degradation when the environmental context was removed from the goal image. These results indicate that environmental context in the image background provides useful but limited clues. We believe that objects with their arrangement in each room play a critical role in our FGPrompt.
E Comparison with memory-based methods
In Table 7, we directly compare our FGPrompt-EF with two different memory-based image navigation methods on the evaluation split B [3]. The reported results are averaged on both straight and curved path types. Although we do not use an additional memory module to store past agent states and model their relationship using the graph neural network, our method still shows priority over the memory-based methods. In our future work, we will discuss the effectiveness of our goal prompting module by involving it with the memory-based methods.
Table 6: Importance of background context. Removing background context in goal image leads to a performance decrease.
Method	Backbone	Pretrain	Sensor(s)	Memory	Split	SPL	SR
VGM [5]	ResNet18	N/A	4 RGB	✓	B	55.1%	75.3%
TSGM [4]	ResNet18	N/A	4 RGB	✓	B	76.9%	85.4%
FGPrompt-EF (Ours)	ResNet9	N/A	4 RGB	×	B	78.3%	96.4%
Table 7: Comparison with memory-based methods on eval split B.
F Additional visualization results
Training curve vs. validation curve. We visualize the training and validation curve in both Success Rate and SPL metrics. As shown in Figure 3, our agent does not overfit the training scenes and has a consistent performance on both the training and validation episodes.
Visualizing FGPrompt-SF. In Figure 5, we visualize the matching results of the skip fusion mechanism of our proposed FGPrompt-SF. The first row shows a successfully matched example that numerous keypoints are detected and paired. The second and last rows show failure cases that the keypoint matching module makes incorrect predictions or failed to find corresponding regions, respectively. In the case of the agent's observation completely different from the goal image, this
Figure 3: Training and validation curve in Success Rate and SPL.
(a) goal image
(b) observation image
(c) paired keypoints in goal image and observation image
Figure 4: Visualization of the matching results of FGPrompt-SF. Paired keypoints are connected with green lines in the last two columns.
matching module does not contribute to the navigation policy, which is particularly significant in the longer episodes that start far from the goal position.
Visualizing FGPrompt-EF. We show the visualization result of the early fusion mechanism of our proposed FGPrompt-SF in Figure 5. The last column present EigenCAM [9] visualized activation maps from the first layer of the joint encoder backbone. Guided by the fine-grained goal prompts in the early fusion scheme, the visual backbone focuses more on regions related to the goal image.
(b) observation image
(c) Joint encoder act. (after fusion)
Figure 5: Visualization of the activation map of FGPrompt-EF. We use EigenCAM [9] to reveal where the model pays more attention.
[1] N. Dalal and B. Triggs. Histograms of oriented gradients for human detection. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2005. 3
[2] D. DeTone, T. Malisiewicz, and A. Rabinovich. Superpoint: Self-supervised interest point detection and description. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 1, 3
[3] M. Hahn, D. S. Chaplot, S. Tulsiani, M. Mukadam, J. M. Rehg, and A. Gupta. No rl, no simulation: Learning to navigate without navigating. In Neural Information Processing Systems (NeurIPS), 2021. 2, 4
[4] N. Kim, O. Kwon, H. Yoo, Y. Choi, J. Park, and S. Oh. Topological semantic graph memory for image-goal navigation. In Conference on Robot Learning (CoRL), pages 393-402. PMLR, 2023. 4
[5] O. Kwon, N. Kim, Y. Choi, H. Yoo, J. Park, and S. Oh. Visual graph memory with unsupervised representation for visual navigation. In IEEE International Conference on Computer Vision (ICCV), pages 15890-15899, 2021. 4
[6] D. G. Lowe. Object recognition from local scale-invariant features. In IEEE International Conference on Computer Vision (ICCV), 1999. 3
[7] A. Majumdar, G. Aggarwal, B. Devnani, J. Hoffman, and D. Batra. Zson: Zero-shot object-goal navigation using multimodal goal embeddings. In Neural Information Processing Systems (NeurIPS), 2022. 2
[8] L. Mezghani, S. Sukhbaatar, T. Lavril, O. Maksymets, D. Batra, P. Bojanowski, and K. Alahari. Memory-augmented reinforcement learning for image-goal navigation. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2022. 2
[9] M. B. Muhammad and M. Yeasin. Eigen-cam: Class activation map using principal components. In International Joint Conference on Neural Networks (IJCNN), 2020. 5, 6
[10] P. Sarlin, D. DeTone, T. Malisiewicz, and A. Rabinovich. Superglue: Learning feature matching with graph neural networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 1, 3
[11] K. Yadav, R. Ramrakhya, A. Majumdar, V.-P. Berges, S. Kuhar, D. Batra, A. Baevski, and O. Maksymets. Offline visual representation learning for embodied navigation. In International Conference on Learning Representations (ICLR), 2022. 2
[12] S. K. R. Ziad Al-Halah and K. Grauman. Zero experience required: Plug & play modular transfer learning for semantic visual navigation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 1, 2