SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We extend our FGPrompt-EF to the panoramic view setting (4 RGB) for direct comparison with some memory-based methods [29, 25, 22] and pre-trained method [47]. We found that our FGPrompt-EF outperforms these memory-based methods by at least  $13.1\%$  in success rate and  $7.8\%$  in SPL, even without additional external memory module and pre-training phase. Besides, we also provide a comparison result on the non-mainstream testing episodes (split B) following [19]. Compared with the self-supervised method NRNS [19] that pretrained on passive videos, our FGPrompt-EF brings  $58.1\%$  improvement in success rate and  $69.0\%$  in SPL, which shows a great advantage by learning to understand the scene based on goal prompting through interacting with the environment.
Cross-domain evaluation on out-of-domain datasets. In Table 2, we report the cross-domain evaluation results on the unseen scenes in the Matterport3D (MP3D) [5] and HM3D [34] to verify the generalization ability. Following [52], we directly transfer our model trained on Gibson to these two new datasets, without any tuning. Since there exists a very large domain gap between these datasets (e.g. more complex and larger scenes in MP3D and diverse scene types in HM3D), this setting is extremely challenging. We leverage the testing episodes released by ZER [52]. Compared with the baseline method ZER, our fine-grained goal prompting method brings  $7 \times$  improvements in the success rate, which shows the generalization ability of our method.
In this section, we first compare the effectiveness of different variants of our method on the ImageNav task. Then we present the detailed ablation of each method to empirically discover their best implementation. For convenience and fairness, all variants in the ablation study are trained for 50M steps on the Gibson dataset.
Comparing different goal prompting methods. We first compare the proposed goal prompting variants on the image-goal navigation task. As shown in Table 3, the Skip Fusion (FGPrompt-SF) variant, integrated fine-grained information by simply adding a keypoint matching-based fusion module to the baseline, performs significantly better (from  $14.0\%$  to  $41.4\%$ ). This reveals that fine-grained goal prompting is important. However, this heuristic method does not work when there is no matched area in the observation. The other two variants further tackle this problem by learning a joint-modeling framework. In detail, the Mid Fusion (FGPrompt-MF) mechanism leverages the intermediate activation maps with varied resolutions to perform goal prompting. As a result, this variant further increases the navigation success rate by  $27.2\%$ . Besides, as a simplified version of our proposed Mid Fusion mechanism, the Early Fusion mechanism enables an implicit fusion process through jointly modeling the goal and observation images. In Table 3, this simple but ingenious design brings a further improvement ( $4.3\%$  in SPL) compared to the Mid Fusion mechanism which is well-designed and ablated. We attribute this to its adaptive and learnable fusion fashion.
Setting	SPL	SR
3D stack	17.3%	20.5%
Edge concat	37.2%	54.8%
Channel concat	54.7%	78.9%
Table 6: How to perform early fusion? A naive concatenation at the channel dimension works the best.
Method	SPL	SR
Baseline (no fusion)	10.5%	12.1%
FGPrompt-EF (Ours)	38.5%	64.6%
FGPrompt-MF (Ours)	42.5%	70.2%
Table 8: Evaluation on the augmented image navigation episodes. Our mid-fusion mechanism is more robust under the dynamic camera parameter setting.
Setting	SPL	SR
Separate modeling	11.2%	13.0%
Tied modeling	12.3%	14.6%
Joint modeling	54.7%	78.9%
Table 7: Does joint modeling works? Yes, it greatly boosts navigation performance compared to the baseline and another similar approach.
Method	SPL	SR
Baseline (no fusion)	0.2%	0.6%
FGPrompt-EF (Ours)	0.8%	3.4%
FGPrompt-MF (Ours)	2.8%	9.9%
Table 9: Evaluation on the instance image navigation dataset. Our mid-fusion mechanism performs better than the baseline image navigation method and the early-fusion variant.
Ablation on the Mid Fusion mechanism. We further investigate the detailed setting of our proposed Mid Fusion mechanism. We conduct ablation studies on the design of FiLM layers in Table 4. We design two different mapping methods to map the activation map into the affine factors in Equation 2, namely Semantic Mapping and Fine-grained High-resolution Mapping. Specifically, for the former, we average pool the activation map in each layer to remove the fine-grained information. For the second method, we keep the spatial resolution of the original activation maps, hence preserving the fine-grained information. We initialize two convolution layers with  $1 \times 1$  stride to learn a mapping function. Not surprisingly, only taking the coarse-grained input from the goal encoder as a condition leg a lot behind, as it lose lots of details that might serve as possible cues during the pooling.
Another important question is how deep the network layers should be considered to perform fusion. Since the perception field grows as the feature map resolution reduces in deeper layers, the information about objects and scenes in these layers could be more and more coarse-grained. We design an ablation study that integrates a different number of network layers to perform fusion. As shown in Table 5, we found that fusing the first two network layers (each layer indicates an entire Resnet block) performs well, indicating that fine-grained information in the early layers is important for goal prompting. When the fusion depth increases to 4 layers, the navigation performance slightly degrades, as considering more prompting layers increases the learning difficulty.
Ablation on the Early Fusion mechanism. Finally, we conduct an ablation study to find out how to perform early fusion on the goal image and observation image. There exists a naive approach to merging them at the pixel level. In particular, we try to concatenate these two images on different dimensions, as shown in Table 6, where concatenation on the channel dimension performs better than on edges (e.g.H and W). We conjecture that aligning and modeling the goal and observation images enables better spatial reasoning. We also investigate stacking images at an additional axis and performing 3D convolution to embed them together. Interestingly, results show that this variant failed to learn an effective fusion process, although it aligns both images in the spatial dimension.
We then compare the early fusion scheme with a similar approach that shares the same parameters between the goal encoder and observation encoder following [29], namely Tied Modeling. In Table 7 we directly compare them with a baseline that learns a goal encoder and an observation encoder separately. We observe that the Tied Modeling variant performs worse similar to the Separate Modeling baseline. Though using shared parameters to encode both goal and observation images, this architecture does not enable goal-prompted learning to focus on the goal-relevant regions and thus failed to effectively reason the goal position.
4.3 Analysis and Qualitative Visualizations
Robustness under dynamic camera parameter setting. We first test the agent trained onImagenav dataset under the dynamic camera parameter setting. We randomly augment the camera height, pitch angle, and HFOV of the goal image in Gibson ImageNav eval episodes. Specifically, we follow the
Method	Success↑	FixedStrict↑	E↓
ResNet18+IL Basline [45]	1.89	4.92	1.32
Ours	10.2 (+439%)	24.9 (+406%)	0.81
Table 10: Results on AI2THOR 1-Phase Rearrangement Challenge. We apply our proposed FGPrompt method on an imitation learning baseline with a ResNet18 backbone. Surprisingly, we found that our inserted module significantly improved the agent's performance on the 1-Phase track of the visual rearrangement task.
Figure 3: EigenCAM visualization of the activation map in the fusion layer of FGPrompt-MF. Images in different rows illustrate results in different testing episodes in Gibson. The Mid Fusion mechanism learns to focus on the objects that are relevant to the goal image.
distribution of these parameters in the instance imagenav paper [24], sampling goal camera height  $h \sim \mathcal{U}(0.8m, 1.5m)$ , pitch delta from  $\mathcal{U}(-5^{\circ}, 5^{\circ})$ , and HFOV from  $\mathcal{U}(60^{\circ}, 120^{\circ})$ . In Table 8, we find that the mid-fusion mechanism performs the best in this scenario.
We further conduct experiments on the instance image navigation (iin) dataset collected by [24]. The episodes in this dataset cover a wide range of object instances in the environment and are much harder to finish. We train three agents on the HM3D ImageNav dataset and evaluate them on the test split of the iin dataset. In Table 9, the baseline model performs poorly in this task with a very low success rate (less than  $1\%$ ). The agents with our proposed fusion mechanisms both perform better. We also observed that the mid-fusion variant outperforms early fusion in this scenario, as its delicately designed activation deformation module yields explicit and adaptive guidance from the goal image. All these results reveal the robustness of mid-fusion in harder tasks.
From Table 9, the performance of our methods on the instanceImagenav task is relatively low compared to the ImageNav task. We speculate that the extremely different perspective of goal images that haven't been seen during training and a longer episode length undermine the performance of our method. This result hints that our method could make a further improvement when combined with memory-based methods [25, 22] to achieve more efficient large-scope exploration.
From the experimental results above, we observed a trade-off between two different fusion schemes. The early-fusion scheme is somehow an interesting finding in that it performs competitively and has a simpler architecture. However, though performs well on the default setting, it does not generalize well to other scenarios where the goal camera parameters don't match with the agent's one. In contrast, our delicately designed mid-fusion mechanism performs better in this case. These results indicate that a carefully designed mid-fusion scheme with more inductive bias is necessary.
Transfer to the visual rearrangement task. To see whether our FGPrompt have wide application scenarios, we extend our method to visual rearrangement, another embodied challenge, which aims to move the objects to a correct position in the environment according to unshuffled images. We conduct experiments on the 1-Phase track of the ai2thor-rearrangement challenge and find our method useful in this task. We start from a ResNet18+IL baseline that separately encodes the unshuffled image and agent's current observation without a fusion mechanism and learn from expert actions. Then we introduce our proposed FGPrompt-EF module into the baseline model by fusing the observation with the unshuffled image in an early stage, resulting in one jointly modeled ResNet encoder. We train and test both methods on 2023 dataset and follow [45] to report the testing metrics of the best checkpoints in Table 10. Our proposed module brings  $400\%$  relative improvement compared to the baseline. We believe it helps the agent to locate correspondent or inconsistent objects in the environment.
How does the fine-grained goal prompting work? We visualize the activation maps using Eigen-CAM [30] before and after the fusion layers of our mid fusion goal prompting method (FGPrompt-MF) to find out how it works in the image navigation task. Illustrations are presented in Figure 3. Prompted with the fine-grained and high-resolution activation map from the goal image, the agent is able to find out the relevant objects in the current observation and pay more attention to them, as shown in the activation visualization in the last column. Interestingly, we found that even though the agent is far away from the goal position, the mid fusion mechanism still guided the observation encoder to focus on relevant objects or explore some candidate regions that may contain the target objects (see the kitchen bar in the last row). We also provide visualization and analysis of the other two goal prompting methods in Appendix.
Performance versus model size. To discuss the feasibility of application on real-world robot systems with resource-limited devices (e.g., household robots), we investigate and compare the model size of our models with previous ones. We report the agent's number of parameters, as well as the ImageNav success rate on Gibson, and visualize them on the same coordinate system. As shown in Figure 1b, our FGPrompt-EF model outperforms existing methods by a large margin with a much smaller model size, indicating its promising ability on applying to real-world robot systems.
Limitation and future work Although our proposed FGPrompt achieved great improvements on different ImageNav datasets, we still need a comprehensive study to find out if this method is applicable to real-world robots. In the future, we will investigate how to deploy this visual navigation methodology to a real-world robot system, to perform sim-to-real transformation.
Conclusion In this paper, we propose a novel fine-grained goal prompting (FGPrompt) method for visual navigation. In particular, we design a Mid Fusion architecture via FiLM Layers conditioning (FGPrompt-MF), which leverages the high-resolution activation maps from the goal encoder to perform an affine transformation on the observation encoder. Furthermore, we rethink it and condense it into an Early Fusion mechanism via joint modeling (FGPrompt-EF), with implicit learning of the fusion process. Experimental results on the Image-goal Navigation task show our method has excellent performance, concise architecture design, and strong generalization ability to unseen environments.
This work was partially supported by the National Natural Science Foundation of China (Grant No. 62072190 & 62376099 & 62072186), the Guangdong Basic and Applied Basic Research Foundation (Grant No. 2019B1515130001), the Program for Guangdong Introducing Innovative and Entrepreneurial Teams 2017ZT07X183, the CCF-Tencent Open Fund RAGR20220108.
[48] B. Yamauchi. A frontier-based approach for autonomous exploration. In IEEE International Symposium on Computational Intelligence in Robotics and Automation (CIRA), 1997. 3
[49] A. Yu and R. J. Mooney. Using both demonstrations and language instructions to efficiently learn robotic tasks. arXiv preprint arXiv:2210.04476, 2022. 3, 4
[50] B. Zhou, D. Bau, A. Oliva, and A. Torralba. Interpreting deep visual representations via network dissection. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2019. 2, 3
[51] Y. Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. Gupta, L. Fei-Fei, and A. Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In IEEE International Conference on Robotics and Automation (ICRA), 2017. 1
[52] S. K. R. Ziad Al-Halah and K. Grauman. Zero experience required: Plug & play modular transfer learning for semantic visual navigation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 2, 3, 5, 6, 7
Appendix for "FGPrompt: Fine-grained Goal Prompting for Image-goal Navigation"
In the appendix, we provide more implementation details and experimental results of our FGPrompt. We organize the appendix as follows.
- In Sec. A, we provide more architecture details on three different types of goal-prompting methods.
- In Sec. B, we provide more experimental details, i.e., training and evaluation settings.
- In Sec. C, we provide more ablation results on the skip fusion mechanism.
- In Sec. D, we provide more ablation results on the mid fusion mechanism.
- In Sec. E, we provide a direct comparison with the memory-based methods.
- In Sec. F, we provide additional visualization results.
A Architecture details
Skip fusion mechanism. As discussed in the previous sections, we design a skip fusion mechanism that utilizes keypoint detection and matching method to provide the agent with low-level prompting. In order to increase the training speed, we abandon the handcrafted local feature descriptor and detector, instead of a deep learning-based keypoint matching pipeline [2, 10], as shown in Figure 1. A convolution-based keypoint detector is adopted to extract keypoints from both the goal image and observation image. After that, a 9-layer graph neural network with attention modules is applied to find the paired points that share similar features.
Figure 1: Architecture details of the skip fusion mechanism.
Mid fusion mechanism. We illustrate the detailed architecture of the proposed mid fusion mechanism in Figure 2. For a Resnet9 backbone, we map the intermediate activation maps of each resnet block (ResBlock) into affine factor  $\beta$  and  $\gamma$  using a  $1 \times 1$  convolution and a fully connected layer. Then the  $\beta$  and  $\gamma$  are injected into the correspondent ResBlock in the observation encoder, guiding the model to focus on goal-relevant regions in the observation.
Early fusion mechanism. The early fusion mechanism is initialized as a single Resnet9 [12] encoder. The stem convolution layer takes a  $128 \times 128 \times 6$  image as input. The following layers have no difference from a standard Resnet encoder. We conduct experiments using both Resnet9 and Resnet50 encoders.
Navigation policy. We initialize the navigation policy network as a 2-layer GRU with an embedding size of 128.
Figure 2: Architecture details of the mid fusion mechanism.
Dataset details. We train our agent on the Gibson dataset and validate the agent on the Gibson, MP3D, HM3D datasets respectively. For the training dataset, there are 72 scenes in total, each scene has 9k episodes, resulting in 648k episodes. The 9k episodes in each scene are evenly divided into three levels according to the distance from the start location to the goal location: easy (1.5 - 3m), medium (3 - 5m), and hard (5 - 10m). For evaluation on Gibson, we use two split, in which split A [8] has 14 scenes with 1.4k episodes per level and split B [3] has 14 scenes with 1k episodes per level. For evaluation on MP3D and HM3d, we use the same test splits as [12], which has 100 scenes with 1k episodes per level and 18 scenes with 1k episodes per level respectively.
Training & validation details. We use the Habitat simulator to train our model on the Gibson dataset using 20 environments running in parallel with  $8 \times 3090$  GPUs. We set the total training time steps to 500M. For one episode, we set the maximum time steps to 500 when performing validation. Other detailed hyperparameters of DD-PPO training follow the recipe of ZER [12].
Reward We use the reward formulation proposed by [12] that consists of three parts, including dense shaping reward  $r_{ds}$ , dense slack reward  $\gamma$ , and sparse success reward  $r_{ss}$ . The dense shaping reward is defined as:
$$
r _ {d s} = r _ {d} \left(d _ {t}, d _ {t - 1}\right) + \left[ d _ {t} \leq d _ {s} \right] r _ {\alpha} \left(\alpha_ {t}, \alpha_ {t - 1}\right), \tag {1}
$$
$$
\text {w h e r e} [ A ] = \left\{ \begin{array}{l l} 1, & \text {i f A i s T r u e} \\ 0, & \text {i f A i s F a l s e} \end{array} \right. \tag {2}
$$
where  $r_d$  is the reduced distance to the goal from the current position relative to the previous one, and  $r_\alpha$  is the reduced angle in radians to the goal view from the current view relative to the previous one. This reward function not only encourages the agent to approach the goal as much as possible, but also encourages the agent to rotate to a view as similar as possible to the goal view when the agent is close enough to the goal. At each time step  $t$ , the agent receives a reward  $r_t$  composed of shape reward and slack reward:
$$
r _ {t} = r _ {d s} - \gamma \tag {3}
$$
where  $\gamma = 0.01$  is the slack penalty that encourages planning a shorter path to the goal. Once predicted a STOP action, the agent will receive a sparse success reward  $r_{ss}$  which is determined by its distance and angle to the goal:
$$
r _ {s s} = 5 \times \left(\left[ d _ {t} \leq d _ {s} \right] + \left[ d _ {t} \leq d _ {s} \text {a n d} \alpha_ {t} \leq \alpha_ {s} \right]\right). \tag {4}
$$
Following [12], we set success distance  $d_{s} = 1\mathrm{m}$  and  $\alpha_{s} = 25^{\circ}$ . As proven by [12, 7, 11], this reward enables the agent to learn to associate between observation  $v_{o}$  and goal image  $v_{g}$ . draw the association between its observation  $o$  and the goal IG. Specifically, The agent will get a sparse reward  $r_{ss} = 5$  if it is within  $d_{s} = 1\mathrm{m}$  from the goal, and 10 points if it is also within  $\alpha_{s} = 25^{\circ}$  from the goal view. Otherwise, it will get a zero reward.
4 RGB setting. To compare with some methods that take panoramic images as input, we equip our agent with 4 pairwise orthogonal cameras to obtain a panoramic view. To reduce the computation cost, we only take the front image of these cameras as the goal image. During training and inference, each RGB image is combined with the goal image and input to our FGPrompt-EF model, and we concatenate all outputs as the visual-motor feature.
C More ablation study on skip fusion mechanism
As discussed in previous sections, we introduce an image feature matching module to provide the agent with fine-grained low-level goal prompts. A straightforward approach is applying the handcrafted local descriptors [6, 1] to detect and match paired image regions. It first detects the representative keypoints in the image and then matches the paired keypoints. The matched keypoints represent similar regions in two images that are scale-invariant, for example, the corner of a table or a part of a unique texture on a closet. However, computing these handcrafted features is time-consuming, as it requires computing Gaussian differences on different pyramid scales. In practice, this operation does not support high concurrency when training in the simulator and results in low FPS. To tackle this issue, we utilize a deep learning-based keypoint detector and matcher, called SuperPoint [2] and SuperGlue [10], in order to achieve batch inference on GPU devices. We provide the speed comparison in Table 1. To comprehensively explore how can we leverage this low-level information to perform goal prompting, we provide detailed ablation on this module. In Table 2, we compare different representation methods of the matched keypoints, where position denotes combining the normed pixel coordinate of each paired keypoint and descriptors means averaging the 256-dimension feature of each paired keypoint. We found that simply providing the agent with the location of matched points works the best.
Matching method	Device	FPS
SIFT	CPU	20
SuperPoint + SuperGlue	GPU	400
Table 1: Comparing the forward speed of different image matching methods. We report the frame per second (FPS) metric during training in the simulator.
Method	SPL	SR
Position	37.1%	52.5%
Descriptors	22.9%	38.1%
Descriptors + Position	24.2%	43.2%
Table 2: Comparing different representations of the matched keypoints. Directly combining the position of paired keypoints performs the best.
D More ablation study on mid fusion mechanism
Fusion layer. We have verified the effectiveness of fusing low-level information using low-level handcrafted descriptors in previous studies. As discussed in previous literature, the intermediate features in the earlier layer of deep convolution networks contain low-level information (e.g., shape, texture, color, etc.). In the above ablation studies, we found that fusing these intermediate features using FiLM layers into observation encoder layers works. We further provide a detailed ablation study on the choice of the fusion layer. From Table 3, fusing later layers with coarse-grained contents performs worse than the first layer. These results show the importance of our proposed fine-grained goal prompting method.