FGPrompt: Fine-grained Goal Prompting for Image-goal Navigation
Authors: Xinyu Sun
Peihao Chen
Jugang Fan
Jian Chen
Thomas H. Li
Mingkui Tan
Venue: NeurIPS 2023 poster (2023)
Keywords: Visual Navigation
Image-Goal Navigation
Embodied AI
TL;DR: We proposed a fine-grained goal prompting method for image-goal navigation. It significantly outperforms baselines, surpassing SOTA by 8% in success rate with 1/50 model size.

Abstract: Learning to navigate to an image-specified goal is an important but challenging task for autonomous systems like household robots. The agent is required to well understand and reason the location of the navigation goal from a picture shot in the goal position. Existing methods try to solve this problem by learning a navigation policy, which captures semantic features of the goal image and observation image independently and lastly fuses them for predicting a sequence of navigation actions. However, these methods suffer from two major limitations. 1) They may miss detailed information in the goal image, and thus fail to reason the goal location. 2) More critically, it is hard to focus on the goal-relevant regions in the observation image, because they attempt to understand observation without goal conditioning. In this paper, we aim to overcome these limitations by designing a Fine-grained Goal Prompting (\sexyname) method for image-goal navigation. In particular, we leverage fine-grained and high-resolution feature maps in the goal image as prompts to perform conditioned embedding, which preserves detailed information in the goal image and guides the observation encoder to pay attention to goal-relevant regions. Compared with existing methods on the image-goal navigation benchmark, our method brings significant performance improvement on 3 benchmark datasets (\textit{i.e.,} Gibson, MP3D, and HM3D). Especially on Gibson, we surpass the state-of-the-art success rate by 8\% with only 1/50 model size.

FGPrompt: Fine-grained Goal Prompting for Image-goal Navigation
Xinyu Sun
Jugang Fan' Thomas H. Li* Jian Chen' Mingkui Tan 12,5+
Peihao Chen'
'South China University of Technology, "Pazhou Laboratory.
Information Technology R&D Innovation Center of Peking University,
*Peking University Shenzhen Graduate School, Key Laboratory of Big Data and Intelligent Robot, Ministry of Education, csxinyusu graii com, mingkuitan @ssut.edu.cn Project Page & Videos: https: /xwpyyiunuuunnnm/// gi wtw.tttuu.....teeeeeeeeeeeeeeeee wtt.op/oopptppaaeeeeeeeeeeeeeeee pagess
Abstract
Learning to navigate to an image-specified goal is an important but challenging task for autonomous systems. The agent is required to reason the goal location from where a picture is shot. Existing methods try to solve this problem by learning navigation policy, which captures semantic features of the goal image and observation image independently and lastly fuses them for predicting a sequence of navigation actions. However, these methods suffer from two major limitations. 1) They may miss detailed information in the goal image, and thus fail to reason the goal location. 2) More critically,
hard focus on the goal-relevant regions in the observation image, because they attempt to understand observation without goal conditioning. In this paper, we aim to overcome these limitations by designing a Fine- -re-ggained Goal Prompting (FGPrompt) method for image-goal navigation. In particular, we leverage fine-grained and high-resolution feature maps in the goal image as prompts to perform conditioned embedding, which preserves detailed information in the goal image and guides the observation encoder to pay attention to goal-relevant regions. Compared with existing methods on the image-goal navigation benchmark, our method brings significant performance improvement on 3 benchmark datasets (i.e., Gibson, MP3D, and HM3D). Especially on Gibson, we surpass the state-of-thed of-the-art success rate by 8% with only 1/50 model size.
Introduction
We focus on the image- moage-goal navigation (ImageNav) task [51] that requires an agent to navigate to an image-specified goal position and face the same orientation as where the photo is taken. In this task, the agent needs to explore the environment and try to find the objects with their surroundings that best match the ones specified the goal image. Though humans prefer to share information using language, image is much clearer and more detailed description to specify goal location intermediate landmark for some household robots [23] or self self-driving vehicles.
Despite its wide applications. this task is still very challenging for the embodied agent due to the following two aspects. First, compared object-goal navigation which assigns goal descriptions with specific semantic categories, it requires the agent to perceive the visual observation as well as the goal image and make comprehensive understanding the scene in order identify goal-relevant objects. Second, objects share similar semantic meanings within one environment, making it challenging to accurately find out the desired object instance.
"Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

100.0
100.0
PGPrompt (Ours)
ZER (Baseline)
Success Rate (%)
Success Rate (%)
80.0
75.0
60.0
50.0
FGProcags MP (Ous))
40.0
OVBL:
OVRI
25.0
Mam-Ang
20.0
ZSON
0.0
Gibson
MP3D (undeen)HM33 (unseen)
0
10
20
30
40
50
Eval Dataset
Number of Parameters (M)
(a) Success rate comparison with baseline (ZER [52]) on three different datasets. Our method performs ef- ficiently and robustly both seen (i.e., Gibson) and unseen (i.. MP3D and HM3D) environments.
(b) Comparison with SOTA both success rate and the number of parameters. the FGPrompt-EI an early fusion variant of our method, achieved 90.4% success rate with only 1/50 model size compared to SOTA.
Figure 1: Main results of our proposed FGPrompt on the image navigation task.
Previous methods [31, 7, 19, 8, 37, 6, 2] seek to solve this task by decomposing the navigation system into several modules in isolation. In general, they tend to adopt efficient exploration skills to build a map incrementally as the understanding of the scene, and further predict waypoint to navigate to. However, these map-based methods require depth maps or the agent's 's GPS position to build the occupancy map or topological map. The latest methods 29, 52, 28, 47, 46] instead try to learn a navigation policy in an end-to-end manner using reinforcement learning. These methods set up two different encoders to obtain semantic embeddings from goal and observation images independently. Subsequently, recurrent model takes these embeddings as input to predict possible action sequence. However, they suffer from two major limitations: 1) As the details in the goal image are gradually overlooked as it goes through deeper network layers, it is harder to find useful cues for reasoning and finding the goal location. 2) Existing methods leave the goal image apart from the observation when performing encoding, it is hard for the agent to focus on the goal-relevant regions in the observation since there is no goal prompting to guide the agent to understand the observation.
When people try to find place captured in an image, they must look for the contextual cues presented with objects, shapes, colors, and textures in both the goal images and current visual observation. Spatial reasoning based on this information plays a critical role in understanding the scene, as people always compare and identify similarities, in order to consider the relative position of various elements and gain insights into the current position and the target location. Motivated by this fact, instead of considering only semantic features of goal and observation images, we propose a novel fine-grained goal prompting (FGPrompi) architecture to learn observation embeddings conditioned on the fine-grained and high-resolution features of the goal image.
Specifically, we implement the goal prompting scheme as fusion process between the goal and observation images and design a mid fusion (FGPrompt-MF) mechanism. This mechanism leverages fine-grained and high-resolution feature maps in the intermediate goal network layers as the prompts, which are proven to contain informative object details [20, 50]. Hereafter, conditioned on these feature maps, we utilize FiLM [32] layers to learn a transform function to adjust the observation activations to focus on goal-relevant objects. In addition, we also design an early fusion (FGPrompt- EF) mechanism by concatenating the goal and observation images at the pixel level. We then use a neural network to perform implicit information exchange. Experimental results show that our proposed method significantly outperforms
thee---f-therrtt methods, as shown in Figure 1.
To sum up, our contributions are as follows: 1) We propose a fine-grained image goal prompting (FGPrompt) architecture to explicitly exchange fine- fine-graines grained information between goal image and observation image, reaching new SOTA of the ImageNav task and also showing great potential in some other embodied tasks including instance image navigation and visual rearrangement tasks. 2) We empower the agent with fine- me-ggained information exchangeability through a simple channel concatenation technique. This scheme is parameter efficient yet shows an absolute advantage on the ImageNav task, even compared to some complex memory graph-based methods. 3) We dedicately design mid-fusion scheme through a novel fine-grained FiLM mapping module to perform more robust information exchange. This scheme shows superior performance in more practical scenarios where the goal image possesses different camera parameters from the observation.

---

2 Related Work
Modular methods. Modular methods leverage strictly defined modules that are handcrafted [37, 23] or learnable [7, 19, 18, 8,37, 2, 10] to address the image-goal navigation task step by step. Classical modular methods typically combine the exploration [48] component, simultaneous localization and mapping (SLAM [16, 43J component, and path planning component to achieve the navigation goal. In order to localize the agent in an unknown environment, some approaches build an explicit metric map of the environment [7, 19], while others propose to obtain an implicit latent map [18] like topological map [8, 37] or simply adopt object detectors without mapping [35]. Chaplot el al. [6] and Avraham et al. [2] train supervised deep models to tackle the sub-tasks, which require a lot of annotated data. Although off-the-shelf modules can be used with zero fine-tuning [23], they still heavily rely on pose and depth sensors, which greatly limits their applicability in the real world.
RL-based navigation. Another pipeline for ImageNav is to directly learn from interactions with the environment using reinforcement learning (RL). RL-based navigation tends to learn end-to-end reward-driven policy that maps observation to action [47, 46, 52, 28, 29, 12] and shows great potential in this task. However, these methods still face the challenge of the sparse reward mechanism and poor generalization performance. To address these issues, previous works [15, 52, 28, 26] propose different methods encourage the agent to explore more efficiently. Yu et al. [15] combines RL policy and visual representation learning model in a min-max game way to incentivize the agent to explore its environment. Al-Halah et al. [52] proposes zero-shot transfer learning approach with novel reward for its semantic search policy. Similarly, Majumdar et al. [28] uses CLIP model pre- treitrained in supervieed manner [33, 41, 9] to enhance image embedding. To tackle the long- horizon planning problem, an external memory module has been proposed by [29, 17, 3, 37, 25, 22, 11] that learns a topological graph [17, 3, 37, 25, 22] or attention map [29] online. Self-supervised learning paradigm has also been explored by Yadav et al. [47, 46] to endow the navigation model with better representation ability. Different from existing approaches, we proposed prompprr observation understanding method that learns to focus on geal-relevant objects through fine fine-grained goal prompts.
Coal-conditioned learning. Existing RL-based navigation methods can be interpreted as learning a goal-conditioned policy, since they only perform fusion on the latent goal embedding and observation embedding. Only semantic-level information can be exchanged during fusing. Some embodied robot planning methods [4, 40, 21, 49] Iearn coal-conditioned observation encoder by injecting the goal embedding into it. Stone et al. [40] and Brohan et al. [4] only consider the language as the goal description, while Jang et al. [21] and Yu et al. [49] try to fuse the goal image with the intermediate feature maps of observation encoder using an affine transformation proposed by FiLM [32]. However, they still focus on the latent embedding of goal images and neglect the fine- ineigne information in high-resolution activation maps. In this paper, we propose to make use of the intermediate activations in the goal encoder as informative guidance to condition the learning of the observation encoder.
3 Image Goal Navigation using Fine-Grained Goal Prompting
3.1 Task definition
Image-goal navigation (ImageNav) requires an agent to navigate to a goal position that matches where the goal image vg was shot. Specifically, the agent starts at a random location po and only receives a goal image og from the environment. At each time step I,, the agent receives an egocentric RGB image U2 captured by RGB sensor fixed on its body, and executes an action a, conditioned on Dt and Uy. In RLsba methods, the action a; is selected based on the learned policy. After performing the action a,, the agent will be assigned a reward T, that encourages the agent to reach the goal position as soon as possible. A more detailed definition of our setup can be found in Section 4.
Existing RL-based methods tackle the ImageNav problem by learning an observation encoder and goal encoder separately, and then fusing their output embeddings together. As shown in Figure (a). this fusion module is commonly equipped on most of the baseline methods. However, those embeddings preserve little detailed information, e.g., shape, texture, and spatial relationship. to promote finding and comparing objects relevant to the goal image [50, 20]. To tackle this challenge, we propose to leverage fine-grained information from lower-level goal image features as prompts to promote the agent's 's ability to focus more on goal-relevant objects.
3

---

Environment
Ohservation Image (P,)
Policy Network
Resnel
%g
Action
3jusion
Actor
GRU
Goal Image (iy)
Reanet
Fusion Baseline
Critic
Pa
Resnet
2/asion
ResBlock
Yome
Bari
ResBlack
Reilu
Resnet
Zg
ResBlock
ReeBlock
Cory
Bind
$
RecBlock
ResBlock3
Fil M
Resmet
ReeBlock
ResBlock
ReLU
PC
Syuston
3
Ayurian
Zp
(b) Early Fusion via Joint Modeling
(d) Mid Fusion via FiLM Layers
(d) Skip Fusion via Keypoint Matching
Figure 2: Illustration of baseline fusion (a) and our goal prompting (b, c, d) for image-goal navigation. All these methods take observation and goal images input and output fused features.
3.2 Fine-grained Goal Prompting
We design and explore three different fine-grained goal prompting methods that vary from fusion mechanism, namely Early Fusion, Mid Fusion, and Skip Fusion. For the first early fusion mech- anism, we investigate injecting fine-grained information from the goal encoder through a simple but effective channel concatenation. After that, we delicately design an explicit information flow through a novel fine-grained FiLM mapping module. Finally, we replace the learnable modules with a heuristic one that injects goal-relation features using feature matching.
Early Fusion via Joint Modeling. naive solution exchange information in images is to directly concatenate them together before input to the encoder. In this way, we are able to fuse fine-grained image details in the very early stage and jointly model them using the same encoder. In particular, we concatenate the goal image with the observation image on the RGB channel dimension, resulting an input tensor shaped 128 128 x 6. This concatenated tensor then fed into ResNet encoder that takes the 6-channel image as input. In this case, the fusion mechanism can be written as:
fusion fo ① Ug)
This simple design yields a promising performance on the image navigation benchmark. Detailed ablation on this early fusion operation can be found in Section 4.2.
Mid Fusion via FiLM Layers. However, as the early fusion mechanism enables spatial reasoning between two images using identical convolution kernel, it is difficult handle the situation when the orientation of the goal camera is noisy. To alleviate this, we further propose an active fusion scheme, utilizing the adaptability of novel fine- fine-grained FiLM mapping module. Previous literature 121, 49] inputs the goal embedding into the ResNet visual backbone via FiLM [32] layers, which adapt learnable affine transformation conditioned the input embedding to the intermediate activation maps in each residual blocks. Through these layers, we can easily connect the intermediate layers in both the goal encoder and the observation encoder to perform mid fusion.
Different from the existing approaches that leverage abstract language embedding as a global condition for all layers, we propose to the hierarchical representations from the intermediate goal encoder layers. This allows us to make good use of the mne-ggained information in high-resolution feature maps. Specifically, we perform FiLM affine transformation the resnet blocks of the observation encoder, where the affine factors 8i,, 7f, in block are conditioned on the shaped activation map = from the correspondent block of the goal encoder. This process can be formulated
% fc(2g)
8 (2gg)
(2)
2 1020 + 8
(3)

---

where 2 denotes a transformed activation map in block i and denotes the cd feature of the feature map. The functions f and h learn to map the condition variable into the affine factors. In practice, we implement them as x convolutions to maintain the same resolution between the input and target activation map. Section 4.2 further investigates the choices of the mapping function and the number of FiLM layers. The output from the conditioned observation encoder fo can then be viewed as the fused feature 2fusion, as shown in Figure 2 (c). The fused feature can be written as:
2 fusion = So(volvg))
Our experiments in Section 4.3 reveal that the mid fusion scheme performs more robustly when the configuration of the goal camera and the observation camera is not perfectly matched.
Skip Fusion via Keypoint Matching. In order to evaluate the importance of the addition of the fusion modules, We finally replaced the aforementioned learnable modules with heuristic one. To achieve this, we follow the idea of Wasserman et al. [44] that attach an additional low-level fusion module using handcrafted keypoint matching methods [27, 36], an improvement of the Late Fusion baseline. We name this mechanism Skip Fusion as it fuses the goal image and observation image in the both early and later stages but skip the others, as shown in Figure 2 (b).
Keypoint matching, which aims to discover representative keypoints in an image and then describe and match them with the most similar ones in another image. As these points are detected based on the low-level statistic [27, 13] of image pixels, we leverage them to play a role as low-level fusion. This scheme is handcrafted as it is not learnable during training. To enable batch inference, we leverage deep learning- based keypoint detecting [14] and a matching [36] method to obtain matched keypoint between the goal image and the observation image. Hereafter, we select top-k matched points according to their matching score to compose a variable 23 and concatenate them together with 2g and 2, as the fusion result:
Ffusion 2g 20 FC(2))
where = = (ar1, y1, "1 1/1 -1..... Tk, yk, r v is a flattened vector of k keypoints and FC denotes to fully connected layer. The default value of vector zạ is set to -1 if the number of matched keypoints less than In Section 4.2, we show the superiority of our proposed Early Fusion and Mid Fusion schemes against this heuristic fusion baseline.
3.3 Navigation Policy
Based on the fused embedding fstin of the goal image and observation image, we train a navigation policy π using reinforcement learning (RL):
St T (2fusion @ (4-1(hb-1)
(6)
where s is the embedding of the agent's current state. h denotes hidden state of the recurrent layers in policy π from previous step. Following previous methods [52, 28], we adopt an actor-critic network to predict state value e and action a using 8 and train it end-to-end using PPO [39]. We utilize the ZER reward [52] to encourage the agent to not only reach the goal position but also face the goal orientation. More details can be found in Appendix.
4 Experiments
Datasets. As for smage-goal navigation, we use the Habitat simulator [38, 42] and train our agent on the Gibson dataset with 72 training scenes and 14 testing scenes under the standard setting. We use the training episodes provided by [29] and train our agent for 500M steps. We report results under multiple datasets to allow direct comparison to various prior works. On the Gibson dataset, we validate our agent on split A generated by [29], and split B generated by [19]. On the MP3D and HM3D, we use the test episodes collected by [52], as well as the instance image navigation dataset released by [24]. We also extend our method to another embodied task named visual rearrangement, where we use the iTHOR simulator and retroor-earranggement 2023 dataset with 80 training scenes, 20 validation scenes and 20 test scenes. Following [45], we train our agent for 75M steps and finally test the best validation checkpoint the test set.
5

---

Method
Backbone
Pretrain
Sensor(s)
Memory
Split
SPL
SR
NTS [8]
ResNer9
N/A
RGBD+Posc
A
43.0%
63.0%
Act-Neur-SLAM [6]
ResNer9
N/A
RGB+Pose
A
23.0%
35.0%
SPTM [37]
ResNet9
N/A
RGB+Pose
27.0% A
51.0%
ZER [52]
ResNe19
N/A
RGB
A
21.6%
29.2%
ZSON [28]
ResNet50
OSD
RGB
A
28.0%
36.9%
OVRL [47]
ResNet50
OSD
RGB
X
A
27.
54.2%
V2RL-V2 [46]
ViT-Base
HGSP
RGB
58.7%
82.0%
PGPrompt-MP (Ours)
ResNet9
N/A
RGB
62. 1%
90.7%
FGPrompt-EF (Ours)
ResNe19
N/A
RGB
66.5%
90.4%
FGPrompt-EF (Ours)
ResNet50
N/A
RGB
68.5 %
92.3% %
Mem-Aug [29]
ResNet18
N/A
RRGB
A
56.0%
69.0%
VGM [25]
ResNet18
N/A
4 RGB
A
64.0%
76,0%
OVRL [47]
ResNet50
OSD
4 RGB
AAR
62.5%
79.8%
TSGM [22]
ResNet18
N/A
RGB
67.2%
81.1%
FGPrompt-EF (Ours)
ResNet9
N/A
RGB
A 75.0% %
94.2% %
NRNS [19] FGPrompt-ER (Ours)
ResNet18
N/A
RGBD
B
12.4%
24.0% 93.0 %
ResNet9
N/A
RGB
B
70.5 %
Table 1:
Comparison with sate-ooftthearre methods on Gibson. All methods are trained and evaluated both on the Gibson dataset.
MP3D
HM3D
Methods
Backbone
SPL
SR
SPI.
SR
Mem-Aug [29]
Resnet18
3.9%
6.9%
3.5%
1.9%
NRNS [19]
Resnet18
5.2%
9.3%
4.3%
6.6%
ZER [52]
Resnet9
10.8%
14.6%
6.3%
9.6%
FGPrompt-MF (Ours)
Resne19
50.4%
77.6%
49.6%
%
Table 2:
Cross-domain evaluation MP3D and HM3D. The agent is trained in Gibson environ- ments and directly transferred to new environments for evaluation.
Agent configuration. We follow the recipe of previous trails [52, 28, 47] initialize agent equipped with only RGB cameras 128 x 128 resolution and 90 FOV. When compared with methods that panoramic input, initialize four RGB sensors to the front, left, right, and back directions of the agent, following [29, 47]. The agent's ' action space is comprised of four discrete actions, including MOVE FORWARD, TURN LEFT, TURN RIGHT, STOP. The minimum units of rotation and forward movement are 30 and 0.25m respectively.
Evaluation metries. We report the success rate (SR) and Success weighted by Path Length (SPL) [1], which takes into account path efficiency of the navigation process. An episode is con- sidered successful if the agent stops within 1.0m Euclidean distance from the goal location and the maximum number of steps in an episode is set to 500 as the default setting.
4.1 Comparison with State-of-the-art Methods
Evaluation on Gibson. In Table 1, we report the ImageNav results on Gibson averaged over three random seeds (the variances of all random seed results are less than lc-4.). We compare our methods with state-of-the-an art methods in two different settings, one takes only one RGB sensor as input following [52, 28, 47] and another one takes RGB sensors to assemble panoramic view following [29, 47]. For the SLAM-based methods in the first three rows, we report the number reproduced by Mezghani al. [29]. We found that our proposed FGPrompt-MF and FGPrompt- EF methods take an absolute advantage compared with all previous methods. Even compared to OVRL-V2 [46], method that utilizes a much larger visual backbone (ViT-B) pre-trained on an in-domain image dataset, we still achieved large performance gains on both SR (92.3% vs. 82.0%) and SPI. (6.5% vs. 58.7%) in the absence of additional pose sensor input. This finding reveals the effectiveness and efficiency of our proposed method.

---

Setting
SPL
SR
Later Fusion (baseline)
11.2%
13.0%
Skip Fusion via keypoint matching (FGPrompt-SF)
24.7%
41.6%
Mid Fusion via FiLM layers (PGPrompt-MF)
50.4%
77.3%
Early Fusion via joint modeling (FGPrompt-EFF) 54.7%
78.9%
Table 3: Comparison of different goal prompting methods Gibson ImageNav task. Fusing the fine-grained goal prompts with the observation instead of directly concatenating their semantic embeddings yield significant improvement.
Mapping Method
SPL
SR
Depth
SPL
SR
N/A
11.2%
13.0%
50.4
77.3%
Semantic Mapping
24.0%
32.0%
2
49.3%
77.6%
FG/HR Mapping 50.4%
77.3%
4
50.2%
71.4%
Table 4: How to map activation into affine factors? Using Fine-grained High-resolution (FG/HR) mapping performs significantly better.
Table 5: How deep should the Mid Fusion perform? Performing Mid Fusion the early layers works better than on all layers.
We extend our FGPrompt- to the panoramic view setting (4 RGB) for direct comparison with some neeeor-basedd methods [29, 25, 22] and pre-trained method [47]. We found that our FGPrompt-EF outperforms these memory-based methods by at least 13.1% in success rate and 7.8% in SPL, even without additional external memory module and pre-training phase. Besides, we also provide comparison result on the non-mainstream testing episodes (split B) following [19]. Compared with the self-supervised method NRNS [19] that pretrained on passive videos, our FGPrompt-EF brings 58. 1 % improvement in success rate and 69.0% in SPI., which shows great advantage by learning to understand the scene based on goal prompting through interacting with the environment.
Cross-domain evaluation on out-of-domain datasets. Table 2, we report the cross- doms-domain evaluation results on the unseen scenes in the Matterport3D (MP3D) [5] and HM3D [34] to verify the generalization ability. Following [52], we directly transfer our model trained on Gibson to these two new datasets, without any tuning. Since there exists very large domain gap between these datasets (e.g. (e.g.more complex and larger scenes in MP3D and diverse scene types HM3D), this setting is extremely challenging. We leverage the testing episodes released by ZER [52]. Compared with the baseline method ZER, our fine-grained goal prompting method brings 7x improvements in the success rate, which shows the generalization ability of our method.
4.2 Ablation Study
In this section, first compare the effectiveness of different variants of our method the ImageNav task. Then we present the detailed ablation of each method to empirically discover their best implementation. For convenience and fairesss, all variants in the ablation study trained for 50M steps on the Gibson dataset.
Comparing different goal prompting methods. We first compare the proposed goal prompting variants on the image-goal navigation task. As shown in Table 3, the Skip Fusion (FGPrompt-SF) variant, integrated fine-grained information by simply adding a keypoint matching-based fusion module the baseline, performs significantly better (from 14.0% 41.4%). This reveals that fine-grained goal prompting is important, However, this heuristic method does not work when there no matched area in the observation. The other two variants further tackle this problem by learning modd-mn framework. In detail, the Mid Fusion (FGPrompt-MP mechanism leverages the intermediate activation maps with varied resolutions variant further increases the navigation success rate by 27.2%. Besides, as simplified version of our proposed Mid Fusion mechanism, the Early Fusion mechanism enables an implicit fusion process through jointly modeling the goal and observation images. In Table 3, this simple but ingenious design brings a further improvement (4.3% in SPL.) compared the Mid Fusion mechanism which well-designed and ablated. We attribute this to its adaptive and learnable fusion fashion.
perform goal prompting. As a result, this
7

---

Setting
SPL
SR
Setting
SPL
SR
3D stack
17.3%
20.5%
Separate modeling
11.2%
13.0%
Edge concat
37.2%
54.8%
Tied modeling 12.3%
14.6%
Channel concat
54.7%
78.9%
Joint modeling
54.7%
78.9% %
Table 6: How to perform early fuuiio? A naive concatenation at the channel dimension works the best.
Table 7: Does joint modeling works? Yes, greatly boosts navigation performance compared the baseline and another similar approach.
Method
SPL
SR
Method
SPL
SR
Baseline (no fusion)
10.5%
12.1%
Baseline (no fusion)
0.2%
0.6%
FGPrompt- (Ours)
38.5%
64.6% %
FGPrompt- (Ours)
0.8%
3.4%
FGPrompt-No MF (Ours) 42.5%
70.2%
FGPrompt- MF (Ours)
2.8%
9.9%
Table Evaluation on the augmented image navigation episodes. Our mid-fusion mecha- nism is more robust under the dynamic camera parameter setting.
Table 9: Evaluation on the instance image nav- igation dataset. Our mid-fusion mechanism per- forms better than the baseline image navigation method and the early-fusion variant.
Ablation the Mid Fusion mechanism. We further investigate the detailed setting of our proposed Mid Fusion mechanism. We conduct ablation studies on the design of FiLM layers in Table 4. We design two different mapping methods to map the activation map into the affine factors in Equation 2, namely Semantic Mapping and Fine-grained High-resolution Mapping. Specifically, for the former. average pool the activation map in each layer to remove the fine-graines information. For the second method, we keep the spatial resolution of the original activation maps, hence preserving the fine-grained information. We initialize function. Not surprisingly, only taking the coarse coarse-grained input from the goal encoder as a condition leg lot behind, as it lose lots of details that might
convolution layers with 1 x stride to learn mapping
as possible cues during the pooling.
Another important question is how deep the network layers should considered to perform fusion. Since the perception field grows the feature map resolution reduces in deeper layers, the information about objects and scenes in these layers could be more and more coarse-grand grained. We design an ablation study that integrates different number of network layers to perform fusion. As shown in Table 5, we found that fusing the first two network layers (cach layer indicates an entire Resnet block) performs well, indicating that Ene-grained information in the early layers is important for goal prompting. When the fusion depth increases to layers, the navigation performance slightly degrades, considering more prompting layers increases the learning difficulty.
Ablation on the Early Fusion mechanism. Finally, we conduct an ablation study to find out how to perform early fusion on the goal image and observation image. There exists naive approach to merging them at the pixel level. In particular, we try to concatenate these two images on different dimensions, shown in Table 6, where concatenation on the channel dimension performs better than on edges (e.g.H H and W). We conjecture that aligning and modeling the goal and observation images enables better spatial reasoning. We also investigate stacking images performing 3D convolution to embed them together. Interestingly, results show that this variant failed learn effective fusion process, although it aligns both images in the spatial dimension.
additional axis and
We then compare the early fusion scheme with a similar approach that shares the same parameters between the goal encoder and observation encoder following [29]. namely Tied Modeling. In Table 7 we directly compare them with baseline that learns a goal encoder and an observation encoder separately. We observe that the Tied Modeling variant performs worse similar to the Separate Modeling bascline. Though using shared parameters to encode both goal and observation images, this architecture does not enable goal-pront prompeed learning focus on the goal-relevant regions and thus failed to effectively reason the goal position.
4.3 Analysis and Qualitative Visualizations
Robustness under dynamic camera parameter setting. We first test the agent trained on imagenav dataset under the dynamic camera parameter setting. We randomly augment the camera height, pitch angle, and HFOV of the goal image in Gibson ImageNav eval episodes. Specifically, follow the

---

Method
Successt
FixedStriet?
E1
ResNet18+1l. Basline [45]
1.89
4.92
1.32
Ours
10.2 (+439%)
24.9 (+406%%
0.81
Table 10: Results A12THOR 1-Phase Rearrangement Challenge. We apply our proposed FGPrompt method on an imitation learning baseline with a ResNet18 backbone. Surprisingly, we found that inserted module significantly improved the agent's performance the 1-Phase track of the visual rearrangement task.
7 1T
7 f1
UT
1T
goal image
(b) observation image
(c) goal activation
observation (e) observation (after fusion)
(before fusion)
Figure 3: CAM visualization the activation map in the fusion layer of FGPrompt-MF. Images in different rows illustrate results in different testing episodes Gibson. The Mid Fusion mechanism learns to focus on the objects that are relevant to the goal image.
distribution of these parameters in the instance imagenav paper [24], sampling goal camera height h (10..mm 1.5m), pitch delta from 41--5, 5"), and HFOV from 2/(60" 120%). In Table 8, find that the mid-fusion mechanism performs the best in this scenario.
We further conduct experiments the instance image navigation (iin) dataset collected by [24], The episodes in this dataset cover a wide range of object instances the environment and are much harder to finish. We train three agents on the HM3D ImageNav dataset and evaluate them on the test split of the iin dataset. Table 9, the baseline model performs poorly in this task with a very low success rate (less than 1%). The agents with proposed fusion mechanisms both perform better. We also observed that the mid-fusion variant outperforms early fusion in this scenario, as its delicately designed activation deformation module yields explicit and adaptive guidance from the goal image. All these results reveal the robustness of mid-fusion in harder tasks.
From Table 9, the performance of our methods on the instance imagenav task is relatively low compared to the ImageNav task. speculate that the extremely different perspective of goal images that haven't been seen during training and longer episode length undermine the performance of our method. This result hints that our method could make further improvement when combined with nemory-based methods [25, 22] to achieve more efficient large-scope exploration.

---

From the experimental results above, we observed a trade-off between two different fusion schemes. The sarly-fusion scheme is somehow an interesting finding that it performs competitively and has a simpler architecture. However, though performs well on the default setting, it does not generalize well to other scenarios where the goal camera parameters don't 't match with the agent's s one. contrast, our delicately designed mid-fusion mechanism performs better in this case. These results indicate that a carefully designed mid-fusion scheme with more inductive bias is necessary.
Transfer to the visual rearrangement task. To see whether our FGPrompt have wide application scenarios, we extend our method to visual rearrangement, another embodied challenge, which aims to move the objects to a correct position in the environment according to unshuffled images. We conduct experiments the 1-Phase track of the aizthor-rearrangement challenge and find our method useful in this task. We start from a ResNet18+1D 8+IL baseline that separately encodes the unshuffled image and agent's 's current observation without fusion mechanism and Iearn from expert actions. Then we introduce our proposed FGPrompt-EF module into the baseline model by fusing the observation with the unshuffled image in an early stage, resulting in one jointly modeled ResNet encoder. We train and test both methods on 2023 dataset and follow [45] to report the testing metrics of the best checkpoints in Table 10. Our proposed module brings 400% relative improvement compared to the baseline. We believe it helps the agent to locate correspondent or inconsistent objects in the environment.
How does the fine-grained goal prompting work? We visualize the activation maps using Eigen- CAM [30] before and after the fusion layers of our mid fusion goal prompting method (FGPrompt-MF) to find out how it works in the image navigation task. Illustrations with the fine-grained and high-resolution activation map from the goal image, the agent is able to find out the relevant objects in the current observation and pay more attention to them, shown in the activation visualization in the last column. Interestingly, we found that even though the agent is far away from the goal position, the mid fusion mechanism still guided the observation encoder to focus on relevant objects or explore some candidate regions that may contain the target objects (see the kitchen bar in the last row). We also provide visualization and analysis of the other two goal prompting methods in Appendix.
presented in Figure 3. Prompted
Performance versus model size. To discuss the feasibility of application on real-world robot systems with resource-limited devices (e.g., household robots), we investigate and compare the model size of our models with previous ones. We report the agent's number of parameters, well the ImageNav success rate on Gibson, and visualize them on the same coordinate system. As shown in Figure b,, our FGPrompt-EF model outperforms existing methods by large margin with much smaller model size, indicating its promising ability on applying to real-world robot systems.
Discussion
Limitation and future work Although our proposed FGPrompt achieved great improvements on different ImageNav datasets, still need a comprehensive study find out if this method is applicable to real-world robots. In the future, we will investigate how to deploy this visual navigation methodology real- real-world robot system, to perform sim-to-real transformation.
Conclusion In this paper, we propose a novel fine-grained goal prompting (FGPrompt) method for visual navigation. In particular, we design a Mid Fusion architecture via FiLM Layers conditioning (FGPrompt-MF), which leverages the high-resolution activation maps from the goal encoder to perform an affine transformation the observation encoder. Furthermore, we rethink it and condense it into an Early Fusion mechanism via joint modeling (FGPrompt-EFP with implicit learning of the fusion process. Experimental results on the Image-goal Navigation task show method has excellent performance, concise architecture design, and strong generalization ability to unseen environments.
Acknowledgments
This work was partially supported by the National Natural Science Foundation of China (Grant No. 62072190 & 62376099 & 62072186) 86), the Guangdong Basic and Applied Basic Research Foundation (Grant No. 2019B15151 300001, the Program for Guangdong Introducing Innovative and Enterprencurial Teams 20172T07X183 the CCF-Tencent Open Fund RAGR20220108.
10

---

References
[1] P. Anderson, A. X. Chang, D. S. Chaplot, A. Dosovitskiy, S. Gupta, V. Koltun, J. Kosecka, J. Malik, R. Mottaghi, M. Savva, and A. R. Zamir. On evaluation of embodied navigation agents. arXiv preprint arXiv: arXiv: /80.006777 2018. 6
[2] G. Avraham, Y. Zuo, T. Dharmasiri, and T. Drummond. Empnet: Neural localisation and mapping using embedded memory points. In JEEE International Conference Computer Vixion (ICCV), 2019. 2,
[3] E. Beeching, J. Dibangoye, O. Simonin, and C. Wolf. Learning to plan with uncertain topological maps. In The European Conference on Computer Vision (ECCV), pages 473-490, 2020. 3
[4] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar. J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman,
A. Herzog, J. Hsu. et al. Rt-1: Robotics trasfformer for real-world control at scale. arXiv preprint
(eXiv:2212.06817 2022. 3
[5] A. Chang, A. Dai, T. Funkhouser, M. Halber, M. Niessner, M. Savva, S. Song. A. Zeng, and Y. Zhang. Matterport3d: Learning from rgb-d data in indoor environments. arXiv preprint arXiv: 1709: 1709 06158, 2017. 7
[6] D. S. Chaplot, D. Gandhi, S. Gupta, A. Gupta, and R. Salakhutdinov. Learning to explore using active neural SLAM. In International Conference on Learning Representations (ICLR), 2020. 2, 3,6
[7] D. S. Chaplot, E. Parisotto, and R. Salakhutdinov. Active neural localization. International Conference
on Learning Representations (ICLR), 2018. 2,3
[8] D. Chaplot, R. Salakhutdinov, Gupta, and S. Gupta. Neural topological SLAM for visual navigation.
In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 2, 3,
[9] P. Chen, D. Huang, D. He, X. Long, R. Zeng, S. Wen, M. Tan, and C. Gan. Rspnet: Relative speed perception for unsupervised video representation learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021. 3
[10] P. Chen, D. Ji, K. Lin, W. Hu, W. Huang, T. Li, M. Tan, and C. Gan. Learning active camera for multi-object navigation. Advances in Neural Information Processing Systems, 2022. 3
[11] P. Chen, D. Ji, K. Lin, R. Zeng, T. Li, M. Tan, and C. Gan. Weakly--uperriseed qulti-graamularity map learning for vision-and-language navigation. Advances in Neural Information Processing Systems, 2022. 3
[12] P. Chen, X. Sun, H. Zhi, R. Zeng. H. Li, G. Liu, M. Tan, and Gan. A² Nav: Action-aware zero- shot robot navigation by exploiting vision-and and-aanuuage ability of foundation models. arXiv preprint arXiv:2308.0m 07997, 2023. 3
[13] N. Dalal and B. Triggs, Histograms of oriented gradients for human detection. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2005. 5
[14] D. DeTone, T. Malisiewicz, and A. Rabinovich. Superpoint: Self superviid interest point detection and description. IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2018.
[15] Y. Du, C. Gan, and P. Isola. Curious representation learning for embodied intelligence. In IEEE Interna- tional Conference Computer Vision (ICCV), 2021. 2,
[16] H. Durrant- Whyte and T. Bailey. Simultancous localization and mapping: part 1. JEEE Robotics and Automation Magazine (RAM), 2006. 3
[17] K. Fang, A. Toshev, I.. Fei-Fei, and S. Savarese. Scene memory transformer for embodied agents in long-horizon tasks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages
538-547, 2019. 3
[18] S. Gupta, J. Davidson, S. Levine, R. Sukthankar, and J. Malik. Cognitive mapping and planning for visual navigation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[19] M. Hahn, D. S. Chaplot, S. Tuusiani, M. Mukadam, J. M. Rehg, and A. Gupta. No rl, no simulation: Learning to navigate without navigating. In Neural Information Processing Systems (NeurIPS), 2021. 2,
5. 6, 7
[20] M. A. Islam, M. Kowal, P. Esser, S. Jia, B. Ommer, K. G. Derpanis, and N. Bruce. Shape texture: Understanding discriminative features in cnns. In ICLR, 2022. 2,
[21] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. BC-Z: zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning (CoRL), 2021. 3,
[22] N. Kim. O. Kwon, H. Yoo, Y. Choi, Park. and Oh. Topological semantic graph memory for image-goal navigation. In Conference on Robot Learning (CoRL), pages 393-402. PMLR, 2023. 3, 6, 7,9
[23] J. Krantz, T. Gervet, K. Yadav, A. Wang, C. Paxton, R. Mottaghi, D. Batra, J. Malik, S. Lee, and D. S. Chaplot. Navigating to objects specified by images. arXiv preprint arXiv:2304.01 01 192, 2023. 1.3
[24] J. Krantz, S. Lee, J. Malik, D. Batra, and D. S. Chaplot. cestance-specifie image goal navigation: Training embodied agents to find object instances. arXiv preprint arXiv:2211.18877 2022. 5.9
11

---

[25] O. Kwon, N. Kim. Y. Choi. H. Yoo, J. Park. and S. Oh. Visual graph memory with unsupervised representation for visual navigation. In IEEE International Conference on Computer Vision (ICCV), pages
15890-15899, 2021. 3. 6, 7.9
[26] K. Lin, P. Chen, D. Huang. T. H. Li, M. Tan, and C. Gan. Learning vision-and-language navigation from youtube videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023. 3
[27] D. G. Lowe. Object recognition from local scale-invariant features. IEEE International Conference on Computer Vision (ICCV), 1999. 5
[28] A. Majumdarr G. Aggarwal, B. Devnani, J. Hoffman, and D. Batra, Zson: Zero-shot object-goal navigation using multimodal goal embeddings. In Neural Information Processing Systems (NeurlPS), 2022. 2, 3, 5,
[29] L. Mezghani, S. Sukhbaatar, T. Lavril, O. Maksymets, D. Batra, P. Bojanowski, and K. Alahari. Memory- augmented reinforcement learning for image-goal navigation. In JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2022. 2, 3,5, 7,8
[30] M. B. Muhammad and M. Yeasin. Eigen-cam: Class activation map using principal components. In International Joint Conference on Neural Networks (JCNN), 2020. 10
[31] R. Mur-Artal, J. M. M. Montiel, and D. Tardós. ORB-SLAM: A versatile and accurate monocular SLAM system. IEEE Transactions on Robotics (T-RO), 2015.
[32] E. Perez, F. Strub, H. de Vries, V. Dumoulin, and A. C. Courville. Film: Visual reasoning with general conditioning layer. In AAAI Conference on Artificial Intelligence (AAAI), 2018. 3,4
[33] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, 2021. 3
[34] S. K. Ramakrishnan, A. Gokaslan, E. Wijmans, O. Maksymets, A. Clegg. J. Turner, E. Undersander, W. Galuba, A. Westbury, A. X. Chang, et al. Habitat matterprt 3d dataset (hm3d); 1000 large-scale 3d
environments for embodied ai. arXiv preprint arXiv:2109.08255 2021. 7
[35] R. Ramrakhya, F. Undersander, D. Batra, and A. Das. Habitat-web: Learning embodied object-search strategies from human demonstrations scale. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[36] P. Sarlin, D. DeTone, T. Malisiewicz, and A. Rabinovich. Superglue: Learning feature matching with graph neural networks. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2020. 5
[37] N. Savinov, A. Dosovitskiy, and V. Koltun. Semi- cani-paraeetric topological memory for navigation. In International Conference on Learning Representations (ICLR). 2018. 3,
[38] M. Savva, J. Malik, D. Parikh, D. Batra, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, and V. Koltun. Habitat: A platform for embudied Al research. In IEEE International Conference on Computer Vision (ICCV), 2019.
[39] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint Xrv: 1exxvv:700 7707,06347, 2017. 5
[40] A. Stone, T. Xiao, Y. Lu, K. Gopalakrishnan, K.-H. Lee, Q. Vuong, P. Wohlhart, B. Zitkovich, F. Xia, C. Finn, et al. Open-world object manipulation using pre-trained vision-language models. arXiv preprint
arXiv:2303.00900 2023. 3
[41] X. Sun, P. Chen, I.. Chen, C. H. Li, M. Tan, and Gan. Masked motion encoding for self-supervised video representation learning. In The IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 3
[42] A. Szot, A. Clegg, E. Undersander, E. Wijmans, Y. Zhao, J. Turner, N. Macstre, M. Mukadam, D. S. Chaplot, O. Maksymets, A. Gokaslan, V. Vondrus, S. Dharur, F. Meier, W. Galuba, A. X. Chang, Z. Kira, V. Koltun, J. Malik, M. Savva, and D. Batra. Habitat 2.0: Training home assistants to rearrange their habitat. In Neural Information Processing Systems (NewrIPS), 2021. 5
[43] S. Thrun. Probabilistic robotics. Commun. ACM, 45(3): 57, 2002. 3
[44] J. Wasserman, K. Yadav, G. Chowdhary, A. Gupta, and U. Jain. List-mile embodied visual navigation. In Conference on Robot Learning, 2023. 5
[45] L. Weihs, M. Deitke, A. Kembhavi, and R. Mottaghi. Visual room rearrangement. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 5, 9,
[46] K. Yadav, A. Majumdar, R. Ramrakhya, N. Yokoyama, A. Baevski, Z. Kira, O. Maksymets, and D. Batra.
Ovrl-v2: A simple state-of-art bascline for imagenav and objectnav. arXiv preprint arX70:2303.07778
2023. 2, 3.6
[47] K. Yadav, R. Ramrakhya, A. Majumdar, V.PP. Berges, S. Kuhar, D. Batra, A. Baeyski, and O. Maksymets. Offline visual representation learning for embodied navigation. In International Conference on Learning Representations (ICLR), 2022. 2, 6.
12

---

[48] B. Yamauchi. A frontier-based approach for autonomous exploration. In IEEE International Symposion Computational Intelligence in Robotics and Automation (CIRA), 1997. 3
[49] A. Yu and R. J. Mooney. Using both demonstrations and language instructions to efficiently learn robotic tasks, arXiv preprint arXiv:2210. 04476, 2022. 3,4
[50] B. Zhou, D. Bau. Oliva, and A. Torralba. Interpreting deep visual representations network dissection. IEEE Transactions on Pattern Analysis and Machine Intelligence 2011. 2,
[51] Y. Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. Gupta, L. Fei-Fei, and A. Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In IEEE International Conference on Robotics and Automation (ICRA), 2017. 1
[52] S. K. R. Ziad Al-Halah and K. Grauman. Zero experience required: Plug & play modular transfer learning for semantic visual navigation. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2022. 3, 5, 7
13

---

Appendix for
"FGPrompt: Fine-grained Goal Prompting for Image-goal Navigation"
In the appendix, we provide more implementation details and experimental results our FGPrompt. We organize the appendix follows.
Sec. A, we provide more architecture details on three different types of goom-pnng methods.
In Sec. B, provide more experimental details, i.e., training and evaluation settings.
In Sec. C, we provide more ablation results on the skip fusion mechanism.
Sec. D, we provide more ablation results on the mid fusion mechanism.
In Sec. E, provide direct comparison with the memory- -aaned methods.
Sec. F, we provide additional visualization results.
A Architecture details
Skip fusion mechanism. As discussed in the previous sections, we design a skip fusion mechanism that utilizes keypoint detection and matching method to provide the agent with low-level prompting. In order to increase the training speed, abandon the handerafted local feature descriptor and detector, instead of deep learning-bar based keypoint matching pipeline [2. 10], as shown in Figure 1. A convolution- based keypoint detector is adopted to extract keypoints from both the goal image and observation image. After that, a 9-layer graph neural network with attention modules applied to find the paired points that share similar features.
Resnet
Zyuston
Vg
Resnet
CNN
GNN
CNN
FC
Keypoints position
SuperPoint
SuperGlue
Figure 1: Architecture details of the skip fusion mechanism.
Mid fusion mechanism. We illustrate the detailed architecture of the proposed mid fusion mecha- nism in Figure 2. For Resnet9 backbone, we map the intermediate activation maps of each resnet block (ResBlock) into affine factor 3 and y using 1 x convolution and a fully connected layer. Then the B and are injected into the correspondent ResBlock in the observation encoder, guiding the model focus on goal-relevant regions in the observation.
Early fusion mechanism. The early fusion mechanism initialized as single Resnet9 [12] encoder. The stem convolution layer takes 128 128 x 6 image as input. The following layers have no difference from a standard Resnet encoder. We conduct experiments using both Resnet9 and Resnet50 encoders.
Navigation policy. We initialize the navigation policy network as a 2-layer GRU with an embedding size of 128.

---

Conv
Vg
Vo
ReI.U
Conv
Conv
ResBlock
conv
B
ResBlock 1
ReLU
BNN
B
Conv
ResBlock
conv
ResBlock 2
ReLU
BN
+
ResBlock
conv
ResBlock 3
R. V.
FiLM
Conv
Boo
ResBlock
conv
ResBlock 4
ReLUU
FC
Y-
Zg
2uusion
Zo
Figure 2: Architecture details of the mid fusion mechanism.
B Experimental details
Dataset details. We train our agent on the Gibson dataset and validate the agent on the Gibson, MP3D, HM3D datasets respectively. For the training dataset, there are 72 scenes in total, each scene has 9k episodes, resulting in 648k episodes. The 9k episodes in each scene are evenly divided into three levels according to the distance from the start location to the goal location: easy (1.5 3m), medium (3 5m). and hard has 14 scenes with 1.4k episodes per level and split B [3] has 14 scenes with 1k episodes per level. For evalution on MP3D and HM3d, we use the same test splits as [12], which has 100 scenes with 1 episodes per level and 18 scenes with 1k episodes per level respectively.
10m). For evalution on Gibson, we use two split, in which split [8]
Training & validation details. We use the Habitat simulator to train our model on the Gibson dataset using 20 environments running in parallel with 8x3090 GPUs. We set the total training time steps to 500M. For one episode, we set the maximum time steps to 500 when performing validation. Other detailed hyperparameters of DD-PPO training follow the recipe ZER [12].
Reward We use the reward formulation proposed by [12] that consists of three parts, including dense shaping reward ras. dense slack reward and sparse success reward Tss- The dense shaping reward is defined as:
dds = ra(de, d ) + [dg S delralar, a2-1),
1, if A is True
where [A]
(2)
0, if is False
where ra the reduced distance the goal from the current position relative the previous one, and To is the reduced angle in radians to the goal view from the current view relative to the previous one. This reward function not only encourages the agent to approach the goal as much possible, but also encourages the agent to rotate to a view as similar as possible to the goal view when the agent close enough to the goal. At each time step t, the agent receives reward r; composed of shape reward and slack reward:
lds Y
where 7 = 0.01 the slack penalty that encourages planning shorter path to the goal. Once predicted a STOP action, the agent will receive sparse success reward rea which is determined by its distance and angle to the goal:
Tss x ([de S d.] + [d. S d, and 0r S ocx))-
Following [12], we set success distance d, 1m and a. 25. As proven by [12, 7, 11], this reward enables the agent to learn to associate between observation v, and goal image Vg. draw association between its observation ot and the goal IG. Specifically, The agent will get sparse reward Tes = it is within d,=In =Im from the goal, and 10 points it is also within a,=25 from the goal view. Otherwise, it will get zero reward.

---

4 RGB setting, To compare with some methods that take panoramic images input, we equip our agent with 4 pairwise orthogonal cameras obtain a panoramic view. To reduce the computation cost, we only take the front image of these cameras as the goal image. During training and inference, each RGB image is combined with the goal image and input to our FGPrompt-FF model, and we concatenate all outputs as the visual-motor feature.
C More ablation study on skip fusion mechanism
As discussed in previous sections, we introduce an image feature matching module to provide the agent with fine-grained low-level goal prompts. A straightforward approach is applying the handcrafted local descriptors [6, 1] to detect and match paired image regions. It first detects the representative keypoints in the image and then matches the paired keypoints. The matched keypoints represent similar regions in two images that are scale-invarian for example, the corner of table or a part of a unique texture on closet. However, computing these handcrafted features is time- consuming, it requires computing Gaussian differences different pyramid scales. In practice, this operation does not support high concurrency when training in the simulator and results low FPS. To tackle this issue, we utilize deep tearning-based keypoint detector and matcher, called SuperPoint [2] and SuperGlue [10]. in order to achieve batch inference on GPU devices. We provide the speed comparison in Table 1. To comprehensively explore how can we leverage this low-level information to perform goal prompting. we provide detailed ablation on this module. In Table 2, compare different representation methods of the matched keypoints, where position denotes combining the normed pixel coordinate of each paired keypoint and descriptors means averaging the 256-dimension feature of each paired keypoint. We found that simply providing the agent with the location of matched points works the best.
Matching method
Device
FPS
Method
SPL
SR
SIFT
CPU
20
Position
37.1 %
52.5%
SuperPoint + SuperGlue
GPU
400
Descriptors
22.9%
38.1%
Deseriptors Position
24.2%
43.2%
Table 1:
Comparing the forward speed of dif- ferent image matching methods. We report the frame per second (FPS) metric during training in the simulator.
Table 2:
Comparing different representa- tions of the matched keypoints. Directly combining the position of paired keypoints per- forms the best.
D More ablation study on mid fusion mechanism
Fusion layer. We have verified the effectiveness of fusing low-level information using low-level handcrafted descriptors previous studies. As discussed in previous literature, the intermediate features in the earlier layer of deep convolution networks contain low-level information (e.g., shape, texture, color, etc.). In the above ablation studies, we found that fusing these intermediate features using FiLM layers into observation encoder layers works. We further provide a detailed ablation study on the choice of the fusion layer. From Table 3, fusing later layers with coarse-graines contents performs worse than the first layer. These results show the importance of our proposed fine- fine-grained goal prompting method.
Comparing more mapping schemes. In the previous sections, we have shown the priority of our proposed fine-grained goal prompting that mapping the intermediate high-resolution activation maps into the affine factors. To further verify the necessity of fine-grained and high-resolution mapping in the mid fusion mechanism, we provide detailed ablation on the semantic mapping methods, where we shift the source of the semantic goal prompt from the average pooled feature of each activation map to high- high-ddmensson feature in the last layer. The poor performance of these two variants in Table 4 further indicates the importance of fine- grained and high-resolution mapping.
Comparing FiLM with self-attention. We conduct experiment that replaces the FiLM module with a self-attention module. Specifically, we project the flattened feature map from the first layer of the goal encoder into the query and the correspondent sequence from the observation encoder
3

---

Layer
SPL
SR
Mapping Method
SPL
1
50.4%
77.3%
FG/HR
50.4%
77.3%
2
44.4%
69.2%
Semantic (each layer)
24.0%
32.0%
3 45.9%
67.3%
Semantic (last layer) 24.4%
32.3%
4
37.1%
52.5%
Table 3: Choice of fusion layer. Early layers contain informative clues for prompting the observation encoder.
Table 4: How to perform semantic mapping? Neither mapping the global mean of each activa- tion layer or semantic-level feature works.
Methods
SPL
SR
Setting
SPL
SR
Self-attention
12.2%
13.9%
Ours w/o background
45.2%
64.4%
Ours
50.4%
77.3%
Ours w/ background
50.4%
77.3%
Table 5: How to perform mid-fusion? Self- attention performs significantly worse than FiLM layers.
Table Importance of background context. Re- moving background context in goal image leads to performance decrease.
into key and value. Then, we utilize the self-attention operation to merge the goal and observation features. Experiment results are shown Table 5.
Importance of environment context in goal image. We leverage the semantic annotation from 145 scenes in HM3D v2 dataset and set the pixel of background (e.g.. uncountable object such as wall and floor) to zero according to the ground truth segmentation map. Numbers are reported in Table 6. From the experimental results, our method suffered from slight degradation when the environmental context was removed from the goal image. These results indicate that environmental context in the image background provides useful limited clues. We believe that objects with their arrangement in each room play critical role in our FGPrompt.
E
Comparison with memory-based methods
In Table 7, we directly compare our FGPrompt-EF with two different memory-based image navigation methods the evaluation split B [3]. The reported results averaged on both straight and curved path types. Although we do not use an additional memory module to store past agent states and model their relationship using the graph neural network. our method still shows priority over the memory-based methods. In our future work, we will discuss the effectiveness of our goal prompting module by involving it with the memory-based methods.
Method
Backbone
Pretrain
Sensor(s)
Memory
Split
SPI.
SR
VGM [5]
ResNet18
N/A
RGB
55.1%
75.3%
TSGM [4]
ResNet18
N/A
RGB
76.9%
85.4%
FGPrompt-EF (Ours)
ResNe19
N/A
4 RGB
x
B
78.3%
96.4% %
Table
Comparison with memory-based methods on eval split B.
Additional visualization results
Training curve vs. validation eurve. We visualize the training and validation curve in both Success Rate and SPL metrics. As shown in Figure 3, our agent does not overfit the training scenes and has consistent performance on both the training and validation episodes.
Visualizing FGPrompt-SE. In Figure 5, we visualize the matching results of the skip fusion mechanism of our proposed FGPrompt-SF. The first row shows successfully matched example that numerous keypoints are detected and paired. The second and last rows show failure cases that the keypoint matching module makes incorrect predictions or failed to find corresponding regions, respectively. In the case of the agent's observation completely different from the goal image, this

---

1.0
1.0
0.8
0.8
Success Rate
(%
oval spi
SPL
0.4
0.4
train spl
0.2
eval st 0.2
train st
0.0
0.0
Steps
1c8
Figure 3: Training and validation curve in Success Rate and SPL.
7
11
(a) goal image
(b) observation image
(c) paired keypoints in goal image and observation image
Figure 4: Visualization of the matching results FGPrompt-SF. Paired keypoints are connected with green lines in the last two columns.
matching module does not contribute to the navigation policy, which is particularly significant in the longer episodes that start far from the goal position.
Visualizing FGPrompt-EF. We show the visualization result of the early fusion mechanism of our proposed FGPrompt-SF in Figure The last column present EigenCAM [9] visualized activation maps from the first layer of the joint encoder backbone. Guided by the fine-grained goal prompts in the early fusion scheme, the visual backbone focuses more regions related to the goal image.
5

---

(a) goal image
(b) observation image
(c) Joint encoder act. (after fusion)
Figure 5: Visualization of the activation map of FGPrompt-EF, We use EigenCAM [9] to reveal where the model pays more attention.

---

References
(1] N. Dalal and B. Triggs, Histograms of oriented gradients for human detection. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2005. 3
[2] D. DeTone, T. Malisiewicz, and A. Rabinovich. Superpoint: Self- Selt-supervised interest point detection and deseription. In IEEF Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 1,
[3] M. Hahn, D. S. Chaplot, S. Tulsiani, M. Mukadam, J. M. Rehg, and A. Gupta. No rl, no simulation: Learning to navigate without navigating. Neural Information Processing Systems (NeurIPS), 2021. 2,
[4] N. Kim. O. Kwon, H. Yoo, Y. Choi, Park, and S. Oh. Topological semantic graph memory for image-goal navigation. In Conference on Robot Learning (CoRL), pages 393 402.402. PMLR, 2023.
[5] O. Kwon, N. Kim, Y. Choi, H. Yoo, J. Park, and S. Oh. Visual graph memory with unsupervised representation visual navigation, In IEEE International Conference on Computer Vision (ICCV), pages 15890-15899 2021. 4
[6] D. Lowe. Object recognition from local scale-invariant features. IEEE International Conference on Computer Vision (ICCV), 1999.
[7] Majumdar. Aggarwal, Devnani, Hoffmann and D. Batra. Zson: Zero-shot object-goal navigation using multimodal goal embeddings. In Neural Information Processing Systems (NeurIPS), 2022. 2
L. Mezghami, S. Sukhbaatar, Lavril, O. Maksymets, Batra, Bojanowski, and Alahari. Memory- augmented reinforcement learning for image-gon goal navigation. In JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2022. 2
[9] M. B. Muhammad and M. Yeasin. Eigen-cam: Class activation map using principal components. In International Joint Conference on Neural Networks (IJCNN), 2020. 5, 6
[10] P. Sarlin, D. DeTone, T. Malisiewicz, and A. Rabinovich. Superglue: Learning feature matching with graph neural networks. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2020. 1,
[11] K. Yadav, R. Ramrakhya, A. Majumdar, V.-P. Berges, S. Kuhar, D. Batra, A. Baevski, and O. Maksymels. Offline visual representation learning for embodied navigation. In International Conference Learning Representations (ICLR), 2022. 2
[12] S. K. R. Ziad Al-Halah and K. Grauman. Zero experience required: Plug & play modular transfer learning for semantic visual navigation. In IEEE Conference Computer Vision and Pattern Recognition (CVPR). 2022. 1,2


Reviews:
[{"review_id": "Yf0cVhliQv", "reviewer": "Reviewer_o71w", "summary": "The authors offer a solution to the image-goal navigation task. The solution focuses on granular feature extraction from the goal image early on in the model pipeline, and usage of the goal image to inform whcih features in the observation the agent should attend to. The paper offers multiple mechanisms to do so: Skip fusion, Mid fusion and Early fusion.", "questions": "* It seems like in Fig 1 graph b, works that solve the the ObjectNav (as opposed to ImageNav) task (ZSON) are also plotted. Is there a reason for that?\n* Maybe its worth specifying that FC -> fully connected MLP for eq (1)\n* Was there a significant difference observed in training time/resources needed compared to the baseline for each of the methods?\n* It seems to me like you are attempting to implement a form of attention. Have you thought about using self attention module but with the observation as the key and goal as the query as a form of fusion?\n* It may be worth running fine-tuning experiments on OOD Datasets\n* Out of curiosity, have you tried evaluating using an agent with a different height of the sensor/camera?\n* I also wonder how big of a role environmental context plays, ie, what would happen if you trained with the goal images where the background was masked out? This is not a priority, just a curiosity.", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* Strong results that seem to outdo the SOTA significantly\n* The method illustration diagram + EigenCAM visuals were well done\n* Plenty of ablations were explored.\n", "weaknesses": "* This isn't necessarily a critique of the contribution of the paper as much as it is of the narrative. The best performing method in the paper (Early Fusion), while having impressive performance, doesn't demonstrate a significantly novel method.(I believe its channel concatenation of the goal and observation before passing through a fully connected MLP. ) As mentioned in the references, image prompting has been used across many other applications as well. It may be worth writing this paper as an establishment of a new baseline of how Image Goal navigation is done, as opposed to a novel contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1. The best performing method in the paper (Early Fusion), while having impressive performance, doesn't demonstrate a significantly novel method.\n\nA1. Thanks for your comments. We would still like to point out the novelty and contribution of our early-fusion method. We empower the ImageNav agent with the **fine-grained information exchange ability** through a simple yet powerful channel concatenation technique on the **very beginning** of the convolution network. The contributions are well recognized by the reviewer 9djz as “the main takeaway from the paper simple but powerful. The fact that channel concatenation for current and goal images works so well on ImageNav is something that should be known more generally.”.\n\n> Q2. As mentioned in the references, image prompting has been used across many other applications as well. It may be worth writing this paper as an establishment of a new baseline of how Image Goal navigation is done, as opposed to a novel contribution.\n\nA2. Existing image prompting methods [A][B] focus on extracting semantic information from conditional input. However, it is insufficient for the image navigation task, since the agent relies on detailed clues from the goal image (e.g., texture details and semantic categories of numerous objects) to infer the goal position. In contrast, **our method focuses on fine-grained promting.** We keep the spatial structure to preserve these fine-grained clues in the feature maps during the fusion schemes.\n\nTo verify the necessity of fine-grained prompting, we introduce a variant that directly prompts the observation encoder using the semantic features of the goal image. In the table below, both our mid-fusion and early-fusion technique with fine-grained prompting showcases notably enhanced performance.\n\n|Setting|SR|SPL|\n|-|-|-|\n| Semantic Prompting [A][B]  | 32.0| 24.0|\n| Our Fine-grained Prompting (MF) | 77.3| 50.4|\n| Our Fine-grained Prompting (EF) | **78.9** | **54.7** |\n\n[A]  BC-Z: zero-shot task generalization with robotic imitation learning. CoRL 2021. \\\n[B] Using both demonstrations and language instructions to efficiently learn robotic tasks. ICLR 2023.\n\n> Q3. It seems like in Fig 1 graph b, works that solve the ObjectNav (as  opposed to ImageNav) task (ZSON) are also plotted.\n\nA3. Sorry for the confusion. ImageNav results has also been reported in ZSON paper. To make a wide comparison among all ImageNav methods, we include it in graph b.\n\n> Q4. Maybe its worth specifying that FC -> fully connected MLP for eq (1).\n\nA4. We have modified the equation in the revised version.\n\n> Q5. Was there a significant difference observed in training time/resources needed compared to the baseline for each of the methods?\n\nA5. Yes. Our early-fusion method significantly reduces the training cost in terms of GPU hours. In comparison, the mid-fusion method slightly increase training cost as it introduces an additional FiLM layer with a convolution operator to obtain affine factors. We provide detailed numbers training a ResNet-50 agent 1x3090 GPU.\n\n| Methods| Frames per second (FPS)↑ | GPU hours↓ |\n|-|-|-|\n| Baseline | 67 | 2070|\n| Ours (MF) | 65  | 2135|\n| Ours (EF) | **126** | **1101**|\n\n\n> Q6. Have you thought about using self attention module?\n\nA6. We conduct an experiment that replaces the FiLM module with a self-attention module. Specifically, we project the flattened feature map from the first layer of goal encoder into query and the correspondent sequence from observation encoder into key and value. Then, we utilize the self-attention operation to merge the goal and observation features. Experiment results are shown below:\n\n|Methods|SR|SPL|\n|-|-|-|\n|Self-attention|13.9|12.2|\n|Ours| **77.3** | **50.4** |\n\nThe results indicate the efficiency of the fine-grained conditioned reasoning through FiLM affine transformation compared to a attention mechanism that is hard to learn.\n\n> Q7. It may be worth running fine-tuning experiments on OOD Datasets\n\nA7. We finetune our FGPrompt agent on HM3D that contains different scenes structures and contents for 100M steps. **As shown in the table below, fine-tuning on HM3D further improve the SR from 76.1% to 81.9%.** Besides, our FGPrompt agent significantly outperforms the baseline by a large margin, demonstrating the effectiveness of our method.\n\n| Methods  | Pre-trained | Fine-tuned | SR   | SPL  |\n|-|-|-|-|-|\n| Baseline | Gibson| -| 9.6  | 6.3  |\n| Baseline | Gibson| HM3D| 20.2 | 17.7 |\n| Ours| Gibson| -| 76.1 | 49.6 |\n| Ours| Gibson| HM3D| **81.9** | **54.5** |\n\n\n> Q8. Have you tried evaluating using an agent with a different height of the sensor/camera?\n\nA8. Yes. We report the evaluation results under different camera heights in the following table. We found that even changing the height of the agent's camera during testing, **our method still outperforms the baseline consistently.**\n\n| Methods  | Training agent height | Eval agent height | SR   | SPL  |\n|-|-|-|-|-|\n| Baseline | 1.25 | 1.25  | 29.2 | 21.6 |\n| Baseline | 1.25 | 1.5| 12.8 | 11.3 |\n| Ours | 1.25 | 1.25| 90.7 | 62.1 |\n| Ours  | 1.25 | 1.5| 76.0 | 46.7 |\n\n\n> Q9. I also wonder how big of a role environmental context plays.\n\nA9. To investigate the importance of environmental context, we train our FGPrompt agent using background-removed goal images. We set the pixel of background (e.g., uncountable objects such as wall and floor) to zero according to the ground truth segmentation map. We leverage the semantic annotation in HM3D v2 dataset to obtain segmentation maps. Numbers are reported as below:\n\n|Setting|SR|SPL|\n|-|-|-|\n|Ours w/o background|64.4|45.2|\n|Ours w/ background|**75.9**|**48.2**|\n\nFrom the experimental results, our method suffered from a slight degradation when environmental context is removed from the goal image. These results indicate that environmental context in image background provides useful but limited clues. We believe that objects with their arrangement in each room play a critical role in our FGPrompt."}, {"review_id": "3o58Mg9Jxe", "reviewer": "Reviewer_9djz", "summary": "This paper introduces FGPrompt (Fine-grained Goal Prompting) for the image-goal navigation task (ImageNav). Existing methods for ImageNav suffer from limitations in capturing detailed goal information and focusing on goal-relevant regions in observation images. FGPrompt tries out three different methods for goal prompting to overcome these limitation, including: 1) keypoint matching, 2) FiLM layers and 3) channel concatenation with 2 and 3 emerging as strong techniques for goal prompting. Experimental results on benchmark datasets demonstrate significant performance improvements compared to existing methods while using much smaller model sizes.", "questions": "While this is not required to be part of the rebuttal, it would be interesting to see how well the trained policy transfers to the real world given its strong transfer on the other scene datasets.\n\nSuggestion:\n1) In line 61 the authors claim that they design the early fusion mechanism by concatenating the goal and observation images. I would suggest that the claim be toned down to say that they try various ways of goal and observation concatenation.\n2) The claim that OVRL-v2 uses pose information in ImageNav is incorrect. The OVRL-v2 paper states that the pose information is only used for the ObjectNav task as mentioned on Page 4.", "limitations": "Since the current FGPrompt technique has the assumption that the current image and goal image will be from cameras with the same parameters (resolution, FOV, etc), I will like to hear from the authors about the limitations of their work where the camera parameters don't match[1]. This is especially important given the community is moving towards these harder tasks.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 2, "contribution": 2, "strengths": "1) I find the main takeaway from the paper simple but powerful. The fact that channel concatenation for current and goal images works so well on ImageNav is something that should be known more generally.\n2) I found the paper easy to read and follow. \n3) The method transfers really well to other scene datasets compared to prior approaches.", "weaknesses": "In my opinion, the paper creates a more complicated story around the various fusion techniques than required. Mid fusion technique is more complicated, requires additional computation, and still performs worse than early fusion. I have two possible suggestions for the authors to make the paper more coherent:\n1) Make the mid-level fusion technique a baseline and focus on understanding the early fusion technique further\n2) Or, show a scenario/task where the mid-fusion technique might be more useful. I believe that the mid-fusion technique might generalize more on the instance imagenav[1] task where the goal image is assumed to be coming from a camera with different parameters to the camera on the robot. \n\n[1] Krantz, J., Gervet, T., Yadav, K., Wang, A., Paxton, C., Mottaghi, R., ... & Chaplot, D. S. (2023). Navigating to Objects Specified by Images. arXiv preprint arXiv:2304.01192.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1. Mid fusion technique is more complicated, requires additional computation, and still performs worse than early fusion. I believe that the mid-fusion technique might generalize more on the instance imagenav task where the goal image is assumed to be coming from a camera with different parameters to the camera on the robot.\n\n\nA1. Sorry for the confusion. **We do find that the mid-fusion technique generalizes better on harder tasks like instance imagenav**, which is difficult due to inconsistent camera parameters. We evaluate three models, namely the baseline model (seperately encode goal image and observation image), our early-fusion agent, and mid-fusion agent, on this task. All these models are trained on the Gibson ImageNav dataset and directly transfer to the HM3D instance imagenav task.\n\n\nIn the table below, the baseline model performs poorly in this task with a very low success rate (less than 1%). The agents with our proposed fusion mechanisms both perform better. **We also observed that the mid-fusion variant actually outperforms early fusion in this scenario**, as its delicately designed activation deformation module yields explicit and adaptive guidance from the goal image to the observation encoder.\n\n\n| Method        | Success | SPL |\n|---------------|---------|-----|\n| Baseline (no fusion)      | 0.6     | 0.2 |\n| Ours (EF)     | 3.4     | 0.8 |\n| Ours (MF)     | **9.9** | **2.8** |\n\nWe also agument the camera height, pitch and HFOV of goal image in Gibson ImageNav eval episodes to evaluate whether these models can handle the situation when goal image and observation image are captured by cameras with different parameters. Specifically, we follow the distribution of these parameters in the instance imagenav paper [A], sampling goal camera height $h\\sim\\mathcal{U}(0.8m,1.5m)$, pitch delta from $\\mathcal{U}(-5^{\\circ},5^{\\circ})$, and HFOV from $\\mathcal{U}(60^{\\circ},120^{\\circ})$.\n\n\nIn the table below, **we also find that the mid-fusion mechanism performs the best in this scenario**. All these results reveal the effectiveness and robustness of mid-fusion in harder tasks.\n\n| Method        | Success  | SPL      |\n|---------------|----------|----------|\n| Baseline (no fusion)     | 12.1     | 10.5     |\n| Ours (EF)     | 64.6     | 38.5     |\n| Ours (MF)     | **78.1** | **52.7** |\n\n[A] Instance-Specific Image Goal Navigation: Training Embodied Agents to Find Object Instances. arXiv 2022.\n\n> Q2. Make the mid-level fusion technique a baseline and focus on understanding the early fusion technique further.\n\nA2. Thanks for the valuable suggestion. For the mid-fusion technique, as analyzed in Q1, it performs the best on the more complicated and practical instance navigation task. We will show its power in instance imagenav task in the revision. \n\nAs for the understanding of early-fusion technique, we have conduct an analysis on it as an extension of the mid-fusion scheme in the section F in the appendix, by means of an EigenCAM visualization. We will update the manuscript to make it more clear.\n\n\n> Q3. While this is not required to be part of the rebuttal, it would be interesting to see how well the trained policy transfers to the real world given its strong transfer on the other scene datasets.\n\nA3. Thanks for your valuable comment. It is an interesting idea to apply our method on a real robot. Since setting up a real robot in a short period is difficult, we left it as our future work.\n\n> Q4. The claim that OVRL-v2 uses pose information in ImageNav is incorrect. The OVRL-v2 paper states that the pose information is only used for the ObjectNav task as mentioned on Page 4.\n\nA4. Thanks for your correction, we will modify the statement in the manuscript.\n\n> Q5. Since the current FGPrompt technique has the assumption that the current image and goal image will be from cameras with the same parameters (resolution, FOV, etc), I will like to hear from the authors about the limitations of their work where the camera parameters don't match.\n\nA5. We agree with the reviewer that the problem definition in ImageNav task has the assumption that the goal images are taken under the same camera setting with the agent and we actually train the agent in these images. As discussed in Q2, we have two main findings on our method: \n\n- As the experiment we have done in Q2, our FGPrompt outperforms baseline methods without fusion mechanism by a large margin. It reveals that our FGPrompt still shows potential in solving this harder task in contrast with baseline methods.\n- The performance of our methods on the instance imagenav task is relatively low compared to the ImageNav task. We speculate that the extremely different perspective of goal images that haven't been seen during training and a longer episode length undermine the performance of our method. This result hints us that our method could make a further improvement when combined with memory-based methods [A] and [B] to achieve more efficient large-scope exploration. We leave this as our future work.\n\n[A] Visual graph memory with unsupervised representation for visual navigation. ICCV 2021. \\\n[B] Topological semantic graph memory for image-goal navigation. CoRL 2023."}, {"review_id": "qrxcLtN5Z1", "reviewer": "Reviewer_mycq", "summary": "The paper proposes different early to middle fusion mechanisms to improve performance of ImageNav tasks thanks to the availability of higher-resolution information in early to intermediate visual encoder layers. The best proposed method (also the simplest in terms of implementation) just concatenates the target and current images and jointly processes them via early fusion, correspondingly using a stem with 6 input channels. The experiments, using three random seeds, are conclusive about the performance of all proposed methods.", "questions": "- The paper states an image used as a goal is is a clearer description than language and shows a wide range of application prospects. Arguably, most of the interaction with robots is likely going to happen through natural language, which is our natural vehicle to share information, so I'd consider adding more context to the introduction.", "limitations": "- They're adequately addressed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Simple design (especially the early fusion proposal) enabling high-resolution spatial reasoning for ImageNav.\n- Excellent performance in comparison to SOTA methods.\n- In-depth analysis for the proposed methods.", "weaknesses": "- I wish the method had also been applied to other task types, e.g. visual rearrangement [1]. Showing good performance in a single task type is a bit limited, even if two variants (panoramic versus limited FOV) are considered.\n- I understand that Mid Fusion can be seen as the most interesting in terms of ablations and visualizations, but I think it feels somewhat strange that a large portion of the discussion in the paper ends up being about the second best proposal.\n\n[1] Weihs et al., Visual room rearrangement, CVPR 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1. I wish the method had also been applied to other task types, e.g. visual  rearrangement. Showing good performance in a single task type is a  bit limited, even if two variants (panoramic versus limited FOV) are considered.\n\nA1. Thanks for your valuable suggestion. We conduct experiments on the 1-Phase track of visual rearrangement task and **find our method useful in this task**. We start from a ResNet18+IL baseline that separately encodes the unshuffled image and agent's current observation(the walkthrough image) without fusion mechanism and learn from expert actions. Then we introduce our proposed FGPrompt into the baseline model by fusing the observation with the unshuffled image in an early stage, resulting in one jointly modeled ResNet encoder. **With our FGPrompt, the agent performs much better than the baseline agent**. We believe it helps the agent to locate correspondent or inconsistent objects in the environment. We report the testing metrics on the visual rearrangement 2023 dataset.\n\n| Method  | Success↑ | FixedStrict↑ | E↓  |\n|---------|----------|--------------|-----|\n| ResNet18+IL Basline [A] |   1.89   |     4.92      | 1.32 |\n| Ours    |   **7.68**    |     **20.1**     | **0.88** |\n\nBesides, we also found that our method is useful on the instance imagenav task regarded to the experiment results in R2Q1. All these results indicate the strong ability of our proposed methods to generalize to a variety of embodied tasks.\n\n[A] Visual room rearrangement, CVPR 2021.\n\n> Q2. I understand that Mid Fusion can be seen as the most interesting in terms of ablations and visualizations, but I think it feels somewhat strange that a large portion of the discussion in the paper ends up being about the second best proposal.\n\n\nA2. Sorry for the confusion. Actually, a trade-off exists between these two methods. \n\n- The early-fusion mechanism is somehow an interesting finding in that it performs competitively and has a simpler architecture. However, though performs well on the default setting, the early-fusion does not generalize well to other scenarios that the goal camera parameters don't match with the agent's one. \n- Our delicately designed mid-fusion mechanism performs better in this case, as evidenced by the attached experimental results from the instance imagenav task in R2Q1. These results indicate that a carefully designed mid-fusion scheme with more inductive bias is necessary.\n\nWe will add more discussion and clarify these findings in the revision.\n\n\n\n> Q3. The paper states an image used as a goal is a clearer description  than language and shows a wide range of application prospects. Arguably,  most of the interaction with robots is likely going to happen through  natural language, which is our natural vehicle to share information, so  I'd consider adding more context to the introduction.\n\nA3. Thanks for your comment. We will modify the statement to make it more precise."}, {"review_id": "QvuYPthJda", "reviewer": "Reviewer_CU6j", "summary": "The paper proposes FGPrompt that conditions the goal image embedding on the observation such that the agent can obtain goal-relevant visual cues during image-goal navigation.\nTo fuse the input and goal image embeddings, FGPrompt introduces two strategies: Mid Fusion by FiLM and Early fusion by encoding the concatenation of the input and goal images.\nThe proposed method achieves a new state-of-the-art by large margins in the Gibson dataset.", "questions": "- Q1: The paper motivates the necessity of fusing the input and goal images but its methodology is directly adopted from FiLM (Mid Fusion). It is unclear which part is novel in the proposed method and what we can learn from the novel part.\n- Q2: The best-performing model (Early Fusion) uses Resnet to jointly encode the input and goal images, resulting in a common CNN-LSTM-based architecture. This seems not surprising as we have already been using neural networks to let them learn how to effectively encode input (in this case, the input and the goal images). Jointly encoding the input and goal images is evidenced by the authors' quantitative analyses but I'm not sure if this is novel.\n- Q3: The best-performing model (Early Fusion) is trained in an end-to-end manner. However, this makes it hard for the model to leverage external knowledge, possibly from large models, as they are not usually trained with the concatenation of images. Can the proposed method leverage such knowledge?", "limitations": "The authors have adequately addressed the limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Mid and early fusing the input and goal images to capture goal-relevant information is well-motivated and sounds sensible.\n- The proposed method achieves substantial improvements over prior arts by large margins, even with fewer parameters.\n- Good illustrations of the proposed method and qualitative examples help better understand the method and its efficacy.", "weaknesses": "- The novelty of the proposed method is a bit weak (see Q1).\n- Early Fusion learns Resnet to jointly encode the input and goal images to capture goal-relevant clues. However, it is not clear what is the difference between a simple CNN-GRU-based policy and the proposed method (see Q2).\n- It is not trivial to leverage prior knowledge (e.g., pretrained large models) (Q3).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1. The paper motivates the necessity of fusing the input and goal images but its methodology is directly adopted from FiLM (Mid Fusion). It is unclear which part is novel in the proposed method and what we can learn from the novel part.\n\n\nA1. We are sorry for the confusion. Nevertheless, **we have to clarify that our method is not directly adopted from FiLM**. We here highlight two main differences between our method and FiLM-based methods[A][B][C]:\n\n- **Different implementation details.** Existing FiLM-based methods first encoder the conditional input (a textual sentence or an image) to a semantic feature and then map this semantic feature into a single affine transformation factor with shape of 1x1xC. In the observation encoder, all its activation values in spatial dimension are affine-transformed using the same factor. In contrast, we calculate pixel-wise affine transformation factors that shape HxWxC from high-resolution feature maps, and thus the activation in different spatial positions can be transformed in a fine-grained manner. \n-  **Different targets**. Existing FiLM-based methods focus on extracting semantic information from conditional input. However, this semantic information is insufficient for the image navigation task, since the agent relies on detailed clues from the goal image (e.g., texture details and semantic categories of numerous objects) to infer the goal position relative to current observation.\n\n\nTo verify the necessity of the pixel-wise affine transformation factors, we introduce a variant that uses the single affine transformation factor inferred from the semantic features of the goal image. In the table below, the FiLM approach incorporating our pixel-wise affine transformation factors showcases notably enhanced performance.\n\n\n| Affine Transormation Factor in FiLM| SR       | SPL      |\n|----------------------------|----------|----------|\n| Single [B]               | 32.0     | 24.0     |\n| Pixel-wise (Ours)               | **77.3** | **50.4** |\n\n\n\n[A] FiLM: Visual Reasoning with a General Conditioning Layer. AAAI 2018. \\\n[B] BC-Z: zero-shot task generalization with robotic imitation learning. CoRL 2021. \\\n[C] Using both demonstrations and language instructions to efficiently learn robotic tasks. ICLR 2023.\n\n\n> Q2. The best-performing model (Early Fusion) uses Resnet to jointly encode  the input and goal images, resulting in a common CNN-LSTM-based  architecture. This seems not surprising as we have already been using  neural networks to let them learn how to effectively encode input (in  this case, the input and the goal images). Jointly encoding the input and goal images is evidenced by the authors' quantitative analyses but  I'm not sure if this is novel.\n\n\nA2. **Our method is fundamentally different from common CNN-LSTM-based architectures [ZER, ZSON, OVRL].** These methods employ two separate CNNs to independently encode the goal image and observation image and then feed the concatenation of these encoded features into LSTM. In contrast, we pursue the integration of information from both the goal image and observation image during the encoding process by concatenating these images in pixel level and using a single CNN to encoder them. \n\nUnlike the prior models where CNNs lack access to information from the goal image while encoding observations, and vice versa, **our approach ensures a more cohesive fusion of these sources of information.** This is critical for the image navigation task because the agent relies on detailed clues from the goal image to infer the goal position relative to current observation. Experimental results in the table below also demonstrate the superiority of our method compared with the common CNN-LSTM-based architecture.\n\n\nBesides, our contributions are also recognized by the reviewer 9djz as \"*the main takeaway from the paper simple but powerful. **The fact that channel concatenation for current and goal images works so well on ImageNav is something that should be known more generally.***\".\n\n\n| Methods | Joint Encoder | SR       | SPL      |\n|---------|---------------|----------|----------|\n| ZER     | No            | 29.2     | 21.6     |\n| ZSON    | No            | 36.9     | 28.0     |\n| OVRL    | No            | 54.2     | 27.0     |\n| **Ours**    | **Yes**           | **92.3** | **68.5** |\n\n\n\n\n\n> Q3. The best-performing model (Early Fusion) is trained in an end-to-end  manner. However, this makes it hard for the model to leverage external  knowledge, possibly from large models, as they are not usually trained  with the concatenation of images. Can the proposed method leverage such  knowledge?\n\nA3. **Yes, our proposed method is compatible with external knowledge in pre-trained models.** To verify this, We initialize the visual encoder using pre-trained large models (i.e., CLIP-RN50) except for the first convolution layer. For the first layer, we copy the original weight and duplicate it on the input channel dimension. Then we finetune the model in an end-to-end manner for 50M steps. Experiment results show the advantage of utilizing external knowledge. We believe that the pre-trained model helps the model to learn faster and potentially provides useful experiences for finding objects in images.\n\n| Method    | External knowledge  | SR   | SPL  |\n|-----------|---------------------|------|------|\n| Ours (EF) | without knowledge | 41.1 | 26.8 |\n| Ours (EF) | with knowledge (CLIP-RN50)          | **64.3** | **29.4** |"}]

Author rebuttal:
We sincerely appreciate all reviewers’ time and efforts in reviewing our paper and for the constructive feedback. In addition to the response to specific reviewers, here we would like to 1) thank reviewers for their acknowledgment of our work, 2) summarize our contributions, and 3) highlight the new results added during the rebuttal: 

1). We are glad that the reviewers appreciate and recognize our contributions.
1. The proposed method achieved substantial improvements. [o71w, 9djz, mycq, CU6j]
2. The main takeaway from our paper is powerful. [9djz, mycq]
3. The proposed method transfer really well. [9djz]
4. The proposed method is well-motivated. [CU6j]
5. The ablation studies are in-depth with good visualization. [o71w, mycq, CU6j]
6. This paper is well-written and easy to follow. [9djz]

2). We summarize our contributions as follows.
- **A novel image goal prompting architecture to solve the ImageNav task.** We propose a fine-grained image goal prompting (FGPrompt) architecture to explicitly exchange fine-grained information between goal image and observation image, reaching a new SOTA of the ImageNav task and also showing great potential in some other embodied tasks including instance image navigation and visual rearrangement tasks.
- **A simple early-fusion scheme to boost the ImageNav performance with very few parameters.** We empower the ImageNav agent with the fine-grained information exchange ability through a simple yet powerful channel concatenation technique. This scheme shows an absolute advantage on the ImageNav task even compared to some complex memory graph-based methods.
- **A generic mid-fusion scheme to address the mismatch between goal camera and observation camera.** We dedicately design a mid-fusion scheme through a novel fine-grained FiLM mapping module to perform more robust information exchange. This scheme shows superior performance in more practical scenarios that the goal image possesses different camera parameters from the observation.
- **An in-depth analysis of the image prompting schemes.** We illustrate the activation maps using EigenCAM and reveal how mid-fusion and early-fusion scheme work, bringing new insights for the embodied AI field.

3). In this rebuttal, we have added more supporting results following the reviewers’ suggestions.
1. Comparison results with an attention-based mid-fusion model. [o71w]
2. Finetuning results on the OOD dataset. [o71w]
3. Training efficiency by means of GPU hours compared with baseline. [o71w]
4. Evaluation under different camera settings. [o71w]
5. Ablation results of background context. [o71w]
6. Evaluation on instance image navigation task. [9djz]
7. Evaluation on visual rearrangement task. [mycq]
8. Performance of our method with pre-trained CLIP initialization. [CU6j]

Meta-review: The paper proposes an approach for the image-goal navigation task. The reviewers have strongly supported the paper since: (1) The method achieves strong results compared to the state-of-the-art approaches. (2) The method is simple and intuitive. (3) The method transfers well to other scene datasets.
(4) The paper is easy to follow and motivates the approach very well. (5) The paper includes in-depth analysis of the models. 

Therefore, the AC follows the recommendation of the reviewers and recommends acceptance.
Decision: Accept (poster)