SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xinyu Sun $^{1,3}$  Peihao Chen $^{1}$  Jugang Fan $^{1}$  Thomas H. Li $^{4}$  Jian Chen $^{1*}$  Mingkui Tan $^{1,2,5*}$
$^{1}$ South China University of Technology,  $^{2}$ Pazhou Laboratory,
<sup>3</sup>Information Technology R&D Innovation Center of Peking University,
$^{4}$ Peking University Shenzhen Graduate School,
<sup>5</sup>Key Laboratory of Big Data and Intelligent Robot, Ministry of Education,
csxinyusu@gmail.com, mingkuitan@scut.edu.cn
Project Page & Videos: https://xinyusun.github.io/fgprompt-pages
Learning to navigate to an image-specified goal is an important but challenging task for autonomous systems. The agent is required to reason the goal location from where a picture is shot. Existing methods try to solve this problem by learning a navigation policy, which captures semantic features of the goal image and observation image independently and lastly fuses them for predicting a sequence of navigation actions. However, these methods suffer from two major limitations. 1) They may miss detailed information in the goal image, and thus fail to reason the goal location. 2) More critically, it is hard to focus on the goal-relevant regions in the observation image, because they attempt to understand observation without goal conditioning. In this paper, we aim to overcome these limitations by designing a Fine-grained Goal Prompting (FGPrompt) method for image-goal navigation. In particular, we leverage fine-grained and high-resolution feature maps in the goal image as prompts to perform conditioned embedding, which preserves detailed information in the goal image and guides the observation encoder to pay attention to goal-relevant regions. Compared with existing methods on the image-goal navigation benchmark, our method brings significant performance improvement on 3 benchmark datasets (i.e., Gibson, MP3D, and HM3D). Especially on Gibson, we surpass the state-of-the-art success rate by  $8\%$  with only 1/50 model size.
We focus on the image-goal navigation (ImageNav) task [51] that requires an agent to navigate to an image-specified goal position and face the same orientation as where the photo is taken. In this task, the agent needs to explore the environment and try to find the objects with their surroundings that best match the ones specified in the goal image. Though humans prefer to share information using language, an image is a much clearer and more detailed description to specify a goal location or an intermediate landmark for some household robots [23] or self-driving vehicles.
Despite its wide applications, this task is still very challenging for the embodied agent due to the following two aspects. First, compared to object-goal navigation which assigns goal descriptions with specific semantic categories, it requires the agent to perceive the visual observation as well as the goal image and make a comprehensive understanding of the scene in order to identify goal-relevant objects. Second, objects share similar semantic meanings within one environment, making it challenging to accurately find out the desired object instance.
(a) Success rate comparison with baseline (ZER [52]) on three different datasets. Our method performs efficiently and robustly in both seen (i.e., Gibson) and unseen (i.e., MP3D and HM3D) environments.
Figure 1: Main results of our proposed FGPrompt on the image navigation task.
(b) Comparison with SOTA both on success rate and the number of parameters. The FGPrompt-EF, an early fusion variant of our method, achieved  $90.4\%$  success rate with only 1/50 model size compared to SOTA.
Previous methods [31, 7, 19, 8, 37, 6, 2] seek to solve this task by decomposing the navigation system into several modules in isolation. In general, they tend to adopt efficient exploration skills to build a map incrementally as the understanding of the scene, and further predict a waypoint to navigate to. However, these map-based methods require depth maps or the agent's GPS position to build the occupancy map or topological map. The latest methods [15, 29, 52, 28, 47, 46] instead try to learn a navigation policy in an end-to-end manner using reinforcement learning. These methods set up two different encoders to obtain semantic embeddings from goal and observation images independently. Subsequently, a recurrent model takes these embeddings as input to predict a possible action sequence. However, they suffer from two major limitations: 1) As the details in the goal image are gradually overlooked as it goes through deeper network layers, it is harder to find useful cues for reasoning and finding the goal location. 2) Existing methods leave the goal image apart from the observation when performing encoding, it is hard for the agent to focus on the goal-relevant regions in the observation since there is no goal prompting to guide the agent to understand the observation.
When people try to find a place captured in an image, they must look for the contextual cues presented with objects, shapes, colors, and textures in both the goal images and current visual observation. Spatial reasoning based on this information plays a critical role in understanding the scene, as people always compare and identify similarities, in order to consider the relative position of various elements and gain insights into the current position and the target location. Motivated by this fact, instead of considering only semantic features of goal and observation images, we propose a novel fine-grained goal prompting (FGPrompt) architecture to learn observation embeddings conditioned on the fine-grained and high-resolution features of the goal image.
Specifically, we implement the goal prompting scheme as a fusion process between the goal and observation images and design a mid fusion (FGPrompt-MF) mechanism. This mechanism leverages fine-grained and high-resolution feature maps in the intermediate goal network layers as the prompts, which are proven to contain informative object details [20, 50]. Hereafter, conditioned on these feature maps, we utilize FiLM [32] layers to learn a transform function to adjust the observation activations to focus on goal-relevant objects. In addition, we also design an early fusion (FGPrompt-EF) mechanism by concatenating the goal and observation images at the pixel level. We then use a neural network to perform implicit information exchange. Experimental results show that our proposed method significantly outperforms state-of-the-art methods, as shown in Figure 1.
To sum up, our contributions are as follows: 1) We propose a fine-grained image goal prompting (FGPrompt) architecture to explicitly exchange fine-grained information between goal image and observation image, reaching a new SOTA of the ImageNav task and also showing great potential in some other embodied tasks including instance image navigation and visual rearrangement tasks. 2) We empower the agent with fine-grained information exchangeability through a simple channel concatenation technique. This scheme is parameter efficient yet shows an absolute advantage on the ImageNav task, even compared to some complex memory graph-based methods. 3) We dedicate design a mid-fusion scheme through a novel fine-grained FiLM mapping module to perform a more robust information exchange. This scheme shows superior performance in more practical scenarios where the goal image possesses different camera parameters from the observation.
Modular methods. Modular methods leverage strictly defined modules that are handcrafted [37, 23] or learnable [7, 19, 18, 8, 37, 6, 2, 10] to address the image-goal navigation task step by step. Classical modular methods typically combine the exploration [48] component, simultaneous localization and mapping (SLAM [16, 43]) component, and path planning component to achieve the navigation goal. In order to localize the agent in an unknown environment, some approaches build an explicit metric map of the environment [7, 19], while others propose to obtain an implicit latent map [18] like a topological map [8, 37] or simply adopt object detectors without mapping [35]. Chaplot et al. [6] and Avraham et al. [2] train supervised deep models to tackle the sub-tasks, which require a lot of annotated data. Although off-the-shelf modules can be used with zero fine-tuning [23], they still heavily rely on pose and depth sensors, which greatly limits their applicability in the real world.
RL-based navigation. Another pipeline for ImageNav is to directly learn from interactions with the environment using reinforcement learning (RL). RL-based navigation tends to learn an end-to-end reward-driven policy that maps observation to action [47, 46, 52, 28, 29, 12] and shows great potential in this task. However, these methods still face the challenge of the sparse reward mechanism and poor generalization performance. To address these issues, previous works [15, 52, 28, 26] propose different methods to encourage the agent to explore more efficiently. Yu et al. [15] combines RL policy and visual representation learning model in a min-max game way to incentivize the agent to explore its environment. Al-Halah et al. [52] proposes a zero-shot transfer learning approach with a novel reward for its semantic search policy. Similarly, Majumdar et al. [28] uses a CLIP model pre-trained in self-supervised manner [33, 41, 9] to enhance image embedding. To tackle the long-horizon planning problem, an external memory module has been proposed by [29, 17, 3, 37, 25, 22, 11] that learns a topological graph [17, 3, 37, 25, 22] or attention map [29] online. Self-supervised learning paradigm has also been explored by Yadav et al. [47, 46] to endow the navigation model with better representation ability. Different from existing approaches, we proposed a goal-prompted observation understanding method that learns to focus on goal-relevant objects through fine-grained goal prompts.
Goal-conditioned learning. Existing RL-based navigation methods can be interpreted as learning a goal-conditioned policy, since they only perform fusion on the latent goal embedding and observation embedding. Only semantic-level information can be exchanged during fusing. Some embodied robot planning methods [4, 40, 21, 49] learn a goal-conditioned observation encoder by injecting the goal embedding into it. Stone et al. [40] and Brohan et al. [4] only consider the language as the goal description, while Jang et al. [21] and Yu et al. [49] try to fuse the goal image with the intermediate feature maps of observation encoder using an affine transformation proposed by FiLM [32]. However, they still focus on the latent embedding of goal images and neglect the fine-grained information in high-resolution activation maps. In this paper, we propose to make use of the intermediate activations in the goal encoder as informative guidance to condition the learning of the observation encoder.
3 Image Goal Navigation using Fine-Grained Goal Prompting
3.1 Task definition
Image-goal navigation (ImageNav) requires an agent to navigate to a goal position that matches where the goal image  $v_{g}$  was shot. Specifically, the agent starts at a random location  $p_0$  and only receives a goal image  $v_{g}$  from the environment. At each time step  $t$ , the agent receives an egocentric RGB image  $v_{t}$  captured by a RGB sensor fixed on its body, and executes an action  $a_{t}$  conditioned on  $v_{t}$  and  $v_{g}$ . In RL-based methods, the action  $a_{t}$  is selected based on the learned policy. After performing the action  $a_{t}$ , the agent will be assigned a reward  $r_{t}$  that encourages the agent to reach the goal position as soon as possible. A more detailed definition of our setup can be found in Section 4.
Existing RL-based methods tackle the ImageNav problem by learning an observation encoder and a goal encoder separately, and then fusing their output embeddings together. As shown in Figure 2 (a), this fusion module is commonly equipped on most of the baseline methods. However, those embeddings preserve little detailed information, e.g., shape, texture, and spatial relationship, to promote finding and comparing objects relevant to the goal image [50, 20]. To tackle this challenge, we propose to leverage fine-grained information from lower-level goal image features as prompts to promote the agent's ability to focus more on goal-relevant objects.
Figure 2: Illustration of baseline fusion (a) and our goal prompting (b, c, d) for image-goal navigation. All these methods take observation and goal images as input and output fused features.
3.2 Fine-grained Goal Prompting
We design and explore three different fine-grained goal prompting methods that vary from fusion mechanism, namely Early Fusion, Mid Fusion, and Skip Fusion. For the first early fusion mechanism, we investigate injecting fine-grained information from the goal encoder through a simple but effective channel concatenation. After that, we delicately design an explicit information flow through a novel fine-grained FiLM mapping module. Finally, we replace the learnable modules with a heuristic one that injects goal-relative features using feature matching.
Early Fusion via Joint Modeling. A naive solution to exchange information in two images is to directly concatenate them together before input to the encoder. In this way, we are able to fuse fine-grained image details in the very early stage and jointly model them using the same encoder. In particular, we concatenate the goal image with the observation image on the RGB channel dimension, resulting in an input tensor shaped  $128 \times 128 \times 6$ . This concatenated tensor is then fed into a ResNet encoder that takes the 6-channel image as input. In this case, the fusion mechanism can be written as:
$$
z _ {f u s i o n} = f _ {o} \left(v _ {o} \oplus v _ {g}\right) \tag {1}
$$
This simple design yields a promising performance on the image navigation benchmark. Detailed ablation on this early fusion operation can be found in Section 4.2.
Mid Fusion via FiLM Layers. However, as the early fusion mechanism enables spatial reasoning between two images using an identical convolution kernel, it is difficult to handle the situation when the orientation of the goal camera is noisy. To alleviate this, we further propose an active fusion scheme, utilizing the adaptability of a novel fine-grained FiLM mapping module. Previous literature [21, 49] inputs the goal embedding into the ResNet visual backbone via FiLM [32] layers, which adapt a learnable affine transformation conditioned on the input embedding to the intermediate activation maps in each residual blocks. Through these layers, we can easily connect the intermediate layers in both the goal encoder and the observation encoder to perform mid fusion.
Different from the existing approaches that leverage abstract language embedding as a global condition for all layers, we propose to use the hierarchical representations from the intermediate goal encoder layers. This allows us to make good use of the fine-grained information in high-resolution feature maps. Specifically, we perform FiLM affine transformation on the resnet blocks of the observation encoder, where the affine factors  $\beta_{i,j}^{i},\gamma_{i,j}^{i}$  in block  $i$  are conditioned on the shaped activation map  $z_{g}^{i}$  from the correspondent block of the goal encoder. This process can be formulated as:
$$
\gamma_ {c} ^ {i} = f _ {c} \left(z _ {g}\right) \quad \beta_ {c} ^ {i} = h _ {c} \left(z _ {g}\right) \tag {2}
$$
$$
\hat {z} _ {o} ^ {i} = \gamma_ {c} ^ {i} z _ {o} ^ {i} + \beta_ {c} ^ {i} \tag {3}
$$
where  $\hat{z}_o^i$  denotes a transformed activation map in block  $i$  and  $c$  denotes the  $c^{th}$  feature of the feature map. The functions  $f$  and  $h$  learn to map the condition variable into the affine factors. In practice, we implement them as  $1\times 1$  convolutions to maintain the same resolution between the input and target activation map. Section 4.2 further investigates the choices of the mapping function and the number of FiLM layers. The output from the conditioned observation encoder  $f_{o}$  can then be viewed as the fused feature  $z_{fusion}$ , as shown in Figure 2 (c). The fused feature can be written as:
$$
z _ {f u s i o n} = f _ {o} \left(v _ {o} \mid v _ {g}\right) \tag {4}
$$
Our experiments in Section 4.3 reveal that the mid fusion scheme performs more robustly when the configuration of the goal camera and the observation camera is not perfectly matched.
Skip Fusion via Keypoint Matching. In order to evaluate the importance of the addition of the fusion modules, We finally replaced the aforementioned learnable modules with a heuristic one. To achieve this, we follow the idea of Wasserman et al. [44] that attach an additional low-level fusion module using handcrafted keypoint matching methods [27, 36], as an improvement of the Late Fusion baseline. We name this mechanism Skip Fusion as it fuses the goal image and observation image in the both early and later stages but skip the others, as shown in Figure 2 (b).
Keypoint matching, which aims to discover representative keypoints in an image and then describe and match them with the most similar ones in another image. As these points are detected based on the low-level statistic [27, 13] of image pixels, we leverage them to play a role as low-level fusion. This scheme is handcrafted as it is not learnable during training. To enable batch inference, we leverage a deep learning-based keypoint detecting [14] and a matching [36] method to obtain matched keypoint between the goal image and the observation image. Hereafter, we select top-k matched points according to their matching score to compose a variable  $z_{k}$  and concatenate them together with  $z_{g}$  and  $z_{o}$  as the fusion result:
$$
z _ {f u s i o n} = z _ {g} \oplus z _ {o} \oplus \operatorname {F C} \left(z _ {k}\right) \tag {5}
$$
where  $z_{k} = (x_{1},y_{1},x_{1}^{\prime},y_{1}^{\prime},\dots,x_{k},y_{k},x_{k}^{\prime},y_{k}^{\prime})$  is a flattened vector of  $k$  keypoints and FC denotes to a fully connected layer. The default value of vector  $z_{k}$  is set to  $-1$  if the number of matched keypoints is less than  $k$ . In Section 4.2, we show the superiority of our proposed Early Fusion and Mid Fusion schemes against this heuristic fusion baseline.
3.3 Navigation Policy
Based on the fused embedding  $z_{fusion}$  of the goal image and observation image, we train a navigation policy  $\pi$  using reinforcement learning (RL):
$$
s _ {t} = \pi \left(z _ {\text {f u s i o n}} \oplus a _ {t - 1} \mid h _ {t - 1}\right) \tag {6}
$$
where  $s_t$  is the embedding of the agent's current state.  $h_{t-1}$  denotes hidden state of the recurrent layers in policy  $\pi$  from previous step. Following previous methods [52, 28], we adopt an actor-critic network to predict state value  $c_t$  and action  $a_t$  using  $s_t$  and train it end-to-end using PPO [39]. We utilize the ZER reward [52] to encourage the agent to not only reach the goal position but also face the goal orientation. More details can be found in Appendix.
Datasets. As for image-goal navigation, we use the Habitat simulator [38, 42] and train our agent on the Gibson dataset with 72 training scenes and 14 testing scenes under the standard setting. We use the training episodes provided by [29] and train our agent for 500M steps. We report results under multiple datasets to allow direct comparison to various prior works. On the Gibson dataset, we validate our agent on split A generated by [29], and split B generated by [19]. On the MP3D and HM3D, we use the test episodes collected by [52], as well as the instance image navigation dataset released by [24]. We also extend our method to another embodied task named visual rearrangement, where we use the iTHOR simulator and ai2thor-rearrangement 2023 dataset with 80 training scenes, 20 validation scenes and 20 test scenes. Following [45], we train our agent for 75M steps and finally test the best validation checkpoint on the test set.
Method	Backbone	Pretrain	Sensor(s)	Memory	Split	SPL	SR
NTS [8]	ResNet9	N/A	RGBD+Pose	×	A	43.0%	63.0%
Act-Neur-SLAM [6]	ResNet9	N/A	RGB+Pose	×	A	23.0%	35.0%
SPTM [37]	ResNet9	N/A	RGB+Pose	×	A	27.0%	51.0%
ZER [52]	ResNet9	N/A	RGB	×	A	21.6%	29.2%
ZSON [28]	ResNet50	OSD	RGB	×	A	28.0%	36.9%
OVRL [47]	ResNet50	OSD	RGB	×	A	27.0%	54.2%
OVRL-V2 [46]	ViT-Base	HGSP	RGB	×	A	58.7%	82.0%
FGPrompt-MF (Ours)	ResNet9	N/A	RGB	×	A	62.1%	90.7%
FGPrompt-EF (Ours)	ResNet9	N/A	RGB	×	A	66.5%	90.4%
FGPrompt-EF (Ours)	ResNet50	N/A	RGB	×	A	68.5%	92.3%
Mem-Aug [29]	ResNet18	N/A	4 RGB	✓	A	56.0%	69.0%
VGM [25]	ResNet18	N/A	4 RGB	✓	A	64.0%	76.0%
OVRL [47]	ResNet50	OSD	4 RGB	×	A	62.5%	79.8%
TSGM [22]	ResNet18	N/A	4 RGB	✓	A	67.2%	81.1%
FGPrompt-EF (Ours)	ResNet9	N/A	4 RGB	×	A	75.0%	94.2%
NRNS [19]	ResNet18	N/A	RGBD	×	B	12.4%	24.0%
FGPrompt-EF (Ours)	ResNet9	N/A	RGB	×	B	70.5%	93.0%
Table 1: Comparison with state-of-the-art methods on Gibson. All methods are trained and evaluated both on the Gibson dataset.
Methods	Backbone	MP3D	HM3D
SPL	SR	SPL	SR
Mem-Aug [29]	Resnet18	3.9%	6.9%	3.5%	1.9%
NRNS [19]	Resnet18	5.2%	9.3%	4.3%	6.6%
ZER [52]	Resnet9	10.8%	14.6%	6.3%	9.6%
FGPrompt-MF (Ours)	Resnet9	50.4%	77.6%	49.6%	76.1%
Table 2: Cross-domain evaluation on MP3D and HM3D. The agent is trained in Gibson environments and directly transferred to new environments for evaluation.
Agent configuration. We follow the recipe of previous trails [52, 28, 47] to initialize an agent equipped with only RGB cameras of  $128 \times 128$  resolution and  $90^{\circ}$  FOV. When compared with methods that use a panoramic input, we initialize four RGB sensors to the front, left, right, and back directions of the agent, following [29, 47]. The agent's action space is comprised of four discrete actions, including MOVE_FORWARD, TURN_LEFT, TURN_RIGHT, STOP. The minimum units of rotation and forward movement are  $30^{\circ}$  and  $0.25\mathrm{m}$  respectively.
Evaluation metrics. We report the success rate (SR) and Success weighted by Path Length (SPL) [1], which takes into account path efficiency of the navigation process. An episode is considered successful if the agent stops within  $1.0\mathrm{m}$  Euclidean distance from the goal location and the maximum number of steps in an episode is set to 500 as the default setting.
4.1 Comparison with State-of-the-art Methods
Evaluation on Gibson. In Table 1, we report the ImageNav results on Gibson averaged over three random seeds (the variances of all random seed results are less than 1e-4.). We compare our methods with state-of-the-art methods in two different settings, one takes only one RGB sensor as input following [52, 28, 47] and another one takes 4 RGB sensors to assemble a panoramic view following [29, 47]. For the SLAM-based methods in the first three rows, we report the number reproduced by Mezghani et al. [29]. We found that our proposed FGPrompt-MF and FGPrompt-EF methods take an absolute advantage compared with all previous methods. Even compared to OVRL-V2 [46], a method that utilizes a much larger visual backbone (ViT-B) pre-trained on an in-domain image dataset, we still achieved large performance gains on both SR (92.3% vs. 82.0%) and SPL (68.5% vs. 58.7%) in the absence of additional pose sensor input. This finding reveals the effectiveness and efficiency of our proposed method.
Setting	SPL	SR
Later Fusion (baseline)	11.2%	13.0%
Skip Fusion via keypoint matching (FGPrompt-SF)	24.7%	41.6%
Mid Fusion via FiLM layers (FGPrompt-MF)	50.4%	77.3%
Early Fusion via joint modeling (FGPrompt-EF)	54.7%	78.9%
Table 3: Comparison of different goal prompting methods on Gibson ImageNav task. Fusing the fine-grained goal prompts with the observation instead of directly concatenating their semantic embeddings yield significant improvement.
Mapping Method	SPL	SR
N/A	11.2%	13.0%
Semantic Mapping	24.0%	32.0%
FG/HR Mapping	50.4%	77.3%
Table 4: How to map activation into affine factors? Using Fine-grained High-resolution (FG/HR) mapping performs significantly better.
Depth	SPL	SR
1	50.4%	77.3%
2	49.3%	77.6%
4	50.2%	71.4%
Table 5: How deep should the Mid Fusion perform? Performing Mid Fusion on the early layers works better than on all layers.