GOMAA-Geo: GOal Modality Agnostic Active Geo-localization
Authors: Anindya Sarkar
Srikumar Sastry
Aleksis Pirinen
Chongjie Zhang
Nathan Jacobs
Yevgeniy Vorobeychik
Venue: NeurIPS 2024 poster (2024)
Keywords: Active Geo-localization
Representation Learning
Remote Sensing
Deep Reinforcement Learning
Primary area: machine_vision
TL;DR: We tackle the task of active geo-localization, where an agent uses a series of visual cues observed during aerial navigation to locate a target specified via various modalities, while the agent is trained solely on data from a single goal modality.

Abstract: We consider the task of active geo-localization (AGL) in which an agent uses a sequence of visual cues observed during aerial navigation to find a target specified through multiple possible modalities. This could emulate a UAV involved in a search-and-rescue operation navigating through an area, observing a stream of aerial images as it goes. The AGL task is associated with two important challenges.  Firstly, an agent must deal with a goal specification in one of multiple modalities (e.g., through a natural language description) while the search cues are provided in other modalities (aerial imagery). The second challenge is limited localization time (e.g., limited battery life, urgency) so that the goal must be localized as efficiently as possible, i.e. the agent must effectively leverage its sequentially observed aerial views when searching for the goal. To address these challenges, we propose GOMAA-Geo -- a goal modality agnostic active geo-localization agent -- for zero-shot generalization between different goal modalities. Our approach combines cross-modality contrastive learning to align representations across modalities with supervised foundation model pretraining and reinforcement learning to obtain highly effective navigation and localization policies. Through extensive evaluations, we show that GOMAA-Geo outperforms alternative learnable approaches and that it generalizes across datasets -- e.g., to disaster-hit areas without seeing a single disaster scenario during training -- and goal modalities -- e.g., to ground-level imagery or textual descriptions, despite only being trained with goals specified as aerial views. Our code is available at: https://github.com/mvrl/GOMAA-Geo.

GOMAA-Geo: GOal Modality Agnostic Active Geo-localization
Anindya Sarkar',, Srikumar Sastry' Aleksis Pirinen 2.3
Chongjie Zhang', Nathan Jacobs', Yevgeniy Vorobeychik
Department of Computer Science and Engineering, Washington University in St. Louis
2RISE Research Institutes of Sweden
Swedish Centre for Impacts of Climate Extremes (climes)
Corresponding authors: {anindya, a. s.aatry)@vusttl
Abstract
We consider the task of active geo-localization (AGL) which an agent uses a sequence of visual cues observed during aerial navigation to find a target specified through multiple possible modalities. This could emulate a UAV involved in a search-and-res
operation navigating through area, observing stream of acrial images as it goes. The AGL task is associated with two important challenges. Firstly, agent must deal with goal specification in one of multiple modalities (e.g., through natural language description) while the search cues are provided other modalities (aerial imagery). The second challenge is limited localization time (e.g., limited battery life, urgency) so that the goal must be localized as efficiently possible, i.e. the agent must effectively leverage its sequentially observed aerial views when searching for the goal. To address these challenges, we propose GOMAA-Geo goal modality agnostic active geo-localization agent for zero- shot generalization between different goal modalities. Our approach combines cross-modality contrastive learning to align representations across modalities with supervised foundation model pretraining and reinforcement learning to obtain highly effective navigation and localization policies. Through extensive evaluations, we show that GOMAA-GE outperforms alternative learnable approaches and that it generalizes across datasets e.g., to disaster-hit areas without seeing single disaster scenario during training and goal modalities e.g., to ground- imagery or textual descriptions, despite only being trained with goals specified as aerial views. Our code is available at: http://githab.com/m/////G.MM
Introduction
common objective among many search-and-reseee (SAR) operations is to locate missing individuals within a defined search area, such as specific neighborhood. To this end, one may leverage indirect information about the location of these individuals that may come from natural language descriptions, photographs, etc, e.g. through social media eyewitness accounts. However, such information may allow to precisely identify actual locations (for example, these may not be provided on social media), and potential GPS information may be unreliable. In such situations, deploying a UAV to explore the area from an aerial perspective can effective for accurate localization and subsequent assistance to those who are missing. However, the field of view of a UAV is typically limited (at least in comparison with the area to be explored), and the inherent urgency of search-and rescuee tasks imposes an effective temporal budget constraint. We refer to this general task and modeling framework as goal modality agnostic active geo-localization. Pirinen et al. 24 recently introduced a
"qqual contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

deep reinforcement learning (DRL) approach for significantly simpler version of this setup, where goals are always assumed to be specified as aerial images. This is severely limiting in practical scenarios, where goal contents are instead often available as ground-level imagery natural language text (e.g., on social media following disaster). Also, since no two geo-localizotion scenarios are alike, zero-shot generalizability is a crucial aspect to consider when developing methods for this task.
To this end, we introduce GOMAA-Geo, a novel framework for tackling the proposed GOal Modality Agnostic Active Geo-localization task. Our framework allows for goals to be specified in several modalities, such as text or ground-level images, whereas search cues are provided as a sequence of aerial images (akin to a UAV with a downwards-facde facing camera). Furthermore, GOMAA-Ges effectively leverages past search information in deciding where to search next. To address the potential misalignment between goal specification and observational modalities, as well as facilitating zero-shot transfer, we develop a cross-modality contrastive learning approach that aligns representation across modalities. We then combine this representation learning with foundation model pretraining and DRL to obtain policies that efficiently localize goals in specification-agnen
way.
Given the scarcity of high-quality datasets that combine aerial imagery with other modalities like natural language text or ground-level imagery, evaluating GOMAA-Ger becomes challenging. To address this limitation, we have created a novel dataset that allows for benchmarking of multi-modal localocali..tion. We then demonstrate that our proposed GOMAA-Geo framework is highly effective at performing active geo-localization tasks across diverse goal modalities, despite being trained exclusively on data from single goal modality (i.e., acrial views).
In summary, we make the following contributions:
We introduce GOMAA-Geo, a novel framework for effectively tackling goal modality agnostic active geo-localization - even when the policy is trained exclusively on data from a single goal modality and perform extensive experiments which demonstrate that GOMAA-Geu significantly outperforms alternative approaches.
two publicly available aerial image datasets,
We create a novel dataset to assess various approaches for active geo- looaliaalintoon across three different goal modalities: aerial images, ground-leve images, and natural language text.
We demonstrate the significance of each component within our proposed GOMAA-Ger Geo framework
through a comprehensive series of quantitative and qualitative ablation analyses.
We assess the zero-shot generalizability of GOMAA-Geo by mimicking real-world disaster scenario, where goals are presented as pre-disaster images and where policy training is done exclusively such pre-disaster data, while the active geo-localization only has access to post- disaster aerial image glimpses during inference.
2 Related Work
Geo-localization. There is extensive prior work one-shot visual geo-localization 41 38 46
43.20 40 34 6 5 47 [2]. Such works aim to infer relationships between images from different perspectives, typically predicting an aerial view corresponding to a ground-level image. This problem is commonly tackled by exhaustively comparing query image with respect to a large reference dataset of aerial imagery. Alternatively, as B a model is trained to directly predict the global reo-coordinnaes of given query image. In contrast, our active geo-localization (AGL) setup aims to localize a target from its location description in one of several modalities by navigating the geographical area containing it.
LLMs for RL. LLMs have been applied RL and robotics for planning 35 42 15
Our work instead focuses on learning goal-modality agnostic zero-shot generalizable agents and aims to leverage LLMs in order learn goal-conditioned history-aware representations to guide RL agents for AGL.
Embodied learning. Our setup is also related to embodied goal navigation 2 48 19, where an agent should navigate (in a first-person perspective) in a 3d environment towards goal specified e.g. as an image. This task may sometimes be more challenging than our setup since the agent needs to explore among obstacles (e.g. walls and furniture). On the other hand, the agent may often observe the goal from far away (e.g. from the opposite side of a recently entered room), while our task is more challenging in that the goal can never be partially observed prior to reaching it.
Autonomous UAVs. There are many prior works about autonomously controlling UAVs 36 18
30 45 25 Many of these (c.g. [36 30 45) revolve around exploring large environments efficiently,

---

so that certain inferences can be accurately performed given only sparse selection of high- righ-resolttion observations. There are also works that are closer to us in terms of task setup 4 18 For example, [24] also considers the task of actively localizing a goal, assuming agent with aerial
TO,
32
31 24
view observations of a scene. However, this work only considers the idealized setting which the goal is specified in terms of top-view observation from the exact same scenario in which the agent operates. In contrast, we allow for flexibly specifying goals in an agnostic manner.
Multi-modal representation learning in remote sensing, Recent studies 17 have shown that satellite image representations can be aligned with the shared embedding space learned by CLIP 22 by using co-located ground-level imagery as an intermediary to link satellite images and language. We utilize such aligned multi-modia embedding to represent goals in modality -agnosit manner.
3 Active Geo-localization Setup
To formalize the active gco-localization (AGL) setup, we consider an agent (e.g. a UAV in -earch-and-rescue scenario) that aims to localize a goal within a pre-defined search area. This area is discretized into X Y grid superimposed on given aerial landscape (larger image), with each grid corresponding to a position (location) and representing the limited field of view of the agent (UAV) i.e., the agent can only observe the aerial content of a sub-image zt corresponding to the grid cell in which it is located at time step t. The agent can move between cells by taking actions E A (up, down, left, right, in canonical birds-eye-view orientation).
The search goal is associated with one of the grid cells, including a description
one of several modalities, and the agent's task is to reach this goal relying solely on visual cues in the form of sequentially observed acrial sub-images within fixed number of steps B. More precisely, let sg be the semantic content and Pa the location (within the search area) of the goal. We use 2gg to denote the provided description of the goal, available to the agent in the form of either (i) natural language text, (ii) ground-level image, or (iii) aerial image (sub-image within the top-view perspective search area). Notably, the true location Py of the goal is not provided to the agent, so it must be inferred by the agent during the search process. The AGL task is deemed accomplished when the agent's current position pr aligns with the goal position Pg. i.c., when p: Pg. An overview is provided Figure
Goal Cormne Poosdedaa Cooaad lmmgg
and of Agns
Irstal Inderiiee testalleiete
Goul Conser: Pontded in
Goal Conamn
Aurtal Image
Langsags ledd
Figure 1: different goal modalities. The agent must navigate to the goal (yellow dot) based on purtial aerial glimpses, i.e. the full area is never observed in its entirety.
Active geo-localization across
We model this problem as a Goal Conditondt Partially Observable Markoy Decision Process (GC- POMDP) and consider family of GC-POMDP environments M {(S" A.. X". T le c} where e is the environment index. Each environment comprises a state space St, shared action space A, observation space Xe, transition dynamics T goal space G C S and discount factor 7 E [0, 1]. The observation x² E 2 is determined by state s E S and the unknown environmental factor 6 F i.c. (s" b%), where F encompasses variations related e.g. disasters, diverse geospatial regions, varying seasons, and so on. We use xf to denote the observation at state se and step t, for domain e.
The primary objective in a GC-POMDP is to learn a history-aware goal conditioned policy T (a112% g)), where 24, (2f, 0,,, -f-1............. Gu, =f) combines all the previous observations and actions up time t, that maximizes the discounted state density function ((x) across all domains e E as follows:
( = E. Ne."" e ~ gr (1 ( (1->)\'pe = - t=
(I)
Here pf = (g represents the probability of reaching the goal g" at step i within domain e under
the policy (. (x g )d) and e and g ~ G refer uniform samples from each set. Throughout the training process, the agent is exposed to a set of training environments {ex).N = Étrain C 6, each identified by its environment index. We also assume during training that the goal content is
3

---

always available to the agent in the form of an aerial image. Our objective is to train history aware goal-conditioned RL agent capable of generalizing across goal modalities (such as natural language text, ground-level images, aerial images) and environmental variations, such as natural disasters.
4 Proposed Framework for Goal Modality Agnostic Active Geo-localization
In this section we introduce GOMAA-Geo, a novel learning framework designed to address the goal modality agnostic active geo-localization (AGL) problem. GOMAA-Geo consists of three components: (i) representation alignment across modalities; (ii) RI.-aligned representation learning using goal aware supervised pretraining of I.MM; and (iii) planning. We next describe in detail each of these components within the proposed framework, and then we explain how these modules are integrated to train a goal-conditioned policy π, capable of generalizing to unseen test environments and unobserved goal modalities after training on Etrain (cf. Section 3
Aligning representations across modalities. As we aspire to learn a goal modality agnostic policy 9, it is essential to ensure that the embedding of the goal content regardless of its modality (such as natural language text) - is aligned with the acrial image modality, as in the AGL setup we assume access only to aerial view glimpses during navigation. To this end, we take motivation from CLIP 28), which is designed to understand ground-level images and text jointly by aligning them in a shared embedding space through contrastive learning. Recent works [17 11] have demonstrated that it is possible to align the representations of aerial images with the shared embedding space learned via CLIP. The key insight is to use co-located internet imagery taken the ground as intermediary for connecting aerial images and language. Following [1 we proceed to train an image encoder tailored for aerial images, aiming align it with CLIP's image encoder by utilizing large-scale dataset comprising paired ground-level and aerial images. To align the embeddings of aerial images with those of ground-level images from CLIP, we employ contrastive learning on the acrial image encoder sæ (parameterized by 0) using the InfoNCE loss (21) in the following manner:
1 N=
exp(s) fa//T)
Calign
-log
ESEN exp(sh f///T)
Here we represent the CLIP image encoder as fg and T denotes the temperature. We optimize the Calign loss (2 to reduce the gap between co-located aerial and ground-level images within the CLIP embedding space. It is important to highlight that the CLIP image encoder remains unchanged throughout training. Therefore, our training methodology essentially permits aerial images to approach images from their corresponding ground-leve scenes and natural language text within the CLIP space. Combining the trained acrial image encoder sy with the CLIP model allows us to achieve embeddings within a unified embedding space for goal contents that span diverse modalities (in particular, aerial images, ground-level images, and natural language text). We refer to this combined model as a CLIP- Multi-Modal Feature Extractor (CLIP-MMFE)
RL-aligned representation learning using goal aware supervised pretraining of LLM. Large Language but recent studies have effectively utilized the capabilities of LLMs to grasp abstract concepts of the world model dynamics in addressing decision-making
Models (LLMs) are in general not proficient planners 33
Input Emboodiings to the LLM time step-3
Relatie Parkioad Erouiin?
Trsinable
Frecuen
GOAL-
<START>
+U>
+51>
Exjecton
Pojuetoo
Levt
challenges [35 42
CLIMASE
However, discrepancies between the knowledge of LLMS and the environment lead to inaccuracies and constrain their functional effectiveness due to insufficient grounding.
+
+
Coner Eharpe Lau
STAR>>
can
Step Generane random trsjectory using policy
Step each sten, the LLM trained to predict "heet" action hased hisorry
[ell,
Figure 2: GASP strategy for pretraining LLMs for AGL.

---

In a GC-POMDP (cf. Section we aim to learn a goal-conditioned latent representation that encompasses the complete history of observed state and action sequences, aiding the agent in decision-making within the partially observed environment. For this purpose, we leverage state-of- the-art LLMss which excel in long-range autoregressive and sequence modeling tasks. However, employing such a model naively is not conducive to obtaining a latent representation that is advantageous for goal-conditioned active geo-localization (AGL). Therefore, we devise Goal- Aware Supervised Pretraining (GASP) strategy that enables learning history-aware goal-conditioned latent representation, which assists in decision-making for the subsequent policy. An overview of the GASP strategy is depicted in Figure 2 and involves two-step training process. First, generate a random sequence of movement actions. Each random sequence of length comprises all the previous observations and actions up to time t. A t-length random sequence is denoted The = [ (o, I1, .1,,...,T,, Second, we train the LLM on a sequence modeling task that involves predicting the optimal actions time that will bring the agent closer to the goal location, based on the observed trajectory data 2h, and the goal content Ty- Accordingly, we train an LLM using the binary cross entropy loss defined
N
LBCE Σ log(p)) + (1 y&) log(1 pi))
P== (L 9)), where |p:l = |AA]
Here N is the length of the random sequence, p; is the predicted probability of actions at time step i when Th;-1 and g are given as the input to LLM,, and yi encodes information regarding the set of actions ATT C A that are considered optimal at time step i, implying that these actions will lead the agent closer to the goal location. Each element of y/ corresponds to an action in the set A, so (2) that y = [ y/ :..:..... ......y; (1A)), where ( if j'th action E A and otherwise y (d) 0. We perform extensive experiments to validate the efficacy of the GASP strategy (see Section 6), and refer
to the appendix section K for details about the GASP architecture and training process.
Planning. So far we have focused on the learning of a history-aware, goal modality agnostic latent representation useful for the AGL task in partially observable environments. Now, we describe the approach for learning an effective policy that leverages the learned latent representation to address this GC-POMDP. We refer to the latent representation obtained from the LLM at time / as ef eL (II,, 9). Formally, we aim to learn a policy æ that maximizes the expected discounted sum of rewards for any given goal g E G. To this end, we use an actor-critic style PPO algorithm [33] that involves learning both an actor (policy network, parameterized by () πī et e... (Ih,, g) + p(A) and a critic (value function, parameterized by 17) Vn, e
+ IR that approximates the true value (.leb) (2n, (zn,, [R( a, .9 + +>V (xe, g)]. We optimize both the actor and Virue (I', g) (,
critic networks using the following loss function:
CP Cplanner (c, = E, [-Colip(C) + Fa (1) - BH [T<( LLM (xN,,g))] [[
Here a and 33 are hyperparameters, and H denotes entropy, so minimizing the final term of encourages the actor to exhibit more exploratory behavior. The Lenit loss is used specifically to optimize the parameters of the critic network and is defined as a squared-error loss, i.c. gertt (Vy(et-l.M (Zh,,)) g)) Virue (z,, g))². The clipped surrogate objective Celip is employed to optimize the parameters of the actor network while constraining the change to a small value e relative to the old actor policy old and is defined as:
(Clip(C) min { Te Mc(-eeLMM (Fh.,))) A', clip (1- 11-c,1+e c,, nold| (. LLLM (1 (wh,, 9)) oold (.le JeL (Ih,, g))
Tc(-leLLM 9))
)
A rt + 7T241 +... 7" x- rT-1 Vn LLM (Mh, g)) (5)
(
After every fixed update step, we copy the parameters of the current policy network के onto the old policy network olld enhance training stability. All hyperparameter details for training the actor and critic network are in the appendix section|K]
There are numerous options for crafting the reward function for the AGL task. One potential approach involves designing a sparse reward signal, where the agent only receives a positive reward upon reaching the goal location and receives either negative reward otherwise. Nevertheless.
5

---

Encoding
ACTOR
LLM
<G0AL>
<START>
<U>
Projection
Projection
GRITIC
Projecton Laver
Projection Layer
Projection Layer
Gunnd
Value
Value
CJP MIFE
CUP-MMIE
Like
Trainable
Frozen
Trainable
Frozen
Planning: GOMAA-GeO at time
Planning: GOMAA-GeO at time
Figure 3: Our proposed GOMAA-Geo framework for goal modality agnostic active geo-localization.
incorporating a denser reward structure has been demonstrated to expedite learning and improve the efficacy of the learned policy (shown in section 6 Specifically, we formulate dense reward function customized for the AGL task as follows:
1, if Ptt1 -
Pp
Pgl Pg Pyll
R(xbe (t, g. Fe+1)
1,
if
P Po 11
J++1 E (mne)
(6)
2,
if Tt+1 = 2ig
In other words, our approach involves penalizing the agent when an action takes it away from the goal when it revisits the same state. Conversely, the agent receives positive reward when its current action brings it closer to the goal, with the highest reward granted when the action results in reaching the goal location.
GOMAA-Geo. Our full GOMAA-Geo framework integrates all the previously introduced components (see Figure 3). Initially, the aerial image encoder sp is trained to align the aerial image and CLIP embeddings. Next, the LLM is trained using the GASP strategy while maintaining 8ø and the CLIP model frozen. Finally, the LLM is also frozen, and only the actor and critic trained using RL.
5 Experiments and Results
Baselines. In this and the subsequent section 6, we evaluate and analyze GOMAA-Geo and compare its performance against the following baseline approaches: (i) Random policy selects action
uniformly at random from the action set A at each time step: (ii) AiRLoc [24] is an RL-based model designed for uni-modal AGL tasks (24). The approach involves training the policy using DRL and encoding the history of state observations using an LSTM [3]. Note that AiRLoc is not agnostic to the goal modality; (iii) PPO policy 3 selects actions based solely on the current observation; (iv)
Decision Transformer (DiT) g is trained using collection of offline optimal trajectories that span from randomly selected start randomly selected goal grids.
Evaluation metric. We evaluate the proposed approaches based on the success ratio (SR), which measured the ratio of the number of successful localizations of the goal within a predefined exploration budget B to the total number of AGL. tasks. We evaluate the SR of GOMAA and the baselines across different distances C from the analyze GOMAA-Gen with a 5 x 5 grid structure, B = 10, and varying start-to-goal distance C (4,5, 6, 8). the appendix section A, we conduct additional experiments across various grid configurations, each employing different values of B with varying C.
the goal location. In the main paper, we
Datasets. We primarily utilize the Massachusetts Buildings (Masa) dataset |20 for both the development and evaluation of GOMAA-Geo aerial imagery. The dataset is split into 70% for training and 15% each for validation and testing.
settings where the goal content provided as
Many existing datasets containing paired aerial and ground-level images lack precise coordinate locations, which are pivotal for the AGL task. Furthermore, the ground-level images typically contain

---

Table I: Evaluation with aerial image goals. GOMAA obtains the highest success ratio (SR).
Evaluation using Masa Dataset
Evaluation using MM-GAG Dataset
Method
C 5
C 6
C= 7
C
C
5
C
C= 7
Random
0,1412
0.U584
0,0640
0.0247
0.0236
0.1412
0,0584
0.0640
0.0247
0.0236
0.1427
0.1775
0.2269
0.2595
0.1489
0.1879
0.2432
DIT
0.2011
0.2956
0.3567
0.4216
0.4559
0.2023
0,2856
0.3516
0.4190
0.4423
AiRLoc
0.1786
0.1561
0,2134
0.2415
0.2393
0.1745
0.1689
0.2019
0.2156
0.2290
GOMAA-Geo
0.4090
0.5056
0.7168
0,8034
0.7854
0.4085
0.5064
0.6638
0.7362
0.7021
little to no meaningful information about the goal (e.g. images of roads, trees, and so on). In partieular, to the best of our knowledge, no open-source dataset is currently available for evaluating the zero-shot generalizability of GOMAA-Geo across diverse goal modalities, such as ground-level images and natural language text. To alleviate this, we have collected dataset from different regions across the world, which allows for specifying the goal content as aerial imagery, ground-level imagery, or natural language text. This dataset allows us to conduct contexts where the goal modality may vary at test time. Note that the data is only used for evaluation training is done on Masa using only aerial imagery as goal modality. We refer to this dataset as Multi-Modal Modal Goal Dataset for Active Geolocalization (MM-GAG), It consists of 73 distinct search areas from different parts of the world. For each area, we select 5 pairs of start and goal locations corresponding to each start-to-goal goag distance C (resulting in 365 evaluation scenarios for each C). We provide more details about the dataset in the appendix section L
AGL experiments in
Finally, to further evaluate the zero-shot generalization capability, we also compare the GOMAA-Geo with baseline approaches using the xBD dataset introduced by Gupta et al. (2019). This dataset includes aerial images from different regions, both before (xBD-pre) and after (xBD-disaster) various natural disasters such as wildfires and floods. It is important to emphasize that all the results we present this work- including the zero-shot generalization settings with the xBD-pre and xBD-post disaster datasets, as well as all across the three different goal modalities of the MM-GAG dataset are evaluated using a model trained exclusively on the Masa datasel (where goals are always specified from an aerial perspective).
Implementation details. We provide comprehensive details on network architectures and training hyperparameeters for each training stage in the appendix section K
Evaluation of GOMAA-Geo. We initiate our evaluation of the proposed methods using the Masa and MM-GAG datasets with aerial image as goal modality. During the evaluation, for each AGL task, randomly select 5 pairs of start and goal locations for each value of C. We report the result in Table [ and observe a significant performance improvement compared to the baseline methods, with success ratio (SR) improvements ranging from 129.00% to 232.67% relative to the baselines across various evaluation settings, showcasing the efficacy of GOMAA-GEO for active geo-localization (AGL) tasks where goals are provided in the form of aerial images. We next evaluate the performance of GOMAA-Geo across various goal modalities using the MM-GAG dataset and present the results in Table 2 For this evaluation, we employ the model trained on the Masa dataset. We see that GOMAA- Geo efficiently performs the AGL task across different goal modalities with comparable performance observed across different modalities despite only being trained with aerial views as goal modality. This highlights the effectiveness of the CLIP-MMFE (cf. Section 4 module in learning modality-invarian representations. Further analyses of CLIP-MMER are provided in Section 6 The experimental outcomes also demonstrate the zero-shot generalization capability of GOMAA-Geo across different goal modalities.
Table 2: GOMAA-Geo generalizes well across goal modalities.
Modality
C
C
Text Ground Image
0,4000
0.4978
0.6766
0.7702
0.6595
0.4383
0,5150
0.6808
0.7489
0.6893
Aerial Imaage
0.4085
0.5064
0.6638
0.7362
0.7021
Zero-shot generalization capabilities of GOMAA-Geo. For additional assessments of GOMAA- Geo's s zero-shot generalizability, we employ trained GOMAA-Geo model exclusively trained on the Masa dataset and evaluate them on both non-disaster data from xBD-pre and disaster data from xBD-disaster. For fair evaluation, we ensure that the training data from Masa depicts geographical areas different from those xBD. Moreover, both pre-and post-disaster evaluation scenarios, the goal content is always presented to the agent as an aerial image captured before the disaster. For
7

---

Goall
image chunch
laa facedl central entrance fanbed windows with suitedd glass Aloss the cramnno, thase is astara ofa figure that appoars relijiuss tean, a repomentation Jeus."
Argnax Policy
Stochsaiie Dolicy ('rial
Stochaarte Helley
Stochartic Palicy (Trial go
Stochaetic Palley
STARE
Figure 4: Example exploration behavior of GOMAA-Get across different goal modalities. The stochastic policy selects actions probabilistically, whereas the argmax policy selects the action with the highest probability.
the xBD-disaster setup, this thus depicts the challenging scenario of localizing goal whose visual description is provided prior to disaster, which may look drastically different when exploring the scene after said disaster. The evaluation dataset comprises 800 distinct search areas from both xBD- and xBD-disaster. These 800 search areas are identical in both xBD-pre and xBD-ddsaatter For each of these areas, we randomly select pairs of start and goal locations corresponding to each value of C. We present the zero-shot generalization results using both xBD-pre and xBD-disaster in Table 3 The results show a substantial performance improvement ranging between 221.15% to 346.83% compared to the baseline approaches and justify the effectiveness of GOMAA-Geo in zero-shot generalization.
Table 3: COMAA-Geu showcases superior zero-shot generalization than the alternatives in all settings.
Evaluation using xBD-pre Dataset
Evaluation using XBD-disaster Dataset
Method
C
C 5
C=
C=7
C= 8
C == 1
C C=
6
C = 7
C 8
Random
0,1412
0.0584
0.0640
0.0247
0.0236
0,1412
0.0584
0.0640
0.0247
0.0236
PPO
0.1237
0.1262
0.1425
0.1737
0.2075
0.1132
0.1146
0.1292
0.1665
0.1953
DIT
0,1132
0,2341
0.3198
0.3664
0.3772
0,1012
0,2389
0.3067
0.3390
0.3543
AiRLoc
0.1191
0.1254
0.1436
0.1676
0.2021
0.1201
0.1298
0.1507
0.1631
0.1989
GOMAA-Geo
0,3825
0.7489
0.7125
0.6553
Further Analyses and Ablation Studies
Effectiveness of the CLIP-MMFE module. In addition to the quantitative results in Table 2 we here conduct qualitative analysis to assess the effectiveness of the CLIP-MMFE module in learning a modality goal representation. For this purpose, we employ the GOMAA- model trained on the Masa dataset. During the evaluation, we maintain fixed search area with identical start and goal locations while varying the goal modality, and then compare the exploration behavior of GOMAA-GE AeeGo each scenario. Our observations (Figure 4 reveal that exploration behaviors are consistent across different goal modalities. This suggests that the learned representation of the goal token remains consistent regardless of the goal modality, given that the other components of the GOMAA-Geo framework remain fixed in each scenario. Additional visualizations are in the section B
Importance of learning a goal conditioned policy. To investigate the significance of goal information in the GOMAA-Geo framework, we assess a GOMAA-Geo variant denoted GOMAAAGMMA where we mask out the goal token and compare its performance against the full GOMAA- Geo. Results on the Masa dataset are presented in Table 4 We observe a substantial drop in performance ranging from 40.40% to 54.50% across different evaluation settings, underscoring the critical role of goal information in learning an effective policy for AGI..
Table 4: Providing goal information is crucial.
Method
C
4
C= 5
C= 6
C= 7
C
Mast-GOMAA
0.2913
0.3566 0.5056
0.4912
0.5200 0.8034
0.5478
GOMAA-Geo
0.4090
0.7168
0.7854
8

---

Angroo Polley
Stochartte Policy
Stochante Policy (ftial at
(fiial
Stochante Policy
START
GOALL
Figure 5: Examples of successful exploration behaviors of GOMAA-Geo.
Importance of the planner module. To evaluate the significance of the planner module, conduct experiments when removing from GOMAA-GeO and compare the performance of this modified version, termed LLM-Geo, with the original GOMAA-Geo. The only distinction between LLM-Geo and GOMAA-Geo is the presence of the planner module in the latter.
We compare their performances across various evaluation settings using the Masa
Table 5: On the importance of the planner module.
dataset, with results presented in Table We observe that the performance of LLM-Geo is significantly inferior GOMAA-Geo with performance gaps ranging from 75.46% to 129.66% across various evaluation settings. The empirical results indicate that relying solely on the LLM is insufficient for solving tasks that involve planning, highlighting the importance of combining an LLM which excels at capturing history with a planning module that learns make decisions while considering future outcomes.
Meshod
==4 4
C= 5
C
C= 7
C 8
L.M-GGo GOMAA-Geo
0.2331
0.2591
0.3121
0.3967
0.4051
0.4090
0.5056
0.7168
0.8034
0,7854
Visualizing the exploration behavior of GOMAA-Geo. In Figure 5
present a series of exploration trajectories generated using the trained stochastic policy for specific start and goal pair. Alongside these trajectories, we include an additional exploration trajectory obtained using the deterministic (argmax) policy. Please refer to the appendix for several additional qualitative results.
Efficacy of GASP pretraining. We assess the effectiveness of GASP (cf. Section 4 by comparing the performance of the original GOMAA-Geo with version of GOMAA-Geo that involves pre- training an LLM using a commonly used input token masking- based autoregressive modeling task tailored for AGL. We call the resulting GOMAA-Geo model Relative Position to Goal aware GOMAA (RPG-GOMAA). Details of the RPG-GOMAA framework, along with the masking-basee LLM pre-training strategy, are provided in the appendix section F We compare the performance of GOMAA-GG -eo with RPG-GOMAA on the Masa dataset and present the results Table 6 Our findings indicate that OPG-GOMAA shows a significant decline in performance compared to GOMAA- Geo as the distance between the start the goal C increases, which highlights the efficacy of GASP in learning history-aware policy for AGL. We also compare the zero-shot generalization capabilities of these methods using the xBD-disaster dataset and report the results in the appendix section
Figure 6: GOMAA-Geu (red)
vs. GOMAA-Ges w/o history (white).
effective
B
Furthermore, we qualitatively assess GASP's effectiveness in learning a history-aware representation for planning. To achieve this, we choose task where GOMAA- Geo successfully locates the goal. We then remove the context by masking out all previously visited states and actions except for the current state s, and goal state Vy. and observe the action selected by the policy. We compare this action to what the policy selects when the entire history provided as input to the L.LM (i.e., the default GOMAA-Geo). From Figure we see that the policy takes optimal actions with context but suboptimal actions without it, which suggests that
Table 6: GASP (bottom) yields the best results overall.
Method
C=4 4
C= 5
C
C= 7
C
=8
RPG GOMAA
0.4116
0.5167
0.6589
0.7643 0.8034
0.7023
GOMAA-Geo
0.4090
0.5056
0.7168
0.7854

---

the LLM trained via GASP effectively learns a history- representation suitable for planning. Additional visualizations are in the appendix section
Failure cases of GOMAA-Geo. Some failure cases are shown in Figure 7 In scenarios where the goal patch is very similar to many other patches in the search space, GOMAA-Geo can become unsuccessful (such scenarios are very confusing even for humans).
Argreaa Pailey
Stochaatic Daliey
Stochantic Dalicy
Stochantic Dolicy (ital g
(Trial
GOAL
Figure 7: Examples of failure cases of GOMAA Geo. Common for each example is that the goal patch is very similar many of the surrounding patches, which makes the search problems particularly challenging.
Effectiveness of proposed dense reward function. Here we investigate the effecctiveness of the proposed reward function (see Equation in the main paper) for learning an efficient policy for active geo-localization tasks. We achieve this by training the planner module with sparse reward function that assigns a positive reward of exclusively upon reaching the desired goal location, and otherwise. Throughout this analysis, we maintain all other components of the original GOMAA-Geo framework and training hyperparameters unchanged. The resulting framework is referred to as Sparse-GOMAA. We proceed compare the performance of GOMAA-Geo and Sparse-GOMAA
Table 7: On the effects of using dense reward. Left: Evaluations Masa. Right: Zero-shot evaluation using xBD-disaster. Using the proposed dense reward (bottom row) yields significantly better results than if training based on sparse reward (top row).
Test with = 10 sotting using Masa
Test with in 5 x setting using xBD-disaster
Method
c=
C =
C C= 6
C =
= 8
C
C ===3
C 6
C = 7
C
Spale-GOMAA
0.3562
0.4312
0.6009
0.7318 0.8034
0.6978
0,3615
0.3842
0.5367 0.6553
0.6455 0.7391
0.6290
GOMAA-Geo
0.4090
0.5056
0.7166
0.7854
0.4002
0.4632
0.6942
under various evaluation settings using the Masa dataset. The findings of this evaluation presented in Table (left). We observe a significant and consistent drop in performance terms of SR across all evaluation settings, ranging from 10.71% 22. 20% The observed outcomes of the experiment suggest that dense reward is indeed helpful for learning an efficient policy for active geo-localization. Additionally, assess the zero-shot generalizability Sparse- GOMAA using xBD-disaster data and compare its performance against GOMAA-Geo (right part of Table 7). We notice that the performance gap widens further, which indicates that the policy learned with the sparse reward function exhibits limited generalization capability.
7 Conclusions
We have introduced GOMAA- Geo, goal modality agnostic active geo-localization agent designed for zero-shot generalization across different goal modalities. Our method integrates cross-modality contrastive learning to align representations across modalities, supervised foundation model pretraining, and reinforcement learning to develop highly effective navigation and localization policies. Extensive evaluations demonstrate that GOMAA-Get outperforms other approaches and generalizes across datasets such as disaster-hi areas without prior exposure during training - and goal modalities (ground-level imagery, as well as textual descriptions), despite being trained solely with aerial view goals. We hope our framework will find applications ranging from search-and-rrscuee to environmental monitoring.
10

---

Acknowledgments
This research was partially supported by the NSF MUS-1195355 IIS-2214141, CNS-2310470, CCF- 2403758), Amazon, NVIDIA, and the Taylor Geospatial Institute.
References
[1] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as i can, not say: Grounding language in robotic affordances. arXiv preprint arXiv:2204001691, 2022.
[2] Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi. Manolis Savva, et al. On evaluation of embodied navigation agents. arXiv preprint arXiv:1807.06755: 2018.
[3] Guillaume Astruc, Nicolas Dufour, Ioannis Siglidis, Constantin Aronssohn, Nacim Bouia, Stephanie Fu, Romain Loiseau, Van Nguyen Nguyen, Charles Raude, Elliot Vincent, al. Openstreetview-5mt The many roads to global visual geolocation. arXiv preprint arXiv:2404. 18873, 2024.
[4] Luca Bartolomei, Lucas Teixeira, and Margarita Chli. Perception-aware path planning for uavs using semantic segmentation. In 2020 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages
IEEE, 2020.
[5] Gabriele Berton, Carlo Masone, and Barbara Caputo. Rethinking visual geo-localization for large-scale applications. arXiv preprint urXiv:2204.022881 2022.
[6] Gabriele Berton, Riccardo Mereu, Gabriele Trivigno, Carlo Masone, Gabriela Csurka, Torsten Sattler, and Barbara Caputo. Deep visual geo-localization benchmark. arXiv preprint arXiv:2204.03444 2022.
[7] Pawel Budzianowski and Ivan Vulić. Hello, it's gpt-2-how can help you? towards the use of pretrained language models for task-oriented dialogue systems. arXiv preprint arXiv:190.005774. 2019. 2019.
[8] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084-15097, 2021.
[9] Yezhen Cong, Samar Khanna, Chenlin Meng, Patrick Liu, Erik Rozi, Yutong He, Marshall Burke, David Lobell, and Stefano Ermon. Satmae: Pre-training transformers for temporal and multi-spectral satellite imagery. Advances in Neural Information Processing Systems, 35:197-211, 2022.
[10] Tung Dang, Christos Papachristos, and Kostas Alexis. Autonomous exploration and simultaneous object search using aerial robots. In 2018 IEEE Aerospace Conference, pages 1-7. IEEE, 2018
[11] Aayush Dhakal, Adeel Ahmad, Subash Khanal, Srikumar Sastry, and Nathan Jacobs. Sat2cap: Mapping fine-graine ane-grained textual descriptions from satellite images. arXiv preprint arXiv:2307. 15904, 2023.
[12] Lena M Downes, Dong-Ki Kim, Ted J Steiner, and Jonathan P How. City-wide street-to-satelltt image geolocalization of mobile ground agent. arXiv preprint rrXiv:2203.05612 2022.
[13] Klaus Greff, Rupesh K Srivastava, Jan Koutník, Bas R Steunebrink, and Jürgen Schmidhuber. Lstm: A search space odyssey. IEEE transactions on neural networks and learning systems, 28(10):2222 22322, 2016.
[14] Farizal Hamzah and Nuraini Sulaiman. Multimodal integration large language models: case study with mistral Ilm.

---

[15] Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, and Jeannette Bohg. Tex(2motion: From natural language instructions feasible plans. Autonomous Robots, 47(8):1345-1365 2023.
[16] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, Advances in neural information processing systems, 36, 2024.
[17] Utkarsh Mall, Cheng Perng Phoo, Meilin Kelsey Liu, Carl Vondrick, Bharath Hariharan, and Kavita Bala. Remote sensing vision-language foundation models without annotations via ground remote alignment. arXiv preprint arXiv:2312.0000 2023.
[18] Ajith Anil Meera, Marija Popović, Alexander Millane, and Roland Siegwart. Obstacle-aware adaptive informative path planning for uav-based target search. In 2019 International Conference on Robotics and Automation (ICRA), pages 718-724. IEEE, 2019.
[19] Lina Mezghani, Sainbayar Sukhbaatar, Thibaut Lavril, Oleksandr Maksymets, Dhruv Batra, Piotr Bojanowski, and Karteek Alahari. Memory-Augmente Reinforcement Learning for Image-Goal Navigation. working paper or preprint, March 2022.
[20] Volodymyr Mnih. Machine Learning for Aerial Image Labeling. PhD thesis, University of Toronto, 2013.
[21] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1:X7vv8807.0377 2018.
[22] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint 20ti::22300...1333, 2023.
[23] Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon I1m: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.0111 2023.
[24] Aleksis Pirinen, Anton Samuelsson, John Backsund, and Kalle Áström. Aerial view localization with reinforcement learning: Towards emulating search-and-rescue. arXiv preprint arXiv:2209.03694 2022.
[25] Marija Popović, Teresa Vidal-Calleja, Gregory Hitz, Jen Jen Chung, Inkyu Sa, Roland Siegwart, and Juan Nieto. An informative path planning framework for uav-based terrain monitoring. Autonomous Robots, 44(6):889-9) 911191. 2020.
[26] Shraman Pramanick, Ewa M Nowara, Joshua Gleason, Carlos D Castillo, and Rama Chellappa. Where in the world is this image? waasformer-basee geo-localization in the wild. arXiv preprint a204 13861, 2022.
[27] Yanyuan Qiao, Zheng Yu, Longteng Guo, Sihan Chen, Zijia Zhao, Mingzhen Sun, Qi Wu, and Jing Liu. Vl-mamba: Exploring state space models for multimodal learning. arXiv preprint arXiv:2403.13600 2024.
[28] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-88 8763. PMLR, 2021.
[29] Konstantinos Roumeliotis, Nikolaos D Tselikas, and Dimitrios K Nasiopoulos. Llama 2: Early adopters' utilization of meta's 's new open-source pretrained model. 2023.
[30] Seyed Abbas Sadat, Jens Wawerla, and Richard Vaughan. Fractal trajectories for online non- uniform aerial coverage. In 2015 IEEE international conference on robotics and automation (ICRA), pages 2971 1-767.. IEEE, 2015.
[31] Anindya Sarkar, Nathan Jacobs, and Yevgeniy Vorobeychik. A partially-supervise reinforcement learning framework for visual active search. Advances in Neural Information Processing Systems, 36:12255--222770 2023.
12

---

[32] Anindya Sarkar, Michael Lanier, Scott Alfeld, Jiarui Feng, Roman Garnett, Nathan Jacobs, and Yevgeniy Vorobeychik. A visual active search framework for geospatial exploration. In Proceedings of the IEEE/CVF Winter Conference on Applicattoos of Computer Vision, pages 8316-8325, 2024.
[33] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06::: 06347, 2017.
[34] Yujiao Shi and Hongdong Li. Beyond cross-view image retrieval: Highly accurate vehicle localization using satellite image. arXiv preprint AXiv:2204.04752. 2022.
[35] Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 11523-11 11530. IEEE, 2023.
[36] Felix Stache, Jonas Westheider, Federico Magistri, Cyrill Stachniss, and Marija Popović. Adaptive path planning for uavs for multi-resolution semantic segmentation. arXiv preprint arXiv:2203.0I1 0.6642. 2022.
[37] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, al. Gemma: Open models based on gemini research and technology. arXiv preprint wrXiv22433008295 2024.
[38] Andrea Vallone, Frederik Warburg, Hans Hansen, Spren Hauberg, and Javier Civera. Danish airs and grounds: A dataset for serial-to-street-leee place recognition and localization. CoRR. abs/2202.01821,, 2022.
[39] Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. On the planning abilities of large language models-a critical investigation. volume 36, 2024.
[40] Tingyu Wang, Zhedong Zheng, Yaoqi Sun, Tat-Seng Chua, Yi Yang, and Chenggang Yan. Multiple-enviromment self-adaptive network for aerial-view geo-localization. arXiv preprint arXiv:2204.08331, 2022.
[41] Daniel Wilson, Xiaohan Zhang, Waqas Sultani, and Safwan Wshah. Visual and object geo- localization: comprehensive survey. arXiv preprint arXiv:2112./5202, 2021.
[42] Jimmy Wu, Rika Antonova, Adam Kan, Marion Lepert, Andy Zeng, Shuran Song, Jeannette Bohg, Szymon Rusinkiewicz, and Thomas Funkhouser. Tidybot: Personalized robot assistance with large language models. Autonomous Robots, 47()):087 2023.
[43] Zelong Zeng, Zheng Wang, Fan Yang, and Shin' Satoh. Geo-localization via ground-to- satellite cross-view image retrieval. IEEE Transactions on Multimedia, pages 1-1, 2022.
[44] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975-11986, 2023.
[45] Leyang Zhao, Li Yan, Xiao Hu, Jinbiao Yuan, and Zhenbao Liu. Efficient and high path quality autonomous exploration and trajectory planning of uav in an unknown environment. ISPRS International Journal of Geo-Information, 10(10):631, 2021.
[46] Runzhe Zhu. Sues-200: A multi-heighh multi-scene cross-view image benchmark across drone and satellite, 2022.
[47] Sijie Zhu, Mubarak Shah, and Chen Chen. Transgeo: Transformer is all you need for cross-view image geo-localization. arXiv preprint arXiv:2204.0009) 2022.
[48] Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning.
In 2017 IEEE international conference on robotics and automation (ICRA), pages 3357-3364.
IEEE, 2017.
13

---

GOMAA-Geo: GOal Modality Agnostic Active Gee-localization (Appendix)
In this supplementary material, we provide additional insights into our GOMAA-Ges framework. Additional visualizations of the exploration behavior of GOMAA-Geo across 3 different goal modalities are presented in Section
B We also evaluate GOMAA-Geo across different grid sizes We then analyze the trade-off in learning modality-specinf vs.
and report the results in Section A modality-invariinn goal representation for active geo-localization in Section C In Section D we provide more visualizations of the exploration behavior of GOMAA-Geo. More qualitative evaluations and zero-shot generalizability of the GASP pre-training strategy are discussed in Section L In Section[] we compare the search performance for different choices of LLM architecture. Furthermore, in Section H. we study the importance of sampling strategy for selecting start-to-gon to-goal distance in policy training. We also evaluate and compare the performance of GOMAA-Geo with varying search budget B while keeping the value of C fixed and present the results in Section G In Section | we present visualizations of the exploration behavior of GOMAA-Geo in disaster-hit regions, even though GOMAA-Geo has been trained exclusively on pre-disaster data (thus these visualizations illustrate examples of GOMAA-Ged zero-shot generalization performance). In Section KJ provide all the implementation details, including training hyperparameters, network architecture, and computing resources used train GOMAA-Geo. Details of our curated dataset MM-GAG is discussed in Section In Section M we present the active geo-localization performance of GOMAA- with the baseline approaches across multiple trials using boxplot. We provide the details of the GPG-GOMMA framework in Section F Finally, we include a brief discussion on the limitations and broader impacts of GOMAA-Geo.
D
along
A Evaluation of GOMAA-Geo across Different Grid Sizes
Here we evaluate the performance of GOMAA-Geo in 10 10 grid setting using the Masa dataset and report the result in Table 8 We also assess the zero-shot generalizability of GOMAA- Geo in a 10 10 grid settings using the xBDtddissster dataset, and depict the findings in Table Note that in all these evaluations on larger grid sizes, we use the GOMAA-Gen model which was trained exclusively on the Masa dataset with a 5 5 grid configuration. Consequently, all results reflect the model's ability to generalize from smaller to larger grid sizes.
Table 8: Evaluations on another grid size, given models trained exclusively on smaller grid sizes. GOMAA-Geo significantly outperforms the compared methods.
Test with 8 20 10 setting
Method
C
C
Random
0.0314
0.0280
0.0157
0.0112
0.0101
DIT GOMAA-Geo
0.0923
0,1015
0.0876
0.0864
0.2427
0.2360
0.2438
0.2685
0.2685
The experimental outcomes suggest that similar to the 5 x settings, we observe significant improvement in performance compared to the baselines ranging from approximately 133.74% to 188.80% across various evaluation settings (including zero- shot generalization performance) in the 10 10 grid setting. Furthermore, results are as expected lower overall in the larger grid setting, which illustrates the difficult nature and motivates further research into our proposed active gco-localization setup.
Table 9: Zero-shot evaluation on another grid size, using xBD-disaster, given models trained exclusively on smaller grid sizes and on another dataset (Masa). GOMAA-Ga significantly outperforms the compared methods here as well.
Test with B
10 setting
Method
C 12
C = 13
C ==
C
C 16
Random
0.0314
0.0280
0.0157
0.0112
0.0101
0.0912
0.0767
GOMAA-Geo
0.2475
0,2162
0.2500
0.3650
0.2487
14

---

B More Visualizations of Exploration Behavior of GOMAA-Geo across different Goal Modalities
In this section, we conduct further qualitative analyses in order to evaluate the effectiveness of the CLIP-MMFE module (cf. Section 4 in learning moodality-agnosii goal representations. Similar to Figure (main paper), here we use the GOMAA-GeO model trained on the Masa dataset thus the agent is trained only in settings where goals are specified as acrial views. During evaluation, we keep the start and goal locations fixed while varying the goal modality, and then compare the exploration behavior GOMAA-
in each scenario. These qualitative results (shown in Figure g reveal that exploration behaviors consistent across different goal modalities. This indicates that the learned representation of the goal token remains consistent regardless of the goal modality, given that the other components of the GOMAA-Geo framework remain fixed in each scenario.
Goal Comrent
"The image shows large, buildinng prominsent contral doese and secles smaller douses towers frunt is large. open square with cobblestone pavemen. There as tall obelisk in the conter of the topped with
Argresa Pulicy
building
Stechsaric Policy (Tiaain
Stuchaaric Palicy (Trial 2)
Stochaaric Policy (Tial 3)
Stnchaasie Doliey (Trial d
STARTT
GOAL
Figure 8: Example exploration behavior GOMAA- Coo Geo across different goal modalities. In this case, the agent successfully reaches the goal for all three goal modalities in the minimum number of steps (red line).
Goal Costent
Gasil Content
"The tmagg abows pizuzenga acena of ciral kin of caral. anched windoos and huloovin, Saturing octane milligs The braidiing has claric unchieernn ando side of caral there smalles bealing wat cn aerbrme uyds.
Argraax Policy
Stochestte Balicy (Trial
Stochastte Pulicy (Trial 2)
Stochastic Pulicy (Trial g)
Stnchantie Pullry (Trial
START
Figure 9:
Example exploration behavior GOMAA Geo across diffcrent goal modalities. In this case, the agent successfully reaches the goal for all three goal modalities in the minimum number of steps (red line).
CAMAA-Geo
C Trade-off: Modality-speciffc vs. Modality-invariant Goal Representation in Active Geolocalization
Learning a modality-agnostic goal representation allows us to address the AGL problem across diverse goal modalities. Specifically, we achieve similar success ratio across all 3 diverse goal modalities as reported Table 2 (main paper). However, aligning representations across different modalities may not fully preserve the representational capabilities of the original modality.
15

---

Argmas Bolley
(fiuliy i
Chriet
Stochente Policy
Stachartic Policy
SENECE
Figure 10: Example exploration behaviors of GOMAA-Geo. The agent successful all cases.
Argmas Dalley
(fiull
Chriel
Stochante Policy (Tital
Soschrtic Palicy
Figure 11: Example exploration behaviors of GOMAA-Geo. The agent successful all cases.
To investigate this hypothesis, we replace the CLIP-MMFE module of the GOMAA-Geo framework with SarMAE 9 (while keeping all other modules of the GOMAA-Geo framework unchanged), which is specifically designed to learn useful latent representations for downstream tasks involving satellite images. This modified framework is referred as SatMAE-Geo.
Table 10: -odality-specific (top) vs. modality-invarian agents (bottom). Despite our GOMAA-Ges being modality- agnostic, it still achieves comparable results to the modality- specific approach SarMAE in the setting which SarMAE- Geo is specifically trained on.
10 setting
Method
C
C 6
C= 7
SaIMAE Geo
0.4329
0.5221
0.7016
0.7914
0.7992
GOMAA-Geo
0.5056
We then compare the performance of our modality-invarian GOMAA-Geo with the modality-specific SatMAE- Geo using the Masa dataset, where the goal content is always provided as an aerial image. The results are presented in Table I0, Interestingly, only slightly inferior compared to SarMAE- Geo (in some evaluation settings, e.g. for C = 5), despite SarMAE-Geo being targeted specifically towards AGL tasks in which goals specified as aerial images. These findings suggest that on the one hand, learning modality-agnostie representations beneficial for addressing active geo-localization problems across diverse goal modalities, on the other hand, they are equally competitive with models that are designed to solve modality-specific active geo-localization tasks, such SatMAE-Geo.
observe that the performance of GOMAA-Geo
D More Visualizations of Exploration Behavior of GOMAA-Geo
In this section, we provide additional visualizations of the exploration behavior of GOMAA-Gea across various geospatial regions. These visualizations are obtained using the Masa dataset with the goal specified as aerial images. Specifically, we present a series of exploration trajectories generated using the trained stochastic policy for specific start and goal pair. Alongside these trajectories, we include an additional exploration trajectory obtained using the deterministic (argmax) policy. We depict the visualizations in Figure TO TT and T2 These visualizations allow us to compare and contrast the behaviors resulting from these different action selection strategics.
16

---

Argmas Bolley
(fiuli i
Chriet
Stochente Policy
Stachartic Palicy
SENECE
Figure 12: Example exploration behavior of GOMAA-Geo. The agent is successful in all cases.
E More Qualitative Evaluation and Zero-Shot generalizability of GASP
We further analyze the effectiveness of GASP strategy in zero-shot generalization setting.
Similar to table 6 in the main paper, here compare the performance of GOMAA- Geo with RPG-GOMAA using the xBD- disaster dataset, while both these competitive approaches are being trained on the Masa dataset. Note that, during evaluation, the goal is presented as pre-disaster top-view imagery.
Table 11: More the efficacy of GASP. Zero-shot evaluation using xBD-disaster.
Teat with B 10 5 setting
Method
C
C 5
C
C
8
RPG-GOMAA
0.4276
0,4519
0.6003
0.6754
0.6398
GOMAA-Geo
0,4002
0.4632
0.6553
0.7391
0.6942
We report the result Table II] We observe
that the RPG-GON DPG-GOMAA method exhibits significant drop in performance across different evaluation setups compared to GOMAA-Geo (specifically, we observe a larger performance gap for higher values of C). The findings presented in table 11 showcase the importance of GASP in learning an efficient zero-shot generalizable policy for AGL.
Similar to Figure in the main paper, in this section, we present a further qualitative evaluation of the efficacy of the GASP strategy for planning. We compare the exploration behavior of GOMAA- Geo, which employs the GASP strategy during LLM pre-training, with the exploration behavior of COMAA-Ger when the context is removed by masking out all previously visited states and actions, leaving only the current state and goal state Zg. We visualize and compare these exploration behaviors with several examples, as depicted in Figure B
Figure 13: Visualization of the efficacy of GASP strategy for planning. GOMAA-Geo (red) outperforms GOMAA- without history (white).
F Details of the RPG-GOMAA Framework
The RPG-GOMAA framework is similar to GOMAA-Geo but differs in how the LLM is trained. In GOMAA- the LLM is trained using GASP. In contrast, the RPG-GOMAA framework employs a different training approach for the L.M. Initially, we generate an optimal sequence of states and actions using random policy. Then, we randomly mask state and action tokens within this sequence. The LLM is trained to predict gradient vectors at each masked state token and the correct action at
17

---

each masked action token in an autoregressive manner. The gradient is calculated as the ratio of vertical to horizontal displacement. We compute all possible gradient vectors for 5x5 grid, resulting in 47 unique gradient configurations. The pretraining strategy for masked state tokens involves predicting the correct gradient category out of these 47 categories, given the history and goal token.
G Performance Comparison of GOMAA-Geo with Varying Search Budget B
In this section, we analyze the performance of GOMAA-Geo for fixed values of C. while varying the search budget B. Specifically, we conduct experiments with C' = 5 and C = 6. For each setting, we vary B follows: B E {C, C 2,C + 4, C 6, C + 8). We perform independent experimental trials for each configuration and present the results using error bars in the boxplot shown in Figure [4 We observe that GOMAA-GES achieves higher success ratio as search budget B increases. We see the exactly same trend for different values of C.
Evaluation using Masa Dataset
B-C+2
B-C+4
D-C+6
B-C+B
Success Rate
Cag
cap
Can
coa
Cog
cas
Figure 14: Performance of GOMAA-Ger across different values of search budget B.
H Importance of Sampling Strategy for Selecting Start and Goal in Policy Training
During the training phase of the planner module, we construct training tasks by uniformly sampling distances between the start and goal locations. We then investigate how this sampling strategy affects policy training. To analyze this, we train the planner module using a curated set training tasks where distances from start to goal are randomly sampled. We evaluate the performance of both
Table 12: On the importance of sampling. Left: Test using Masa. Right: Zero-shot evaluation using xBD-disaster.
Teat with B 10 setting using Masa
Test with 8 10 in 5 setting using xBD-disaster
Sampling
C
C
C
Random
0.4913
0.5567
0.6858
0.7412
0.6908
0.4758
0.4832 0.4632
0.6054 0.6553
0.6532
0.6178
Uniform
0.4090
0.5056
0.7168
0,8034
0.7854
0.4002
0,7391
0.6942
policies using the Masa dataset and compare their zero-shot generalization capabilities using the xBD-disaster dataset, presenting the results in Table [2] Our observations reveal that the policy trained with the uniform sampling strategy performs better on average across all evaluation settings (including the zero-shot generalization setting) compared to the policy trained with random sampling.
18

---

This outcome is intuitive because random sampling tends to bias the training toward certain distance ranges from start to goal. For instance, in a 5x5 grid setting, there are more samples with distances of 4 or 5 compared to samples with distances of 7 or 8. Consequently, the policy trained with random sampling is biased toward distances of 4 to and exhibits poorer performance for distances of 7 To alleviate this issue, during the construction of training tasks, we adopt uniform sampling strategy when selecting the distances between the start and goal for policy training.
I Performance Comparison for Different Choices of LLM Architecture
In this section, we assess the performance of GOMAA-Get for different choices of LLM
architectures: Gemma (37]. Falcon (23], GPT-2 77 Mamba (27). Llama-2 [29), and
Mistral [T.] Note that we specifically choose decoder-only causal language models
for this comparison, aligning with the requirements of the GOMAA-Geo framework. We conduct this assessment using the Masa dataset. Additionally, we examine the zero-shot generalizability of different models using the xBD-disaster dataset and also compare performance across different goal modalities using the MM-GAG dataset. We present the findings in tables 14 Our experimental
Table 13: Results using different LLM models. Here, natural language text is used as goal modality.
Test with B 10 in 5 x setting using MM-GAG
Method
C 5
C C=6 6
C 7
C C=
Gemma
0.5319
0,5659
0.6851
0.5702
GPT-2
0.4936
0.6170
0.7617
0.7532
Mamba
0.4851
0.6085
0.8170
0.7574
Falzon
0.4978
0.6766
0.7702
0.6595
Llama-2
0.3659
0.4212
0.4425
0.3617
15
13
Mistral
0.3276
0,3829
0.4085
0,3829
outcome reveals that Gemma outperforms other models when evaluated using the Masa dataset. On the other hand, Falcon outperforms other models when evaluated using the xBD-disaster dataset in the zero-shot generalization setting, as reported in COMAA-Geo. is notably lower when employing Mistral or Llama-2 as the LLM model, showing the least favorable outcomes across all evaluation settings. We have identified exploring why representation learned via certain LLM models is more effective for planning compared to others
table 14 Conversely, the performance
important area for future research. We also observe that the performance of GOMAA-Ges is superior across different goal modalities when utilizing Gemma, GPT-2, Mamba, or Falcon as the LLM model as shown in table 15 and [3] Given its performance across various evaluation settings, we choose Falcon as the default choice for GOMAA-Geo framework due to its superior generalization capability compared other LLM models.
Table 14: Evaluations using different LLM models. Left: Evaluations on Masa. Right: Zero-shot evaluation using xBD-disaster.
Test with 10 in x 5 setting
Test with = 10 in x 5 setting
LLM
C 5
C c= 6
C 7
C C= 8
C=
C C= 6
C=
C = 8
Gomma
0,5044
0.7191
0.8966
0,8539
0,5075
0.6400
0.7462
0.6287
GPTF2
0.5191
0,6842
0.8247
0.8011
0.4650
0.6450
0.7262
0.6812
Mumba
0.4415
0.5258
0.6326
0.7663
0.4011
0.5393
0.6932
0.6842
Falcon
0.5056
0.7168
0.8034
0.7854
0.4737
0.6850
0.7800
0.7125
Llama-2
0.4393
0,5809
0.6011
0,5460
0.3612
0.4562
0.4013
0,3662
Mistral
0.4168
0.4752
U.5337
0.4853
0.3625
0.4162
0.4287
0.4075
Visualizations of Exploration Behavior of GOMAA-Geo in Disaster-Hi Regions (Zero-shot Generalization Setting)
In this section, we visualize the exploration behavior of GOMAA-Geo in zero-shot generalization setting. We train GOMAA-Get using pre-disaster data and then visualize its exploration behavior with post-disaster data while providing the goal as a pre-disaster aerial image. These visualizations, shown in Figure [5] highlight the effectiveness of GOMAA-Gen in addressing AGL. tasks in zero-shot generalization scenarios.
19

---

Table 15: Evaluations using different LLM models on the MM-GAG Dataset. Left: Goals specified as ground-level images. Right: Goals specified as aerial images.
Test with B 10 in 5 x setting
Test with B
10 5 setring
LLM
C 3
C 6
C
C 8
C
C
C
C
Gomma
0.5702
0.6510
0.7702
0.5519
0.6612
0.7545
0.6890
GPT-2
0.5148
0,5702
0.7957
0.7659
0.5213
0.5543
0.7687
0,7467
0.4619
0.8222
Falcon
0.5150
0.6808
0.7489
0.6893
0.5064
0.6638
0.7362
0,7021
LJama-2
0.3489
0.3744
0.3915
0,3319
0.3287
0.3719
0.4053
0.3542
Mistral
0.3617
0,3829
0.4085
0.3829
0.3745
0.3912
0.4221
0,3980
Goel as Pre Disester Accial Image
Goal as Pre Diuasier
Acrial Imnage
START
Post Dinaster Acrial Emage
Image
Figure 15: Exploration behavior of GOMAA-Geo in zero-shot generalization setting. The agent is successful in all cases.
K Implementation Details
In this section, we detail the training process for GOMAA-Geo. The proposed GOMAA-Geo framework comprises three modules: the CLIP-MMFE module, the GASP-based LLM module, and the planning module. Since each module trained independently. we discuss the training details for each module separately, beginning with the CLIP-MMFE module.
Details of CLIP-MMFE Module The network architecture the satellite encoder (so) used in the CLIP-MMFE module is identical the image encoder of CLIP (fo). As mentioned in Section 4 during the training of the satellite encoder, the CLIP image encoder remains frozen. We utilize pre-trained CLIP encoder available on Hugging Face to guide the training of the satellite encoder.
We choose learning rate of le-4, a batch size of 256, the number of training epochs as 300, and the Adam optimizer to train the parameters 0 of the satellite encoder. The objective function we use to train the satellite encoder is defined in equation] 2
Details of GASp-based LLM Module Next, we provide details of the GASP-based LLM module. We select Falcon, popular LLM architecture, for GASP-based LLM module. We initialize the GASp-based LLM module using a pre-trained Falcon model available on Hugging Face B We attach a multi-modal projection layer to the output of CLIP-MMFE, which transforms the image representations into LLM representation space. Additionally, apart from the <GDAL> token, we add relative position encoding to each state. This helps the GASP module to learn the relative orientations and positions acrial images has seen previously. Note that, relative positions are measured with respect to the top-left position of the search area. Performance comparisons with different L.LM architectures are discussed Section The loss function we use to optimize the GASP-based L.LM module is defined in equation 3 We use learning rate of le-4, batch size of 1, number of training epochs as 300, and the Adam optimizer to train the parameters of the LLM module.
20

---

Planning module Finally, the planner module consists of an actor and a critic network. Both these networks are simple MLPs, each comprising three hidden layers with Tanh non-linear activation layers in between. We also incorporate softmax activation at the final layer of the actor network to output probability distribution over the actions. We use learning rate of le-4, batch size of 1, number of training epochs as 300, and the Adam optimizer to train both the actor and critic network. We use the loss function as defined in equation to optimize both the actor and critic network. We choose the values of and 3 (as defined in equation 4 to be 0.5 and 0.01 respectively. We also choose the clipping ratio (e) to be 0.2. We select discount factor 7 to be 0.99 for the experiments and copy the parameters of onto Told after every 4 epochs of policy training. Note that in this work we consider an action invalid if it causes an agent to move outside the predefined search area. During training, we divide the images into 5 5 non-overlapping pixel grids each of size 300 300.
Compute Resources We single NVidia A100 GPU server with memory of 80 GB for training and a single NVidia V100 GPU server with memory of 32 GB for running the inference. required approximately 24 hours to train our model for about 300 epochs with 830 AGL tasks each with maximum exploration budget (B) of 10, while inference time 24.629 seconds for 890 AGL tasks with C 6 single NVidia V100 GPU.
L MM-GAG Dataset Details
The dataset was built by collecting high-quality geo-tagged images from smartphone devices. After filtering the images, the dataset contained 73 images in total across the globe. The global coverage of the dataset is depicted in Figure 16 For each ground-level image, we downloaded high-resolution satellite image patches at spatial resolution of 0.6m per pixel. To be able to create 5x5 grids, we downloaded 10x 10 satellite image patches size 256x256 pixels. Further, we automatically a multimodal large language model. We used the prompt: "Describe the contents of the image in detail. Be to the point and do not talk about the weather and the sky".
captioned each ground-level image using LLaVA-7A [16] [
Figure 16: Locations of the samples in our MM-GAG dataset.
Sampling locations.
The ground-level images were collected from diverse group of users via small-scale crowdsourcing effort. We made every effort to ensure the images were sourced from wide range of countries (the dataset covers 11 countries). The sampling locations were determined by the GPS information embedded in the EXIF data of the images, not by manual selection. The purpose of using the privately sourced images was avoid leakage into any of the foundation models.
21

---

Data filtering, Initially, we collected 82 images. We applied basic filter to the collected ground- level images, based on the availability of GPS data. Since needed retrieve satellite imagery corresponding to each ground-level image, we required that each image include GPS information in its EXIF data. Images lacking GPS information were excluded We did not apply any further filtering. Finally, our dataset comprises 73 ground-level images.
Regarding potential biases. As mentioned before, did best to ensure diversity among the ground-level images. Our dataset features both indoor and outdoor scenes from II different countries. Furthermore, we report in Table the average pairwise similarity between the images in our dataset, computed using cosine similarity of the corresponding image embeddings from various vision models.
Table 16: Comparison of average pairwise similarity between the images in dataset, computed using different backbones.
Vision Model
Dino V2 [22]
SigLIP [44]
CLIP 28
Avg. Pairwise Similarity
0.10:2.22
0.32
0.56
The low average pairwise similarity suggests that the images in our dataset represent a diverse range of concepts. Finally, we would like to emphasize that single ground-level image can be utilized to generate up to 300 potential start and goal scenarios by spatially adjusting the grid of satellite images. By randomly initializing start and goal locations and averaging the results over 5 different random seeds enabled to robustly evaluate our model.
Link the dataset. Our dataset is publicly available this link
M Analyzing Geo-Locallzztion Performance across Multiple Trials
Here, we compare the search performance of all the baseline approaches along with GOMAA-Ges across 5 different trials for different evaluation settings using boxplot. In Figure Inndd 18 we report the result with B 10 and grid configuration using Masa and MM-GAG dataset respectively.
0.8
0.6
Success Rate
+
lo
0.0
kandoen
Airoc
MD GrataA- Gea
Nandon
ttr
Autric
coaaa-Ben
Rundon
Anros
gratad.oo Deo
HD
Rundden
Abeos
WD Goata--ee
handow
thr
Astrcc
WO
Figure 17: Boxplot of different methods for different choices of using Masa Dataset.
"Note that no GPS information was provided to GOMAA Geo when tackling the active rco-localization task (as we are interested in tackling scenarios where GPS information is lacking or disturbed), i.c. the GPS positions were only used during the dataset creation, match ground-level views satellite imagery.
22

---

C 5
C-m 6
0.6
Success Rate
0.0
Pandhon
Aitoc
GONSA Gaeo
PPO
Ranovna
Dir
Aitos
PRO coMbA Gao
Panobon
Dr
Aitoc
PP
Pandlas
Airoc
PPO Good Dae
Random
Der
Artoc
60R66 Cato
PRO
Figure 18: Boxplot of different methods for different choices of using MM-GAG dataset.
N Limitations
As a work that pioneers the task of modality-invariant active geo-localization, some limitations have been included in order provide streamlined, reproducible and controllable first and experimentation setup. Such limitations include: (i) the ego-pose of the agent is assumed to always be known and noise-free (whereas in real-world deployment, UAV would need to handle this as well); (ii) we have utilized grid-like state-action space as opposed to continuous one (which would be the case in real UAVs); and (iii) the agent's movements are constrained plane, i.e. it cannot change elevation during exploration (however, even in real-world applications, there may be circumstances where a UAV would have to hover at a limited and constant clevation, c.g. to avoid heavy winds during sarchhand-reseuus operation in windy area). We emphasize that these limitations relatively minor when considered relation to the several novel aspects introduced in our work and thus we leave further steps towards real-world deployment for future work.
O Broader Impacts
The development of GOMAA-Geo has the potential to create significant positive downstream impacts across a wide range of applications, most notably enhancing automated search-and- rescue operations and environmental monitoring. These advancements promise to deliver tangible benefits terms of efficiency, accuracy, and overall effectiveness, fundamentally transforming how these critical tasks performed.
serrch-and-escuus operations. GOMAA-Geu has potential to revolutionize search-and-rescue missions by enabling flexible, rapid and precise localization of individuals in distress. In disaster- stricken areas, such as those affected by carthquakes, floods, or wildfires, the ability of GOMAA-GE -Geo to process various forms of data (c.g., acrial images, ground-level photographs, textual descriptions) ensures that rescue teams can quickly locate and assist survivors. This can lead to a significant reduction in the time taken to find missing persons, ultimately saving lives and resources.
Environmental monitoring. Environmental monitoring stands benefit immensely from the capabilities of GOMAA-Geo. By efficiently analyzing and interpreting data from diverse modalities, this technology can be used to enhance the monitoring of wildlife, forests, water bodies, and other ecological assets. This improved monitoring capability can lead to better-informed conservation efforts, more effective management of natural resources, and early detection of environmental hazards, contributing to the preservation of ecosystems and biodiversity.
23

---

Further considerations. While the potential benefits of GOMAA- Geo are substantial, it is imperative consider the ethical implications of its deployment. We strongly advocate for the responsible use of this technology, emphasizing applications that contribute to the common good. Specifically, we advise against and condemn any misuse of GOMAA-GeO in contexts that could harm individuals or societies, such as in warfare surveillance that infringes privacy rights.
P Additional Visualizations of Exploration Behaviors of GOMAA-Geo in Scenes Containing Object Categories Other than Forests and Buildings
In Fig. 19 we present further visualizations GOMAA-Ged exploration behavior in scenes that contain additional land cover types (i.e., beyond buildings and forests). For example, the leftmost figure depicts a visual scene with lake, with the goal located in the middle of the lake. The middle figure also features lake, with the goal situated observe that the GOMAA-Geo agent (red line) successfully locates the goal and follows the optimal path. Finally, the rightmost figure shows a scene with large parking lot, where the goal is situated. Even in this scenario, the GOMAA-Geo agent successfully locates the goal in a minimal number of steps. The exploration behaviors presented in the main paper, appendix, and Fig. 19 demonstrate GOMAA- -eo s zero-shot generalization capability for active geo-localization across diverse scenes.
the shore. In both of these scenarios,
Argmas Polley
Stockastie Palley
Strchestle Rulley 2)
Stocheste (Tril g
(Trisl
STNKY
LEDAL
Figure 19: Exploration behaviors of GOMAA-Geo across different scenes.
Exploration Behavior of GOMAA-Geo with Goal not at Boundary of the Search Area
we present additional visualizations of GOMAA-Geo exploration behavior in scenes where the goal is a boundary. Similar scenarios with the goal at the boundary, observe that GOMAA-Geo successfully locates the goal by following an optimal path in these cases as well. These visualizations, along with those presented the main paper and appendix, demonstrate that GOMAA-Ger is efficient in locating the goal, regardless of its position in the search area.
In Figure 20
Argeas Dalley
Stochasite Belley
Stecheaste belley a)
(Pridl
Belley (Trial
START
Figure 20: Example exploration behaviors of GOMAA-Geo with goals not at border.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Section discusses our proposed framework, while Sections and and the results from our experiments, supporting the claims made in the abstract and introduction. We provide further analysis of our proposed framework throughout the Appendix section, also justifying the claims made in the abstract and introduction.
present
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer:
[Yes]
Justification:
We discuss the "Limitations" in section N of the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
25

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA].
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes].
Justification: We discuss the relevant details of our experimental setup Section 5 aad 6 also discuss the details of the architectures, training hyperparameters corresponding each training stage in Section of the Appendix. We also include code and data that can be used recreate results, including the weights of our proposed model in the supplementary material. Moreover, detailed documentation along with all code, data, and models will be made publicly available upon acceptance, to facilitate easy reproduction of all the results in this work.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
26

---

(d)
reproducible, they We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes].
Justification: We include our code in the supplementary material following the submission guideline. Data could be downloaded separately, detailed in the README file in the root directory of the code repository. We also include the weights our trained model, which can be loaded for evaluation without having run the script train it from scratch.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
Sublic/guidess wttp://ooooonnoiiioooololoee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/nipss.ccpuulic//uune http:/uddiees/eeeeee / https///oooeSubbeiiioooodeoeedee
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes].
Justification: All the necessary details are provided in the main paper in section and The details of the training hyperparameters, type of optimizer, etc. are discussed in Section of the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
27

---

Yes, in section M of the Appendix, we provide the error bars obtained by evaluating our model across different independent trials.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section Kf the Appendix discusses our computing resources and the total CPU time required to run our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttt::///eurrppp httpps///eerrpps.ccppbiiiic
Answer: [Yes].
Justification: We follow the NeurIPS Code of Ethics. Our research does not involve human participants, and our data were obtained in a way that respects the corresponding licenses, and processed to preserve anonymity.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there is a special consideration due to laws regulations in their jurisdiction).
28

---

10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the broader impacts of our work in Section O of the Appendix, which also includes necessary considerations that should be taken into account when using our model.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We judge that our work poses no immediate risk of misuse of our released weights and data.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
29

---

Justification:
All works whose data are used by us are appropriately credited and the corresponding licenses were respected; see, for example, Section 5 in the main paper and Section K in the Appendix.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The README file in our code includes documentation on how to use the data and the weights of our trained policy network.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA].
This work includes no crowdsourcing experiments, nor research with human subjects. However, described in the main paper, we have ourselves collected proof-of- concept dataset of triplets (satellite image, ground-level image, natural language text) for showcasing the ability of our approach generalize across different goal modalities. None of the images or text descriptions contain sensitive information (e.g. about other people).
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
30

---

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our work does not involve human participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "91RzWN9Qpx", "reviewer": "Reviewer_4muN", "summary": "The paper introduces GOMAA-Geo, a novel framework for active geo-localization (AGL) that is capable of zero-shot generalization across different goal modalities. GOMAA-Geo is designed to assist agents, such as UAVs in search-and-rescue operations, to locate targets specified through various modalities (e.g., natural language, ground-level images, or aerial images) using a sequence of visual cues observed during aerial navigation. The framework addresses two main challenges: 1. Dealing with goal specifications in multiple modalities while relying on aerial imagery as search cues. 2. Limited localization time due to constraints like battery life, necessitating efficient goal localization. GOMAA-Geo integrates cross-modality contrastive learning to align representations across modalities, foundation model pretraining, and reinforcement learning to develop effective navigation and localization policies. The framework has been evaluated extensively and shown to outperform alternative approaches, demonstrating its ability to generalize across datasets and goal modalities without prior exposure during training.", "questions": "See the weakness", "limitations": "I suggest the authors to discuss the negative impacts of proposed method to our society like public safety.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper proposes a novel approach that combines cross-modality learning, foundation model pretraining, and reinforcement learning for active geo-localization, which is novel to some extent.\n2. GOMAA-Geo's ability to generalize across different datasets and modalities without prior training exposure, enhancing its applicability in diverse real-world scenarios. The framework has undergone rigorous testing and comparison with alternative methods, demonstrating its effectiveness through quantitative and qualitative analyses.\n3. The authors have created a new dataset to facilitate benchmarking, which is beneficial for future research in the area.", "weaknesses": "While the framework shows promise, it may require further validation in real-world conditions with actual UAVs and under various environmental challenges.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments. We are pleased to see that you found our proposed framework novel and that the new dataset will be valuable for future research in this field. Below, we address all your comments. \n\n> **Q1**:  While the framework shows promise, it may require further validation in real-world conditions with actual UAVs and under various environmental challenges..\n\n**A1**: In this work we have focused on laying the ground work for the novel and real-world relevant task of Global Navigation Satellite System (GNSS)-denied goal localization, which we call *goal modality agnostic active geo-localization*. Evaluations of our main GOMAA-Geo approach, as well as an extensive set of baselines and ablations, are conducted on multiple real-image datasets (as also pointed out in the reviewer's list of the paper's strengths), including under various environmental challenges (see e.g. generalization from pre-disaster to post-disaster scenarios in Table 3). Natural next steps indeed include evaluating these types of methodologies also using real UAVs, as suggested by the reviewer. We plan to explore this in follow-up work, and given that our code and data will be made publicly available, others will be able to explore this as well.\n\n\n> **Q2**: I suggest the authors to discuss the negative impacts of proposed method to our society like public safety.\n\n**A2**: Please refer to “Broader Impacts” in the appendix (page 21), which discusses both positive and negative potential downstream applications. Specifically, under “Further considerations” (Line 804-808), we discuss potential negative downstream applications of these types of methodologies. We are grateful for the reviewer's consideration of these issues, and wholeheartedly agree that it is important to carefully consider both positive and negative potential downstream effects when developing new methodologies.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IzaIfcoKaO", "reviewer": "Reviewer_9JwT", "summary": "This paper introduces GOMAA-Geo, a framework designed to enhance active geo-localization for the UAV, enabling them to find targets specified through various modalities like natural language or imagery. It addresses the challenge of efficient localization in dynamic environments without specific training on disaster scenarios. Utilizing a modality-agnostic approach with cross-modality contrastive learning and a combination of pretraining and reinforcement learning, GOMAA-Geo achieves robust navigation and localization. It conducted extensive experiments across various datasets and demonstrated its effectiveness.", "questions": "What is the full name of the UAV? There is no explanation in the whole paper, which will cause confusion to readers.", "limitations": "It addresses the limitations in the Appendix.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- It proposes a new dataset named Multi-Modal Goal Dataset for Active Geolocalization (MM-GAG) for active geo-localization across different goal modalities: aerial images, ground-level images, and natural language text.\n- This paper introduced a framework named GOMAA-Geo, which utilizes a modality-agnostic approach, combining cross-modality contrastive learning to generalize across different goal specifications and modalities.\n- Extensive experiments are done in this paper, which is quite laborious.", "weaknesses": "- The introduced dataset is the main contribution of the paper, but the details of the dataset are very limited, and there is no more information in the appendix, such as the details of data collection, why the zero-shot generalization performance of the evaluation model is better?\n- The writing needs to be improved, especially the method part, which is quite confusing. In 240 lines, it is claimed that the complete GOMAA-Geo framework integrates all the components introduced before, but it is not easy to find the corresponding relationship from the text to the figure, and the connection between Figure 1, Figure 2, and Figure 3 is also difficult to sort out.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "While we very much appreciate the feedback on our work, we were surprised to find that this reviewer has recommended rejecting the paper, given that the listed \"Strengths\" seem to clearly outweigh concerns listed under \"Weaknesses\" (where no technical concerns about our work were mentioned) nor under \"Questions\" (where only a clarification about an abbreviation is requested). We address the listed concerns and the question below.\n\n> **Q1**: The introduced dataset is the main contribution of the paper.\n\n**A1**: We wish to emphasize that while our dataset is indeed a significant contribution, we do not view it as the *main* contribution. Rather, our main contribution is the introduction and extensive evaluation of a novel framework designed to effectively address goal modality agnostic active geo-localization, even when the policy is trained solely on data from a single goal modality. However, we do agree that our dataset is also an important contribution, both on its own (we carefully considered how to design it so that it is easy to use for future researchers, and as mentioned will release it together with the code upon paper acceptance), and because it allows for assessing and evaluating various methods within our novel task setting (i.e., it is key for validating our main contribution). We summarize our contributions in Lines 55-67.\n\n> **Q2**: The details of the dataset are very limited, and there is no more information in the appendix, such as the details of data collection.\n\n**A2**: We describe the proposed MM-GAG dataset on Lines 263-276 in Section 5 “Experiments and Results”, and provide additional information in the Appendix section titled “MM-GAG Dataset Details”. Together, these parts of the paper describe key dataset details such as (i) locations in the world where the data were captured (Fig. 15); (ii) **details of data collection** / creation procedure (beginning at Line 762 in the appendix); (iii) number of evaluation scenarios covered (Line 273-275). If the reviewer can identify concrete missing details, we would be happy to provide these as well. And, of course, we will release the dataset for future research in this field upon acceptance on GitHub along with detailed documentation. We plan to use the GitHub issue tracker to gather requests for additional information about the dataset to ensure ease of use for future researchers.\n\n> **Q3**: Why the zero-shot generalization performance of the evaluation model is better? \n\n**A3**: We attribute the model's superior zero-shot generalization performance to the CLIP-MMFE module (see Lines 149 - 170 for a detailed discussion on this; we also encourage the reviewer to see Lines 323- 331 where we empirically validate the effectiveness of CLIP-MMFE and provide corresponding visualizations in Fig. 4). One of the most impressive features of the CLIP-MMFE module is its ability to align features across different modalities (such as aerial imagery and text) with the CLIP ground-level image encoder known for its zero-shot generalization capabilities. \n\nMoreover, based on the results in Table 2, one of the unseen modalities during training (\"Ground Image\") appears surprisingly to lead to better results compared to the modality seen during training (\"Aerial Image\"). The reason for this is that the ground-level image task is a little bit easier because it is constrained to be in locations with discriminative information (the available ground-level images typically depict discriminative scenery such as unusual/salient buildings etc), whereas aerial images are not similarly constrained. The way in which we see that \"Ground Image\" gives slightly better results is by computing the average across the different C-values (i.e. across columns) of Table 2 and getting the mean results for Text, Ground Image, and Aerial Image to be 0.6008, 0.6145 and 0.6034, respectively. Thus the results are on average best for Ground Image and worst for Text, with Aerial Image in between. Note, however, that all results are quite similar, which showcases the robustness across the various modalities. We also refer the reviewer to Appendix D, where we discuss this “trade-off” between modality-specific vs. modality-agnostic goal representation in active geo-localization. Our findings suggest that on the one hand, modality-agnostic representations are beneficial for addressing active geo-localization problems across diverse goal modalities, and on the other hand, they are equally competitive with models that are designed to solve modality-specific active geo-localization tasks, such as SatMAE-Geo.\n\n\n> **Q4**: The writing needs to be improved, especially the method part, which is quite confusing. In 240 lines, it is claimed that the complete GOMAA-Geo framework integrates all the components introduced before, but it is not easy to find the corresponding relationship from the text to the figure, and the connection between Figure 1, Figure 2, and Figure 3 is also difficult to sort out.\n\n\n**A4**: We appreciate the reviewer's suggestions on improving readability, including improving the consistency between Fig 1-3. We will make improvements as suggested in the revision.\n\n\n> **Q5**: What is the full name of the UAV?\n\n**A5**: UAV stands for “Unmanned Aerial Vehicle”; we will ensure this is clarified in the revised paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qr2cv0LZK5", "reviewer": "Reviewer_noHa", "summary": "In this paper, the author proposes a direction classification task, which is for the drone path navigation. \nMy main concerns are about the task. \nUsually we have the global satellite-view image, shall we estimate the direction? \nMost works do the location retrieval, which is a common practise in the geolocation.", "questions": "Please see the Weakness.", "limitations": "Please see the Weakness.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "In this paper, the author proposes a direction classification task, which is for the drone path navigation. \nThe method seems sound. \nThe idea is presented clear.", "weaknesses": "1. My main concerns are about the task. \nUsually we have the global satellite-view image, shall we estimate the direction? \nMost works do the location retrieval, which is a common practise in the geolocation. \n\n2. Can the network general to unseen scenarios? \nFor example, there are many similar buildings or similar forests. \n\n3. I am not sure the network could predict the future according to the history. \nBecause there are no global map as input. \n\n4. The drone usually do not go back, so the estimation space is only 3 directions. \nIf we further consider the boundary or corner, there are only 2 / 1 direction. Will it affect the model? \n\n5. I see the samples shown in the paper. \nI see a biases toward the boundary and corner. Could you show more samples with destination at the center? \n\n6. Missing alignment. \nHow about give a destination between the two patches?\n\n7. The direction estimation is not new. Similar methods have been explored in the RL and detection. \n\n8. Typos. i.e. should be i.e.,", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper! We hopefully address all of your current concerns / potential misunderstandings below.\n\n\n> **Q1**: My main concerns are about the task. Usually we have the global satellite-view image, shall we estimate the direction?\n\n**A1**: This appears to be a misunderstanding. We are **_not_** proposing a “direction classification task” (DCT), i.e., the task is not merely to predict a direction pointing toward a specific goal, but to _explore a partially explored environment to reach the goal_. This may imply that the agent needs to also explore places that are not taking it closer to the goal, to reveal contextual information which may later be useful for efficiently moving towards the goal. We denote this novel task as _goal modality agnostic active geo-localization_.\n\nA DCT is _only used as a pre-training task_ (see Sec. 4, L. 171-210, Fig. 2). After pre-training, our full GOMAA-Geo system is refined with RL (see Sec. 4, L. 211-243). In Table 5 we clearly show the benefit of the additional RL-based fine-tuning on top of the DCT warm-up task.\n\n**Comparison to other works:** There indeed exist many works on what we call _static_ geo-localization, which often assume access to a broader satellite view perspective, and where the mapping between a ground-level and top-view (satellite) image is treated as an image retrieval problem. We discuss this in Related Work (see e.g. L. 69-75). However, high-resolution satellite imagery is not available publicly everywhere in the world, and GPS information can sometimes be absent or unreliable (e.g. in search-and-rescue operations after natural disasters or warfare activity). Motivated by this, our work tackles geo-localization in partially observable GPS-denied environments, and we specifically tackle it in a _goal modality agnostic_ setup, since if GPS is not available, an agent must be able to robustly find a given location regardless of how it is specified.\n\n> **Q2**: Can the network generalize to unseen scenarios? For example, there are many similar buildings or similar forests.\n\n**A2**: Absolutely! This is one of the key strengths of our framework, which we validated e.g. by conducting extensive experiments with open-source datasets specifically designed to test GOMAA-Geo's ability to handle a range of unseen pre- and post-disaster real-world scenarios. Despite not being trained on any post-disaster aerial images, GOMAA-Geo performs well on images captured after disasters (Table 3). See also L. 307-321 for a detailed discussion on this.\n\nAdditionally, the experimental section and the appendix contain extensive experimental results regarding generalization to unseen scenarios and goal modalities. See e.g. L. 287-306, Table 2, Appendix B, C, F. See also the **attached PDF** for example exploration behaviors of GOMAA-Geo in scenes containing additional land cover types (i.e., beyond buildings and forests). These will be added to the appendix of the revised version.\n\n> **Q3**: Not sure the network could predict the future according to the history.\n\n**A3**: The agent is _not_ tasked to predict the future based on its history of sequentially observed top-view glimpses and actions, but rather to predict a promising next action that ultimately leads the agent to reach its goal. We discuss this in the main paper (L. 376-387) and show clearly in Fig. 6 and in Appendix F that the full history is useful for this, as opposed to e.g. only basing the decision on the most recent state. \n\n> **Q4**: \"The drone usually do not go back, so the estimation space is only 3 directions. If we further consider the boundary or corner, there are only 2 / 1 direction. Will it affect the model?\"\n\n**A4**: When in a non-boundary position, the agent can always select any of the 4 actions (up, down, left, right), but as it is aware of its relative position in its search area (_not_ global GPS location; see L. 110) and a history of past relative positions, it quickly learns during training -- based on the reward (eq. (6)) that penalizes such abnormal behaviors -- that it is mostly not beneficial to move to its previous position. When it is on a boundary, we invalidate actions that would take the agent outside the search area (see L. 753-754), which is handled by sampling / selecting only from the available actions. We rarely observed abnormal behaviors from our trained GOMAA-Geo model, as evidenced by numerous visual illustrations of exploration behaviors in different scenarios throughout the paper (see e.g. Fig. 4, 5, 6 in the main paper; also Fig. 7-12 in the Appendix).\n\n> **Q5**: Could you show more samples with destination at the center?\n\n**A5**: See the **attached PDF** with additional visual results, where the goal is in the middle parts of the search area (will also be added to the appendix in the revision). Note that cases with the goal closer to a border typically correspond to scenarios where the start-to-goal distance is larger, so the examples that we focused most on in the paper can be expected to be more difficult.\n\n> **Q6**: Missing alignment. How about give a destination between the two patches?\n\n**A6**: We unfortunately have trouble understanding this remark. It seems you are asking if we could represent goal locations that aren't centered within a given cell. Our current framework assumes a discrete domain and doesn't require that goal locations be centered within the target cell, only that they are contained within the field of view.\n\n> **Q7**: The direction estimation is not new. Similar methods have been explored in the RL and detection.\n\n**A7**: This misunderstanding was addressed in A1 above. We emphasize that *we are not proposing direction estimation*. Rather, we propose a novel, more complex task of goal modality agnostic active geo-localization. To the best of our knowledge, this task has not been explored by others and is therefore novel. \n\n> **Q8**: Typos.\n\n**A8**: Thanks, we'll fix the identified typos in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lEJ7p15kX2", "reviewer": "Reviewer_DjP3", "summary": "The paper introduces GOMAA-Geo, a system designed for active geo-localization tasks where the goal can be specified in various modalities, including text, ground-level, or aerial imagery. The framework uses a combination of cross-modality contrastive learning to align representations across different modalities, supervised foundation model pretraining, and reinforcement learning to derive effective navigation and localization policies. It demonstrates strong performance in both same-modality and cross-modality localization tasks, with an emphasis on zero-shot generalization to new, unseen datasets.", "questions": "N/A", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. Modality Agnosticism: The ability to handle goal specifications in various modalities makes GOMAA-Geo highly versatile and applicable in diverse real-world scenarios, particularly in search-and-rescue operations where goal modalities can vary significantly.\n\n2. Zero-Shot Generalization: The framework is effective at generalizing to completely new environments and goal modalities it has never seen during training, which is crucial for practical applications where pre-collected data might not be available.\n\n3. Robust Performance: Extensive testing shows that GOMAA-Geo consistently outperforms other state-of-the-art methods in terms of success rates across multiple datasets and modalities, confirming its efficacy.", "weaknesses": "1. Complex Integration: The integration of multiple complex components—contrastive learning, supervised pretraining, and reinforcement learning—while effective, can be challenging to optimize and maintain, potentially limiting its adaptability.\n\n2. Computational Demands: The processes involved, particularly the training across different modalities and the reinforcement learning steps, might require substantial computational resources, which could be a barrier to deployment in resource-constrained environments.\n\n3. Limited Discussion on Failure Cases: The paper could benefit from a more detailed discussion on scenarios where GOMAA-Geo fails or performs suboptimally, which would be valuable for future improvements and practical deployments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks, we appreciate the reviewer's valuable feedback!\n\n\n> **Q1**: Complex Integration: The integration of multiple complex components—contrastive learning, supervised pretraining, and reinforcement learning—while effective, can be challenging to optimize and maintain, potentially limiting its adaptability.\n\n**A1**: We agree that our approach relies on more than a single “end-to-end learning protocol”, and in that sense, it might be somewhat more laborious to train our method from scratch. However, a similar argument can be made for how the 3-step approach (contrastive learning; supervised pretraining; reinforcement learning) makes the overall system more robust, modular, and adaptable. By this, we mean that, assuming e.g. a SOTA reinforcement learning algorithm emerges in the future, then most of the system can be kept intact and only the third step needs to be replaced with that new RL approach. Finally, note that our work pioneers a new task setup and set of approaches for tackling it, and future work may well result in even more streamlined approaches for the task (which we encourage also via releasing all code and pre-trained models upon paper acceptance).\n\n> **Q2**: Computational Demands: The processes involved, particularly the training across different modalities and the reinforcement learning steps, might require substantial computational resources, which could be a barrier to deployment in resource-constrained environments.\n\n**A2**:  We first want to re-emphasize that our GOMAA-Geo model is _**not**_ trained across different modalities. This is also described e.g., in the abstract (Line 20-21), Section 3 “Active Geo-localization Setup” (Line 137-138), and Section 5 “Experiments and Results” (Line 271-272). We do agree, however, that RL training is typically a time- and/or resource-consuming task, which may indeed be difficult in resource-constrained environments. But, we would also like to highlight here that all models were trained on a single dual GPU workstation, which we believe is well within the budget of many research labs. More on compute resources/requirements are described around Lines 756-760 in the appendix.  Finally, as also listed by you under the strength “2. Zero-Shot Generalization”, our results clearly show that GOMAA-Geo generalizes well to completely new environments and goal modalities, so e.g. research groups/practitioners with less compute available may still be able to successfully use these approaches without further RL training. \n\n> **Q3**: Limited Discussion on Failure Cases: The paper could benefit from a more detailed discussion on scenarios where GOMAA-Geo fails or performs suboptimally, which would be valuable for future improvements and practical deployments.\n\n**A3**: Thank you for pointing this out; we will add several failure case examples with associated discussions to the appendix of the revised version. Please see also the **attached PDF** with examples of failure cases, where one can see that in scenarios where the goal patch is identical to many other patches in the search space, our GOMAA-Geo agent can become unsuccessful (but note that it would be confusing even for humans in these settings, as we also rely on more discriminative aspects when looking for something).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their valuable feedback and insightful comments. We are glad that the majority of the reviewers find the zero-shot generalizability across goal modalities to be a strong contribution of our work, and agree with the reviewers that extensive experiments have been conducted in order to validate various claims and contributions. We considered all the concerns raised by the reviewers carefully and hopefully clarified all of their queries. Furthermore, we have attached a PDF with additional visualizations as requested by the reviewers.

Meta-review: This paper received mixed reviews, including a Weak Accept, two Borderline Accepts, and a Borderline Reject. Reviewers who lean towards acceptance appreciate the novelty of the dataset and the technical contributions. However, concerns were raised regarding the lack of detailed dataset information, insufficient insights into certain results, and an absence of discussion on failure cases. The rebuttal effectively addressed most of these issues, leading three reviewers to agree on accepting the paper.

The reviewer 'noHa' (Borderline Reject) raised several concerns, primarily around the problem formulation, particularly the shift from a traditional retrieval approach to a path-finding problem. While these concerns were largely addressed in the rebuttal, 'noHa' remained cautious. The AC, however, supports the authors' novel framing of the problem, recognizing it as a fresh perspective that could inspire further exploration in the field, as current geolocalization methods largely focus on image retrieval.
Decision: Accept (poster)