GeoCLIP: Clip-Inspired Alignment between Locations and Images for Effective Worldwide Geo-localization
Authors: Vicente Vivanco Cepeda
Gaurav Kumar Nayak
Mubarak Shah
Venue: NeurIPS 2023 poster (2023)
Keywords: Geo-localization
Image-to-GPS retrieval
CLIP
Random Fourier Features

Abstract: Worldwide Geo-localization aims to pinpoint the precise location of images taken anywhere on Earth. This task has considerable challenges due to the immense variation in geographic landscapes. The image-to-image retrieval-based approaches fail to solve this problem on a global scale as it is not feasible to construct a large gallery of images covering the entire world. Instead, existing approaches divide the globe into discrete geographic cells, transforming the problem into a classification task. However, their performance is limited by the predefined classes and often results in inaccurate localizations when an image's location significantly deviates from its class center. To overcome these limitations, we propose GeoCLIP, a novel CLIP-inspired Image-to-GPS retrieval approach that enforces alignment between the image and its corresponding GPS locations. GeoCLIP's location encoder models the Earth as a continuous function by employing positional encoding through random Fourier features and constructing a hierarchical representation that captures information at varying resolutions to yield a semantically rich high-dimensional feature suitable to use even beyond geo-localization. To the best of our knowledge, this is the first work employing GPS encoding for geo-localization. We demonstrate the efficacy of our method via extensive experiments and ablations on benchmark datasets. We achieve competitive performance with just 20% of training data, highlighting its effectiveness even in limited-data settings. Furthermore, we qualitatively demonstrate geo-localization using a text query by leveraging the CLIP backbone of our image encoder. The project webpage is available at: https://vicentevivan.github.io/GeoCLIP

GeoCLIP: Clip-Inspired Alignment between Locations and Images for Effective Worldwide Geo-localization
Vicente Vivanco Cepeda, Gaurav Kumar Nayak, Mubarak Shah
Center for Research in Computer Vision, University of Central Florida, USA wricentee.vivancccer gauravkumar.. .nayak) @ucf edu; shah@crcv. acf.ede
Abstract
Worldwide Geo-localization aims to pinpoint the precise location of images taken anywhere Earth. This task has considerable challenges due the immense variation in geographic landscapes. The image-to-imal fail solve this problem global scale as it not feasible construct a large gallery of images covering the entire world. Instead, existing approaches divide the globe into discrete geographic cells, transforming the problem into classification task. However, their performance is limited by the predefined classes and often results inaccurate localizations when an image's location significantly deviates propose GeoCLIP, novel
aetrieval-basee approaches
from its class center. To overcome these limitations, CLIP-inspired Image-to-GPS retrieval approach that enforces alignment between the image and its corresponding GPS locations. GeoCLIP's location encoder models the Earth as a continuous function by employing positional encoding through random Fourier features and constructing a hierarchical representation that captures information at varying resolutions to yield a semantically rich high- dimensional feature suitable to even beyond gco-localization. To the best of our knowledge, this is the first work employing GPS encoding for geo-localization. We demonstrate the efficacy of our method via extensive experiments and ablations benchmark datasets. We achieve competitive performance with just 20% of training data, highlighting its effectiveness even in limited-data settings. Furthermore, we qualitatively demonstrate geo-localization using text query by leveraging the CLIP backbone of image encoder. The project webpage is available httts::/viceeterr/// wwwpeeiiiiiniiitbubbb.o 1o/GeoCLIP
Introduction
Image aco-localization refers determining the geographical location where photograph is taken from. This problem is of great significance in a wide variety applications, such as navigation, tourism, and security. However, this task poses several challenges, especially for locations that lack distinctive landmarks or are outside tourist hotspots. Typically, image-based geo-localization mainly focuses identifying the GPS locations of images within specified region of interest (e.g., within city). However, worldwide geo-localization [12] 5) is not limited to any specific area but to the entire globe, thus making it significantly more challenging, and hasn't 't been explored extensively.
Geo-localization the global level requires finding the locations (i.c., latitude and longitude) of images from different parts of the world. Many retrieval-based methods [28 [1] 34 29 which perform image-to maage-to-image retrieval limit their search space to specific citics. Extending such approaches for worldwide geo-localization demands maintaining gallery of all possible images of the entire globe to perform a match with the query image. However, creating such a large image gallery is infeasible; thus, these methods inappropriate for retrieval at the global scale. Existing works instead use classification-batt
32
24 20 33
approaches [4] 26 12 5 27 [8] using scenes, hierarchy, queries, and segmentation. These methods divide the Earth into geographic classes (covering particular area) and apply a linear classifier for prediction. an image belonging to a class is far from
37th Conference on Neural Information Processing Systems (NeurIPS 2023).

---

(A) Problem Setup
(8) Approach
ER
ER
p>
GPS Gallery
(60 1128 10000
G
Query
(-S51 NOMe 150 2016
{
(7.6178")
Pairs of (lat, lon)
Location Encoder
(53.9921 1.418")
Retrieva
V(-)
7017 90.19157)
(53.9921
1.541
Ag
Retrieved GPS
CLIP
M
Gallery
Coordinates
aeguaatVV Le)
Similanity Matrix (lat, ton)
Feature
Pairs of (lat, (om)
Query Image
Image Enooder
Figure 1:
The objective of image- image-based geo-localization is to find the GPS coordinate of a given image. (A) We formulate this problem as an Image-to-Ge to-GPS retrieval approach where the image is a query, and the gallery is collection of GPS coordinates. (B) We propose novel method, GeoCLIP, having two major components: location encoder (-) obtain high-dimensional features from the 2D GPS coordinates and CLIPP-bsee image encoder V-)) to ohtain the image features. We perform the matching features a query image against a gallery of GPS embeddings, and the most similar GPS embedding is the predicted GPS coordinates.
its center, it would result in localization error even though the classification prediction is correct. Also, the predefined classes limit the prediction to R 21k possible locations, which not dense enough to predict any location in the world. Moreover, these methods often need large amounts of training data which takes days to train even with high computational resources. Thus, there is a need for new effective and efficient approach that can better model the relationships between images and their corresponding geographic locations. We summarize the problem setup and compare the major differences between existing works and our proposed method in Figure
To address the above limitations of existing approaches, we propose a CL] inspired image-to- GPS retrieval approach (named 'GeoCLIP") where we retrieve the GPS coordinates of an unseen query image by matching with the gallery of GPS coordinates. To facilitate matching, we design location and image encoders to project GPS and image in shared embedding space. Our location encoder transforms a given 2D GPS coordinate into a high-dimensional feature space. The traditional latitude and longitude systems have inherent distortions in representing certain areas (e.g., countries near to poles and equator have inconsistent representations). Hence, we first convert the GPS locations using equal earth projection [17] to minimize such distortion. Another challenge is obtain a high-dimensiooal representation of a 2D GPS coordinate. The direct use of standard MLP leads to spectral bias, they approximate low-frequee fwwffrequency functions and fail capture fighffrequecc details with oimedimesioonal inputs. Previous studies [2] have demonstrated that such spectral bias significantly degrade the performance in various applications, particularly when capturing these details is crucial for the task. To alleviate this, we employ a positional encoding technique using Random Fourier Features (RFF) (2 before processing through feedforward network.
To enrich the location features, we further enforce hierarchical representation in our location encoder by capturing features at different granularities (ranging from coarse to fine fine-grained resolutions) and combining them to obtain a rich figh-ddmensiioal feature. Specifically, we vary the frequencies in RFF by changing sigma values using exponential assignment strategy, enabling us process the 2D GPS coordinates at different resolutions. We employ contrastive loss between features of the image and GPS. It is well established that the incorporation of additional negative samples enhances the performance of contrastive learning. Our gallery is a collection of GPS coordinates instead of images. Hence is easy for to fetch negatives by uniformly sampling the GPS coordinates from the globe. To this end, we implement a queue of GPS values to serve as artificial negatives. Our location encoder learns generic features Moreover, using the CLIPP (15) backbone in our image encoder helps extract meaningful features and allows us compare GPS coordinates with text descriptions, which is not feasible existing works. Besides this, our image-to-GPS retrieval method facilitates a much finer evaluation than previous techniques, resulting in better localization performance.
find them suitable for even beyond reo-localization.
Below we summarize our major contributions as follows:
To the best of our knowledge, we are the first to solve worldwide geo-localization task via image-to-GPS retrieval approach (GeoCIIIP) by explicitly aligning the features of images to corresponding GPS locations.
2

---

Our location encoder incorporates positional encoding with random Fourier features to efficiently encode GPS coordinates and mitigate spectral bias in MLPs. In addition, we use an exponential sigma assignment strategy to facilitate learning hierarchical features at different resolutions (Sec. 3.T.I..
We validate the efficacy of our location encoder in an additional application, highlighting its versatility beyond geo-localization tasks (Sec. 4.5).
Unlike existing works that only use an image query, we also qualitatively demonstrate global gco-localization with text query using GeoCLIP (Sec. 4.4).
We also demonstrate the effectiveness of our approach in limited data settings (Sec. 4.2
2
Related Works
This section briefly discusses the relevant works related to our approach.
12 14], have tackled the problem of global image geo-localization. While the most common localization papers use an image-to-image [2]. this is not yet feasible on the global scale because of the data collection that would be necessary for a global reference dataset to compare against. Instead, Weyand el al. (27 proposed to split the earth into individual classes to predict the location of an image. Through constructing these classes, the prediction can be relatively coarse or fine- une-grained based the geographic size of the classes. Vo et al. (26) utilized multiple levels of geographic classes to learn different granularities, and Sco et al. [8] was the first to combine these levels together for enhanced prediction. Müller et al. [12 additionally uses the scene labels of images to learn by having separate encoders designated to learn each scene's 's features. Recently Clark et al. [5] made use of even more hierarchies and scenes than the previous works, they also decoder to learn different features from every input image that represents each hierarchy and scene. The use of semantic segmentation was introduced by [14] in an attempt to provide additional context and understanding for the model to better geo-localize. They use two transformer encoders, one for the RGB image and the other for the segmented representation of that image, and share information between the encoders to get better location prediction. Unlike existing global image geo-localization works, we instead take an image-to-GPS retrieval approach by directly matching the query image features to the features of GPS data.
Global Image Prediction:
Many previous works [2 18 26
retrieval technique [16 [9 20 25
34
specialized transformer
Learning from GPS Data:
The GPS coordinates of image are used in [2 10
31 3 to
provide additional context when performing classification. They construct GPS embeddings by leveraging image tags, textual descriptors, and information from geospatial databases, which are paired with image embeddings to improve the classification accuracy. We differ from this work by utilizing Random Fourier Features (RFF) (21 with MLPs. RFF is often used in NeRFs represent the location or viewing angle for reconstructing the image or scene at that location or from that view. However, our location encoder uses RFF to represent a specific location on the earth and aligns them with the corresponding image features. We later also show that GPS embeddings of our location encoder is generic and can even be utilized for classification tasks as well, leading to further performance gains and showing its utility beyond gco-localization tasks.
Contrastive Learning: It is used in wide variety of tasks in multiple deep learning- based disciplines. The basic idea is to match the features from a positive pair while pushing away from negative example features. Our method uses two different contrastive learning strategies, SimCLR and (L) SimCLR uses a base image and creates two separately augmented versions of it. It then attempts to match these two versions of the image together and push away the other images that are not from the same base. They show that this creates a robust feature representation of images. On the other hand, CLIP utilizes a large image-text database to learn shared feature space between its image and text encoders. To establish a meaningful and distinct representation for both the text and image, they enforce similarity between the image features and the corresponding text caption. CLIP has been widely used in many tasks since its inception. We utilize the pretrained model from CLIP as backbone to our image encoder, while our augmentation strategy similar to SimCLR. However, instead of matching the image to another image, as in SimCLR, or to text, as in CLIP, we match it to a gallery of GPS features. The additional benefit of using CLIP backbone is the ability to compare GPS coordinates through our location encoder and text from CLIP's text encoder, which provides a very intriguing qualitative analysis that we show later (Sec. 4.4p. Now, we explain our proposed approach in detail in the next section.
3

---

GPS
GR
Froeen
Location Encoder [()))
Random Fourier Fieetures (RIF)
GPS Baton
G
L
GPS
Location Encoder
(lof, 2on)
V(-)
G
L R
(Itt, Jon)
TAV
fula
Batch
E
y
Image Enooder
Fexture Similarity Mattix
Figure 2: (A) Given a batch of GPS coordinates (GB), we generate additional negatives Q using our dynamic queue strategy. Each image in batch (1 is processed using our image encoder V(-) with CLIP backbone. Similarly, we process the GPS coordinates using our location encoder (-).). We train our model to align image features with corresponding GPS embeddings using contrastive loss over feature similarity matrix. (B) (.(.)
transforms 2D coordinates G, into G using equal earth projection (EEP)[ Then obtain hierarchical representations of G², using RFF 21 and MIPs, aggregated to obtain rich high-dimensional representation L,-
3 Proposed Approach
Given the access to training dataset Dtrain {(In, G,)) containing pairs of image Ln and GPS coordinate G,. Our goal is to train a world- wide model W using samples from Dtrain and then use the trained model W to predict the location of unseen K query images of test dataset D test Specifically, our objective is accurately predict the latitude and longitude query images, G = W(T Vk E [I..K] where T E Drest- The images the query set 7ọ can belong to any geographical location of Earth, making it challenging setup.
Worldwide Image Geo-localization Setup:
We formulate this problem as a retrieval problem. Traditionally, retrieval involves matching a query image with images from the gallery, and the GPS coordinates of the closest gallery image would be the desired output. This approach is often used when the search space is limited to specific cities. However, has limitations when applied to retrieval at the global scale, which requires the construction of a gallery containing images covering the entire world. Instead, it is much easier to build a gallery of GPS coordinates. Hence, geo-localization can be framed an image-to-GPS matching problem where the closest GPS match would be the location of the query image.
In order perform matching, the features of the image and GPS location need be extracted. Hence, we have two major components in our architecture, i.e., Location Encoder _((-) and Vision Transformer (-)) as Image Encoder. Now, explain the model architecture containing details of both the encoders, followed by the training and evaluation strategies.
3.1 Model architecture
The architecture of our method GeoCLIP is shown in Figure]
The CLIP 155 model has demonstrated strong generalization performance across a wide range of computer vision tasks. Motivated by it, our image encoder contains a Vision Transformer, which is a pre-trained CLIP ViT(6)-L/14 model. We use it backbone and keep frozen. To adapt this backbone for our gco-localization task, we add two linear trainable layers with dimensions of hn and h2, respectively. These additional layers are employed to fine-tune the learned representation for our task. Hence, our overall image encoder V(-) extracts the features V, (I.)). Vi E []...N] and I; E Dirain-
Image Encoder:
Next, we explicitly encode the GPS coordinates, which has not been tried for the geo-localization task. We now explain our location encoder in detail, which encodes the GPS coordinates of location.
3.1.1 Location Encoder
Encoding 2D GPS coordinates (i.e., G. IR2 &R, E NJ) to high-dimensional representation (RP) where D > 2, is challenging task. A simple use of standard MLPs for such encoding suffers from spectral bias 21l (unable to capture nigh-freeqeecy details). Hence, to handle this challenge and effectively perform this encoding, employ different components in encoder, i.c., representing

---

GPS coordinates using equal earth projection 17], using positional encoding through random Fourier features, and enforeing hierarchical representation various scales. We explain each of them below:
Equal Earth Projection (EEP): To accurately represent 2D GPS coordinates and minimize distor- tions inherent in standard coordinate systems (i.e., countries closer the poles are overrepresented in traditional latitude and longitude systems), we employ the Equal Earth Projection ([Z)) By utilizing EEP, we counterbalance the distortions and ensure more accurate representation of the Earth's s surface. Given G; E IR² (in radians), we transform it into G², by applying the EEP as below:
2/3G cost
G
G
Pa 00 + PP + PP2 PP
(1)
3(9 P1 8 + + 86 +3 32 02 + P1)
where sin 0 After applying the EEP. we scale the resulting longitude in the range 1 to 1, and the latitude values are scaled proportionally.
v3 sin Glat P1 1.340264, P2 = 0.081106, P3 0.000893, P1 0.003796.
2
Random Fourier Features (RFF): To capture rich high-frequency details from low-dimensional GPS input, we apply sinusoidal-basee positional encoding technique to G Specifically, we leverage random Fourier features (2I for this positional encoding. We construct an encoding layer that determines the range of frequencies be encoded. We limit the frequencies using fixed matrix R. whose entries are sampled from a Gaussian distribution with the standard deviation (a). The matrix R is set at the beginning of training and remains unchanged throughout the training process. The RFF operation 7(-) encodes GPS coordinate G¹, as (G)) = [cos(2TRG)) sin(2 where the entries of a m th row and n column of matrix R. are r Im.n ~ (0,
Hierarchical Representation: To construct hierarchical representation capable of capturing information at varying scales, we vary the frequency in RFF to obtain multiple high-dimensional representations spanning from coarse to fine-gres grained. As the frequency range in RFF depends on the a value, we propose an exponential assignment strategy for choosing the a values in our location encoder. Specifically. for given range of values (i.e., [amins o maoz D. we obtain the values for M level hicrarchy using the following:
plogg(o ((tgg//mmm))(()((la)a))) (i-1))
(3/(MM))))))))))
Vi (1..
This exponential assignment of a values enables our location encoders (-)) to effectively process the input 2D coordinates at different resolutions. Consequently, this hierarchical setting allows (-)-) to specialize in capturing features of particular location at various scales. It enforces our model to efficiently learn rich understanding of spatial information, making it suitable for a wide range of geospatial deep learning tasks.
Next, we pass the encoded hierarchical features through feedforward MLP. Each MLP indepen- dently processes the outputs of RFF and yields a vector v; E IRd, We then perform element-wise addition of these resulting vectors to obtain joint representation v = EM Vi-
Below summarize the overall operations performed by our location encoder 2((-) for obtaining rich high-dimensional encoded features (L,) of a GPS 2D coordinate G::
M
L = L(G,) Σ ((G(EEP(G))..))
3.2
Model Training
We train our model using contrastive learning scheme (similar to [15] where we maximize the similarity of features of an image I; (i.c., Vi) with its corresponding location features of GPS coordinate G₁ (i.e., Lj) and minimize its similarity with other GPS coordinates in batch. To further improve our learned representation, we also adopt a training scheme similar to SimCLR 2
We apply set of random augmentations (A) to each image in a batch and create views of an image to improve the quality of learned representations and promote better generalization. We employ the same set of augmentations used in the SimCLR. For each augmented image Iij (ith sample of
Dirain and jth view, we also create variation in its corresponding GPS coordinates G; by injecting random noise 7i to (denoting by G.j) which helps to enforce spatial smoothness in the location encoder. The noise T7 is sampled from a Gaussian distribution with standard deviation of o, (i.c.,
5

---

NO o))) This added noise encourages the location encoder to learn more robust representations that are less sensitive to small variations in the GPS coordinates.
At any point during training, our location encoder (..) can obtain the embedding of any GPS location g (i.e., Iig) without requiring a corresponding image. We exploit this capability generate additional negative samples during training to improve the learned representation with contrastive learning. We append queue Q of GPS locations with a fixed length S every batch of training data Dirain. During training, we generate embeddings for these GPS coordinates (L). We also inject noise (1/ ~ NO0, of.)) to I., such that By o This additional queue of GPS embeddings notably improves the performance of our model, particularly at smaller scales (refer Table B(a).
Hence. for an 1thh sample of batch (1,2 C E Dirain) having P augmented views with temperature T, our overall training objective is to minimize the following loss:
Σ log E exp Lig/T) + o exp(Vi) LL,TT)
exp (Va)= Liy/T)
exp
After the model parameters are updated on a batch of Drrain with batch size [B), we update the queue Q by replacing its oldest coordinates with GPS coordinates from the current batch G where [B] < S = [Q), ensuring an up-to-date reflection of the most recent GPS coordinates encountered (naming it as 'dynamic queue' strategy).
Evaluation: By modeling the Earth as a continuous function, our location encoder can generate an embedding for any GPS coordinate on the planet. Hence, our method allows us to directly compare the embedding of a query image with the embeddings of any chosen GPS location, effectively eliminating the constraint of working within a predetermined set of classes.
Unlike existing works that create classes using the knowledge of the training set, we create gallery of possible GPS coordinates during the evaluation phase by randomly sampling from the training set. The other ways for constructing the GPS gallery are discussed in the supplementary. To obtain the
GPS predictions (C@) for 1tth image in the unseen query set (T²), ) we compare the query image's embedding with the embeddings of GPS coordinates in our gallery. The GPS coordinate with the
highest similarity the query image's embedding indicates the most probable location for the image (i.e., Go arg max of EDies V(I (GG)).
4 Experiments
Datasets and Evaluation details: We perform experiments on publicly available benchmark datasets to have a fair comparison with existing works. For training, we use the MediaEval Placing Tasks 2016 (MP-16) dataset ( which consists of 4.72 million geotagged images from Flickr [.. We test our trained model on several different datasets: Im2GPS3K (7]. the recently introduced Google World Streets 15K dataset (GWS15k) [5]. and YFCC26k (23.. During testing. we conduct image-to-GPS retrieval where the query image from the test datasets is matched against a GPS gallery of 100K (Im2GPS3K) and 500K (GWS15k) coordinates. The performance of GcoCLIP on different gallery sizes is put in supplementary. In our evaluation, similar to (5). we also uses Ten Crop (5 different crops of an image and their flipped counterparts) method where the predictions on these crops are averaged to make a joint prediction. We report our results using threshold metric, similar to the protocol followed in previous works. Specifically, for each query image, we compute the Geodesic distance between GPS coordinates predicted by our model and the respective ground truth. We calculate how many of them (in %) fall within the distance thresholds (1km, 25km, 200km, 750km, and 2500km) and report average performance of model over three runs.
Implementation details: Each MLP f/ used in our location encoder .-(-) is composed of an input layer with 512 dimensions, four hidden layers (each with ReLU activation and 1024 dimensions), and an output layer of 512 dimensions. The RFF of the positional encoding layers have an input size of dimensions and an output size of 512 dimensions. The dynamic queue (QQ) as 4096) is initialized randomly with values drawn from a uniform distribution covering the range of latitude and longitude. We add gaussian noise to GPS coordinates with o, and oi as 150 and 1000 respectively. Further ablations on the noise and queue length are put in supplementary. We use three hierarchies (M as 3) with miin as 20, and Ømaz as 2 respectively (refer Sec. 4.3). Our image encoder VV-) utilizes OpenAl's pre-trained CLIP ViT-L/14 as backbone with two trainable linear layers hy and hz having dimensions of 768 and 512 respectively. We perform image augmentations similar to SimCLR.

---

Training details: Our GeoCLIP model is trained using Adam optimizer with a learning rate of 3 x 10 and a weight decay of x 10 We use learning rate of 10-4 4 to train classifiers with
the CLIP backbone. We employ step decay learning rate scheduler with gamma of 0.87 and a step size of one epoch. The model trained until convergence, which typically occurs around tenth epoch. We use batch size [B| as 512 when training on full dataset. For data-efficient settings with 20%, 10%, and 5% of the data, we use [B] as 256, 256, and 128 respectively. The learnable temperature parameter is initialized to 0.07 (refer eq. 4). The CLIP backbone is frozen during training while location encoder and h, and hi layers are fully trainable. We train our models on an NVIDIA Ampere GPU along with 12 CPUs. Our code is in PyTorch [3] and is shared in supplementary.
4.1
Comparison with State-of-the-an methods
Table I: We compare the performance of GeoCLIP with the state-of-the-art methods on (a) Im2GPS3k 7 and (b) GWS15k SS datasets. Our method yiclds consistent gains across datasets and different distance thresholds.
(a) Results on the Im2GPS3k 7
dataset
(b) Results on the recent GWS15k S
dataset
Street
City
Region
Country
Continent 2500 km
Method
km
25 km
200 km
750 km
Street km
City
Region
Country
Continent
[.JENIN
26
7.2
19.4
26.9
38.9
55.9
Method
25 km
200 km
750 km
2500 km
CPlaNet 18
PlaNet 27
ISNs 12
10.5
10.2
8.5
28.0
24.8
26.5
34.6
34.3
36.6
48.6
49.7
48.4
64.6
64.6
66.0
Translccater
GooDecoder X
ISNs 12]
0,05
0.5
0.7
0.6
1.1
1.5
4.2
8.7
25.5
15.5
26.9
50.5
38.5
Translocator
14
11.8
31.1
46.7
58.9
80.1
GooDeooder
12.8
45.9
61.0
76.1
0.6
3.1
16.9
45.7
Our
14.11
34.47
50.65
69.67
83.82
We perform comparative analysis of GeoCLIP against worldwide Geo-Localization benchmarks, Im2GPS33 and Google World Streets 15k (GWS15K). The results are shown in Table I In all the metrics, our method surpasses the previous otate-of-the-art (SOTA) performance on Im2GPS3K, achieving improvements of +1.31 %. +0.97%, +3.95%, and +8.67% in the 1km, 25km, 200km, 750km, and 2500km thresholds respectively. The results on another dataset (YFCC26K) are put in supplementary, where we also observe similar trend.
Notably, our approach exhibits large gain on the more challenging GWS15k dataset, surpassing the previous SOTA model with significant accuracy improvements of +1.6%. +8.2%, +18.8, and + +23.6% in the 25km, 200km, 750km, and 2500km thresholds respectively. Our performance is nearly double the accuracy of the previous state-of-the -fte--of-the-at model. The GWS15k contains samples that are uniformly sampled across the Earth and are not biased towards any specific geographic location. Moreover, the images in this dataset have large distribution shift compared to the training set, making the geo-localization task tough and challenging. Our substantial improvement can be attributed to the better embeddings achieved by our encoders. Specifically, the hierarchical nature of the location encoder helps focus on multiple resolutions of a location simultancously, and its alignment with corresponding -uitt--ditti images gets further enhanced with the robust features obtained by the CLIP backbone our image encoder.
4.2 Performance of GeoCLIP in Limited Data settings
The worldwide geo-localization methods assume access to a training set containing millions of data samples. Training on such large scale datasets is often highly tone-conuu,ngg even with substantial computational resources at hand, taking days to complete. Moreover, practical limitations may arise, such as restricted access to data due to privacy concerns or proprietary rights held by companies. Hence, we also investigate the potential of our GeoCLIP method in limited-data settings. In the F we present a comprehensive analysis of our method's performance on Im2GPS3k dataset across various data settings: 20%, 10%, 5% of the full dataset. Results on other datasets provided in the supplementary. We can observe that our method maintains a high level of performance, closely approximating the results achieved with the full dataset, even as the available data decreases exponentially. Notably, even with a mere 20% of the data, the performance of GeoCLIP drops by only 0.0%, 0.5%, 1.6%, 1.9%, and 1.0% 2500km, 750km, 200km, 25km, and kkm distance thresholds respectively. These results highlight the efficiency of GeoCLIP in limited-data scenarios, emphasizing its potential to address practical challenges in worldwide geo-localization.
To further demonstrate the better efficacy of our method in limited data settings, we perform the same experiment with classification approach [2] (ISNs). We summarize the comparison of our method
7

---

with the classification approach on different quantities of limited training data in Table It can be easily observed that the classification method's performance degrades much faster than our approach when the amount of training data is reduced, highlighting our method to be more data-efficient.
GeoCLIP: Data Effi ciency
83A
83.6
83.2
80
70-
69-7
692
69.3
67.6
Acuracy
11.1
30
12.1
1.0%
20%
10%
5%
Percentage of Available Data
Figure 3: We vary the amount of training data from 100% to as low as 5%. GeoCLIP achieves respectable accuracy even with only 5% of the training data, demonstrating its utility in limited data scenarios.
Table 2: Comparison of data efficiency of GeoCLIP with respect classification approach when the amount training data reduced drastically. The value inside brackets on each row shows performance degradation compared to the respective method's 's performance with 100% training data.
Im2GPS3K (Amount of Training Data)
Method
100%
20%
10%
5%
2500 km LIe Ours)
83.8
83.8 ([0.0)
83.6 ((U.2)
83.2 (10.6)
2500km (ISNs 12)
66.0
50.1 (115.9)
48.7 (L17.3)
43,9 ((22.1)
750km (GeoCLIP Ours)
69.7
69.2 (J0.5)
69.1 (J0.6)
6T26
750km (ISNs
49.7
31.9 (L17.8)
(L20.5))
(124.6)
200km (GcoCL IP. Ours)
50.6
(31.6)
(C1.7)
48.7 (11.))
200km (ISNs 122
36,6
2016.6)
(119.2)
14.3 (122.3)
25km (GeoCLIP. Ours)
34.5
(21.9)
31.9
31.1 (13.4)
25km (ISNs
12
28.0
14.1 (113.9)
11.9 (L16.I)
(.18.))
1km (GeoCLIP. Ours)
14.1
13.1 (11.0)
12.1 (U2.0)
10.8 (13.3)
1km (ISNs
10.5
5.1 (45.4)
4.5 (16.0)
3.2 (17.3)
4.3 Ablations
Effective encoding of raw GPS coordinates: The direct use of an individual MLP to encode GPS coordinates into feature vectors serves a baseline. As shown in Table the ML.P performs well at larger scales, but it stumbles at finer scales due to distortions in the standard coordinate system and its own spectral bias. To handle the distortions, we apply the Equal Earth Projection (EEP), which boosts performance at 1km, 25km, and 200km thresholds by +0.1%, +6.07%, and +3.93%, respectively. However, the encoding of fine-grained details remains challenge. To address this, we introduce an additional encoding layer using Random Fourier Features (a as 24). This layer enables the MLP learn high-frequency functions necessary to distinguish features finer scales, like streets and cities. This modification nearly doubles the accuracy at 1km and 25km, with increases of +5.11 14% and + 14.95%, respectively. When we include harder negatives via our dynamic queue, improving performance across all scales, with a notable increase of +4.47% at the 1km threshold.
Importance of Hierarchical Learning: We experiment with different a's in eq. 2 (20, 21, 2a) to investigate their influence across various scales. With a single accuracy metric, one might typically seek an optimal sigma. However, we're optimizing for five scale metrics (1km, 25km, 200km, 750km, and 2500km), making the task more difficult. Our findings, highlighted in Table 3b)) and further elaborated in the supplementary material, reveal an interesting trade-off: smaller o excels at larger scales but has issues with smaller scales, while larger increases fine- ine-ggrined accuracy but decreases on larger scales. Building upon this observation. we combine the representations from the encoders with varying as. By merging the outputs from the different encoders, we create joint hierarchical representation that maximizes the location encoder's s capabilities across all scales. This method outperforms the use of individual a at all scales, effectively addressing the varying performance issue related to different as, thereby endorsing the need for hierarchical representation in our architecture.

---

Table 3: We perform ablations on Im2GPS3k show the importance of different components of our location en- coder. (a) Our encoding of GPS coordinates using Equal Earth Projection (EEP), Random Fourier (aaurees(FFF) and Dynamic Queue (DQ) strategy led to significant gains compared to the MLP alone, especially in challenging localization within Ikm and 25 km. (b) Our GeoCLIP approach with hierarchies (GeoCLIP w/ H) leads to further gains and even performs much better than the respective classification with hierarchies C (w/ H) approach. The improvements with respect baseline (a) and C (w/ H) (b) are shown in brackets and highlighted in blue.
(b) Benefits of hierarchical learning
(a) Encoding GPS coordinates
Method
1 km
25 km
City
200 km
Region
Country
750 km
Continent 2500 km
Method
Street
km
25 km
City
200 km
Region
Country
750 km
Continent 2500 km
Cw H)
11.4
34.3
49.4
67.4
80.8
MLP (Baseline)
0.13
11.01
41.68
67.83
83.28
GcoCLiP (Coarse)
1.07
49.18
83.75
+EEP
0.23
17.08
45.61
68.8
82.98
GeoCLIP (Mediam)
9.84
33.1
49.28
69.07
83.35
EEP +RFF
5.37
32.05
49.42
68.4
82.98
GeoCIIP (Fine)
13.61
33.43
47.78
65.1
80.95
+EEP +RFF +DQ
(+9.71) (+22.09)
9.84
33.1
(+7.6)
49.28
(+1.24)
69.07
(+0.07)
83.35
GooCLIP (w/ H)
14.11
34.47
50.65
69.67
83.82
(+2.71) (+0.17)
(+1.25)
(+2.27)
(+2.02)
4.4 Qualitative Results: Geo-localization using Text-based query
We demonstrate GeoCLIP's ability to process text queries, highlighting the semantic richness of our model. As image encoder has a pre-trained CLIP backbone (trained to match visual and text features), our location encoder gets inherent alignment with CLIP's text features, enabling us to map textual descriptions geographical coordinates.
This capability allows us to assign geographical context any given text. Specifically, we can take any text, generate its embedding using CLIP text encoder, process it through our layers hı and h2, and calculate its similarity with a set GPS embeddings. This results in a similarity map, demonstrating the geographical distribution of the text query's 's concept. For instance, using "Desert" as a text query, we compute the cosine similarity between the text query's embedding and the GPS embeddings. The resulting similarity map, (Figure 4 reveals the spatial distribution of desert regions worldwide.
Method
Classifier mAP
Visual (V)
visual
0.234
Text Query: "Desert"
OneFot
geo
HashTag 22
geo
0.163
TagFeature
geo
0.161
GPS2Vec onebot
30
geo
0.100
GPSSVec 30
geo
0.182
GPS2Vs>+
31
geo
0.210
GeoCLIP
geo
0,249
Onellot 3
+ V
Fuston
0.238
Hashilug
fusion
0.261
TagFcature 10 + V
fusion
0,260
GPS2Vec Onchot
fusion
0.277
GPS2Vcc
30)
+
fusion 0.300
Lino
fusion 0.347
GPS2Vec+ 3 V
fusion 0.348
GeoCLTP (Ours)
fusion
0.362
Figure 4: Qualitative demonstration: -colloccaiz with text query using GeoCLIP. Our location can differ- entiate between geographical coordinates and associate semantic meaning from text to geographical space, pro- viding a geographical context textual descriptions.
Table 4: Comparison of methods utilizing GPS data for image classification the geotagged NUSWIDE Dataset. Our method outperforms existing methods using GPS features with or without visual features.
4.5 Utility of our Location Encoder beyond Geo-localization
We investigate the utility of GeoCLIP's location encoder in improving multi-class image classification task. We perform experiments on the geo-tagged test splits used in [31] We conduct two experiments: 1) Classifying image using only its GPS location features, and 2) Classifying an image using a combination of its GPS location features and visual features.
dataset [4] with the same train and
We obtain GPS features from our pretrained location encoder (which is kept frozen), which are normalized and fed through Multilayer Perceptron (MLP) with single hidden layer of 2048 dimensions, followed by a dropout layer for regularization. For classification incorporating visual context, we adopt the Bag-of- Tag-of-Visual- Visual-Words (BovW) representation based on SIFT descriptors as the visual feature for a fair comparison with existing works. We normalize location and visual features, concatenate them, and then pass them through the MLP.

---

Our results in Table indicate that GeoCLIP achieves state-of of-the-art performance on both approaches. This particularly significant, highlights that our location encoder has learned semantically rich representation across the Earth, being able to generalize beyond geo-localization and can therefore be extended to assist other location-aware methods to improve performance without any significant changes to their architecture.
4.6 GeoCLIP with Prior Geographical Knowledge
In practical applications of geo-localization, users often possess some level of prior geographical context. For instance, while the exact GPS coordinates of an image may be unknown, the general originating area, such as the continent, country, state, or city, may be known. This additional information can effectively narrow the search space, thereby improving the model's s accuracy due to the reduced number of potential locations for consideration. To assess GeoCLIP's s adaptability in these situations, we perform an experiment where the gallery of potential GPS coordinates is limited within a predefined radius of known geographical knowledge of each image. We choose the radius as per the assumed level of prior knowledge (higher search radius when coarse information is known and smaller radius with the finer knowledge): for a continent, coordinates within 2500km are taken; for a country, within 750km; for a state, within 200km; and for city, within 25km. Results across these different levels of granularity are shown in Table
Table 5: Performance of GeoCLIP when the granularity of search space varied. The performance improves as we search from coarse to finer granularity leveraging prior knowledge.
Street
City
Region 200 km
Country 750 km
Continent 2500 km
Granularity
1 km
25 km
Continent
15.02
37.70
55.39
78.11
100.0
Country 15.85
41.18
63.33
100.0
100.0
State
18.82
52.12
100.0
100.0
100.0
City
26.15
100.0
100.0
100.0
100.0
The performance of image-to-GPP retrieval improves as we search from coarse to finer granularity, which is quite intuitive. For instance, if we have prior knowledge of the city, it's much easier to geo-localize the true location than to have prior knowledge about the continent. The model's capacity adapt restricted search spaces affirms its applicability in real-world scenarios where partial geographic context is commonly available.
5 Conclusion
We addressed the challenge of worldwide geo-localization by formulating it as an image-to-GPS retrieval problem. We evaluated the efficacy of our method (GeoCLIP) not only in full training data scenarios but also in limited data settings. The performance of GeoCLIP was competitive even when the training data was reduced significantly. GeoCLIP could even perform geo-localization using text queries by leveraging the CLIP backbone of our image encoder. Moreover, our location encoder (using equal earth projection 17). RFF(1], and MLPs) better encoded GPS coordinates compared to the direct use of MLPs. However, coarse localizing either large or small areas. Our hierarchical learning overcame this issue by capturing the location features various scales, leading to further performance boost. Also, our location encoder is not tied to geo-localization and helps to aid in classification problems. In the future, we plan to further investigate the utility of our location encoder in assisting other computer vision problems. Limitations: As we use a CI.IP-based backbone in our image encoder, precomputing the image features on the large training dataset (each image of dimension 224 x 224 3) is cime-consuming. But once the features are precomputed, the training of our GeoCLIP model is relatively fast. We also observed that single a value used in RFF is insufficient perform well on both small and large areas. However, our hierarchical learning is a promising direction to overcome this issue.
fine-grat aneeggraneed used in RFF performed better in
Broader Impacts: Our image-based reo-localization can act as a substitute for tion in scenarios where GPS signals are unreliable, blocked, or inaccurate. Combining image image-based geo-tagging with GPS-based positioning systems can help to improve stability and safety applica- tions like autonomous driving and navigation. Also, it can benefit digital forensics, but avoiding its unethical use that could mislead location detection systems or violate privacy important. More discussion on ethical issues and possible mitigation approaches are provided in the supplementary.
localiza-
10

---

Acknowledgments and Disclosure of Funding
This work was supported by the US Army contract W911NF-2120192. We would like to extend our gratitude to all the reviewers for their valuable suggestions. We would also like to thank Brandon Clark for his helpful discussions on this project.
References
[1] Flickr: http://wwww.ficccccooom
[2] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[3] Gordon A. Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6172-61 6780,610,.200 2017.
[4] Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yantao Zheng. Nus-wide: a real-world web image database from national university of singapore. In ACM International Conference on Image and Video Retrieval, 2009.
[5] Brandon Clark, Alec Kerrigan, Parth Parag Kulkarni, Vicente Vivanco Cepeda, and Mubarak Shah. Where we are and what we're looking at: Query based worldwide image geo-localization using hierarchies and scenes. arXiv preprint arXiv:2303. 200.0:2303.04249,, 2023.
[6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020. [7] James Hays and Alexei A. Efros. Im2gps: estimating geographic information from single image. In 2008 IEEE Conference on Computer Vision and Pattern Recognition, pages 1-8, 2008.
[8] Mike Izbicki, Evangelos Papalexakis, and Vassilis J Tsotras. Exploiting the earth's spherical geometry to geolocate images. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2019, Würzburg, Germany, September 16-20, 2019, Proceedings. Part 11, pages 1- Springer. 2020.
[9] Martha Larson, Mohammad Soleymani, Guillaume Gravier, Bogdan Ionescu, and Gareth J.F. Jones. The benchmarking initiative for multimedia evaluation: Mediaeval 2016. JEEE MultiMe-
dia, 24(1):93-96, 2017.
[10] Shuai Liao, Xirong Li, Heng Tao Shen, Yang Yang, and Xiaoyong Du Tag features for geo-aware image classification. IEEE Transactions on Multimedia, 17(7):058-1106 2015. [11] Liu Liu and Hongdong Li. Lending orientation to neural networks for cross-view geo- localization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5624-5633, 2019.
[12] Eric Muller-Budack, Kader Pustu-Iren, and Ralph Ewerth. Geolocation estimation of photos using a hierarchical model and scene classification. In Proceedings of the European conference on computer vision (ECCV), pages 563-579, 2018
[13] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024 8035. Curran Associates, Inc., 2019.
[14] Shraman Pramanick, Ewa M Nowara, Joshua Gleason, Carlos D Castillo, and Rama Chellappa. Where in the world is this image? transformer-basee puter Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Puri XXXVIII, pages 196-215. Springer. 2022.
csollocalizatinn in the wild. In Com-
[15] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[16] Krishna Regmi and Mubarak Shah. Bridging the domain gap for ground-to-aerins image matching. Proceedings of the IEFF/CVF International Conference on Computer Vision pages 470-479, 2019.
11

---

[17]
Bojan Savrič, Tom Patterson, and Bernhard Jenny. The equal earth map projection. International Journal of Geographical Information Science. 33(3):454 465, 2019.
[18] Paul Hongsuck Seo, Tobias Weyand, Jack Sim, and Bohyung Han. Cplanet: Enhancing
image geolocalization by combinatorial partitioning of maps. In Proceedings of the European Conference on Computer Vision (ECCV), pages 536-551, 2018.
[19] Yujiao Shi, Liu Liu, Xin Yu, and Hongdong Li. Spatial- feature aggregation for image based cross-view gco-localization. Advances in Neural Information Processing Systems, 32. 2019.
[20] Yujiao Shi, Xin Yu, Dylan Campbell, and Hongdong Li. Where am i looking at? joint location and orientation estimation by cross-view matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4064-4072, 2020.
[21] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 33:7537-7554
2020.
[22] K. Tang, M. Paluri, L. Fei-Fei, R. Fergus, and L. Bourdev. Improving image classification with location context. In 2015 JEEE International Conference on Computer Vision (ICCV), pages 1008-1016, Los Alamitos, CA, USA, dec 2015. IEEE Computer Society.
[23] Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfec100m: The new data in multimedia research. Communications the ACM, 5942)664-73, 2016.
[24] Yicong Tian, Chen Chen, and Mubarak Shah. Cross-view image matching for geo-localization in urban environments. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 3608-3616, 2017.
[25] Aysim Toker, Qunjie Zhou, Maxim Maximov, and Laura Leal-Taixé. Coming down to earth: Satellite-to-street view synthesis for geo-localization. In Proceedings of the IEEE/CVE Confer- ence on Computer Vision and Pattern Recognition, pages
2021.
[26] Nam Vo. Nathan Jacobs, and James Hays. Revisiting im2gps in the deep learning era. In Proceedings of the IEEE international conference on computer vision, pages 2621-2663 2017. [27] Tobias Weyand, Ilya Kostrikov, and James Philbin. Planet-phote geolocation with convolutional neural networks. In Computer Vision-ECCC 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14, pages 37-55. Springer, 2016. [28] Scott Workman, Richard Souvenir, and Nathan Jacobs. Wide-area image geolocalization with aerial reference imagery. In Proceedings of the IEEE International Conference on Computer Vision, pages 3961-3969, 2015.
[29] Hongji Yang, Xiufan Lu, and Yingying Zhu. Cross-view geo-localization with layer-to-layer transformer. Advances in Neural Information Processing Systems, 34:29009- 2021. [30] Yifang Yin, Zhenguang Liu, Ying Zhang. Sheng Wang, Rajiv Ratn Shah, and Roger Zimmer- mann. Gps2vec: Towards generating worldwide gps embeddings. Proceedings of the 27th ACM International
SnnesatiAAnneeraattnooanla Conference on Advances in Geographic Information Systems, 2019. [31] Yifang Yin, Ying Zhang, Zhenguang Liu, Yuxuan Liang, Sheng Wang, Rajiv Ratn Shah, and Roger Zimmermann. Learning multi-contexs aware location representations from large-scoe scale geotagged images. In Proceedings of the 29th ACM International Conference on Multimedia, MM '21, page 899-907, New York, NY. USA, 2021. Association for Computing Machinery.
[32] Sijie Zhu, Mubarak Shah, and Chen Chen. Transgeo: Transformer is all you need for cross- image geo-localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1162-1171, 2022.
cross-view view
[33] Sijie Zhu, Linjie Yang. Chen Chen, Mubarak Shah, Xiaohui Shen, and Heng Wang. r² former: Unified retrieval and reranking transformer for place recognition. arXiv preprint arXiv:2304.nm 033400 10, 2023.
[34] Sijie Zhu, Taojiannan Yang, and Chen Chen. Vigor: Cross-view image geo-localization beyond one-to-one retrieval. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 3640-3649, 2021.
12


Reviews:
[{"review_id": "8WrbtLAqKV", "reviewer": "Reviewer_vH7S", "summary": "For global localization task for a query photo, the authors propose a CLIP-inspired image-to-GPS retrieval approach (named ‘GeoCLIP’) where we retrieve the GPS coordinates of an unseen query image by matching it with the gallery of GPS coordinates.  Regarding location encoder, some new ideas such as Random Fourier Features and Hierarchical Representation were introduced. The experimental results showed the effectiveness of the proposed method by outpeforming the existing baselines.\n", "questions": "* This approach is interesting. However, it seems to assume that one location corresponds to one image feature. Is it always true ?  For example, in restraurants, there are many kinds of meal menus. In some places, the view might be changed greatly with viewing direction change or even with a small location movement.\n\n* When searching the location of a query image, matching it with a collection of GPS coordinates is needed. The needed gallery size of the GPS coordinate seems to be large. How long does it take to estimate the location of one image in case of 100K (Im2GPS3k) and 500K (GWS15k) coordinates.? It seems to need a relatively long time. How about comparison to the baselines regarding search time ?\nOf course, all search-based methods has this disadvantage as well. However, for image search, many fast search methods has been proposed such as hash-based search. Is it possible to use any fast search methods for the proposed method ?\nHow about search time and computational cost compared with the classification-based methods ?\n", "limitations": "A limitation was mentioned regarding image encoding at times of training. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Matching GPS location embeddings and image embeddings (GeoCLIP) are introduced into an image localization problem. This is novel.  \n\n* It is very interesting that the location encoder gets inherent alignment with CLIP’s text features, enabling us to map textual descriptions to geographical coordinates.\n\n* The proposed location encoding method employing Random Fourier Features is much improved compared with the straightforward method which is the direct use of an individual MLP to encode GPS coordinates into feature vectors.\n\n* The experimental results showed that the proposed method outperformed the existing baselines which employs image search methods or grid classification methods. ", "weaknesses": "* Matching an image embedding with many location embedding (100K (Im2GPS3k) and 500K (GWS15k) coordinates) is needed to estimate a GPS location. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for finding our method GeoCLIP (“matching of GPS location embeddings and image embeddings”) as “novel”, for appreciating our location encoder having “new ideas such as Random Fourier Features and Hierarchical Representation”, and also for mentioning our method’s ability to “map textual descriptions to geographical coordinates” as “interesting”. We reply to the queries below.\n\n**One location corresponding to one image feature**: We agree that the view can change drastically by changing the viewing direction. In traditional Geo-localization literature formulation, an image is assigned the GPS location of the camera with which the picture was taken. If the GPS of the camera taking the picture does not change regardless of viewpoint, the same GPS is assigned. We follow the traditional formulation, which is also followed by the existing works. Thus, same GPS location can have images of different views which might have different CLIP embeddings. That’s why instead of directly using CLIP embeddings, we add two trainable linear layers on top of our CLIP image backbone. Thereby, enforcing different views to match the same GPS embedding via our trainable layers, the image encoder will inherently learn view invariant features corresponding to one location. \n\n**Time taken to estimate the location of one image in case of 100K (Im2GPS3k) and 500K (GWS15k) coordinates**: Below, we provide the location estimation time (in seconds).\n| GPS Gallery           | Time per Batch (1 image) | Time per Batch (128 images)|\n|-----------------------|------------------|---------------------|\n| 100k                  | 0.0033            | 0.0745               |\n| 500k                  | 0.0152       | 0.1043               |\n \n**Comparison of search time with baselines**: The traditional image-to-image retrieval matches query image embedding with the reference gallery of image embeddings. We also use an image as a query, but our reference gallery consists of GPS embeddings. The time taken to obtain the image embeddings for the query remains the same in both approaches (image-to-image and image-to-GPS). Also, our location encoder employs the same size of GPS embedding vector as the size of image embedding. The gallery embeddings in the image-to-image retrieval approach require all the gallery images to pass through the image encoder, which can be a compute-intensive process. However, the time taken to get embeddings for the reference gallery is significantly lower in our method, as our location encoder is a lightweight network containing MLPs of 4 hidden layers and one output layer. Once the embeddings are obtained, search time would remain the same in both cases.\n\n**Fast search methods**: We can use any search algorithm to match the query image embedding with the reference gallery GPS embeddings. Yes, any fast search method can also be used. \n\n**Comparison with classification-based methods**: The classification methods divide the globe into discrete geographical cells (used as classes), where an image is classified to identify which cell it belongs to. Thereby the center coordinates of the geographic cell are used as the prediction for the GPS location of the query image. Though they can obtain fast predictions on a given query, the classification strategy results in coarse predictions (the predefined classes limit localization performance), and classes have to be pre-defined during training. On the other hand, the retrieval-based methods match an input query with the geo-tagged gallery images, and the GPS of the matching gallery image is assigned to the query image. We directly match the input query image with the gallery of GPS coordinates. Even though the prediction time would be higher compared to classification, retrieval-based methods can be applied to any size gallery, and the gallery can be updated anytime without re-training, and also, these methods enable fine-grain geo-localization.\n\n\n"}, {"review_id": "KWFLTnSV3e", "reviewer": "Reviewer_AH57", "summary": "This work learns an image-to-GPS retrieval approach where the image is a query, and the gallery\nis a collection of GPS coordinates. The authors propose GeoCLIP, which consists of a location encoder, and an image encoder. The retrieval then is based given a new query image against a collection of gps embeddings. Experimental results show that this work outperforms previous SOTA at the Im2GPS3k and GWS15k datasets, across all distance thresholds. ", "questions": "I have stated my questions above in the weaknesses, so this is what I would like to get answers from the authors on. ", "limitations": "The authors do address this in their work. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This is a well written paper, with good motivation, writing and experimental section. \n* The extra results on \"Utility of our location encoder beyond geo-localization\" provide some more arguments about the effectiveness of their location encoder\n*  The discussion about encoding GPS with an MLP is very interesting and is backed by good experiments. \n * Hierarchical learning experiments are solid and seem to provide good justification for the design choices. ", "weaknesses": "* For Figure 3, I have no baseline to compare against. How would other methods compare in terms of how fast the curve degrades?\n* For the GPS encoding, how come the \"city\" scale is the one with the biggest rise? Could it be something related to the choice of parameters for the Random Fourier Features? It might make sense if some of the discussion from the supplementary material can be transferred to the main paper here since this is a key part of the method. \n* For the hierarchical method, how much does the merging of the outputs from the different encoders affect the overall computational needs?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback on our work: “well-written paper, with good motivation”, “hierarchical learning experiments are solid”, and “good justification for design choices”. We are also thankful to the reviewer for the encouraging remarks about our section ‘Utility of our location encoder beyond geo-localization’ and for appreciating our discussion and experiments on GPS embedding with MLP. We reply to the queries below.\n\n**Figure 3 (baseline to compare)**: In Figure 3, we showed the efficacy of our proposed method GeoCLIP in limited data settings where we observed respectable performance even with 5% training data. As suggested, we performed the same experiment with the classification baseline approach (ISNs [12]). We summarize the comparison of our method with the classification approach in the table below. The value inside the brackets on a particular row shows the performance degradation compared to the respective method’s performance with 100% training data.\n\n| Im2GPS3k                | **100 %** |    **20%**   |    **10%**    |    **5%**    |\n|-------------------------|:---------:|:------------:|:-------------:|:------------:|\n| 2500 km (GeoCLIP, Ours) |    83.8   |  83.8 (-0.0) |  83.6 (-0.2)  |  83.2 (-0.6) |\n| 2500km  (ISN [12])      |    66.0   | 50.1 (**-15.9**) |  48.7 (**-17.3**) | 43.9 (**-22.1**) |\n| 750km (GeoCLIP, Ours)   |    69.7   |  69.2 (-0.5) |  69.1 (-0.6)  |  67.6 (-2.1) |\n| 750km  (ISN [12])       |    49.7   | 31.9 (**-17.8**) | 29.2 (**-20.5**)) | 25.1 (**-24.6**) |\n| 200km (GeoCLIP, Ours)   |    50.6   |  49.0 (-1.6) |  48.9 (-1.7)  |  48.7 (-1.9) |\n| 200km  (ISN [12])       |    36.6   | 20.0 (**-16.6**) |  17.4 (**-19.2**) | 14.3 (**-22.3**) |\n| 25km (GeoCLIP, Ours)    |    34.5   |  32.6 (-1.9) |  31.9 (-2.6)  |  31.1 (-3.4) |\n| 25km  (ISN [12])        |    28.0   | 14.1 (**-13.9**) |  11.9 (**-16.1**) |  9.6 (**-18.4**) |\n| 1km (GeoCLIP, Ours)     |    14.1   |  13.1 (-1.0) |  12.1 (-2.0)  |  10.8 (-3.3) |\n| 1km  (ISN [12])         |    10.5   |  5.1 (**-5.4**)  |   4.5 (**-6.0**)  |  3.2 (**-7.3**)  |\n\nIt can be easily observed that the classification method's performance degrades much faster than our approach when the amount of training data is reduced, highlighting our method to be more data-efficient. \n\n**Performance improvement in city scale**: The embeddings of 2D GPS coordinates obtained using direct MLPs suffer from spectral bias, i.e., they only capture low-frequency information and fail to capture high-frequency details (Lines 54-59, 162-164). Thus, their performance (Table 2a first row) is good only on large-scale distance thresholds (2500 km). Particularly on the small scale distance thresholds such as street and city, require features to capture fine-grained high-frequency details. MLP performance is very poor on them. To improve the performance on smaller scales, we applied Random Fourier Features (RFF). \n\nWe found $\\sigma$ to be an important parameter in RFF (eq. 2 and 3), which can control the frequency range. As shown in Figure 2 of the supplementary, the small $\\sigma$ value performs well on coarser granularity (200km, 750km, and 200 km), while the large $\\sigma$ performs well on fine-grained levels (1km and 25 km). One intuitive way to overcome this trade-off is to select an intermediate $\\sigma$ value ($2^4$) which turned out to perform very well on a city scale (Table 2(a)) compared to the MLP baseline. Yes, the performance improvement can be attributed to the choice of a particular $\\sigma$ value (parameter of RFF). Moreover, our hierarchical strategy provides further gains on all distance threshold metrics (last row of Table 2(b)) compared to the performance obtained with the individual best $\\sigma$ value (last row of Table 2(a)). As suggested, we will add more discussion in the main paper to avoid any confusion.\n\n**Merging outputs of different encoders of hierarchical method**: We merge the GPS embeddings by simple addition operation (Lines 194-195). These are constant operations performed on M different vectors for M level hierarchy, which is fixed across the experiments on different datasets, maintaining a similar time complexity as any individual encoder. Besides this, our location encoder is just a small-sized shallow network containing MLPs of 4 hidden layers and one output layer, which is computationally lightweight compared to popular deep network architectures. \n\n\n\n\n"}, {"review_id": "G1od9CaFyU", "reviewer": "Reviewer_UZhc", "summary": "This paper proposes an image-to-GPS retrieval method, with an image encoder from the pre-trained CLIP and a tailored GPS encoder.\n\nThe GPS encoder is implemented by using:  i) the Equal Earth Projection to mitigate the GPS distortion; ii) Random Fourier transform with multiple MLPs to obtain high-frequency `positional encoding'.\n\nExperiments on standard benchmarks demonstrate the effectiveness of the proposed method.", "questions": "Please refer to weaknesses.", "limitations": "Yes.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1) This paper is well-written and easy to follow;\n\n2) The idea of this paper is interesting and new;\n\n3) The experimental evaluations are strong, particularly the interesting Section 4.4.", "weaknesses": "I only have some minor comments:\n\n1) Though it is interesting to adopt the CLIP pre-trained encoders, I would like to the experiments substituting the CLIP image encoder with Swin encoder (as GeoDecoder [5]), or substituting the Swin encoder in [5] with the CLIP image encoder.\n\nThis would help readers to clearly understand the contribution of the CLIP image encoder.\n\n2) I would like to see an experiment under the following setting:\n\na) Evenly partition the earth into patches, and use the patch center GPS as the database;\n\nb) perform the proposed image-to-GPS retrieval and report the performance;\n\nc) Repeat a) and b) with coarse-to-fine patch partitions.\n\n3) Please explicitly list the image-encoding time and the image-to-GPS retrieval time;\n\n4) Please tone down the claim of \"However, this approach is not practical as it becomes infeasible to construct a gallery containing images covering the entire world.\" The reason is that we have a world satellite image gallery.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the encouraging feedback on our section 4.4 and for finding our work to be “well-written and easy to follow”, and our idea to be “interesting and new”. Also, we are thankful to the reviewer for mentioning - “experimental evaluations are strong”. We reply to the queries below. \n\n**Contribution of CLIP image encoder**: We adopted CLIP pretrained encoder as it has been trained on 400 million (image, text) pairs and has shown strong generalization performance across different downstream tasks. Besides this, the muli-modal model architecture allowed us also to process text queries. The CLIP text features get inherent alignment with our location encoder, enabling us also to perform worldwide geo-localization using text queries. As suggested, we performed an experiment by substituting our CLIP image encoder with the Swin encoder, whose results are presented below (please note that the end-to-end training on large-scale geo-localization training dataset (MP-16) is computationally expensive and time consuming, and the experiment is still running).\n\n|     Im2GPS3k    | 1km |  25km | 200km | 750km | 2500km |\n|:---------------:|:---:|:-----:|:-----:|:-----:|:------:|\n| Swin (3 epochs) | 4.9 | 17.65 | 27.76 | 43.54 |  64.4  |\n\n**Results on coarse-to-fine patch partitions**: As suggested, we evenly partitioned the earth into patches using the Fibonacci sphere and subsequently used patch centers as GPS coordinates. Our coarse partitioning contained 10k patches, medium partitioning had 100k patches and the fine-grained consisted of 1M patches. Apart from evenly partitioning the whole earth, we also performed the experiments by evenly partitioning the land areas. Below we present the results of our image-to-GPS retrieval approach (GeoCLIP) on two different datasets (Im2GPS3k and GWS15k).\n\n|            Im2GPS3k           |  1km |  25km | 200km | 750km | 2500km |\n|:-----------------------------:|:----:|:-----:|:-----:|:-----:|:------:|\n| Coarse Patch  (10K All Earth) | 0.00 |  2.57 |  19.0 | 42.08 |  64.53 |\n| Medium Patch (100K All Earth) | 0.00 |  3.27 | 25.53 | 50.18 |  71.91 |\n|   Fine Patch (1M All Earth)   | **0.03** | **9.18** | **33.47** | **55.32** |  **75.34** |\n| Coarse Patch  (10K Land Only) | 0.00 |  2.14 | 20.12 | 45.65 |  68.17 |\n| Medium Patch (100K Land Only) | 0.00 |  7.24 | 30.16 | 53.99 |  74.64 |\n|   Fine Patch (1M Land Only)   | **0.02** | **12.85** | **37.07** | **59.39** |  **77.78** |\n\n|             GWS15k            |  1km | 25km | 200km | 750km | 2500km |\n|:-----------------------------:|:----:|:----:|:-----:|:-----:|:------:|\n| Coarse Patch  (10K All Earth) | 0.00 | 0.35 |  9.30 | 37.16 |  72.08 |\n| Medium Patch (100K All Earth) | 0.00 | 0.48 | 11.54 | 40.52 |  73.63 |\n|   Fine Patch (1M All Earth)   | **0.00** | **1.06** | **12.86** | **41.75** | **73.59** |\n| Coarse Patch  (10K Land Only) | 0.00 | 0.66 | 10.99 | 40.35 |  73.59 |\n| Medium Patch (100K Land Only) | 0.00 | 0.73 | 13.16 | 43.63 |  74.99 |\n|   Fine Patch (1M Land Only)   | **0.00** | **1.12** | **13.89** | **43.36** |  **74.91** |\n\nWe observe better performance on the fine patch partitions across datasets, as it allows finer search on the GPS embeddings leading to better localization performance. \n\n**Image-encoding time and image-to-GPS retrieval time**: The time taken (in seconds) for encoding image and overall time for image-to-GPS retrieval (sum of image encoding time, GPS encoding time, and search time) is shown below for different gallery sizes on Im2GPS3k dataset. \n\n|  GPS Gallery |   Batch of 1 image  |                             | Batch of 128 images |                             |\n|--------------|:-------------------:|:---------------------------:|:-------------------:|:---------------------------:|\n|              | Image encoding Time | Image-to-GPS retrieval time | Image encoding Time | Image-to-GPS retrieval time |\n|     10k      |       0.00033       |            0.0006           |       0.06000       |            0.0674           |\n|     100k     |       0.00033       |            0.0033           |       0.06042       |            0.0745           |\n|     500k     |       0.00033       |            0.0152           |       0.06049       |            0.1043           |\n\nWhen the gallery size is varied, the image encoding time remains almost the same, while image-to-GPS retrieval time shows an increasing trend with the increase in gallery size.\n\n**Tone down the sentence**: Thank you for pointing this out. We will edit the sentence accordingly in the final version.\n\n\n\n\n"}, {"review_id": "LOR930fHpY", "reviewer": "Reviewer_AAT4", "summary": "The paper addresses the problem of geo-locating an image, i.e. estimating the latitude and longitude of the where the image was taken. \n\nPrevious works treat the problem as a classification problem (of cells distributed across the globe). In contrast the this work proposes to retrieve the GPS coordinates from a collection of possible locations. In order to do so the authors propose a neural network architecture that encodes the (ground truth) GPS location and the image in a joint feature space and train it via a contrastive learning scheme. During the inference the query image is encoded and the GPS coordinate closest in feature space is taken as the predicted location.\n\nThe main contribution of the paper is the positional encoding of GPS coordinates via random fourier features [21], which are demonstrated to help represent higher frequency information and hence improve localization at higher spatial resolution (Tab. 2).\n\nExperimental results demonstrate state-of-the-art retrieval performance on 3 different datasets.", "questions": "**Q1** We do you keep the CLIP backbone frozen?\n\n**Q2** Why is the gallery of GPS coordinates sampled from the *training* set? Is this on the dataset from [9] which is used for training? Did you consider covering the globe, e.g. uniformly?\nThe dependency of the geographic locations on the training set implies that the test set needs to be covered by the training set. This seems to prevent correct out-of-distribution predictions.", "limitations": "The ability to accurately locate any photo on the earth represents a security and privacy issue. E.g., even if a user ops out of GPS tagging their photos, the presented approach enables to infer the user location from their images -- and the location trace of the user over time.\nThis surveillance capability should be stated as negative societal impact or potential harmful consequence of the approach, when deployed in public. ", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "**S1** The encoding of GPS coordinates via random fourier features (RFF) is novel. While RFF have been demonstrated to be useful to capture higher frequency properties for 2d image or 3d coordinates [21], e.g. in NeRF, to the best of my knowledge they have not been used to encode lat/lng coordinates for geo-localization. The authors also demonstrate that an equal earth projection (EEP) (which preserves the area covered) is beneficial for encoding 2d coordinates on the globe, especially for higher accuracy localization (< 1km in Tab. 2).\n\n**S2** Good ablation studies that demonstrate the usefulness of the chosen solutions, especially Tab. 2 for EEP and RFF, but also the ablations provided in the sup. material in Sec. 3. This helps the reproducibility of the approach.\n\n**S3** The place recognition performance improves considerably over state-of-the-art, even though the taken approach is relatively simple. This can be attributed to the ability of the model to learn an understand of spatial information.", "weaknesses": "**W1** The usage of the CLIP frozen backbone is not well motivated and evaluated. As such it remains unclear why CLIP features are needed for the place recognition task, rather than a network trained from scratch or fine-tuned on the task. Experimental evaluation demonstrates how the learned feature is able to relate textual information to geographic locations, but the influence of this capability on predicting lat/lng locations for a query image is not evaluated. \n\n**W2** The evaluation of the text-to-location encoding in Sec. 4.5 is very limited. The encoded location is used for image classification, but the paper falls too short in outlining the goal and metric of the chosen classification task. As such the evaluation can not stand by it's own but the reader needs to resort to [4, 31] for context.\n\n**W3** The size of the gallery (the quantization of possible GPS locations) is tuned per dataset. Instead a single size across all dataset would foster broader applicability of the approach. Sec. 3.1 of the sup. material suggests that this would in fact be possible, e.g. with a size of ~250k. (See also Q2)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging our GPS encoding method (“encoding GPS coordinates via random fourier features”) to be “novel”, for mentioning our representation of 2D GPS coordinates using equal earth projection to be “beneficial”, for finding our ablation studies to be “useful”, and for appreciating our ablations in Table 2 and in the supplementary material (Sec. 3). We reply to the queries below.\n\n**Frozen CLIP backbone**: We use CLIP backbone in our image encoder, which is kept frozen during training. We also add two linear layers (h1 and h2) to adapt the backbone to our geo-localization task, where these layers are kept trainable (Lines 153-157). The reasons for keeping the CLIP backbone frozen are as follows:\n* The standard training dataset of worldwide geo-localization (MP-16) contains 4.72 million geotagged images (Lines 237-238). Training from scratch or finetuning the whole image encoder can definitely be a viable option. However, it requires a huge training time, especially when the training dataset is so large, this is a big issue in general for research being conducted at Universities with limited resources. This expensive training had also been an issue with existing worldwide geo-localization methods. To minimize the training cost, we only train the added linear layers. \n* We have tried finetuning the CLIP backbone also. But we observed it did not perform better compared to the case where a frozen backbone is used. This may be the consequence of catastrophic forgetting of original CLIP weights while training, i.e., the knowledge of a huge training dataset of the CLIP foundation model (400 million image-text pairs) stored in the form of learned weights gets lost while updating them. Based on experimental observations, keeping the frozen CLIP backbone and only training linear layers (h1 and h2) was found to better adapt to our image-to-GPS retrieval task.\n* Another benefit of using pretrained CLIP weights is that they can be precomputed, i.e., the CLIP features on the training dataset can be saved beforehand. We observed doing this drastically reduced our training time. For instance, the training time of one epoch on MP-16 dataset was reduced substantially from $\\approx27$hrs to $\\approx15$ minutes.\n* As CLIP is a multimodal architecture (its image and text encoders are aligned), when we align the pretrained image encoder of CLIP with our location encoder, the pretrained text encoder of CLIP also gets implicitly aligned with the location encoder (Lines 325-327). As a result, without separate training on text, our method, GeoCLIP can also process text queries enabling geo-localization using text. \n\n**Sampling of gallery GPS coordinates**: The traditional image-to-image retrieval methods use a reference gallery comprising images from the test set. In our case, if we follow the same protocol, i.e., creating a reference gallery with the GPS coordinates of the test set, then it will become an easier task. For instance, in the case of the Im2GPS3k test dataset, the number of GPS coordinates would be 3000. Our method GeoCLIP achieves strong localization performance via image-to-GPS retrieval in such cases. Below we present the results when the gallery of GPS coordinates is formed using a test set.\n\n|  Dataset |   1km  |  25km  |  200km |  750km | 2500km |\n|:--------:|:------:|:------:|:------:|:------:|:------:|\n| Im2GPS3k | 29.66 | 47.64 | 59.05 | 72.53 | 85.78 |\n\nThe experimental results reported in the paper were performed on a more difficult task. The training set MP-16 dataset contains 4.72 million images. Sampling 100k or 500k samples from it to construct reference gallery GPS coordinates requires a test image to be searched on a larger gallery size which is tougher than searching on a small gallery obtained using a test set. \n\nEven if the test set contains images that are out of distribution (different from the training data distribution), our method still performs reasonably well. For instance, the recent Google World Streets 15K (GWS15k) is a challenging dataset [5], with locations that have either not been densely covered in the training dataset or that are at a considerable distance from covered regions. As a consequence, existing methods do not perform well on this dataset. Our method comfortably beats state-of-the-art methods on it (Table 1(b)).\n\nThe reason for creating a reference gallery of GPS coordinates using the training set is to utilize prior knowledge of the dataset. This knowledge helps to restrict our search to regions where pictures are mostly taken, and to narrow down our search to geographic places of interest. Even classification-based methods divide the globe into discrete, non-uniform geographic cells (classes) by using the knowledge of the training set. However, our method can also achieve decent performance without using such a dataset prior. This is shown in the results below, where we uniformly sample across the Earth or sample evenly spaced coordinates using a Fibonacci sphere.\n\n|            Sampling           | 1km  |  25km | 200km | 750km | 2500km |\n|:-----------------------------:|:----:|:-----:|:-----:|:-----:|:------:|\n| Uniform  (1 M, Land Only)   | 0.70 | 11.95 | 37.04 | 59.09 |  78.14 |\n| Evenly Spaced (1M Land Only ) | 0.02 | 12.85 | 37.07 | 59.39 |  77.78 |\n| Uniform  (1 M All Earth)   |    0.10  |  7.17     |   32.03    |    54.28   |   75.17     |\n| Evenly Spaced  (1M All Earth) | 0.03 |  9.18 | 33.47 | 55.32 |  75.34 |\n\n**Negative societal impact**: Thank you for the suggestion. In our current draft, we had briefly mentioned avoiding unethical use that could lead to misleading detections or violate privacy (Lines 372-373). We will include the reviewer's suggested points to further strengthen our broader impacts section.\n\n\n\n\n\n\n"}, {"review_id": "LKcpxPTMex", "reviewer": "Reviewer_SMLq", "summary": "This paper tackles the image geolocalization problem. Instead of formulating the task as image retrieval or classification over a set of geospatial bins as is typical, the authors frame it as image-to-GPS retrieval by matching image features with location features. The proposed training strategy leverages Random Fourier Features (RFF) to encode geospatial location across multiple scales and align them with features from a CLIP visual encoder, with several training enhancements. The resulting method achieves state of the art performance on three benchmarks, across all scales, and showcases interesting applications in text geolocalization and classification.", "questions": "My initial rating is weak accept. This paper introduces a method for training a high-quality geospatial location encoder that interacts with, widely used, CLIP features and demonstrates its use for localization and several applications. This has the potential to benefit the wider community.\n\nQuantitatively, how does the proposed approach compare in terms of runtime and memory footprint with existing approaches?\n\nHow does the text encoding perform at smaller scales (region or city)?\n\nCan the text encoding benefit the image-based geolocalization?", "limitations": "Limitations addressed in the manuscript.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Addresses an important and interesting problem. Image localization is a fundamental problem in vision that continues to receive attention in the literature. \n\n- The approach has technical novelty and is of the first to formulate the localization task as an image-to-GPS retrieval problem. The key idea is to relate images directly to locations via aligning image features to location embeddings using contrastive learning. Such an approach uses a continuous model of location whereas traditional geolocalization approaches depend on a large reference database of images with known locations to support image-level matching (via image retrieval or classification) and inferring location from the closest match. Instead, the authors propose the concept of a GPS gallery.  Though training still requires geotagged images (image + location), the authors show the proposed approach requires less training data.  Further, at inference, the GPS gallery can theoretically be any set of locations to query against, instead of being limited by the reference image set. Geospatial location is represented using NeRF-style location embeddings (Random Fourier Features with an MLP).  Results show that infusing the location representations with multiple spatial scales improves geolocalization performance. \n\n- Evaluation is extensive and compelling. State-of-the-art (SOTA) performance on multiple datasets at different threshold scales (street, city, region, country, and continent). Near SOTA performance on Im2GPS3k training using only 20% of the training data.\n\n- Intriguing applications. The results on text geolocalization and geotagged image classification showcase the utility of the location encoding beyond image-based geolocalization. ", "weaknesses": "- A central idea of this paper is the concept of modeling location continuously. I was disappointed to see that during evaluation, the GPS gallery (the set of possible locations) was limited to locations sampled from the training set (L230). Though I understand why this was done from an evaluation perspective, it significantly hurts the \"global localization\" theme. At the very least, I would have liked to see an evaluation of how selecting the GPS gallery at inference (choice of sampling distribution, granularity, etc.) impacts performance. This also raises the issue that the location encoder is not truly global, it is limited by where images have been captured on the globe (as training requires image/location pairs). \n\n- Some minor details missing. Section 4.2 does not mention which dataset the experiment was performed on. Comparing the numbers from Figure 3 and Table 1a, it can be assumed to have been conducted on Im2GPS3k, but the dataset should still be mentioned to improve clarity. Similarly, section 4.3 does not mention which dataset(s) was/were used for the ablation study.\n\n- The application for text query-based geolocalization is interesting, but the results are lacking. It is only demonstrated qualitatively on a continent scale using a single keyword (desert) in the main paper. While this is interesting, it leaves a lot to be desired in terms of understanding the full or even partial capability of the text encoding for tasks such as geolocalization or digital forensics. \n\n- Quality of the presentation could be improved. For example, the introduction could provide additional motivation for the underlying problem. In its current form, the introduction mentions that there are a variety of applications for geolocalization, but does not support this claim with any citations or descriptions of how geolocalization benefits them. Instead, the majority of content is related to why the problem is difficult and how existing approaches are inadequate, repeating much of the related work section content (e.g., binning, classification, and scaling methods). The conclusion section, in particular, lacks flow and could benefit from additional editing.  Unclear mathematical notation on L138. What is K?  Minor grammatical errors (L49, L107, L111, L372, etc.).\n\n- Word choice in the conclusion. L353 suggests that this work “solved worldwide geolocalization”. Of course this problem is far from solved! ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging our problem as “important and interesting”, for finding our approach and our problem formulation to be “novel”, for finding our evaluation to be “extensive and compelling”, and for mentioning our demonstrated applications as “intriguing”. We reply to the queries below.\n\n**Results on different selection choices for GPS gallery**: As suggested, we perform evaluations when the GPS gallery is formed by different sampling distributions and present the results on Im2GPS3k dataset below.\n\n| GPS Gallery                        |  1km  |  25km | 200km | 750km | 2500km |\n|------------------------------------|:-----:|:-----:|:-----:|:-----:|:------:|\n| Sampling from Train Set (lat, lon) | 14.11 | 34.47 | 50.65 | 69.67 |  83.82 |\n| Test set (lat, lon)                | 29.66 | 47.64 | 59.05 | 72.53 |  85.78 |\n| Evenly Spaced (1M Land Only )      |  0.02 | 12.85 | 37.07 | 59.39 |  77.78 |\n| Evenly Spaced (1M All Earth)       |  0.03 |  9.18 | 33.47 | 55.32 |  75.34 |\n\nSimilar to traditional image-to-image retrieval methods, when the image gallery is constructed using the test set (in our case, test set GPS coordinates), we achieve the best performance. To have a fair comparison with existing worldwide geo-localization methods (classification-based approaches), which use training set to create geographical cells, we used the GPS coordinates of the training dataset to construct the reference GPS gallery. \n\nWithout using any dataset prior knowledge of geographical places of interest, reference gallery can also be constructed by evenly sampling GPS coordinates across the globe. But, it is not a good way to create GPS gallery as there can be many places on Earth (like oceans and polar regions), which may not be relevant places of interest (also evidenced by results). To accurately predict within 1km radius, GPS coordinates must lie in the reference gallery.  By uniformly sampling one million coordinates across Earth's surface, a total area of 1,000,000 km² is encompassed. But, considering Earth's land area $\\approx$ 148,326,000 km², the likelihood of having the true coordinates in reference gallery is low. Thus, if true GPS coordinates fall outside the available GPS options of the gallery (for instance, not within a 1km proximity to the true location), then image-to-GPS performance within 1km distance threshold would suffer. As we observe in higher scales, competitive performance may be achieved through uniform sampling, given the increased likelihood of true coordinates lying within the larger distance thresholds from reference gallery coordinates. Given the vast search space, applying prior knowledge and alternative search methods can enhance efficiency and improve performance.\n\nWe also perform evaluations when granularity of sampling is varied and show the results below.\n\n| Granularity |  1km  |  25km | 200km | 750km | 2500km |\n|:-----------:|:-----:|:-----:|:-----:|:-----:|:------:|\n|  Continent  | 15.02 | 37.70 | 55.39 | 78.11 |  100.0 |\n|   Country   | 15.85 | 41.18 | 63.33 | 100.0 |  100.0 |\n|    State    | 18.82 | 52.12 | 100.0 | 100.0 |  100.0 |\n|     City    | 26.15 | 100.0 | 100.0 | 100.0 |  100.0 |\n\nWe considered four different granularity levels. The performance of image-to-GPS retrieval improves as we search from coarse to finer granularity, which is quite intuitive. For instance, if we have prior knowledge of the city, it's much easier to geo-localize the true location than to know the continent.\n\n**Missing minor details**: Thank you for pointing this out. Yes, results in sec. 4.2 and 4.3 are on Im2GPS3k dataset. We will mention it in the final version to improve clarity and avoid confusion.\n\n **Mathematical notation and minor grammatical errors**: K is the total number of query images in $D_{test}$. We will mention it and correct the grammatical errors as suggested.\n\n**Word choice in conclusion**: We will edit that line as suggested.\n\n**Comparison of runtime and memory footprint**: Below we provide runtime and memory footprint of baseline methods (ISNs, Translocator) and compare our method. \n\n| Methods           | Memory footprint (parameter count) | Runtime (batch of 128 images) |\n|-------------------|:----------------------------------:|:-----------------------------:|\n| ISNs [12]         |             257,830,177            |             0.0997            |\n| Translocator [14] |             478,125,882            |             0.2374            |\n| GeoCLIP (Ours)    |             314,400,770            |             0.1043            |\n\nThough our method has higher storage and runtime than ISNs, is much lower than Translocator. Note that, GeoCLIP has better accuracy than both approaches across all distance threshold metrics. In general, classification-based methods are expected to have lower runtime as no searching is performed on gallery. But, recent methods such as Translocator use auxiliary information such as scenes to get better performance by formulating a multi-task learning framework using dual branch vision transformers, which can lead to higher runtime apart from higher parameters. Our method does not depend on additional information such as scenes and has a lightweight location encoder containing MLPs of 4 hidden layers and one output layer.\n\n**Performance of text encoders at smaller scales (region or city)**: We qualitatively demonstrate the localization of a city and landmarks in the figure of attached pdf. Please note GeoCLIP using text queries on a few specific places is also provided in supplementary section 4.3.  \n\n**Text encoding to benefit image-based geolocalization**: Different modality information, such as text, can benefit image-based geolocalization. One simple way could be to fuse text and image features, then use resultant features for matching GPS embeddings. However, our current training dataset of geo-localization does not have (image, text) pairs. But this idea has great potential and can be an interesting future work."}]

Author rebuttal:
We sincerely thank all the reviewers for their valuable and encouraging feedback on our work. We are happy to see that all the reviewers have appreciated our work. The reviewers vH7S, AAT4, and SMLq even explicitly mentioned our approach to be novel. Moreover, we are very thankful that our demonstrated applications were well appreciated by reviewers vH7S, AH57, UZhc, and SMLq. It is also very encouraging that SMLq mentions that our work has the potential to benefit the wider community. We reply to the queries of each reviewer individually. We also attach a pdf containing figures, which we appropriately refer to while answering specific questions. 


Meta-review: All reviewers unanimously recommend acceptance, claiming that the papertackles an important problem, has technical novelty and presents extensive and compelling evaluations. The paper further passed an ethics review, that also deemed the paper fit for publication after the authors add some discussion on defense strategies against actors misusing the method.

The authors should take into account all the feedback from technical and ethics reviewers when preparing their final version.
Decision: Accept (poster)