Title: GeoCLIP: Clip-Inspired Alignment between Locations and Images for Effective Worldwide Geo-localization
Venue: NeurIPS 2023
Accepted: yes
Keywords: Geo-localization

Image-to-GPS retrieval

CLIP

Random Fourier Features
Abstract: Worldwide Geo-localization aims to pinpoint the precise location of images taken anywhere on Earth. This task has considerable challenges due to the immense variation in geographic landscapes. The image-to-image retrieval-based approaches fail to solve this problem on a global scale as it is not feasible to construct a large gallery of images covering the entire world. Instead, existing approaches divide the globe into discrete geographic cells, transforming the problem into a classification task. However, their performance is limited by the predefined classes and often results in inaccurate localizations when an image's location significantly deviates from its class center. To overcome these limitations, we propose GeoCLIP, a novel CLIP-inspired Image-to-GPS retrieval approach that enforces alignment between the image and its corresponding GPS locations. GeoCLIP's location encoder models the Earth as a continuous function by employing positional encoding through random Fourier features and constructing a hierarchical representation that captures information at varying resolutions to yield a semantically rich high-dimensional feature suitable to use even beyond geo-localization. To the best of our knowledge, this is the first work employing GPS encoding for geo-localization. We demonstrate the efficacy of our method via extensive experiments and ablations on benchmark datasets. We achieve competitive performance with just 20% of training data, highlighting its effectiveness even in limited-data settings. Furthermore, we qualitatively demonstrate geo-localization using a text query by leveraging the CLIP backbone of our image encoder. The project webpage is available at: https://vicentevivan.github.io/GeoCLIP

GeoCLIP: Clip-Inspired Alignment between Locations and Images for Effective Worldwide Geo-localization

Abstract

Worldwide Geo-localization aims to pinpoint the precise location of images taken anywhere on Earth. This task has considerable challenges due to the immense variation in geographic landscapes. The image-to-image retrieval-based approaches fail to solve this problem on a global scale as it is not feasible to construct a large gallery of images covering the entire world. Instead, existing approaches divide the globe into discrete geographic cells, transforming the problem into a classification task. However, their performance is limited by the predefined classes and often results in inaccurate localizations when an image's location significantly deviates from its class center. To overcome these limitations, we propose GeoCLIP, a novel CLIP-inspired Image-to-GPS retrieval approach that enforces alignment between the image and its corresponding GPS locations. GeoCLIP's location encoder models the Earth as a continuous function by employing positional encoding through random Fourier features and constructing a hierarchical representation that captures information at varying resolutions to yield a semantically rich highdimensional feature suitable to use even beyond geo-localization. To the best of our knowledge, this is the first work employing GPS encoding for geo-localization. We demonstrate the efficacy of our method via extensive experiments and ablations on benchmark datasets. We achieve competitive performance with just 20% of training data, highlighting its effectiveness even in limited-data settings. Furthermore, we qualitatively demonstrate geo-localization using a text query by leveraging the CLIP backbone of our image encoder. The project webpage is available at: https://vicentevivan.github.io/GeoCLIP

1 Introduction

Image geo-localization refers to determining the geographical location where a photograph is taken from. This problem is of great significance in a wide variety of applications, such as navigation, tourism, and security. However, this task poses several challenges, especially for locations that lack distinctive landmarks or are outside tourist hotspots. Typically, image-based geo-localization mainly focuses on identifying the GPS locations of images within a specified region of interest (e.g., within a city). However, worldwide geo-localization [12, 14, 5] is not limited to any specific area but to the entire globe, thus making it significantly more challenging, and hasn't been explored extensively.

Geo-localization at the global level requires finding the locations (i.e., latitude and longitude) of images from different parts of the world. Many retrieval-based methods [28, 11, 34, 29, 32, 24, 20, 33] which perform image-to-image retrieval limit their search space to specific cities. Extending such approaches for worldwide geo-localization demands maintaining a gallery of all possible images of the entire globe to perform a match with the query image. However, creating such a large image gallery is infeasible; thus, these methods are inappropriate for retrieval at the global scale. Existing works instead use classification-based approaches [14, 26, 12, 5, 27, 8, 18] using scenes, hierarchy, queries, and segmentation. These methods divide the Earth into geographic classes (covering a particular area) and apply a linear classifier for prediction. If an image belonging to a class is far from

Feature

GPS Gallery

(B)  Approach

Image

Pairs of

GPS Gallery

Retrieved GPS

Coordinates

Query

Retrieve

Pairs of

CLIP

ER2

ERD

D>2

()万

(40.7128°,—74.0060°)

（-33.8688°,151.2093°)

（55.7558°,37.6176°)

(53.9921°,

1.5418°)

(25.7617°,

80.1918°)

(53.9921°,

-1.5418°)

L()

V()

L1

L2

L3

Ln

Vi

V·LV-LV·L3

V-Ln

(lat,lon)

its center, it would result in a localization error even though the classification prediction is correct. Also, the predefined classes limit the prediction to ≈ 21 k possible locations, which is not dense enough to predict any location in the world. Moreover, these methods often need large amounts of training data which takes days to train even with high computational resources. Thus, there is a need for a new effective and efficient approach that can better model the relationships between images and their corresponding geographic locations. We summarize the problem setup and compare the major differences between existing works and our proposed method in Figure 1.

To address the above limitations of existing approaches, we propose a CLIP[15]-inspired image-toGPS retrieval approach (named 'GeoCLIP') where we retrieve the GPS coordinates of an unseen query image by matching it with the gallery of GPS coordinates. To facilitate matching, we design location and image encoders to project GPS and image in a shared embedding space. Our location encoder transforms a given 2D GPS coordinate into a high-dimensional feature space. The traditional latitude and longitude systems have inherent distortions in representing certain areas (e.g., countries near to poles and equator have inconsistent representations). Hence, we first convert the GPS locations using equal earth projection [17] to minimize such distortion. Another challenge is to obtain a high-dimensional representation of a 2D GPS coordinate. The direct use of standard MLP leads to spectral bias, as they approximate low-frequency functions and fail to capture high-frequency details with low-dimensional inputs. Previous studies [21] have demonstrated that such spectral bias can significantly degrade the performance in various applications, particularly when capturing these details is crucial for the task. To alleviate this, we employ a positional encoding technique using Random Fourier Features (RFF) [21] before processing it through a feedforward network.

To enrich the location features, we further enforce hierarchical representation in our location encoder by capturing features at different granularities (ranging from coarse to fine-grained resolutions) and combining them to obtain a rich high-dimensional feature. Specifically, we vary the frequencies in RFF by changing sigma values using our exponential assignment strategy, enabling us to process the 2D GPS coordinates at different resolutions. We employ contrastive loss between features of the image and GPS. It is well established that the incorporation of additional negative samples enhances the performance of contrastive learning. Our gallery is a collection of GPS coordinates instead of images. Hence it is easy for us to fetch negatives by uniformly sampling the GPS coordinates from the globe. To this end, we implement a queue of GPS values to serve as artificial negatives. Our location encoder learns generic features as we find them suitable for use even beyond geo-localization. Moreover, using the CLIP [15] backbone in our image encoder helps extract meaningful features and allows us to compare GPS coordinates with text descriptions, which is not feasible in existing works. Besides this, our image-to-GPS retrieval method facilitates a much finer evaluation than previous techniques, resulting in better localization performance.

Below we summarize our major contributions as follows:

To the best of our knowledge, we are the first to solve worldwide geo-localization task via image-to-GPS retrieval approach (GeoCLIP) by explicitly aligning the features of images to corresponding GPS locations.

Our location encoder incorporates positional encoding with random Fourier features to efficiently encode GPS coordinates and mitigate spectral bias in MLPs. In addition, we use an exponential sigma assignment strategy to facilitate learning hierarchical features at different resolutions (Sec. 3.1.1).

We validate the efficacy of our location encoder in an additional application, highlighting its versatility beyond geo-localization tasks (Sec. 4.5).

Unlike existing works that only use an image as query, we also qualitatively demonstrate global geo-localization with text query using GeoCLIP (Sec. 4.4).

We also demonstrate the effectiveness of our approach in limited data settings (Sec. 4.2).

2 Related Works

This section briefly discusses the relevant works related to our approach.

Global Image Prediction: Many previous works [27, 18, 26, 12, 14], have tackled the problem of global image geo-localization. While the most common localization papers use an image-to-image retrieval technique [16, 19, 20, 25, 34, 32], this is not yet feasible on the global scale because of the data collection that would be necessary for a global reference dataset to compare against. Instead, Weyand et al. [27] proposed to split the earth into individual classes to predict the location of an image. Through constructing these classes, the prediction can be relatively coarse or fine-grained based on the geographic size of the classes. Vo et al. [26] utilized multiple levels of geographic classes to learn different granularities, and Seo et al. [18] was the first to combine these levels together for an enhanced prediction. Müller et al. [12] additionally uses the scene labels of images to learn by having separate encoders designated to learn each scene's features. Recently Clark et al. [5] made use of even more hierarchies and scenes than the previous works, they also use a specialized transformer decoder to learn different features from every input image that represents each hierarchy and scene. The use of semantic segmentation was introduced by [14] in an attempt to provide additional context and understanding for the model to better geo-localize. They use two transformer encoders, one for the RGB image and the other for the segmented representation of that image, and share information between the encoders to get a better location prediction. Unlike existing global image geo-localization works, we instead take an image-to-GPS retrieval approach by directly matching the query image features to the features of GPS data.

Learning from GPS Data: The GPS coordinates of an image are used in [22, 10, 30, 31, 3] to provide additional context when performing classification. They construct GPS embeddings by leveraging image tags, textual descriptors, and information from geospatial databases, which are paired with image embeddings to improve the classification accuracy. We differ from this work by utilizing Random Fourier Features (RFF) [21] with MLPs. RFF is often used in NeRFs to represent the location or viewing angle for reconstructing the image or scene at that location or from that view. However, our location encoder uses RFF to represent a specific location on the earth and aligns them with the corresponding image features. We later also show that GPS embeddings of our location encoder is generic and can even be utilized for classification tasks as well, leading to further performance gains and showing its utility beyond geo-localization tasks.

Contrastive Learning: It is used in a wide variety of tasks in multiple deep learning-based disciplines. The basic idea is to match the features from a positive pair while pushing away from negative example features. Our method uses two different contrastive learning strategies, SimCLR[2] and CLIP[15]. SimCLR uses a base image and creates two separately augmented versions of it. It then attempts to match these two versions of the image together and push away the other images that are not from the same base. They show that this creates a robust feature representation of images. On the other hand, CLIP utilizes a large image-text database to learn a shared feature space between its image and text encoders. To establish a meaningful and distinct representation for both the text and image, they enforce similarity between the image features and the corresponding text caption. CLIP has been widely used in many tasks since its inception. We utilize the pretrained model from CLIP as a backbone to our image encoder, while our augmentation strategy is similar to SimCLR. However, instead of matching the image to another image, as in SimCLR, or to text, as in CLIP, we match it to a gallery of GPS features. The additional benefit of using CLIP backbone is the ability to compare GPS coordinates through our location encoder and text from CLIP's text encoder, which provides a very intriguing qualitative analysis that we show later (Sec. 4.4). Now, we explain our proposed approach in detail in the next section.

Location Encoder

(A)

(B)

GPS

ER2

∈RD[D>2]

：Frozen

Queue

O

RandomFourier

MLPs

Features(RFF)

Batch

GPS

fi()

GB

L()

Location

thGPS

72()

f2(-)

(lat,lon)

Encoder

L2

Ln

L

L2

Gi

个

N

02

38 8

V1

V-LV·L

V·LV-LV-L

7-4

Batch

Image

發

CLIP

V2

V·LV-L

V-Ln

V-L

V-L

V-L

(lat,lon)

ER2

YM()

M

fM(-)

IB

OM

Va·LnV·L

V·Ls

ImageEncoder

FeatureSimilarityMatrix

：Frc

3 Proposed Approach

Worldwide Image Geo-localization Setup : Given the access to training dataset D train = { ( I n , G n ) } N n =1 containing pairs of image I n , and GPS coordinate G n . Our goal is to train a worldwide model W using samples from D train and then use the trained model W to predict the location of unseen K query images of test dataset D test . Specifically, our objective is to accurately predict the latitude and longitude of query images, G Q k = W ( I Q k ) , ∀ k ∈ [1 ..K ] where I Q k ∈ D test . The images in the query set I Q can belong to any geographical location of Earth, making it a challenging setup.

We formulate this problem as a retrieval problem. Traditionally, retrieval involves matching a query image with images from the gallery, and the GPS coordinates of the closest gallery image would be the desired output. This approach is often used when the search space is limited to specific cities. However, it has limitations when applied to retrieval at the global scale, which requires the construction of a gallery containing images covering the entire world. Instead, it is much easier to build a gallery of GPS coordinates. Hence, geo-localization can be framed as an image-to-GPS matching problem where the closest GPS match would be the location of the query image.

In order to perform matching, the features of the image and GPS location need to be extracted. Hence, we have two major components in our architecture, i.e., Location Encoder L ( · ) and Vision Transformer V ( · ) as Image Encoder. Now, we explain the model architecture containing details of both the encoders, followed by the training and evaluation strategies.

3.1 Model architecture

The architecture of our method GeoCLIP is shown in Figure 2.

Image Encoder : The CLIP [15] model has demonstrated strong generalization performance across a wide range of computer vision tasks. Motivated by it, our image encoder contains a Vision Transformer, which is a pre-trained CLIP ViT[6]-L/14 model. We use it as a backbone and keep it frozen. To adapt this backbone for our geo-localization task, we add two linear trainable layers with dimensions of h 1 and h 2 , respectively. These additional layers are employed to fine-tune the learned representation for our task. Hence, our overall image encoder V ( · ) extracts the features V i = V ( I i ) , ∀ i ∈ [1 . . . N ] and I i ∈ D train .

Next, we explicitly encode the GPS coordinates, which has not been tried for the geo-localization task. We now explain our location encoder in detail, which encodes the GPS coordinates of a location.

3.1.1 Location Encoder

Encoding 2D GPS coordinates (i.e., G i ∈ R 2 , ∀ i ∈ [1 . . . N ] ) to a high-dimensional representation ( R D ) where D ≫ 2 , is a challenging task. A simple use of standard MLPs for such encoding suffers from spectral bias [21] (unable to capture high-frequency details). Hence, to handle this challenge and effectively perform this encoding, we employ different components in our encoder, i.e., representing

GPS coordinates using equal earth projection[17], using positional encoding through random Fourier features, and enforcing hierarchical representation at various scales. We explain each of them below:

Equal Earth Projection (EEP) : To accurately represent 2D GPS coordinates and minimize distortions inherent in standard coordinate systems (i.e., countries closer to the poles are overrepresented in traditional latitude and longitude systems), we employ the Equal Earth Projection (EEP)[17]. By utilizing EEP, we counterbalance the distortions and ensure a more accurate representation of the Earth's surface. Given G i ∈ R 2 (in radians), we transform it into G ′ i by applying the EEP as below:

where sin θ = √ 3 2 sin G lat i , P 1 = 1 . 340264 , P 2 = -0 . 081106 , P 3 = 0 . 000893 , P 4 = 0 . 003796 . After applying the EEP, we scale the resulting longitude in the range -1 to 1 , and the latitude values are scaled proportionally.

Random Fourier Features (RFF) : To capture rich high-frequency details from low-dimensional GPS input, we apply a sinusoidal-based positional encoding technique to G ′ i . Specifically, we leverage random Fourier features (RFF)[21] for this positional encoding. We construct an encoding layer that determines the range of frequencies to be encoded. We limit the frequencies using a fixed matrix R , whose entries are sampled from a Gaussian distribution with the standard deviation ( σ ). The matrix R is set at the beginning of training and remains unchanged throughout the training process. The RFF operation γ ( · ) encodes GPS coordinate G ′ i as γ ( G ′ i ) = [cos(2 π R G ′ i ) , sin(2 π R G ′ i )] T , where the entries of a m th row and n th column of matrix R are r m,n ∼ N (0 , σ ) .

Hierarchical Representation : To construct a hierarchical representation capable of capturing information at varying scales, we vary the frequency in RFF to obtain multiple high-dimensional representations spanning from coarse to fine-grained. As the frequency range in RFF depends on the σ value, we propose an exponential assignment strategy for choosing the σ values in our location encoder. Specifically, for a given a range of σ values (i.e., [ σ min , σ max ]), we obtain the σ values for M level hierarchy using the following:

This exponential assignment of σ values enables our location encoders L ( · ) to effectively process the input 2D coordinates at different resolutions. Consequently, this hierarchical setting allows L ( · ) to specialize in capturing features of a particular location at various scales. It enforces our model to efficiently learn a rich understanding of spatial information, making it suitable for a wide range of geospatial deep learning tasks.

Next, we pass the encoded hierarchical features through a feedforward MLP. Each MLP f i independently processes the outputs of RFF and yields a vector v i ∈ R d . We then perform element-wise addition of these resulting vectors to obtain a joint representation v = ∑ M i =1 v i .

Below we summarize the overall operations performed by our location encoder L ( · ) for obtaining rich high-dimensional encoded features ( L i ) of a GPS 2D coordinate G i :

3.2 Model Training

We train our model using a contrastive learning scheme (similar to [15]) where we maximize the similarity of features of an image I i (i.e., V i ) with its corresponding location features of GPS coordinate G i (i.e., L i ) and minimize its similarity with other GPS coordinates in a batch. To further improve our learned representation, we also adopt a training scheme similar to SimCLR [2].

We apply a set of random augmentations ( A ) to each image in a batch and create P views of an image to improve the quality of learned representations and promote better generalization. We employ the same set of augmentations used in the SimCLR. For each augmented image I ij ( i th sample of D train and j th view, we also create variation in its corresponding GPS coordinates G i by injecting random noise η j to it (denoting it by G i j ) which helps to enforce spatial smoothness in the location encoder. The noise η j is sampled from a Gaussian distribution with a standard deviation of σ η (i.e.,

η ∼ N (0 , σ 2 η ) . This added noise encourages the location encoder to learn more robust representations that are less sensitive to small variations in the GPS coordinates.

At any point during training, our location encoder L ( · ) can obtain the embedding of any GPS location g (i.e., L g ) without requiring a corresponding image. We exploit this capability to generate additional negative samples during training to improve the learned representation with contrastive learning. We append a queue Q of GPS locations with a fixed length S to every batch of training data D train . During training, we generate embeddings for these GPS coordinates ( ˜ L ). We also inject noise ( η ′ ∼ N (0 , σ 2 η ′ ) ) to ˜ L , such that σ η ′ ≥ σ η . This additional queue of GPS embeddings notably improves the performance of our model, particularly at smaller scales (refer Table 3(a)).

Hence, for an i th sample of a batch ( I B i , G B i ∈ D train ) having P augmented views with temperature τ , our overall training objective is to minimize the following loss:

After the model parameters are updated on a batch of D train with batch size | B | , we update the queue Q by replacing its oldest coordinates with GPS coordinates from the current batch G B where | B |≪ S = | Q | , ensuring an up-to-date reflection of the most recent GPS coordinates encountered (naming it as 'dynamic queue' strategy).

Evaluation : By modeling the Earth as a continuous function, our location encoder can generate an embedding for any GPS coordinate on the planet. Hence, our method allows us to directly compare the embedding of a query image with the embeddings of any chosen GPS location, effectively eliminating the constraint of working within a predetermined set of classes.

Unlike existing works that create classes using the knowledge of the training set, we create a gallery of possible GPS coordinates during the evaluation phase by randomly sampling from the training set. The other ways for constructing the GPS gallery are discussed in the supplementary. To obtain the GPS predictions ( G Q k ) for a k th image in the unseen query set ( I Q k ), we compare the query image's embedding with the embeddings of GPS coordinates in our gallery. The GPS coordinate with the highest similarity to the query image's embedding indicates the most probable location for the image (i.e., G Q k is arg max G Q i ∈ D test V ( I Q k ) · L ( G Q i ) ).

4 Experiments

Datasets and Evaluation details : We perform experiments on publicly available benchmark datasets to have a fair comparison with existing works. For training, we use the MediaEval Placing Tasks 2016 (MP16 ) dataset [9], which consists of 4 . 72 million geotagged images from Flickr [1]. We test our trained model on several different datasets: Im2GPS3k [7], the recently introduced Google World Streets 15 K dataset (GWS15k) [5], and YFCC26k [23]. During testing, we conduct image-to-GPS retrieval where the query image from the test datasets is matched against a GPS gallery of 100 K (Im2GPS3k) and 500 K (GWS15k) coordinates. The performance of GeoCLIP on different gallery sizes is put in supplementary. In our evaluation, similar to [5], we also uses a Ten Crop ( 5 different crops of an image and their flipped counterparts) method where the predictions on these crops are averaged to make a joint prediction. We report our results using a threshold metric, similar to the protocol followed in previous works. Specifically, for each query image, we compute the Geodesic distance between GPS coordinates predicted by our model and the respective ground truth. We calculate how many of them (in % ) fall within the distance thresholds ( 1 km, 25 km, 200 km, 750 km, and 2500 km) and report average performance of model over three runs.

Implementation details : Each MLP f i used in our location encoder L ( · ) is composed of an input layer with 512 dimensions, four hidden layers (each with ReLU activation and 1024 dimensions), and an output layer of 512 dimensions. The RFF of the positional encoding layers have an input size of 2 dimensions and an output size of 512 dimensions. The dynamic queue ( | Q | as 4096 ) is initialized randomly with values drawn from a uniform distribution covering the range of latitude and longitude. We add gaussian noise to GPS coordinates with σ η and σ η ′ as 150 and 1000 respectively. Further ablations on the noise and queue length are put in supplementary. We use three hierarchies ( M as 3 ) with σ min as 2 0 , and σ max as 2 8 respectively (refer Sec. 4.3). Our image encoder V ( · ) utilizes OpenAI's pre-trained CLIP ViT-L/14 as backbone with two trainable linear layers h 1 and h 2 having dimensions of 768 and 512 respectively. We perform image augmentations similar to SimCLR.

Training details : Our GeoCLIP model is trained using Adam optimizer with a learning rate of 3 × 10 -5 and a weight decay of 1 × 10 -6 . We use a learning rate of 3 × 10 -4 to train classifiers with the CLIP backbone. We employ a step decay learning rate scheduler with a gamma of 0 . 87 and a step size of one epoch. The model is trained until convergence, which typically occurs around tenth epoch. We use batch size | B | as 512 when training on full dataset. For data-efficient settings with 20% , 10% , and 5% of the data, we use | B | as 256 , 256 , and 128 respectively. The learnable temperature parameter τ is initialized to 0 . 07 (refer eq. 4). The CLIP backbone is frozen during training while location encoder and h 1 and h 2 layers are fully trainable. We train our models on an NVIDIA Ampere GPU along with 12 CPUs. Our code is in PyTorch [13] and is shared in supplementary.

4.1 Comparison with State-of-the-art methods

Table 1: We compare the performance of GeoCLIP with the state-of-the-art methods on (a) Im2GPS3k [7] and (b) GWS15k [5] datasets. Our method yields consistent gains across datasets and different distance thresholds.

We perform a comparative analysis of GeoCLIP against worldwide Geo-Localization benchmarks, Im2GPS3k and Google World Streets 15k (GWS15k). The results are shown in Table 1. In all the metrics, our method surpasses the previous state-of-the-art (SOTA) performance on Im2GPS3k, achieving improvements of +1 . 31% , +0 . 97% , +3 . 95% , and +8 . 67% in the 1 km, 25 km, 200 km, 750 km, and 2500 km thresholds respectively. The results on another dataset (YFCC26k) are put in supplementary, where we also observe a similar trend.

Notably, our approach exhibits a large gain on the more challenging GWS15k dataset, surpassing the previous SOTA model with significant accuracy improvements of +1 . 6% , +8 . 2% , +18 . 8% , and +23 . 6% in the 25 km, 200 km, 750 km, and 2500 km thresholds respectively. Our performance is nearly double the accuracy of the previous state-of-the-art model. The GWS15k contains samples that are uniformly sampled across the Earth and are not biased towards any specific geographic location. Moreover, the images in this dataset have a large distribution shift compared to the training set, making the geo-localization task tough and challenging. Our substantial improvement can be attributed to the better embeddings achieved by our encoders. Specifically, the hierarchical nature of the location encoder helps focus on multiple resolutions of a location simultaneously, and its alignment with corresponding out-of-distribution images gets further enhanced with the robust features obtained by the CLIP backbone of our image encoder.

4.2 Performance of GeoCLIP in Limited Data settings

The worldwide geo-localization methods assume access to a training set containing millions of data samples. Training on such large scale datasets is often highly time-consuming, even with substantial computational resources at hand, taking days to complete. Moreover, practical limitations may arise, such as restricted access to data due to privacy concerns or proprietary rights held by companies. Hence, we also investigate the potential of our GeoCLIP method in limited-data settings. In the Figure 3, we present a comprehensive analysis of our method's performance on Im2GPS3k dataset across various data settings: 20% , 10% , 5% of the full dataset. Results on other datasets are provided in the supplementary. We can observe that our method maintains a high level of performance, closely approximating the results achieved with the full dataset, even as the available data decreases exponentially. Notably, even with a mere 20% of the data, the performance of GeoCLIP drops by only 0 . 0% , 0 . 5% , 1 . 6% , 1 . 9% , and 1 . 0% on 2500 km , 750 km, 200 km, 25 km, and 1 km distance thresholds respectively. These results highlight the efficiency of GeoCLIP in limited-data scenarios, emphasizing its potential to address practical challenges in worldwide geo-localization.

To further demonstrate the better efficacy of our method in limited data settings, we perform the same experiment with classification approach [12] (ISNs). We summarize the comparison of our method

with the classification approach on different quantities of limited training data in Table 2. It can be easily observed that the classification method's performance degrades much faster than our approach when the amount of training data is reduced, highlighting our method to be more data-efficient.

100%

20%

10%

5%

Percentage of Available Data

0

10

20

30

40

50

60

70

80

90

100

Accuracy (%)

14.1

13.1

12.1

10.8

34.5

32.6

31.9

31.1

50.6

49.0

48.9

48.7

69.7

69.2

69.1

67.6

83.8

83.8

83.6

83.2

GeoCLIP: Data Efficiency

1 km (Street)

25 km (City)

200 km (Region)

750 km (Country)

2500 km (Continent)

4.3 Ablations

Effective encoding of raw GPS coordinates : The direct use of an individual MLP to encode GPS coordinates into feature vectors serves as a baseline. As shown in Table 3(a), the MLP performs well at larger scales, but it stumbles at finer scales due to distortions in the standard coordinate system and its own spectral bias. To handle the distortions, we apply the Equal Earth Projection (EEP), which boosts performance at 1 km, 25 km, and 200 km thresholds by +0 . 1% , +6 . 07% , and +3 . 93% , respectively. However, the encoding of fine-grained details remains a challenge. To address this, we introduce an additional encoding layer using Random Fourier Features ( σ as 2 4 ). This layer enables the MLP to learn high-frequency functions necessary to distinguish features at finer scales, like streets and cities. This modification nearly doubles the accuracy at 1 km and 25 km, with increases of +5 . 14% and +14 . 95% , respectively. When we include harder negatives via our dynamic queue, improving performance across all scales, with a notable increase of +4 . 47% at the 1 km threshold.

Importance of Hierarchical Learning : We experiment with different σ 's in eq. 2 ( 2 0 , 2 4 , 2 8 ) to investigate their influence across various scales. With a single accuracy metric, one might typically seek an optimal sigma. However, we're optimizing for five scale metrics ( 1 km, 25 km, 200 km, 750 km, and 2500 km), making the task more difficult. Our findings, highlighted in Table 3(b) and further elaborated in the supplementary material, reveal an interesting trade-off: smaller σ excels at larger scales but has issues with smaller scales, while larger σ increases fine-grained accuracy but decreases on larger scales. Building upon this observation, we combine the representations from the encoders with varying σ s. By merging the outputs from the different encoders, we create a joint hierarchical representation that maximizes the location encoder's capabilities across all scales. This method outperforms the use of individual σ at all scales, effectively addressing the varying performance issue related to different σ s, thereby endorsing the need for hierarchical representation in our architecture.

(a) Encoding GPS coordinates

(b) Benefits of hierarchical learning

4.4 Qualitative Results: Geo-localization using Text-based query

We demonstrate GeoCLIP's ability to process text queries, highlighting the semantic richness of our model. As our image encoder has a pre-trained CLIP backbone (trained to match visual and text features), our location encoder gets inherent alignment with CLIP's text features, enabling us to map textual descriptions to geographical coordinates.

This capability allows us to assign a geographical context to any given text. Specifically, we can take any text, generate its embedding using CLIP text encoder, process it through our layers h 1 and h 2 , and calculate its similarity with a set of GPS embeddings. This results in a similarity map, demonstrating the geographical distribution of the text query's concept. For instance, using "Desert" as a text query, we compute the cosine similarity between the text query's embedding and the GPS embeddings. The resulting similarity map, (Figure 4) reveals the spatial distribution of desert regions worldwide.

Text Query: 'Desert'

4.5 Utility of our Location Encoder beyond Geo-localization

We investigate the utility of GeoCLIP's location encoder in improving multi-class image classification task. We perform experiments on the geo-tagged NUS-WIDE dataset [4] with the same train and test splits as used in [31]. We conduct two experiments: 1) Classifying an image using only its GPS location features, and 2) Classifying an image using a combination of its GPS location features and visual features.

We obtain GPS features from our pretrained location encoder (which is kept frozen), which are normalized and fed through a Multilayer Perceptron (MLP) with a single hidden layer of 2048 dimensions, followed by a dropout layer for regularization. For classification incorporating visual context, we adopt the Bag-of-Visual-Words (BovW) representation based on SIFT descriptors as the visual feature for a fair comparison with existing works. We normalize location and visual features, concatenate them, and then pass them through the MLP.

Our results in Table 4 indicate that GeoCLIP achieves state-of-the-art performance on both approaches. This is particularly significant, as it highlights that our location encoder has learned a semantically rich representation across the Earth, being able to generalize beyond geo-localization and can therefore be extended to assist other location-aware methods to improve performance without any significant changes to their architecture.

4.6 GeoCLIP with Prior Geographical Knowledge

In practical applications of geo-localization, users often possess some level of prior geographical context. For instance, while the exact GPS coordinates of an image may be unknown, the general originating area, such as the continent, country, state, or city, may be known. This additional information can effectively narrow the search space, thereby improving the model's accuracy due to the reduced number of potential locations for consideration. To assess GeoCLIP's adaptability in these situations, we perform an experiment where the gallery of potential GPS coordinates is limited within a predefined radius of known geographical knowledge of each image. We choose the radius as per the assumed level of prior knowledge (higher search radius when coarse information is known and smaller radius with the finer knowledge): for a continent, coordinates within 2500km are taken; for a country, within 750km; for a state, within 200km; and for a city, within 25km. Results across these different levels of granularity are shown in Table 5.

The performance of image-to-GPS retrieval improves as we search from coarse to finer granularity, which is quite intuitive. For instance, if we have prior knowledge of the city, it's much easier to geo-localize the true location than to have prior knowledge about the continent. The model's capacity to adapt to restricted search spaces affirms its applicability in real-world scenarios where partial geographic context is commonly available.

5 Conclusion

We addressed the challenge of worldwide geo-localization by formulating it as an image-to-GPS retrieval problem. We evaluated the efficacy of our method (GeoCLIP) not only in full training data scenarios but also in limited data settings. The performance of GeoCLIP was competitive even when the training data was reduced significantly. GeoCLIP could even perform geo-localization using text queries by leveraging the CLIP[15] backbone of our image encoder. Moreover, our location encoder (using equal earth projection[17], RFF[21], and MLPs) better encoded GPS coordinates compared to the direct use of MLPs. However, coarse σ or a fine-grained σ used in RFF performed better in localizing either large or small areas. Our hierarchical learning overcame this issue by capturing the location features at various scales, leading to a further performance boost. Also, our location encoder is not tied to geo-localization and helps to aid in classification problems. In the future, we plan to further investigate the utility of our location encoder in assisting other computer vision problems. Limitations : As we use a CLIP-based backbone in our image encoder, precomputing the image features on the large training dataset (each image of dimension 224 × 224 × 3 ) is time-consuming. But once the features are precomputed, the training of our GeoCLIP model is relatively fast. We also observed that a single σ value used in RFF is insufficient to perform well on both small and large areas. However, our hierarchical learning is a promising direction to overcome this issue.

Broader Impacts : Our image-based geo-localization can act as a substitute for GPS-based localization in scenarios where GPS signals are unreliable, blocked, or inaccurate. Combining image-based geo-tagging with GPS-based positioning systems can help to improve stability and safety in applications like autonomous driving and navigation. Also, it can benefit digital forensics, but avoiding its unethical use that could mislead location detection systems or violate privacy is important. More discussion on ethical issues and possible mitigation approaches are provided in the supplementary.

Acknowledgments and Disclosure of Funding

This work was supported by the US Army contract W911NF-2120192. We would like to extend our gratitude to all the reviewers for their valuable suggestions. We would also like to thank Brandon Clark for his helpful discussions on this project.

References

Flickr : https://www.flickr.com.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PMLR, 2020.

Gordon A. Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6172-6180, 2017.

Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yantao Zheng. Nus-wide: a real-world web image database from national university of singapore. In ACM International Conference on Image and Video Retrieval , 2009.

Brandon Clark, Alec Kerrigan, Parth Parag Kulkarni, Vicente Vivanco Cepeda, and Mubarak Shah. Where we are and what we're looking at: Query based worldwide image geo-localization using hierarchies and scenes. arXiv preprint arXiv:2303.04249 , 2023.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

James Hays and Alexei A. Efros. Im2gps: estimating geographic information from a single image. In 2008 IEEE Conference on Computer Vision and Pattern Recognition , pages 1-8, 2008.

Mike Izbicki, Evangelos E Papalexakis, and Vassilis J Tsotras. Exploiting the earth's spherical geometry to geolocate images. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2019, Würzburg, Germany, September 16-20, 2019, Proceedings, Part II , pages 3-19. Springer, 2020.

Martha Larson, Mohammad Soleymani, Guillaume Gravier, Bogdan Ionescu, and Gareth J.F. Jones. The benchmarking initiative for multimedia evaluation: Mediaeval 2016. IEEE MultiMedia , 24(1):93-96, 2017.

Shuai Liao, Xirong Li, Heng Tao Shen, Yang Yang, and Xiaoyong Du. Tag features for geo-aware image classification. IEEE Transactions on Multimedia , 17(7):1058-1067, 2015.

Liu Liu and Hongdong Li. Lending orientation to neural networks for cross-view geolocalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 5624-5633, 2019.

Eric Muller-Budack, Kader Pustu-Iren, and Ralph Ewerth. Geolocation estimation of photos using a hierarchical model and scene classification. In Proceedings of the European conference on computer vision (ECCV) , pages 563-579, 2018.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32 , pages 8024-8035. Curran Associates, Inc., 2019.

Shraman Pramanick, Ewa M Nowara, Joshua Gleason, Carlos D Castillo, and Rama Chellappa. Where in the world is this image? transformer-based geo-localization in the wild. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVIII , pages 196-215. Springer, 2022.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Krishna Regmi and Mubarak Shah. Bridging the domain gap for ground-to-aerial image matching. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 470-479, 2019.

Bojan Šavriˇ c, Tom Patterson, and Bernhard Jenny. The equal earth map projection. International Journal of Geographical Information Science , 33(3):454-465, 2019.

Paul Hongsuck Seo, Tobias Weyand, Jack Sim, and Bohyung Han. Cplanet: Enhancing image geolocalization by combinatorial partitioning of maps. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 536-551, 2018.

Yujiao Shi, Liu Liu, Xin Yu, and Hongdong Li. Spatial-aware feature aggregation for image based cross-view geo-localization. Advances in Neural Information Processing Systems , 32, 2019.

Yujiao Shi, Xin Yu, Dylan Campbell, and Hongdong Li. Where am i looking at? joint location and orientation estimation by cross-view matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4064-4072, 2020.

Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems , 33:7537-7547, 2020.

K. Tang, M. Paluri, L. Fei-Fei, R. Fergus, and L. Bourdev. Improving image classification with location context. In 2015 IEEE International Conference on Computer Vision (ICCV) , pages 1008-1016, Los Alamitos, CA, USA, dec 2015. IEEE Computer Society.

Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. Communications of the ACM , 59(2):64-73, 2016.

Yicong Tian, Chen Chen, and Mubarak Shah. Cross-view image matching for geo-localization in urban environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3608-3616, 2017.

Aysim Toker, Qunjie Zhou, Maxim Maximov, and Laura Leal-Taixé. Coming down to earth: Satellite-to-street view synthesis for geo-localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6488-6497, 2021.

Nam Vo, Nathan Jacobs, and James Hays. Revisiting im2gps in the deep learning era. In Proceedings of the IEEE international conference on computer vision , pages 2621-2630, 2017.

Tobias Weyand, Ilya Kostrikov, and James Philbin. Planet-photo geolocation with convolutional neural networks. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14 , pages 37-55. Springer, 2016.

Scott Workman, Richard Souvenir, and Nathan Jacobs. Wide-area image geolocalization with aerial reference imagery. In Proceedings of the IEEE International Conference on Computer Vision , pages 3961-3969, 2015.

Hongji Yang, Xiufan Lu, and Yingying Zhu. Cross-view geo-localization with layer-to-layer transformer. Advances in Neural Information Processing Systems , 34:29009-29020, 2021.

Yifang Yin, Zhenguang Liu, Ying Zhang, Sheng Wang, Rajiv Ratn Shah, and Roger Zimmermann. Gps2vec: Towards generating worldwide gps embeddings. Proceedings of the 27th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems , 2019.

Yifang Yin, Ying Zhang, Zhenguang Liu, Yuxuan Liang, Sheng Wang, Rajiv Ratn Shah, and Roger Zimmermann. Learning multi-context aware location representations from large-scale geotagged images. In Proceedings of the 29th ACM International Conference on Multimedia , MM'21, page 899-907, New York, NY, USA, 2021. Association for Computing Machinery.

Sijie Zhu, Mubarak Shah, and Chen Chen. Transgeo: Transformer is all you need for cross-view image geo-localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1162-1171, 2022.

Sijie Zhu, Linjie Yang, Chen Chen, Mubarak Shah, Xiaohui Shen, and Heng Wang. r 2 former: Unified retrieval and reranking transformer for place recognition. arXiv preprint arXiv:2304.03410 , 2023.

Sijie Zhu, Taojiannan Yang, and Chen Chen. Vigor: Cross-view image geo-localization beyond one-to-one retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3640-3649, 2021.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the image geolocalization problem. Instead of formulating the task as image retrieval or classification over a set of geospatial bins as is typical, the authors frame it as image-to-GPS retrieval by matching image features with location features. The proposed training strategy leverages Random Fourier Features (RFF) to encode geospatial location across multiple scales and align them with features from a CLIP visual encoder, with several training enhancements. The resulting method achieves state of the art performance on three benchmarks, across all scales, and showcases interesting applications in text geolocalization and classification.\n\nSTRENGTHS:\n- Addresses an important and interesting problem. Image localization is a fundamental problem in vision that continues to receive attention in the literature. \n\n- The approach has technical novelty and is of the first to formulate the localization task as an image-to-GPS retrieval problem. The key idea is to relate images directly to locations via aligning image features to location embeddings using contrastive learning. Such an approach uses a continuous model of location whereas traditional geolocalization approaches depend on a large reference database of images with known locations to support image-level matching (via image retrieval or classification) and inferring location from the closest match. Instead, the authors propose the concept of a GPS gallery.  Though training still requires geotagged images (image + location), the authors show the proposed approach requires less training data.  Further, at inference, the GPS gallery can theoretically be any set of locations to query against, instead of being limited by the reference image set. Geospatial location is represented using NeRF-style location embeddings (Random Fourier Features with an MLP).  Results show that infusing the location representations with multiple spatial scales improves geolocalization performance. \n\n- Evaluation is extensive and compelling. State-of-the-art (SOTA) performance on multiple datasets at different threshold scales (street, city, region, country, and continent). Near SOTA performance on Im2GPS3k training using only 20% of the training data.\n\n- Intriguing applications. The results on text geolocalization and geotagged image classification showcase the utility of the location encoding beyond image-based geolocalization.\n\nWEAKNESSES:\n- A central idea of this paper is the concept of modeling location continuously. I was disappointed to see that during evaluation, the GPS gallery (the set of possible locations) was limited to locations sampled from the training set (L230). Though I understand why this was done from an evaluation perspective, it significantly hurts the \"global localization\" theme. At the very least, I would have liked to see an evaluation of how selecting the GPS gallery at inference (choice of sampling distribution, granularity, etc.) impacts performance. This also raises the issue that the location encoder is not truly global, it is limited by where images have been captured on the globe (as training requires image/location pairs). \n\n- Some minor details missing. Section 4.2 does not mention which dataset the experiment was performed on. Comparing the numbers from Figure 3 and Table 1a, it can be assumed to have been conducted on Im2GPS3k, but the dataset should still be mentioned to improve clarity. Similarly, section 4.3 does not mention which dataset(s) was/were used for the ablation study.\n\n- The application for text query-based geolocalization is interesting, but the results are lacking. It is only demonstrated qualitatively on a continent scale using a single keyword (desert) in the main paper. While this is interesting, it leaves a lot to be desired in terms of understanding the full or even partial capability of the text encoding for tasks such as geolocalization or digital forensics. \n\n- Quality of the presentation could be improved. For example, the introduction could provide additional motivation for the underlying problem. In its current form, the introduction mentions that there are a variety of applications for geolocalization, but does not support this claim with any citations or descriptions of how geolocalization benefits them. Instead, the majority of content is related to why the problem is difficult and how existing approaches are inadequate, repeating much of the related work section content (e.g., binning, classification, and scaling methods). The conclusion section, in particular, lacks flow and could benefit from additional editing.  Unclear mathematical notation on L138. What is K?  Minor grammatical errors (L49, L107, L111, L372, etc.).\n\n- Word choice in the conclusion. L353 suggests that this work “solved worldwide geolocalization”. Of course this problem is far from solved!\n\nQUESTIONS:\nMy initial rating is weak accept. This paper introduces a method for training a high-quality geospatial location encoder that interacts with, widely used, CLIP features and demonstrates its use for localization and several applications. This has the potential to benefit the wider community.\n\nQuantitatively, how does the proposed approach compare in terms of runtime and memory footprint with existing approaches?\n\nHow does the text encoding perform at smaller scales (region or city)?\n\nCan the text encoding benefit the image-based geolocalization?\n\nLIMITATIONS:\nLimitations addressed in the manuscript.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the problem of geo-locating an image, i.e. estimating the latitude and longitude of the where the image was taken. \n\nPrevious works treat the problem as a classification problem (of cells distributed across the globe). In contrast the this work proposes to retrieve the GPS coordinates from a collection of possible locations. In order to do so the authors propose a neural network architecture that encodes the (ground truth) GPS location and the image in a joint feature space and train it via a contrastive learning scheme. During the inference the query image is encoded and the GPS coordinate closest in feature space is taken as the predicted location.\n\nThe main contribution of the paper is the positional encoding of GPS coordinates via random fourier features [21], which are demonstrated to help represent higher frequency information and hence improve localization at higher spatial resolution (Tab. 2).\n\nExperimental results demonstrate state-of-the-art retrieval performance on 3 different datasets.\n\nSTRENGTHS:\n**S1** The encoding of GPS coordinates via random fourier features (RFF) is novel. While RFF have been demonstrated to be useful to capture higher frequency properties for 2d image or 3d coordinates [21], e.g. in NeRF, to the best of my knowledge they have not been used to encode lat/lng coordinates for geo-localization. The authors also demonstrate that an equal earth projection (EEP) (which preserves the area covered) is beneficial for encoding 2d coordinates on the globe, especially for higher accuracy localization (< 1km in Tab. 2).\n\n**S2** Good ablation studies that demonstrate the usefulness of the chosen solutions, especially Tab. 2 for EEP and RFF, but also the ablations provided in the sup. material in Sec. 3. This helps the reproducibility of the approach.\n\n**S3** The place recognition performance improves considerably over state-of-the-art, even though the taken approach is relatively simple. This can be attributed to the ability of the model to learn an understand of spatial information.\n\nWEAKNESSES:\n**W1** The usage of the CLIP frozen backbone is not well motivated and evaluated. As such it remains unclear why CLIP features are needed for the place recognition task, rather than a network trained from scratch or fine-tuned on the task. Experimental evaluation demonstrates how the learned feature is able to relate textual information to geographic locations, but the influence of this capability on predicting lat/lng locations for a query image is not evaluated. \n\n**W2** The evaluation of the text-to-location encoding in Sec. 4.5 is very limited. The encoded location is used for image classification, but the paper falls too short in outlining the goal and metric of the chosen classification task. As such the evaluation can not stand by it's own but the reader needs to resort to [4, 31] for context.\n\n**W3** The size of the gallery (the quantization of possible GPS locations) is tuned per dataset. Instead a single size across all dataset would foster broader applicability of the approach. Sec. 3.1 of the sup. material suggests that this would in fact be possible, e.g. with a size of ~250k. (See also Q2)\n\nQUESTIONS:\n**Q1** We do you keep the CLIP backbone frozen?\n\n**Q2** Why is the gallery of GPS coordinates sampled from the *training* set? Is this on the dataset from [9] which is used for training? Did you consider covering the globe, e.g. uniformly?\nThe dependency of the geographic locations on the training set implies that the test set needs to be covered by the training set. This seems to prevent correct out-of-distribution predictions.\n\nLIMITATIONS:\nThe ability to accurately locate any photo on the earth represents a security and privacy issue. E.g., even if a user ops out of GPS tagging their photos, the presented approach enables to infer the user location from their images -- and the location trace of the user over time.\nThis surveillance capability should be stated as negative societal impact or potential harmful consequence of the approach, when deployed in public.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper proposes an image-to-GPS retrieval method, with an image encoder from the pre-trained CLIP and a tailored GPS encoder.

The GPS encoder is implemented by using:  i) the Equal Earth Projection to mitigate the GPS distortion; ii) Random Fourier transform with multiple MLPs to obtain high-frequency `positional encoding'.

Experiments on standard benchmarks demonstrate the effectiveness of the proposed method.

STRENGTHS:
1) This paper is well-written and easy to follow;

2) The idea of this paper is interesting and new;

3) The experimental evaluations are strong, particularly the interesting Section 4.4.

WEAKNESSES:
I only have some minor comments:

1) Though it is interesting to adopt the CLIP pre-trained encoders, I would like to the experiments substituting the CLIP image encoder with Swin encoder (as GeoDecoder [5]), or substituting the Swin encoder in [5] with the CLIP image encoder.

This would help readers to clearly understand the contribution of the CLIP image encoder.

2) I would like to see an experiment under the following setting:

a) Evenly partition the earth into patches, and use the patch center GPS as the database;

b) perform the proposed image-to-GPS retrieval and report the performance;

c) Repeat a) and b) with coarse-to-fine patch partitions.

3) Please explicitly list the image-encoding time and the image-to-GPS retrieval time;

4) Please tone down the claim of "However, this approach is not practical as it becomes infeasible to construct a gallery containing images covering the entire world." The reason is that we have a world satellite image gallery.

QUESTIONS:
Please refer to weaknesses.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
This work learns an image-to-GPS retrieval approach where the image is a query, and the gallery
is a collection of GPS coordinates. The authors propose GeoCLIP, which consists of a location encoder, and an image encoder. The retrieval then is based given a new query image against a collection of gps embeddings. Experimental results show that this work outperforms previous SOTA at the Im2GPS3k and GWS15k datasets, across all distance thresholds.

STRENGTHS:
* This is a well written paper, with good motivation, writing and experimental section. 
* The extra results on "Utility of our location encoder beyond geo-localization" provide some more arguments about the effectiveness of their location encoder
*  The discussion about encoding GPS with an MLP is very interesting and is backed by good experiments. 
 * Hierarchical learning experiments are solid and seem to provide good justification for the design choices.

WEAKNESSES:
* For Figure 3, I have no baseline to compare against. How would other methods compare in terms of how fast the curve degrades?
* For the GPS encoding, how come the "city" scale is the one with the biggest rise? Could it be something related to the choice of parameters for the Random Fourier Features? It might make sense if some of the discussion from the supplementary material can be transferred to the main paper here since this is a key part of the method. 
* For the hierarchical method, how much does the merging of the outputs from the different encoders affect the overall computational needs?

QUESTIONS:
I have stated my questions above in the weaknesses, so this is what I would like to get answers from the authors on.

LIMITATIONS:
The authors do address this in their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nFor global localization task for a query photo, the authors propose a CLIP-inspired image-to-GPS retrieval approach (named ‘GeoCLIP’) where we retrieve the GPS coordinates of an unseen query image by matching it with the gallery of GPS coordinates.  Regarding location encoder, some new ideas such as Random Fourier Features and Hierarchical Representation were introduced. The experimental results showed the effectiveness of the proposed method by outpeforming the existing baselines.\n\nSTRENGTHS:\n* Matching GPS location embeddings and image embeddings (GeoCLIP) are introduced into an image localization problem. This is novel.  \n\n* It is very interesting that the location encoder gets inherent alignment with CLIP’s text features, enabling us to map textual descriptions to geographical coordinates.\n\n* The proposed location encoding method employing Random Fourier Features is much improved compared with the straightforward method which is the direct use of an individual MLP to encode GPS coordinates into feature vectors.\n\n* The experimental results showed that the proposed method outperformed the existing baselines which employs image search methods or grid classification methods.\n\nWEAKNESSES:\n* Matching an image embedding with many location embedding (100K (Im2GPS3k) and 500K (GWS15k) coordinates) is needed to estimate a GPS location.\n\nQUESTIONS:\n* This approach is interesting. However, it seems to assume that one location corresponds to one image feature. Is it always true ?  For example, in restraurants, there are many kinds of meal menus. In some places, the view might be changed greatly with viewing direction change or even with a small location movement.\n\n* When searching the location of a query image, matching it with a collection of GPS coordinates is needed. The needed gallery size of the GPS coordinate seems to be large. How long does it take to estimate the location of one image in case of 100K (Im2GPS3k) and 500K (GWS15k) coordinates.? It seems to need a relatively long time. How about comparison to the baselines regarding search time ?\nOf course, all search-based methods has this disadvantage as well. However, for image search, many fast search methods has been proposed such as hash-based search. Is it possible to use any fast search methods for the proposed method ?\nHow about search time and computational cost compared with the classification-based methods ?\n\nLIMITATIONS:\nA limitation was mentioned regarding image encoding at times of training.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}