SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Unlike existing works that create classes using the knowledge of the training set, we create a gallery of possible GPS coordinates during the evaluation phase by randomly sampling from the training set. The other ways for constructing the GPS gallery are discussed in the supplementary. To obtain the GPS predictions  $(G_k^Q)$  for a  $k^{th}$  image in the unseen query set  $(I_k^Q)$ , we compare the query image's embedding with the embeddings of GPS coordinates in our gallery. The GPS coordinate with the highest similarity to the query image's embedding indicates the most probable location for the image (i.e.,  $G_k^Q$  is  $\arg \max_{G_i^Q \in D_{test}} \mathcal{V}(I_k^Q) \cdot \mathcal{L}(G_i^Q)$ ).
Datasets and Evaluation details: We perform experiments on publicly available benchmark datasets to have a fair comparison with existing works. For training, we use the MediaEval Placing Tasks 2016 (MP-16) dataset [9], which consists of 4.72 million geotagged images from Flickr [1]. We test our trained model on several different datasets: Im2GPS3k [7], the recently introduced Google World Streets 15K dataset (GWS15k) [5], and YFCC26k [23]. During testing, we conduct image-to-GPS retrieval where the query image from the test datasets is matched against a GPS gallery of  $100\mathrm{K}$  (Im2GPS3k) and  $500\mathrm{K}$  (GWS15k) coordinates. The performance of GeoCLIP on different gallery sizes is put in supplementary. In our evaluation, similar to [5], we also uses a Ten Crop (5 different crops of an image and their flipped counterparts) method where the predictions on these crops are averaged to make a joint prediction. We report our results using a threshold metric, similar to the protocol followed in previous works. Specifically, for each query image, we compute the Geodesic distance between GPS coordinates predicted by our model and the respective ground truth. We calculate how many of them (in  $\%$  ) fall within the distance thresholds (1km, 25km, 200km, 750km, and  $2500\mathrm{km}$ ) and report average performance of model over three runs.
Implementation details: Each MLP  $f_{i}$  used in our location encoder  $\mathcal{L}(\cdot)$  is composed of an input layer with 512 dimensions, four hidden layers (each with ReLU activation and 1024 dimensions), and an output layer of 512 dimensions. The RFF of the positional encoding layers have an input size of 2 dimensions and an output size of 512 dimensions. The dynamic queue  $(|Q|$  as 4096) is initialized randomly with values drawn from a uniform distribution covering the range of latitude and longitude. We add gaussian noise to GPS coordinates with  $\sigma_{\eta}$  and  $\sigma_{\eta'}$  as 150 and 1000 respectively. Further ablations on the noise and queue length are put in supplementary. We use three hierarchies  $(M$  as 3) with  $\sigma_{min}$  as  $2^{0}$ , and  $\sigma_{max}$  as  $2^{8}$  respectively (refer Sec. 4.3). Our image encoder  $\mathcal{V}(\cdot)$  utilizes OpenAI's pre-trained CLIP ViT-L/14 as backbone with two trainable linear layers  $h_1$  and  $h_2$  having dimensions of 768 and 512 respectively. We perform image augmentations similar to SimCLR.
Training details: Our GeoCLIP model is trained using Adam optimizer with a learning rate of  $3 \times 10^{-5}$  and a weight decay of  $1 \times 10^{-6}$ . We use a learning rate of  $3 \times 10^{-4}$  to train classifiers with the CLIP backbone. We employ a step decay learning rate scheduler with a gamma of 0.87 and a step size of one epoch. The model is trained until convergence, which typically occurs around tenth epoch. We use batch size  $|B|$  as 512 when training on full dataset. For data-efficient settings with  $20\%$ ,  $10\%$ , and  $5\%$  of the data, we use  $|B|$  as 256, 256, and 128 respectively. The learnable temperature parameter  $\tau$  is initialized to 0.07 (refer eq. 4). The CLIP backbone is frozen during training while location encoder and  $h_1$  and  $h_2$  layers are fully trainable. We train our models on an NVIDIA Ampere GPU along with 12 CPUs. Our code is in PyTorch [13] and is shared in supplementary.
4.1 Comparison with State-of-the-art methods
Table 1: We compare the performance of GeoCLIP with the state-of-the-art methods on (a) Im2GPS3k [7] and (b) GWS15k [5] datasets. Our method yields consistent gains across datasets and different distance thresholds.
(a) Results on the Im2GPS3k [7] dataset
Method	Street 1 km	City 25 km	Region 200 km	Country 750 km	Continent 2500 km
[L]kNN, σ = 4 [26]	7.2	19.4	26.9	38.9	55.9
PlaNet [27]	8.5	24.8	34.3	48.4	64.6
CPlaNet [18]	10.2	26.5	34.6	48.6	64.6
ISNs [12]	10.5	28.0	36.6	49.7	66.0
Translocator [14]	11.8	31.1	46.7	58.9	80.1
GeoDecoder [5]	12.8	33.5	45.9	61.0	76.1
Ours	14.11	34.47	50.65	69.67	83.82
(b) Results on the recent GWS15k [5] dataset
Method	Street 1 km	City 25 km	Region 200 km	Country 750 km	Continent 2500 km
ISNs [12]	0.05	0.6	4.2	15.5	38.5
Translocator [14]	0.5	1.1	8.0	25.5	48.3
GeoDecoder [5]	0.7	1.5	8.7	26.9	50.5
Ours	0.6	3.1	16.9	45.7	74.1
We perform a comparative analysis of GeoCLIP against worldwide Geo-Localization benchmarks, Im2GPS3k and Google World Streets 15k (GWS15k). The results are shown in Table 1. In all the metrics, our method surpasses the previous state-of-the-art (SOTA) performance on Im2GPS3k, achieving improvements of  $+1.31\%$ ,  $+0.97\%$ ,  $+3.95\%$ , and  $+8.67\%$  in the 1km, 25km, 200km, 750km, and 2500km thresholds respectively. The results on another dataset (YFCC26k) are put in supplementary, where we also observe a similar trend.
Notably, our approach exhibits a large gain on the more challenging GWS15k dataset, surpassing the previous SOTA model with significant accuracy improvements of  $+1.6\%$ ,  $+8.2\%$ ,  $+18.8\%$ , and  $+23.6\%$  in the 25km, 200km, 750km, and 2500km thresholds respectively. Our performance is nearly double the accuracy of the previous state-of-the-art model. The GWS15k contains samples that are uniformly sampled across the Earth and are not biased towards any specific geographic location. Moreover, the images in this dataset have a large distribution shift compared to the training set, making the geo-localization task tough and challenging. Our substantial improvement can be attributed to the better embeddings achieved by our encoders. Specifically, the hierarchical nature of the location encoder helps focus on multiple resolutions of a location simultaneously, and its alignment with corresponding out-of-distribution images gets further enhanced with the robust features obtained by the CLIP backbone of our image encoder.
4.2 Performance of GeoCLIP in Limited Data settings
The worldwide geo-localization methods assume access to a training set containing millions of data samples. Training on such large scale datasets is often highly time-consuming, even with substantial computational resources at hand, taking days to complete. Moreover, practical limitations may arise, such as restricted access to data due to privacy concerns or proprietary rights held by companies. Hence, we also investigate the potential of our GeoCLIP method in limited-data settings. In the Figure 3, we present a comprehensive analysis of our method's performance on Im2GPS3k dataset across various data settings:  $20\%$ ,  $10\%$ ,  $5\%$  of the full dataset. Results on other datasets are provided in the supplementary. We can observe that our method maintains a high level of performance, closely approximating the results achieved with the full dataset, even as the available data decreases exponentially. Notably, even with a mere  $20\%$  of the data, the performance of GeoCLIP drops by only  $0.0\%$ ,  $0.5\%$ ,  $1.6\%$ ,  $1.9\%$ , and  $1.0\%$  on  $2500\mathrm{km}$ ,  $750\mathrm{km}$ ,  $200\mathrm{km}$ ,  $25\mathrm{km}$ , and  $1\mathrm{km}$  distance thresholds respectively. These results highlight the efficiency of GeoCLIP in limited-data scenarios, emphasizing its potential to address practical challenges in worldwide geo-localization.
To further demonstrate the better efficacy of our method in limited data settings, we perform the same experiment with classification approach [12] (ISNs). We summarize the comparison of our method
with the classification approach on different quantities of limited training data in Table 2. It can be easily observed that the classification method's performance degrades much faster than our approach when the amount of training data is reduced, highlighting our method to be more data-efficient.
Figure 3: We vary the amount of training data from  $100\%$  to as low as  $5\%$ . GeoCLIP achieves respectable accuracy even with only  $5\%$  of the training data, demonstrating its utility in limited data scenarios.
Table 2: Comparison of data efficiency of GeoCLIP with respect to a classification approach when the amount of training data is reduced drastically. The value inside brackets on each row shows the performance degradation compared to the respective method's performance with  $100\%$  training data.
Method	Im2GPS3k (Amount of Training Data)
100%	20%	10%	5%
2500 km (GeoCLIP, Ours)	83.8	83.8 (↓0.0)	83.6 (↓0.2)	83.2 (↓0.6)
2500km (ISNs [12])	66.0	50.1 (↓15.9)	48.7 (↓17.3)	43.9 (↓22.1)
750km (GeoCLIP, Ours)	69.7	69.2 (↓0.5)	69.1 (↓0.6)	67.6 (↓2.1)
750km (ISNs [12])	49.7	31.9 (↓17.8)	29.2 (↓20.5))	25.1 (↓24.6)
200km (GeoCLIP, Ours)	50.6	49.0 (↓1.6)	48.9 (↓1.7)	48.7 (↓1.9)
200km (ISNs [12])	36.6	20.0 (↓16.6)	17.4 (↓19.2)	14.3 (↓22.3)
25km (GeoCLIP, Ours)	34.5	32.6 (↓1.9)	31.9 (↓2.6)	31.1 (↓3.4)
25km (ISNs [12])	28.0	14.1 (↓13.9)	11.9 (↓16.1)	9.6 (↓18.4)
1km (GeoCLIP, Ours)	14.1	13.1 (↓1.0)	12.1 (↓2.0)	10.8 (↓3.3)
1km (ISNs [12])	10.5	5.1 (↓5.4)	4.5 (↓6.0)	3.2 (↓7.3)
4.3 Ablations
Effective encoding of raw GPS coordinates: The direct use of an individual MLP to encode GPS coordinates into feature vectors serves as a baseline. As shown in Table 3(a), the MLP performs well at larger scales, but it stumbles at finer scales due to distortions in the standard coordinate system and its own spectral bias. To handle the distortions, we apply the Equal Earth Projection (EEP), which boosts performance at  $1\mathrm{km}$ ,  $25\mathrm{km}$ , and  $200\mathrm{km}$  thresholds by  $+0.1\%$ ,  $+6.07\%$ , and  $+3.93\%$ , respectively. However, the encoding of fine-grained details remains a challenge. To address this, we introduce an additional encoding layer using Random Fourier Features ( $\sigma$  as  $2^{4}$ ). This layer enables the MLP to learn high-frequency functions necessary to distinguish features at finer scales, like streets and cities. This modification nearly doubles the accuracy at  $1\mathrm{km}$  and  $25\mathrm{km}$ , with increases of  $+5.14\%$  and  $+14.95\%$ , respectively. When we include harder negatives via our dynamic queue, improving performance across all scales, with a notable increase of  $+4.47\%$  at the  $1\mathrm{km}$  threshold.
Importance of Hierarchical Learning: We experiment with different  $\sigma$ 's in eq. 2 ( $2^0, 2^4, 2^8$ ) to investigate their influence across various scales. With a single accuracy metric, one might typically seek an optimal sigma. However, we're optimizing for five scale metrics (1km, 25km, 200km, 750km, and 2500km), making the task more difficult. Our findings, highlighted in Table 3(b) and further elaborated in the supplementary material, reveal an interesting trade-off: smaller  $\sigma$  excels at larger scales but has issues with smaller scales, while larger  $\sigma$  increases fine-grained accuracy but decreases on larger scales. Building upon this observation, we combine the representations from the encoders with varying  $\sigma$ s. By merging the outputs from the different encoders, we create a joint hierarchical representation that maximizes the location encoder's capabilities across all scales. This method outperforms the use of individual  $\sigma$  at all scales, effectively addressing the varying performance issue related to different  $\sigma$ s, thereby endorsing the need for hierarchical representation in our architecture.
Table 3: We perform ablations on Im2GPS3k to show the importance of different components of our location encoder. (a) Our encoding of GPS coordinates using Equal Earth Projection (EEP), Random Fourier Features(RFF), and Dynamic Queue (DQ) strategy led to significant gains compared to the MLP alone, especially in challenging localization within  $1\mathrm{km}$  and  $25\mathrm{km}$ . (b) Our GeoCLIP approach with hierarchies (GeoCLIP w/ H) leads to further gains and even performs much better than the respective classification with hierarchies C (w/ H) approach. The improvements with respect to baseline in (a) and C (w/ H) in (b) are shown in brackets and highlighted in blue.
(a) Encoding GPS coordinates
Method	Street 1 km	City 25 km	Region 200 km	Country 750 km	Continent 2500 km
MLP (Baseline)	0.13	11.01	41.68	67.83	83.28
+EEP	0.23	17.08	45.61	68.8	82.98
+EEP +RFF	5.37	32.03	49.42	68.4	82.98
+EEP +RFF +DQ	9.84	33.1	49.28	69.07	83.35
(+9.71)	(+22.09)	(+7.6)	(+1.24)	(+0.07)
(b) Benefits of hierarchical learning
Method	Street 1 km	City 25 km	Region 200 km	Country 750 km	Continent 2500 km
C (w/ H)	11.4	34.3	49.4	67.4	80.8
GeoCLIP (Coarse)	1.07	24.66	49.18	69.47	83.75
GeoCLIP (Medium)	9.84	33.1	49.28	69.07	83.35
GeoCLIP (Fine)	13.61	33.43	47.78	65.1	80.95
GeoCLIP (w/ H)	14.11 (+2.71)	34.47 (+0.17)	50.65 (+1.25)	69.67 (+2.27)	83.82 (+2.02)
4.4 Qualitative Results: Geo-localization using Text-based query
We demonstrate GeoCLIP's ability to process text queries, highlighting the semantic richness of our model. As our image encoder has a pre-trained CLIP backbone (trained to match visual and text features), our location encoder gets inherent alignment with CLIP's text features, enabling us to map textual descriptions to geographical coordinates.
This capability allows us to assign a geographical context to any given text. Specifically, we can take any text, generate its embedding using CLIP text encoder, process it through our layers  $h_1$  and  $h_2$ , and calculate its similarity with a set of GPS embeddings. This results in a similarity map, demonstrating the geographical distribution of the text query's concept. For instance, using "Desert" as a text query, we compute the cosine similarity between the text query's embedding and the GPS embeddings. The resulting similarity map, (Figure 4) reveals the spatial distribution of desert regions worldwide.
Figure 4: Qualitative demonstration: Geo-localization with text query using GeoCLIP. Our location can differentiate between geographical coordinates and associate semantic meaning from text to geographical space, providing a geographical context to textual descriptions.
Method	Classifier	mAP
Visual (V)	visual	0.234
OneHot [3]	geo	0.066
HashTag [22]	geo	0.163
TagFeature [10]	geo	0.161
GPS2Vec onehot [30]	geo	0.130
GPS2Vec [30]	geo	0.182
GPS2Vec+ [31]	geo	0.210
GeoCLIP (Ours)	geo	0.249
OneHot [3] + V	fusion	0.238
HashTag [22] + V	fusion	0.261
TagFeature [10] + V	fusion	0.260
GPS2Vec Onehot [30] + V	fusion	0.277
GPS2Vec [30] + V	fusion	0.300
Liao et al. [10]	fusion	0.347
GPS2Vec+ [31] + V	fusion	0.348
GeoCLIP (Ours) + V	fusion	0.362
Table 4: Comparison of methods utilizing GPS data for image classification on the geotagged NUSWIDE Dataset. Our method outperforms existing methods using our GPS features with or without visual features.
4.5 Utility of our Location Encoder beyond Geo-localization
We investigate the utility of GeoCLIP's location encoder in improving multi-class image classification task. We perform experiments on the geo-tagged NUS-WIDE dataset [4] with the same train and test splits as used in [31]. We conduct two experiments: 1) Classifying an image using only its GPS location features, and 2) Classifying an image using a combination of its GPS location features and visual features.
We obtain GPS features from our pretrained location encoder (which is kept frozen), which are normalized and fed through a Multilayer Perceptron (MLP) with a single hidden layer of 2048 dimensions, followed by a dropout layer for regularization. For classification incorporating visual context, we adopt the Bag-of-Visual-Words (BovW) representation based on SIFT descriptors as the visual feature for a fair comparison with existing works. We normalize location and visual features, concatenate them, and then pass them through the MLP.
Our results in Table 4 indicate that GeoCLIP achieves state-of-the-art performance on both approaches. This is particularly significant, as it highlights that our location encoder has learned a semantically rich representation across the Earth, being able to generalize beyond geo-localization and can therefore be extended to assist other location-aware methods to improve performance without any significant changes to their architecture.
4.6 GeoCLIP with Prior Geographical Knowledge
In practical applications of geo-localization, users often possess some level of prior geographical context. For instance, while the exact GPS coordinates of an image may be unknown, the general originating area, such as the continent, country, state, or city, may be known. This additional information can effectively narrow the search space, thereby improving the model's accuracy due to the reduced number of potential locations for consideration. To assess GeoCLIP's adaptability in these situations, we perform an experiment where the gallery of potential GPS coordinates is limited within a predefined radius of known geographical knowledge of each image. We choose the radius as per the assumed level of prior knowledge (higher search radius when coarse information is known and smaller radius with the finer knowledge): for a continent, coordinates within  $2500\mathrm{km}$  are taken; for a country, within  $750\mathrm{km}$ ; for a state, within  $200\mathrm{km}$ ; and for a city, within  $25\mathrm{km}$ . Results across these different levels of granularity are shown in Table 5.
Table 5: Performance of GeoCLIP when the granularity of search space is varied. The performance improves as we search from coarse to finer granularity leveraging prior knowledge.
Granularity	Street
1 km	City
25 km	Region
200 km	Country
750 km	Continent
2500 km
Continent	15.02	37.70	55.39	78.11	100.0
Country	15.85	41.18	63.33	100.0	100.0
State	18.82	52.12	100.0	100.0	100.0
City	26.15	100.0	100.0	100.0	100.0
The performance of image-to-GPS retrieval improves as we search from coarse to finer granularity, which is quite intuitive. For instance, if we have prior knowledge of the city, it's much easier to geo-localize the true location than to have prior knowledge about the continent. The model's capacity to adapt to restricted search spaces affirms its applicability in real-world scenarios where partial geographic context is commonly available.
We addressed the challenge of worldwide geo-localization by formulating it as an image-to-GPS retrieval problem. We evaluated the efficacy of our method (GeoCLIP) not only in full training data scenarios but also in limited data settings. The performance of GeoCLIP was competitive even when the training data was reduced significantly. GeoCLIP could even perform geo-localization using text queries by leveraging the CLIP[15] backbone of our image encoder. Moreover, our location encoder (using equal earth projection[17], RFF[21], and MLPs) better encoded GPS coordinates compared to the direct use of MLPs. However, coarse  $\sigma$  or a fine-grained  $\sigma$  used in RFF performed better in localizing either large or small areas. Our hierarchical learning overcame this issue by capturing the location features at various scales, leading to a further performance boost. Also, our location encoder is not tied to geo-localization and helps to aid in classification problems. In the future, we plan to further investigate the utility of our location encoder in assisting other computer vision problems.
Limitations: As we use a CLIP-based backbone in our image encoder, precomputing the image features on the large training dataset (each image of dimension  $224 \times 224 \times 3$ ) is time-consuming. But once the features are precomputed, the training of our GeoCLIP model is relatively fast. We also observed that a single  $\sigma$  value used in RFF is insufficient to perform well on both small and large areas. However, our hierarchical learning is a promising direction to overcome this issue.
Broader Impacts: Our image-based geo-localization can act as a substitute for GPS-based localization in scenarios where GPS signals are unreliable, blocked, or inaccurate. Combining image-based geo-tagging with GPS-based positioning systems can help to improve stability and safety in applications like autonomous driving and navigation. Also, it can benefit digital forensics, but avoiding its unethical use that could mislead location detection systems or violate privacy is important. More discussion on ethical issues and possible mitigation approaches are provided in the supplementary.
This work was supported by the US Army contract W911NF-2120192. We would like to extend our gratitude to all the reviewers for their valuable suggestions. We would also like to thank Brandon Clark for his helpful discussions on this project.