SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Vicente Vivanco Cepeda, Gaurav Kumar Nayak, Mubarak Shah
Center for Research in Computer Vision, University of Central Florida, USA {vicente.vivancocepheda, gauravkumar.nayak}@ucf.edu; shah@crvc.ucf.edu
Worldwide Geo-localization aims to pinpoint the precise location of images taken anywhere on Earth. This task has considerable challenges due to the immense variation in geographic landscapes. The image-to-image retrieval-based approaches fail to solve this problem on a global scale as it is not feasible to construct a large gallery of images covering the entire world. Instead, existing approaches divide the globe into discrete geographic cells, transforming the problem into a classification task. However, their performance is limited by the predefined classes and often results in inaccurate localizations when an image's location significantly deviates from its class center. To overcome these limitations, we propose GeoCLIP, a novel CLIP-inspired Image-to-GPS retrieval approach that enforces alignment between the image and its corresponding GPS locations. GeoCLIP's location encoder models the Earth as a continuous function by employing positional encoding through random Fourier features and constructing a hierarchical representation that captures information at varying resolutions to yield a semantically rich high-dimensional feature suitable to use even beyond geo-localization. To the best of our knowledge, this is the first work employing GPS encoding for geo-localization. We demonstrate the efficacy of our method via extensive experiments and ablations on benchmark datasets. We achieve competitive performance with just  $20\%$  of training data, highlighting its effectiveness even in limited-data settings. Furthermore, we qualitatively demonstrate geo-localization using a text query by leveraging the CLIP backbone of our image encoder. The project webpage is available at: https://vicentevivan.github.io/GeoCLIP
Image geo-localization refers to determining the geographical location where a photograph is taken from. This problem is of great significance in a wide variety of applications, such as navigation, tourism, and security. However, this task poses several challenges, especially for locations that lack distinctive landmarks or are outside tourist hotspots. Typically, image-based geo-localization mainly focuses on identifying the GPS locations of images within a specified region of interest (e.g., within a city). However, worldwide geo-localization [12, 14, 5] is not limited to any specific area but to the entire globe, thus making it significantly more challenging, and hasn't been explored extensively.
Geo-localization at the global level requires finding the locations (i.e., latitude and longitude) of images from different parts of the world. Many retrieval-based methods [28, 11, 34, 29, 32, 24, 20, 33] which perform image-to-image retrieval limit their search space to specific cities. Extending such approaches for worldwide geo-localization demands maintaining a gallery of all possible images of the entire globe to perform a match with the query image. However, creating such a large image gallery is infeasible; thus, these methods are inappropriate for retrieval at the global scale. Existing works instead use classification-based approaches [14, 26, 12, 5, 27, 8, 18] using scenes, hierarchy, queries, and segmentation. These methods divide the Earth into geographic classes (covering a particular area) and apply a linear classifier for prediction. If an image belonging to a class is far from
Figure 1: The objective of image-based geo-localization is to find the GPS coordinate of a given image. (A) We formulate this problem as an image-to-GPS retrieval approach where the image is a query, and the gallery is a collection of GPS coordinates. (B) We propose a novel method, GeoCLIP, having two major components: a location encoder  $\mathcal{L}(\cdot)$  to obtain high-dimensional features from the 2D GPS coordinates and a CLIP-based image encoder  $\mathcal{V}(\cdot)$  to obtain the image features. We perform the matching of features of a query image against a gallery of GPS embeddings, and the most similar GPS embedding is the predicted GPS coordinates.
its center, it would result in a localization error even though the classification prediction is correct. Also, the predefined classes limit the prediction to  $\approx 21k$  possible locations, which is not dense enough to predict any location in the world. Moreover, these methods often need large amounts of training data which takes days to train even with high computational resources. Thus, there is a need for a new effective and efficient approach that can better model the relationships between images and their corresponding geographic locations. We summarize the problem setup and compare the major differences between existing works and our proposed method in Figure 1.
To address the above limitations of existing approaches, we propose a CLIP[15]-inspired image-to-GPS retrieval approach (named 'GeoCLIP') where we retrieve the GPS coordinates of an unseen query image by matching it with the gallery of GPS coordinates. To facilitate matching, we design location and image encoders to project GPS and image in a shared embedding space. Our location encoder transforms a given 2D GPS coordinate into a high-dimensional feature space. The traditional latitude and longitude systems have inherent distortions in representing certain areas (e.g., countries near to poles and equator have inconsistent representations). Hence, we first convert the GPS locations using equal earth projection [17] to minimize such distortion. Another challenge is to obtain a high-dimensional representation of a 2D GPS coordinate. The direct use of standard MLP leads to spectral bias, as they approximate low-frequency functions and fail to capture high-frequency details with low-dimensional inputs. Previous studies [21] have demonstrated that such spectral bias can significantly degrade the performance in various applications, particularly when capturing these details is crucial for the task. To alleviate this, we employ a positional encoding technique using Random Fourier Features (RFF) [21] before processing it through a feedforward network.
To enrich the location features, we further enforce hierarchical representation in our location encoder by capturing features at different granularities (ranging from coarse to fine-grained resolutions) and combining them to obtain a rich high-dimensional feature. Specifically, we vary the frequencies in RFF by changing sigma values using our exponential assignment strategy, enabling us to process the 2D GPS coordinates at different resolutions. We employ contrastive loss between features of the image and GPS. It is well established that the incorporation of additional negative samples enhances the performance of contrastive learning. Our gallery is a collection of GPS coordinates instead of images. Hence it is easy for us to fetch negatives by uniformly sampling the GPS coordinates from the globe. To this end, we implement a queue of GPS values to serve as artificial negatives. Our location encoder learns generic features as we find them suitable for use even beyond geo-localization. Moreover, using the CLIP [15] backbone in our image encoder helps extract meaningful features and allows us to compare GPS coordinates with text descriptions, which is not feasible in existing works. Besides this, our image-to-GPS retrieval method facilitates a much finer evaluation than previous techniques, resulting in better localization performance.
Below we summarize our major contributions as follows:
- To the best of our knowledge, we are the first to solve worldwide geo-localization task via image-to-GPS retrieval approach (GeoCLIP) by explicitly aligning the features of images to corresponding GPS locations.
- Our location encoder incorporates positional encoding with random Fourier features to efficiently encode GPS coordinates and mitigate spectral bias in MLPs. In addition, we use an exponential sigma assignment strategy to facilitate learning hierarchical features at different resolutions (Sec. 3.1.1).
- We validate the efficacy of our location encoder in an additional application, highlighting its versatility beyond geo-localization tasks (Sec. 4.5).
- Unlike existing works that only use an image as query, we also qualitatively demonstrate global geo-localization with text query using GeoCLIP (Sec. 4.4).
- We also demonstrate the effectiveness of our approach in limited data settings (Sec. 4.2).
This section briefly discusses the relevant works related to our approach.
Global Image Prediction: Many previous works [27, 18, 26, 12, 14], have tackled the problem of global image geo-localization. While the most common localization papers use an image-to-image retrieval technique [16, 19, 20, 25, 34, 32], this is not yet feasible on the global scale because of the data collection that would be necessary for a global reference dataset to compare against. Instead, Weyand et al. [27] proposed to split the earth into individual classes to predict the location of an image. Through constructing these classes, the prediction can be relatively coarse or fine-grained based on the geographic size of the classes. Vo et al. [26] utilized multiple levels of geographic classes to learn different granularities, and Seo et al. [18] was the first to combine these levels together for an enhanced prediction. Müller et al. [12] additionally uses the scene labels of images to learn by having separate encoders designated to learn each scene's features. Recently Clark et al. [5] made use of even more hierarchies and scenes than the previous works, they also use a specialized transformer decoder to learn different features from every input image that represents each hierarchy and scene. The use of semantic segmentation was introduced by [14] in an attempt to provide additional context and understanding for the model to better geo-localize. They use two transformer encoders, one for the RGB image and the other for the segmented representation of that image, and share information between the encoders to get a better location prediction. Unlike existing global image geo-localization works, we instead take an image-to-GPS retrieval approach by directly matching the query image features to the features of GPS data.
Learning from GPS Data: The GPS coordinates of an image are used in [22, 10, 30, 31, 3] to provide additional context when performing classification. They construct GPS embeddings by leveraging image tags, textual descriptors, and information from geospatial databases, which are paired with image embeddings to improve the classification accuracy. We differ from this work by utilizing Random Fourier Features (RFF) [21] with MLPs. RFF is often used in NeRFs to represent the location or viewing angle for reconstructing the image or scene at that location or from that view. However, our location encoder uses RFF to represent a specific location on the earth and aligns them with the corresponding image features. We later also show that GPS embeddings of our location encoder is generic and can even be utilized for classification tasks as well, leading to further performance gains and showing its utility beyond geo-localization tasks.
Contrastive Learning: It is used in a wide variety of tasks in multiple deep learning-based disciplines. The basic idea is to match the features from a positive pair while pushing away from negative example features. Our method uses two different contrastive learning strategies, SimCLR[2] and CLIP[15]. SimCLR uses a base image and creates two separately augmented versions of it. It then attempts to match these two versions of the image together and push away the other images that are not from the same base. They show that this creates a robust feature representation of images. On the other hand, CLIP utilizes a large image-text database to learn a shared feature space between its image and text encoders. To establish a meaningful and distinct representation for both the text and image, they enforce similarity between the image features and the corresponding text caption. CLIP has been widely used in many tasks since its inception. We utilize the pretrained model from CLIP as a backbone to our image encoder, while our augmentation strategy is similar to SimCLR. However, instead of matching the image to another image, as in SimCLR, or to text, as in CLIP, we match it to a gallery of GPS features. The additional benefit of using CLIP backbone is the ability to compare GPS coordinates through our location encoder and text from CLIP's text encoder, which provides a very intriguing qualitative analysis that we show later (Sec. 4.4). Now, we explain our proposed approach in detail in the next section.
Figure 2: (A) Given a batch of GPS coordinates  $(G^{B})$ , we generate additional negatives  $Q$  using our dynamic queue strategy. Each image in batch  $(I^{B})$  is processed using our image encoder  $\mathcal{V}(\cdot)$  with CLIP[15] backbone. Similarly, we process the GPS coordinates using our location encoder  $\mathcal{L}(\cdot)$ . We train our model to align image features with corresponding GPS embeddings using contrastive loss over feature similarity matrix. (B)  $\mathcal{L}(\cdot)$  transforms 2D coordinates  $G_{i}$  into  $G_{i}^{\prime}$  using equal earth projection (EEP)[17]. Then we obtain hierarchical representations of  $G_{i}^{\prime}$  using RFF [21] and MLPs, aggregated to obtain a rich high-dimensional representation  $L_{i}$ .
Worldwide Image Geo-localization Setup: Given the access to training dataset  $D_{train} = \{(I_n, G_n)\}_{n=1}^N$  containing pairs of image  $I_n$ , and GPS coordinate  $G_n$ . Our goal is to train a worldwide model  $W$  using samples from  $D_{train}$  and then use the trained model  $W$  to predict the location of unseen  $K$  query images of test dataset  $D_{test}$ . Specifically, our objective is to accurately predict the latitude and longitude of query images,  $G_k^Q = W(I_k^Q)$ ,  $\forall k \in [1..K]$  where  $I_k^Q \in D_{test}$ . The images in the query set  $I^Q$  can belong to any geographical location of Earth, making it a challenging setup.
We formulate this problem as a retrieval problem. Traditionally, retrieval involves matching a query image with images from the gallery, and the GPS coordinates of the closest gallery image would be the desired output. This approach is often used when the search space is limited to specific cities. However, it has limitations when applied to retrieval at the global scale, which requires the construction of a gallery containing images covering the entire world. Instead, it is much easier to build a gallery of GPS coordinates. Hence, geo-localization can be framed as an image-to-GPS matching problem where the closest GPS match would be the location of the query image.
In order to perform matching, the features of the image and GPS location need to be extracted. Hence, we have two major components in our architecture, i.e., Location Encoder  $\mathcal{L}(\cdot)$  and Vision Transformer  $\mathcal{V}(\cdot)$  as Image Encoder. Now, we explain the model architecture containing details of both the encoders, followed by the training and evaluation strategies.
3.1 Model architecture
The architecture of our method GeoCLIP is shown in Figure 2.
Image Encoder: The CLIP [15] model has demonstrated strong generalization performance across a wide range of computer vision tasks. Motivated by it, our image encoder contains a Vision Transformer, which is a pre-trained CLIP ViT[6]-L/14 model. We use it as a backbone and keep it frozen. To adapt this backbone for our geo-localization task, we add two linear trainable layers with dimensions of  $h_1$  and  $h_2$ , respectively. These additional layers are employed to fine-tune the learned representation for our task. Hence, our overall image encoder  $\mathcal{V}(\cdot)$  extracts the features  $V_i = \mathcal{V}(I_i), \forall i \in [1 \dots N]$  and  $I_i \in D_{train}$ .
Next, we explicitly encode the GPS coordinates, which has not been tried for the geo-localization task. We now explain our location encoder in detail, which encodes the GPS coordinates of a location.
3.1.1 Location Encoder
Encoding 2D GPS coordinates (i.e.,  $G_{i} \in \mathbb{R}^{2}, \forall i \in [1 \dots N]$ ) to a high-dimensional representation  $(\mathbb{R}^D)$  where  $D \gg 2$ , is a challenging task. A simple use of standard MLPs for such encoding suffers from spectral bias [21] (unable to capture high-frequency details). Hence, to handle this challenge and effectively perform this encoding, we employ different components in our encoder, i.e., representing
GPS coordinates using equal earth projection[17], using positional encoding through random Fourier features, and enforcing hierarchical representation at various scales. We explain each of them below:
Equal Earth Projection (EEP): To accurately represent 2D GPS coordinates and minimize distortions inherent in standard coordinate systems (i.e., countries closer to the poles are overrepresented in traditional latitude and longitude systems), we employ the Equal Earth Projection (EEP)[17]. By utilizing EEP, we counterbalance the distortions and ensure a more accurate representation of the Earth's surface. Given  $G_{i} \in \mathbb{R}^{2}$  (in radians), we transform it into  $G_{i}^{\prime}$  by applying the EEP as below:
$$
G _ {i} ^ {\prime l a t} = \frac {2 \sqrt {3} G _ {i} ^ {l o n} c o s \theta}{3 (9 P _ {4} \theta^ {8} + 7 P _ {3} \theta^ {6} + 3 P _ {2} \theta^ {2} + P _ {1})}; G _ {i} ^ {\prime l o n} = P _ {4} \theta^ {9} + P _ {3} \theta^ {7} + P _ {2} \theta^ {3} + P _ {1} \theta \quad (1)
$$
where  $\sin \theta = \frac{\sqrt{3}}{2}\sin G_i^{lat}$ ,  $P_1 = 1.340264$ ,  $P_2 = -0.081106$ ,  $P_3 = 0.000893$ ,  $P_4 = 0.003796$ . After applying the EEP, we scale the resulting longitude in the range  $-1$  to  $1$ , and the latitude values are scaled proportionally.
Random Fourier Features (RFF): To capture rich high-frequency details from low-dimensional GPS input, we apply a sinusoidal-based positional encoding technique to  $G_{i}^{\prime}$ . Specifically, we leverage random Fourier features (RFF)[21] for this positional encoding. We construct an encoding layer that determines the range of frequencies to be encoded. We limit the frequencies using a fixed matrix  $\mathbf{R}$ , whose entries are sampled from a Gaussian distribution with the standard deviation  $(\sigma)$ . The matrix  $\mathbf{R}$  is set at the beginning of training and remains unchanged throughout the training process. The RFF operation  $\gamma(\cdot)$  encodes GPS coordinate  $G_{i}^{\prime}$  as  $\gamma(G_{i}^{\prime}) = [\cos(2\pi \mathbf{R}G_{i}^{\prime}), \sin(2\pi \mathbf{R}G_{i}^{\prime})]^{\mathrm{T}}$ , where the entries of a  $m^{th}$  row and  $n^{th}$  column of matrix  $\mathbf{R}$  are  $\mathbf{r}_{m,n} \sim \mathcal{N}(0,\sigma)$ .
Hierarchical Representation: To construct a hierarchical representation capable of capturing information at varying scales, we vary the frequency in RFF to obtain multiple high-dimensional representations spanning from coarse to fine-grained. As the frequency range in RFF depends on the  $\sigma$  value, we propose an exponential assignment strategy for choosing the  $\sigma$  values in our location encoder. Specifically, for a given a range of  $\sigma$  values (i.e.,  $[\sigma_{min},\sigma_{max}]$ ), we obtain the  $\sigma$  values for  $M$  level hierarchy using the following:
$$
\sigma_ {i} = 2 ^ {\log_ {2} \left(\sigma_ {m i n}\right) + (i - 1) \left(\log_ {2} \left(\sigma_ {m a x}\right) - \log_ {2} \left(\sigma_ {m i n}\right)\right) / (M - 1)}, \quad \forall i \in \{1 \dots M \}. \tag {2}
$$
This exponential assignment of  $\sigma$  values enables our location encoders  $\mathcal{L}(\cdot)$  to effectively process the input 2D coordinates at different resolutions. Consequently, this hierarchical setting allows  $\mathcal{L}(\cdot)$  to specialize in capturing features of a particular location at various scales. It enforces our model to efficiently learn a rich understanding of spatial information, making it suitable for a wide range of geospatial deep learning tasks.
Next, we pass the encoded hierarchical features through a feedforward MLP. Each MLP  $f_{i}$  independently processes the outputs of RFF and yields a vector  $\mathbf{v}_i \in \mathbb{R}^d$ . We then perform element-wise addition of these resulting vectors to obtain a joint representation  $\mathbf{v} = \sum_{i=1}^{M} \mathbf{v}_i$ .
Below we summarize the overall operations performed by our location encoder  $\mathcal{L}(\cdot)$  for obtaining rich high-dimensional encoded features  $(L_{i})$  of a GPS 2D coordinate  $G_{i}$ :
$$
L _ {i} = \mathcal {L} \left(G _ {i}\right) = \sum_ {i = 1} ^ {M} f _ {i} \left(\gamma \left(E E P \left(G _ {i}\right), \sigma_ {i}\right)\right) \tag {3}
$$
3.2 Model Training
We train our model using a contrastive learning scheme (similar to [15]) where we maximize the similarity of features of an image  $I_{i}$  (i.e.,  $V_{i}$ ) with its corresponding location features of GPS coordinate  $G_{i}$  (i.e.,  $L_{i}$ ) and minimize its similarity with other GPS coordinates in a batch. To further improve our learned representation, we also adopt a training scheme similar to SimCLR [2].
We apply a set of random augmentations  $(\mathcal{A})$  to each image in a batch and create  $P$  views of an image to improve the quality of learned representations and promote better generalization. We employ the same set of augmentations used in the SimCLR. For each augmented image  $I_{ij}$  ( $i^{th}$  sample of  $D_{train}$  and  $j^{th}$  view, we also create variation in its corresponding GPS coordinates  $G_{i}$  by injecting random noise  $\eta_{j}$  to it (denoting it by  $G_{i}j$ ) which helps to enforce spatial smoothness in the location encoder. The noise  $\eta_{j}$  is sampled from a Gaussian distribution with a standard deviation of  $\sigma_{\eta}$  (i.e.,
$\eta \sim \mathcal{N}(0,\sigma_{\eta}^{2})$ . This added noise encourages the location encoder to learn more robust representations that are less sensitive to small variations in the GPS coordinates.
At any point during training, our location encoder  $\mathcal{L}(\cdot)$  can obtain the embedding of any GPS location  $g$  (i.e.,  $L_{g}$ ) without requiring a corresponding image. We exploit this capability to generate additional negative samples during training to improve the learned representation with contrastive learning. We append a queue  $Q$  of GPS locations with a fixed length  $S$  to every batch of training data  $D_{train}$ . During training, we generate embeddings for these GPS coordinates  $(\tilde{L})$ . We also inject noise  $(\eta' \sim \mathcal{N}(0, \sigma_{\eta'}^2))$  to  $\tilde{L}$ , such that  $\sigma_{\eta'} \geq \sigma_{\eta}$ . This additional queue of GPS embeddings notably improves the performance of our model, particularly at smaller scales (refer Table 3(a)).
Hence, for an  $i^{th}$  sample of a batch  $(I_i^B,G_i^B\in D_{train})$  having  $P$  augmented views with temperature  $\tau$ , our overall training objective is to minimize the following loss:
$$
\mathcal {L} _ {i} = - \sum_ {j = 1} ^ {P} \log \frac {\exp \left(V _ {i j} \cdot L _ {i j} / \tau\right)}{\sum_ {i = 0} ^ {| B |} \exp \left(V _ {i j} \cdot L _ {i j} / \tau\right) + \sum_ {i = 0} ^ {S} \exp \left(V _ {i j} \cdot \tilde {L} _ {i} / \tau\right)}. \tag {4}
$$
After the model parameters are updated on a batch of  $D_{train}$  with batch size  $|B|$ , we update the queue  $Q$  by replacing its oldest coordinates with GPS coordinates from the current batch  $G^{B}$  where  $|B| \ll S = |Q|$ , ensuring an up-to-date reflection of the most recent GPS coordinates encountered (naming it as 'dynamic queue' strategy).
Evaluation: By modeling the Earth as a continuous function, our location encoder can generate an embedding for any GPS coordinate on the planet. Hence, our method allows us to directly compare the embedding of a query image with the embeddings of any chosen GPS location, effectively eliminating the constraint of working within a predetermined set of classes.