Year: 2021

Abstract:
Location encoding is valuable for a multitude of tasks where both the absolute positions and local contexts (image, text and other types of metadata) of spatial objects are needed for accurate predictions. However, most existing approaches do not leverage unlabeled data, which is crucial for use cases with limited labels. Furthermore, the availability of large-scale real-world GPS coordinate data demand representation and prediction at global scales. Existing location encoding models assume that the input coordinates are in Euclidean space, which can lead to modeling errors due to distortions introduced when mapping coordinates from other manifolds (e.g., spherical surfaces) to Euclidean space. We introduce Sphere2Vec, a location encoder, which can directly encode spherical coordinates while preserving spherical distances. Sphere2Vec is trained with a self-supervised learning framework which pre-trains deep location representations from unlabeled geotagged images with contrastive losses, and then fine-tunes to perform supervised geographic object classification tasks. Sphere2Vec achieves the performances of state-of-the-art results on various image classification tasks ranging from species, Point of Interest (POI) facade, to remote sensing. The self-supervised pretraining significantly improves the performance of Sphere2Vec especially when the labeled data is limited.

heading: 1 INTRODUCTION
text: Spatial information has become an important component to many machine learning tasks that address the environmental, economic, and societal needs for sustainable development. Just to name a few, these tasks include geo-aware species fine-grained recognition (Chu et al., 2019; Mac Aodha et al., 2019), Point of Interest (POI) facade image classification (Chu et al., 2019; Yan et al., 2018), remote sensing image classification (Christie et al., 2018; Ayush et al., 2020), poverty prediction (Jean et al., 2016; 2019), point cloud classification and semantic segmentation (Qi et al., 2017a;b), traffic forecasting (Li et al., 2018; Cai et al., 2020), and geographic question answering (Mai et al., 2019; 2020a). Developing a general model for vector space representations of any point in space, termed a location encoder, would pave the way for many future applications that improves our well-being. Such a location encoder can operate on a 2D space or on other manifolds; of particular interest is the spherical surface of the Earth.
Here we use geo-aware image classification shown in Figure 1 and 2 as an example. During training we assume a dataset D “ tpIi,xi, yiq|i “ 1, ..., Nu, where Ii is an image, yi P 1, ...., C is the corresponding class label, xi represents the location (longitude and latitude) and optionally the time the image was taken1. The image information is often processed by an state-of-the-art image classification model (e.g., InceptionV3) to predict the class labels as shown in Figure 1. Although the image itself is the most critical for image classification, sometimes it can be insufficient – the image of two distinct classes may look very similar both for the central object and its surrounding environment. For example, Figure 2a and 2f shows example images of two different fox species: Arctic fox (Vulpes lagopus) and bat-eard fox (Otocyon megalotis), for which the image model can be confused due to the high visual similarity of the two species. Fortunately, the geo-location information of an image may provide a clue to their real classes. Figure 2b, 2g show that these two species have distinct spatial distribution patterns, and the image locations provide complementary information to the images. This
1In this study we focus on the location information and leave the time aspect of modeling to future.
insight motivates the development of hybrid classification models, which take both the images and their locations as input (Mac Aodha et al., 2019; Mai et al., 2020b).
In spite of their effectiveness, hybrid models have an issue when representing points in non-Euclidean spaces. The demand on representation and prediction learning at a global scale grows dramatically due to emerging global scale issues, such as the transition path of the latest pandemic (Chinazzi et al., 2020), long lasting issue for malaria (Caminade et al., 2014), under threaten global biodiversity(Di Marco et al., 2019; Ceballos et al., 2020), and numerous ecosystem and social system responses for climate change (Hansen & Cramer, 2015). However, when applying the state-of-the-art 2D space location encoders (Gao et al., 2019; Chu et al., 2019; Mac Aodha et al., 2019; Mai et al., 2020b; Zhong et al., 2020) to large-scale real-world GPS coordinate datasets such as species images taken all over the world, a map projection distortion problem (Williamson & Browning, 1973) emerges. These 2D location encoders are designed for preserving distance in 2D (or 3D) Euclidean space, while GPS coordinates are in fact on a spherical manifold, e.g., point x “ pλ, φq on a sphere with longitude λ P r´π, πq and latitude φ P r´π{2,´π{2s (See Figure 1). Directly applying these 2D location encoders on spherical coordinates will yield a large distortion in the polar regions. Note that map projection distortion is unavoidable when projecting spherical coordinates into 2D space. This emphasizes the importance of calculating on a round planet (Chrisman, 2017). See Appendix 9.1 for more discussion.
From a modeling perspective another key challenge is how to design the self-supervised training objective that effectively transfers the attribute information (e.g., image features) at different locations into a location encoder. This is very important in practice since collecting and labeling geo-tagged images is particularly costly as annotations often require domain expertise. However, most existing location encoding approaches (Chu et al., 2019; Mac Aodha et al., 2019; Mai et al., 2020b) are developed and trained in a supervised learning framework while massive unlabeled geographic data cannot be used. Ayush et al. (2020) proposed a geographic self-supervised learning framework, but they mainly focus on training image encoders while the usage of geo-location information is limited to data augmentation in the pre-training stage. In contrast, in this work, we focus on training location encoders in a self-supervised manner so that geo-location information is explicitly use as part of the model input.
In this work, we propose Sphere2Vec as shown in Figure 1, which directly encodes point coordinates on a spherical surface while avoiding the map projection distortion. The multi-scale encoding method utilizes Double Fourier Sphere basis (OpS2q terms) or a subset (OpSq terms) of it while still being able to correctly measure the spherical distance. Furthermore, we develop a self-supervised learning framework which pre-trains deep location representations from unlabeled geo-tagged images by predicting image features or image identities base on their geo locations. We compare several self-supervised learning objectives including Mean Square Error loss (MSE), contrastive binary classification loss (BI), and contrastive multi-classification loss (MC). The pre-trained Sphere2Vec model can be fine-tuned (with an additional output layer) to perform supervised geographic objects classification tasks. We demonstrate the effectiveness of Sphere2Vec on geo-aware image classification tasks including fine-grain species recognition (Chu et al., 2019; Mac Aodha et al., 2019; Mai et al., 2020b), POI fascade image classification (Tang et al., 2015; Mac Aodha et al., 2019), and remote sensing image recognition (Christie et al., 2018; Ayush et al., 2020).

heading: In summary, the contributions of our work are:
text: 1. We develop a unified view of distant preserving location encoding methods on spheres based on Double Fourier Sphere (DFS) (Merilees, 1973; Orszag, 1974).
2. We propose an effective pre-training strategy for location encoders from unlabeled geotagged images by predicting image features and image identities based on their geo locations.
3. Sphere2Vec improves the performances of state-of-the-art results on four species recognition datasets (BirdSnap, NABirds, iNat2017, iNat2018), one POI fascade image classification dataset (YFCC), and one remote sensing image classification dataset (fMoW). It also demonstrate significant advantage in few shot learning settings.

heading: 2 PROBLEM FORMULATION
text: Following previous work (Mai et al., 2020b), Euclidean location encoding can be formulated as follows. Given a set of points (e.g., POI) P “ tpiu in L-D space (L “ 2, 3), we define a function eP,θpxq : RL Ñ Rd (L ! d), which is parameterized by θ and maps any coordinate x in space to a vector representation of d dimension. Each point (e.g., a restaurant) pi “ pxi,viq is associated with a location xi and attributes vi (i.e., POI features such as types, names, capacity, images, etc.). The function eP,θpxq encodes the information about Point x which is useful for certain tasks such as predicting a label y. Attributes and coordinates of points can be seen as analogies to words and word positions in commonly used word embedding models.
Similarly, we can formulate spherical location encoding as follows. Given a set of points P “ tpiu on the surface of a sphere S2, e.g., species occurrences all over the world, where pi “ pxi,viq and xi “ pλi, φiq P S2 indicates a point with longitude λi P r´π, πq and latitude φi P r´π{2, π{2s. Define a function eP,θpxq : S2 Ñ Rd, which is parameterized by θ and maps any coordinate x in a spherical surface S2 to a vector representation of d dimension. See Appendix 9.1 for more discussions about the map projection distortion problem.

heading: 3 METHOD
text: The geo-aware image classification task (Chu et al., 2019; Mac Aodha et al., 2019) can be formulated as follows. Given an image I taken from location/point x, we estimate which category y it belongs to by P py|I,xq. If we assume I and x are conditionally independent given y, then based on Bayes’
theorem, we have P py|I,xq 9 P py|xqP py|Iq (See Mac Aodha et al. (2019)). P py|Iq can be given by any state-of-the-art image classifier such as Inception V3 (Szegedy et al., 2016) for the species recognition task or MoCo-V2+TP (Ayush et al., 2020) for the RS image classification task. Figure 1 illustrates the whole training and inference process by using species recognition as the example task. P py|Iq and P py|xq are given by the image encoder (the green box) and location encoder (the blue box) respectively and are multiplied together for final prediction. In this work, we focus on estimating the geographic prior distribution of class y over the spherical surface P py|xq 9 σpepxqT:,yq where σp9q is a sigmoid activation function. T P Rdˆc is a class embedding matrix where the yth column T:,y P Rd indicates the class embedding for class y. In the following we introduce the two main contributions of our work – location encoding on spherical surfaces (the blue box in Figure 1), and a self-supervised training framework for location representation model (the orange box in Figure 1). In Section 3.1, we will discuss the design of spherical distance-kept location encoder epxq, Sphere2Vec. We developed a unified view of distancereserving encoding on spheres based on Double Fourier Sphere (DFS) (Merilees, 1973; Orszag, 1974). In Section 3.2 we discuss training strategies for Sphere2Vec including self-supervised pre-training given unlabeled images, and supervised fine tuning. We use eP,θpxq to indicate the spherical location encoding neural network with parameters θ, and will refer it by epxq for simplicity.
3.1 Sphere2Vec ENCODER
We want to find a function Gpsphereqpxq which does a one-to-one mapping from each point xi “ pλi, φiq P S2 to a multi-scale representation with S as the total number of scales such that it satisfies the following requirement:
xGpsphereqpx1q, Gpsphereqpx2qy “ fp∆Dq,@x1,x2 P S2, (1)
where ∆D P r0, πRs is the spherical surface distance between x1,x2, R is the radius of this sphere, and fpxq is a strictly monotonically decreasing function for x P r0, πRs. In other words, we expect to find a function Gpsphereqpxq such that the resulting multi-scale representation of x preserves the spherical surface distance.
Similar to the case for Euclidean space (Mai et al., 2020b), any point x “ pλ, φq P S2 with longitude λ P r´π, πq and latitude φ P r´π
2 , π 2 s, can be explicitly encoded in a multi-scale fashion in the
form of epxqpxq “ NNffnpGp˚qpxqq where NNffnpq is a learnable multi-layer perceptron with h hidden layers and k neurons per layer. Gp˚qpxq is a concatenation of multi-scale spherical spatial features of S levels. p˚q indicates different types of spherical location encoders. In the following, we call epxqpxq location encoder and its component Gp˚qpxq position encoder. Let rmin, rmax be the minimum and maximum scaling factor, and g “ rmaxrmin . 2
sphereDFS Double Fourier Sphere (DFS) (Merilees, 1973; Orszag, 1974) is a simple yet successful pseudospectral method, which has been applied to efficient analysis of large scale phenomenons such as weather (Sun et al., 2014) and blackholes (Bartnik & Norton, 2000). Our first intuition is to use the base functions of DFS to help decompose x “ pλ, φq into a high dimensional vector:
GpsphereDFSqpxq “ S´1 ď
n“0 rsinφn, cosφns Y
S´1 ď m“0 rsinλm, cosλmsY
S´1 ď
n“0
S´1 ď m“0 rcosφn cosλm, cosφn sinλm, sinφn cosλm, sinφn sinλms.
(2)
where @s “ 0, 1, ..., S ´ 1, we define λs “ λfs , φs “ φ fs , fs “ rmin ¨ gs{pS´1q. Y means vector concatenation and
ŤS´1 s“0 indicates vector concatenation through different scales. It basically lets all
the scales of φ terms interact with all the scales of λ terms in the encoder. This would introduce a position encoder whose output has OpS2q dimensions and increase the memory burden in training and hurts generalization (Figure 3a).
2In practice we fix rmax “ 1 meaning no scaling of λ, φ.
An encoder might achieve better result by only using a subset of these terms. In comparison, the state of art encoder such as grid and wrap (Mac Aodha et al., 2019) define: 3
Gpgridqpxq “ S´1 ď
s“0 rsinφs, cosφs, sinλs, cosλss. (3)
We can see that grid employs a subset of terms from sphereDFS (Figure 3b) and performs poorly at a global scale due to its inability to preserve spherical distances (See Appendix 9.3 for proof).
We explore different subsets of DFS terms while achieving two goals: 1) efficient representation with OpSq dimensions 2) preserving distance measures on spheres. See Appendix 9.2 for the detail definition of sphereC, sphereC`, sphereM , sphereM`, and Figure 3 for a unified view. We hypothesize that encoding these terms in the multi-scale representation would make the training of the encoder easier and the order of output dimension is still OpSq. In location encoding, the uniqueness of encoding (no two points on sphere having the same position encoding) is very important, Gp˚qpxq in the five proposed methods are one-to-one mapping. Theorem 1. @˚ P tsphereC, sphereC`, sphereM, sphereM`, sphereDFSu, Gp˚qpxq is an injective function. See the proof in Appendix 9.5.
3.2 TRAINING Sphere2Vec
Self-Supervized Pretraining (the yellow box in Figure 1). We consider two contrastive objectives. The first is a binary classification loss, or so called noise contrastive estimation (Gutmann & Hyvärinen, 2010), which avoids calculation of the partition function and has been successfully used in word embeddings (Mikolov et al., 2013) and language modeling (Mnih & Teh, 2012):
LBIpP,N q “ ´Epa,bq„P rlog σpspa,bqqs ´ Epa,b´q„N rlogp1´ σpspa,b´qqqs (4)
where BI stands for “binary”, P is a set of positive pairs (denoted as pa,bq), N is a set of negative pairs (denoted as pa,b´q), sp¨, ¨q is a similarity function that assigns higher values when the inputs are more similar (such as cosinepq), and σpvq “ ev{p1` evq is the sigmoid function. The second objective function is the multi-class classification loss with temperature, which has been successfully used in unsupervised learning for images (He et al., 2020) and text (Gao et al., 2021):
LMCpP,N , τq “ Epa,bq„P
«
exppspa,bq{τq exppspa,bq{τq ` ř
b´PN pb|aq exppspa,b´q{τq
ff
(5)
where MC stands for “multi-class”, N pb|aq obtains a negative pair with first entry being a; P and sp¨, ¨q are defined as earlier. The temperature scaling parameter τ determines how soft the softmax is (Hinton et al., 2015). In practice it help with the trade off between top ranked classes (precision) versus reset of the classes (recall).
In order to learn useful representations, we need to choose appropriate distributions for positive pairs P and negative pairs N . During the location encoder pretraining stage we use a geo-tagged but unlabeled image set Xunlabeled, each image is represented by a embedding generated from a pretrained image extract extractor F4 such as InceptionV3 or MoCo-V2+TP (Ayush et al., 2020).
Given an unlabeled image px, Iq P Xunlabeled, we use the following positive and negative instances: 3As a multi-scale encoder, grid degenerates to wrap when the number of scales S “ 1. 4F should not see the current image labels during pretraining stage.
• Geo-tagged positive Px “ tpepxq,WFpIqqu where W is a linear projection layer. • In-batch negatives N b “ tpepxq,WFpI´q|px´, I´q P Bpx, Iqztpx, Iquu where Bpx, Iq
represents the batch of examples for which px, Iq is part of during training. • Sampled negative locations N s “ tpepx´q,WFpIqqu of size N , where x´ is one of the N evenly sampled locations from the surface of the sphere for each example x. • Dropout positive Pd “ tpepxq, e1pxqqu, where we pass the same input x to the encoder epq
twice and obtain two embeddings as “positive pairs” by applying independently sampled dropout masks. This is a data augmentation strategy (so called SimCSE), which has been very successful in generating sentence embeddings (Gao et al., 2021). We use e1pxq to denote the embedding from the second mask. • Dropout negative N d “ tpepxq, e1px´qq|px´, I´q P Bpx, Iqztpx, Iqu, where e1p¨q has the
same meaning as that defined in Pd and Bpx, Iq indicates the same as N b.
We define two versions of contrastive losses, which both have three components: in-batch, negative location, and SimCSE. The self-supervized binary classification loss LBI is defined as
LBIpXq “ LbatchBI pXq ` β1L negloc BI pXq ` β2L simcse BI pXq
“ LBIpPx,N bq ` β1LBIpH,N sq ` β2LBIpPd,N dq (6)
where β1 and β2 control the contribution of the last two loss components.
The self-supervized multi-class classification loss LMC is defined as
LMCpXq “ LbatchMC pXq ` α1L negloc MC pXq ` α2L simcse MC pXq
“ LMCpPx,N b, τ0q ` α1LMCpPx,N s, τ1q ` α2LMCpPd,N d, τ2q (7)
where α1 and α2 are hyper-parameters. Supervised Fine-Turning (the blue box in Figure 1). We directly use image labels in the training objective. Following Mac Aodha et al. (2019), we used a presence-absence loss function which converts the multi-class labels into binary multi-labels. Given a set of training samples Xlabeled “ tpx, yqu, the loss function LsupervizedpXq is defined as:
LsupervizedpXq “ βLBIpP l,Hq ` LBIpH,N lq ` LBIpH,N lsq (8) Here, β is a hyperparameter to enlarge the weight of positive samples, and the following positive and negative samples are used:
• Labeled positives P l “ tpepxq,T:,yq|px, yq P Xu. • Labeled negatives N l “ tpepxq,T:,iq|px, yq P X, i P t1..cuztyuu. • Sampled negative locations N ls “ tpepx´q,T:,iq|px, yq P X, i P t1..cuu, where x´ is
one of the N evenly sampled locations from the surface of the sphere for each example x.

heading: 4 RELATED WORK
text: Multi-Scale Position/Location Encoding on Spheres Recent works in various applications (Zhong et al., 2020; Mai et al., 2020b; Mildenhall et al., 2020; Tancik et al., 2020) found that a simple sinusoidal mapping of input coordinates (“positional encoding”) allows multilayer perceptrons (MLP) to represent higher frequency content. More specifically fθ is decomposed into fθ “ NNθpPEpxqq, where PE is a deterministic function mapping x from Rd into higher dimensional space and NNθ is simply a MLP. This strategy is compelling since MLPs can be orders of magnitude more compact than grid-sampled representations in order to achieve similar order of details (Tancik et al., 2020). They show that passing input points through a simple Fourier feature mapping enables a MLP to learn high-frequency functions in low-dimensional problem domains. This development is consistent with Rahaman et al. (2019a) who show that deep networks are biased towards learning lower frequency functions, and mapping the inputs to a higher dimensional space using high frequency functions enables better fitting of data that contains high frequency details. In this work we extend the success of this line of work to none Euclidean spaces such as spherical surface. We leverage truncated Fourier transformation on spheres to achieve computation efficiency while avoiding the error caused by projection distortion.
Unsupervised Representation Learning Unsupervised text encoding models such as transformer (Vaswani et al., 2017; Devlin et al., 2019) has been effectively utilized in many Natural Language Processing (NLP) tasks. At its core, a trained model encodes words into vector space representations based on their positions and context in the text. Following the success in NLP, there has been significen recent progress in unsupervised image pretraining (He et al., 2020; Caron et al., 2020; Ayush et al., 2020). Interestingly almost all of them are based on certain form of contrastive learning (Hadsell et al., 2006), which helps to construct unsupervised classification objectives from continuous inputs such as images. He et al. (2020) proposes Momentum Contrast (MoCo) for unsupervised visual representation learning. To increase the number of negative examples in contrastive training, they uses a queue of multiple mini-batches. Similar strategy has been adopted in NLP (Gao et al., 2021). To improve the encoding inconsistency between mini batches, they make the target image encoder parameterizes a moving average of the query image encoder. In this work we are focusing on the pretraining of location encoder with a frozen image encoder. Our approach is very memory efficient (easily scaling up to 1024 batch size) and therefore avoid the need of multi-batch training. Ayush et al. (2020) combines a geo-location classification loss and a contrastive image loss similar to MoCo during pre-training. For geo-location classification, a deep image network is trained to predict a coarse geo-location of where in the world the image might come from. Since the geo-location is not part of the classifier’s input, it is hard for the model to directly leverage it in the prediction process. Zhai et al. (2018) learn location representation from unlabeled image-location pairs for image localization. They apply cross entropy loss to discretized location (or time), which cannot leverage the continuity of the approximated function. One main difference and also one of our main contributions is our contrastive self-supervised loss, which avoids the need of discretization.

heading: 5 EXPERIMENT

heading: 5.1 THE EFFECTIVENESS OF SPHERICAL LOCATION ENCODING
text: To test the effectiveness of Sphere2Vec we first conduct geo-aware image classification experiments in a full supervised setting (without the self-supervised pretraining step) on seven large-scale real-world datasets including 5 (animal) species datasets, one POI dataset (YFCC), and one remote sensing dataset (fMoW). Please refer to Mac Aodha et al. (2019) and Christie et al. (2018) for the dataset descriptions. Table 1 compares the result of five variants of Sphere2Vec models against nine baseline models (see their detailed descriptions in Appendix 9.6. We can see that the Sphere2Vec models outperform baselines on all seven datasets, and the variants with linear number of DFS terms works as well as or even better than sphereDFS. Figure 2c-2e and 2h-2j show the predicted species distributions from different models for two fox species. We can see that wrap and grid have overgeneralization issue in data sparse area (e.g., Africa) and the North Pole area. The distributions produced by sphereC` are more compact and generalizable. To help understand the impact of Sphere2Vec , we conduct detailed analysis on iNat2017 and iNat2018. Figure 4a shows the image locations in iNat2017 validation dataset. We split this dataset into different latitude bands and compare the ∆MRR between each model to grid. Figure 4b and 4c show the number of samples and ∆MRR in each band while Figure 4f shows that the contrast between these two variables for different models. We can see that Sphere2Vec models have larger ∆MRR at the North Pole region (Figure 4c). Moreover, Sphere2Vec has advantages on bands with less data samples, e.g. φ P r´30˝,´20˝q. We also compare sphereC` and grid at a quantized spatial resolution - in latitude-longitude cells(Figure 4d and 4e). sphereC` has more advantages in North Pole and data sparse cells. For more details please refer to Appendix 9.12, 9.13, and 9.14.

heading: 5.2 THE EFFECTIVENESS OF SELF-SUPERVISED PRETRAINING
text: Next, we investigate the impact of self-supervised contrastive pretraining (see Section 3.2). For species recognition and POI image classification task, we use the pretrained Inception-V3 model provided by PyTorch as image feature extractor F in the yellow box of Figure 1. As for fMoW dataset, we use the unsupervised pretrained MoCo-V2+TP (Ayush et al., 2020) as F. Both models are supervised fine-tuned on the respective dataset to act as gpFp¨qq in the green box of Figure 1. We consider three self-supervised training objectives - mean square error loss MSE between the location embedding epxq and the image feature FpIq, noise contrastive binary classification loss BI (see Equation 6), and contrastive multi-class classification loss MC (See Equation 7). In order to
simulate the situations with limited labeled data we first perform self-supervised pretraining for epxq with one of the three objectives using 100% of the unlabeled dataset Xunlabeled (e.g., iNatlist 2018). Then we perform stratified sampling to obtain Γ portion of the labeled data, denoted as XΓ, while keeping its label distribution the same as the full training set. Finally we fine-tune the pretrained epxq on XΓ with the supervised learning objective (Equation 8).
MC indicate training sphereM` with three different self-supervised
objectives before supervised training. (b) Comparing the performance of sphereM` and theory (the 2nd best model on iNat2018 dataset) in both supervised only (Sup.) and self-supervised plus supervised (Sup.&MC) settings. We can see that in both settings, sphereM` is able to outperform theory under different ratio Γ.
Figure 5a compares the model performance of sphereM` on iNat2018 with different training objectives L. Each curve indicates the model performance of one training process in different supervised label ratios Γ with identical epxq. From Figure 5a, we can see that compared with Sup. setting, all three self-supervised objectives can improve the model performances. The largest improvements happen when Γ is small (i.e., few shot learning setting), and we get a 3.9% performance improvement when adding MC self-supervised training stage for Γ “ 0.05. BI and MC show competitive performance while MC is slightly better. MSE is less effective than the two. Furthermore, in order to show the impact of self-supervised training on different epxq, we compare the performance of sphereM` and theory (the 2nd best model on iNat2018 ) under different L (See Figure 5b). We can see that MC can improve the performance of both epxq while sphereM` can outperform theory in either supervised or self-supervised + supervised settings.
We also perform a series of ablation studies on these self-supervised training objectives. Please refer to Appendix 9.10 for detail analysis. Moreover, we also compare MC and Sup. on the fMoW dataset and find that MC self-supervised loss is also effective in the label limited learning setting on the fMoW dataset (more details in Figure 8 and Appendix 9.11).

heading: 6 CONCLUSION
text: In this work, we propose a multi-scale spherical location encoder - Sphere2Vec which can encode any location on the spherical surface into a high dimension vector. Moreover, we propose an selfsupervised learning framework which pretrains location encoders based on unlabeled geo-tagged images. Experiment results on seven large-scale geo-aware image classification datasets shows that Sphere2Vec can outperform multiple strong baselines. Moreover, we show that by adding the self-supervised pretraining stage, the performance of Sphere2Vec can be further improved, especially when labeled data is limited. In the future, we would like to explore the effectiveness of Sphere2Vec on other machine learning tasks such as image geolocalization, geographic question answering, and so on. Moreover, we only pretrain the location encoder while keeping the image encoder freeze during the self-supervised training stage. In the future, we would like to explore ways to pretrain dual encoders at the same time or early fusion of image and location data.

heading: 7 ETHICS STATEMENT
text: All datasets utilized in this work are publicly available. We only do some data preprocessing such as image feature extraction. No new information is collected.
We find out that the geographic coverage of most datasets used in this work such as BirdSnap, NABird, iNat2017, iNat2018, YFCC, and fMoW are mostly concentrated on North American, Europe while they have poor geographic coverage on other regions and countries such as Russia, China, Africa, and so on (See Figure 4a). This kind of geographic bias will also affect the machine learning models trained on them as shown in Figure 4d. This paper shows the our Sphere2Vec is more robust than the state-of-the-art models for such bias. However, how to systematically handle such geographic bias should be an interesting research direction.

heading: 8 REPRODUCIBILITY STATEMENT
text: The proposed models as well as preprocessed datasets in this paper will be made public available through GitHub.

heading: 9 APPENDIX

heading: 9.1 THE MAP PROJECTION DISTORTION PROBLEM
text: In fact there is no map projection, which can preserve distances in all directions. The so-called equidistant projection can only preserve distance on one direction, e.g., the longitude direction for the equirectangular projection (See Figure 6d), while the conformal map projections (See Figure 6a) can preserve directions while resulting in a large distance distortion. For a comprehensive overview of map projections and their distortions, see Mulcahy & Clarke (2001). This is a well recognized problem in Cartography which shows the importance of calculating on a round planet Chrisman (2017). Due to the limitations of these 2D location encoders, there is an urgent need for a location encoding method which preserves the spherical distance (e.g., great circle distance5) between two points. The multi-scale encoding method utilizes Double Fourier Sphere basis (OpS2q terms) or a subset (OpSq terms) of it while still being able to correctly measure the spherical distance. This inspires us to explore the most effective subset of Fourier bases on spheres.

heading: 9.2 SPHEREC,SPHEREC+,SPHEREM,SPHEREM+
text: sphereC Inspired by the fact that any point px, y, zq in 3D Cartesian coordinate can be expressed by sin and cos basis of spherical coordinates (λ, φ plus radius) 6, we define the basic form of Sphere2Vec, namely sphereC encoder, for scale s as
GpsphereCqpxq “ S´1 ď
s“0 rsinφs, cosφs cosλs, cosφs sinλss. (9)
It can be shown that when S “ 1,GpsphereCqpxq directly satisfies our expectation in Equation 1 where fpxq “ cosp xR q. See more detailed analysis and comparison to the grid encoder in Appendix 9.3.
sphereM Considering the fact that many geographical features are more sensitive to either latitude (e.g., temperature, sunshine duration) or longitude (e.g., timezones, geopolitical borderlines), we might want to focus on increasing the resolution of either φ or λ while the other is hold relatively at large scale. Therefore, we introduce a multi-scale position encoder sphereM , where interaction terms between φ and λ always have one of them fixed at top scale:
GpsphereMqpxq “ S´1 ď
s“0 rsinφs, cosφs cosλ, cosφ cosλs, cosφs sinλ, cosφ sinλss. (10)
This new encoder ensures that the φ term interact with all the scales of λ terms and λ term interact with all the scales of φ terms. Note that when S “ 1, GpsphereMq is equivalent to GpsphereCq. Both sphereC and sphereM are multi-scale versions of a spherical distance-kept encoder (See Equation 12) and keep that as the main term in their multi-scale representation.
5https://en.wikipedia.org/wiki/Great-circle_distance 6https://en.wikipedia.org/wiki/Spherical_coordinate_system
sphereC+ and sphereM+ From the analysis of the two proposed encoders and the state-of-the-art grid encoders (Appendix 9.3), we know that grid pays more attention to the sum of cos difference of latitudes and longitudes, while our proposed encoders pay more attention to the spherical distances. In order to capture both information, we consider merging grid with each proposed encoders to get more powerful models that encode geographical information from different angles.
GpsphereC`qpxq “ GpsphereCqpxq YGpgridqpxq, GpsphereM`qpxq “ GpsphereMqpxq YGpgridqpxq.
(11)
9.3 ANALYSIS OF sphereC AND ITS COMPARISON TO THE grid ENCODER
To illustrate that sphereC is good at capturing spherical distance, we take a close look at its basic case S “ 1 (define s “ 0 and fs “ 1), where the multi-scale encoder degenerates to
pGpsphereCqpxq “ rsinpφq, cospφq cospλq, cospφq sinpλqs. (12) These three terms are included in the multi-scale version (S ą 1) and serve as the main terms at the largest scale and also the lowest frequency (when s “ S ´ 1). The high frequency terms are added to help the downstream neuron network to learn the point-feature more efficiently. Interestingly, pGpsphereCq captures the spherical distance in a very explicit way: Theorem 2. Let x1, x2 be two points on the same sphere with radius R, then
x pGpsphereCqpx1q, pGpsphereCqpx2qy “ cosp ∆D
R q, (13)
where ∆D is the great circle distance between x1 and x2. Under this metric,
} pGpsphereCqpx1q ´ pGpsphereCqpx2q} “ 2 sinp ∆D
2R q. (14)
Moreover, } pGpsphereCqpx1q ´ pGpsphereCqpx2q} « ∆DR ,when ∆D is small w.r.t. R.
See the proof in Appendix 9.4. Since the central angle ∆δ “ ∆DR P r0, πs and cospxq is strictly monotonically decrease for x P r0, πs, Theorem 2 shows that pGpsphereCqpxq directly satisfies our expectation in Equation 1 where fpxq “ cosp xR q. In comparison, when S “ 1, the inner product in the output space of grid encoder is
x pGpgridqpx1q, pGpgridqpx2qy “ cospφ1 ´ φ2q ` cospλ1 ´ λ2q, (15) which models the latitude difference and longitude difference of x1 and x2 separately rather than spherical distance. This introduces problems in encoding. For instance, consider data pairs x1 “ pλ1, φq and x2 “ pλ2, φq, the distance between them in output space of grid, } pGpgridqpx1q ´ pGpgridqpx2q}2 “ 2´ 2 cospλ1 ´ λ2q stays as a constant in terms of φ. However, when φ varies from ´π2 to π 2 , the actual spherical distance changes in a wide range, e.g., the actual distance between the data pair at φ “ ´π2 (South Pole) is 0 while the distance between the data pair at φ “ 0 (Equator), gets the maximum value. This issue in measuring distances also has a negative impact on grid’s ability to model distributions in areas with sparse sample points because it is hard to learn the true spherical distance. We observe that grid reaches peak performance at much smaller rmin than that of Sphere2Vec encodings. Moreover, sphereC outperforms grid near polar regions where grid claims large distance though the spherical distance is small (A, B in Figure 1).

heading: 9.4 PROOF OF THEOREM 2
text: Proof. Since pGpsphereCqpxiq “ rsinpφiq, cospφiq cospλiq, cospφiq sinpλiqs for i “ 1, 2, the inner product
x pGpsphereCqpx1q, pGpsphereCqpx2qy “ sinpφ1q sinpφ2q ` cospφ1q cospλ1q cospφ2q cospλ2q ` cospφ1q sinpλ1q cospφ2q sinpλ2q “ sinpφ1q sinpφ2q ` cospφ1q cospφ2q cospλ1 ´ λ2q “ cosp∆δq “ cosp∆D{Rq, (16)
where ∆δ is the central angle between x1and x2, and the spherical law of cosines is applied to derive the second last equality. So,
} pGpx1q ´ pGpx2q}2
“ x pGpx1q ´ pGpx2q, pGpx1q ´ pGpx2qy “ 2´ 2 cosp∆D{Rq “ 4 sin2p∆D{2Rq.
(17)
So } pGpx1q ´ pGpx2q} “ 2 sinp∆D{2Rq since ∆D{2R P r0, π2 s. By Taylor expansion, } pGpx1q ´ pGpx2q} « ∆D{R when ∆D is small w.r.t. R.

heading: 9.5 PROOF OF THEOREM 1
text: @˚ P tsphereC, sphereC`, sphereM, sphereM`u, Gp˚qpx1q “ Gp˚qpx2q implies
sinpφ1q “ sinpφ2q, (18)
cospφ1q sinpλ1q “ cospφ2q sinpλ2q, (19)
cospφ1q cospλ1q “ cospφ2q cospλ2q, (20)
from s “ 0 terms. Equation 18 implies φ1 “ φ2. If φ1 “ φ2 “ π{2, then both points are at North Pole, λ1 “ λ2 equal to whatever longitude defined at North Pole. If φ1 “ φ2 “ ´π{2, it is similar case at South Pole. When φ1 “ φ2 P p´π2 , π 2 q, cospφ1q “ cospφ2q ‰ 0. Then from Equation 19 and 20,
sinλ1 “ sinpλ2q, cospλ1q “ cospλ2q, (21)
which shows that λ1 “ λ2. In summary, x1 “ x2, so Gp˚q is injective. If ˚ “ sphereDFS, Gp˚qpx1q “ Gp˚qpx2q implies
sinpφ1q “ sinpφ2q, cospφ1q “ cospφ2q, sinpλ1q “ sinpλ2q, cospλ1q “ cospλ2q, (22)
which proves x1 “ x2 and Gp˚q is injective directly.

heading: 9.6 BASELINES
text: In order to understand the advantage of sphereical-distance-kept location encoders, we compare different versions of Sphere2Vec with multiple baselines:
• No Prior indicates a image classifier without using any location information, i.e., predicting image labels purely based on image information P py|Iq. • tile divides the study area A (e.g., the earth surface) into grids with equal intervals along
the latitude and longitude direction. Each grid has an embedding to be used as the encoding for every location x fall into this grid. This is a common practice by many previous work when dealing with coordinate data (Berg et al., 2014; Adams et al., 2015; Tang et al., 2015).
• wrap is a location encoder model introduced by Mac Aodha et al. (2019). Given a location x “ pλ, φq, it uses a coordinate wrap mechanism to convert each dimension of x into 2 numbers - Gpwrapqpxq “ rsinpπ λ
180˝ q, cospπ λ 180˝ q, sinpπ φ 90˝ q, cospπ φ 90˝ qs. Then the
results are passed through a multi-layered fully connected neural network NNpwrapqpq which consists of an initial fully connected layer, followed by a series of h residual blocks, each consisting of two fully connected layers (k hidden neurons) with a dropout layer in between. We adopt the official code of Mac Aodha et al. (2019)7 for this implementation.
7http://www.vision.caltech.edu/~macaodha/projects/geopriors/
• wrap ` ffn is similar to wrap except that it replaces NNpwrapqpq with NNffnpq, a simple learnable multi-layer perceptron with h hidden layers and k neurons per layer as that Sphere2Vec has. wrap ` ffn is used to exclude the effect of different NNpq on the performance of location encoders. In the following, all location encoder baselines use NNffnpq as the learnable neural network component so that we can directly compare the effect of different position encoding Gp¨q on the model performance.
• xyz first converts xi “ pλi, φiq P S2 into 3D Cartesian coordinates px, y, zq centered at the sphere center by following Equation 23 before feeding into a multilayer perceptron NNpq. Here, we let px, y, zq to locate on a unit sphere with radius R “ 1. xyz can be treated as a special case of sphereC where S “ 1.
Gpxyzqpxq “ rz, x, ys “ rsinφ, cosφ cosλ, cosφ sinλs (23)
• rbf randomly samples M points from the training dataset as RBF anchor points
{xanchorm ,m “ 1...M}, and use gaussian kernels exp `
´ ‖ xi ´ x anchor m ‖2
2σ2 ˘ on each anchor points, where σ is the kernel size. Each point pi has a M -dimension RBF feature vector which is fed into NNffnpq to obtain the location embedding. This is a strong baseline for representing floating number features in machine learning models. • rff , i.e., Random Fourier Features (Rahimi & Recht, 2008; Nguyen et al., 2017), first en-
codes location x into aD dimension vector -Gprffqpxq “ ϕpxq “ ?
2? D rcos pωTi x` biqsDi“1
where ωi i.i.d„ N p0, δ2Iq and bi is uniformly sampled from r0, 2πs. I is an identity matrix.
Each component of ϕpxq first projects x into a random direction ωi and makes a shift by bi. Then it wraps this line onto the unit cirle in R2 with the cosine function. Rahimi & Recht (2008) show that ϕpxqTϕpx1q is an unbiased estimate of the Gaussian kernal Kpx,x1q. ϕpxq is consist of D different estimates to produce a further lower variance approximation. To make rff comparable to other baselines, we feed ϕpxq into NNffnpq to produce the final location embedding. • grid is a multi-scale location encoder on 2D Euclidean space proposed by Mai et al. (2020b).
Here, we simply treat x “ pλ, φq as 2D coordinate. It first useGpgridqpxq shown in Equation 3 to encode location x into multi-scale representation and then feed it into NNffnpq to produce the final location embedding. • theory is another multi-scale location encoder on 2D Euclidean space proposed by Mai
et al. (2020b). It use a position encoder Gptheoryqpxq shown in Equation 24.. Here, xs “ rλs, φss “ r λfs , φ fs s and a1 “ r1, 0sT ,a2 “ r´1{2, ? 3{2sT ,a3 “ r´1{2,´ ? 3{2sT P R2 are three unit vectors which orient 2π{3 apart from each other. The encoding results are feei.nto NNffnpq to produce the final location embedding.
Gptheoryqpxq “ S´1 ď
s“0
3 ď j“1 rsinpxxs,ajyq, cospxxs,ajyqs. (24)

heading: 9.7 MODEL IMPLEMENTATION DETAIL
text: All types of Sphere2Vec as well as all baseline models we compared share the same model set up - epxqpxq “ NNffnpGp˚qpxqq. Here, Gp˚qp¨q is a position encoder which preprocesses the input coordinates x “ pλ, φq P S2 and outputs a position embedding Gp˚qpxq. There are no learnable parameter in Gp˚qpxq. The position encoders Gp˚qp¨q used for different Sphere2Vec are discussed in detail in Section 3.1 while Gp˚qp¨q of all baselines are discussed in Appendix 9.6. We can see that Gp˚qp¨q of grid, theory and different types of Sphere2Vec encode the input coordinates in a multi-scale fashion by using different sinusoidal functions with different frequencies. Many previous work call this practice “Fourier input mapping” (Rahaman et al., 2019b; Tancik et al., 2020; Basri et al., 2020; Anokhin et al., 2021). The difference is that grid and theory use the Fourier features from 2D Euclidean space while our Sphere2Vec uses all or the subset of Double Fourier Sphere Features to take into account the sphereical geometry and the distance distortion it brings.
Except for wrap, NNffnp¨q used by all the other location encoders is a learnable multi-layer perceptron (MLP) with h hidden layers and k neurons per layer. We use the original MLP implementation provided by Mai et al. (2020b). Here, each layer of NNffnpq is a linear layer followed by a dropout layer, a nonlinear activation function, a layer normalization layer (Ba et al., 2016), and a skip connection. Different from other location encoders, wrap uses a different implementation of the full connected layer NNwrapffn p¨q. Please refer to Appendix 9.6 and Mac Aodha et al. (2019) for the detailed implementation of NNwrapffn p¨q.
All models are implemented in PyTorch. We use the original implementation of wrap from Mac Aodha et al. (2019) and the implementation of grid and theory from Mai et al. (2020b). We train/evaluate each model on a Ubuntu machine with 2 GeForce GTX Nvidia GPU cores, each of which has 10GB memory.

heading: 9.8 MODEL TRAINING DETAIL AND HYPERPARAMETER TUNING
text: To train different types of Sphere2Vec as well as baseline models, we use Adam as the optimizer in both unsupervised and supervised training phase. Basically, we first do unsupervised training with an initial learning rate ηunsuper by following either Equation 6 or 7 as the unsupervised training objective. In this setting, we predict the corresponding image embedding based on the input location. Only location encoder is trained during the unsupervised training phase while we keep the image encoder freezed as shown in Figure 1. Here, we just use a image encoder pretrained on other image datasets such as ImageNet and use it as a image feature extractor F. After the model is convergence, we use a new learning rate ηsuper to do supervised training by following Equation 8.

heading: 9.8.1 FULL SUPERVISED TRAINING SETTING
text: In order to test the effectiveness of our spherical encoding method – Sphere2Vec, we train all location encoders in a fully supervised setting. In order to find the best hyperparameter combinations for each model on each dataset, we use grid search to do hyperparameter tuning including supervised training learning rate ηsuper “ r0.01, 0.005, 0.002, 0.001, 0.0005, 0.00005s, the number of scales S “ r16, 32, 64s, the minimum scaling factor rmin “ r0.10.050.020.010.0050.0010.0001s, the number of hidden layers and number of neurons used in NNffnp¨q – h “ r1, 2, 3, 4s and k “ r256, 512, 1024s, the dropout rate in NNffnp¨q – dropout “ r0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7s. We also test multiple options for the nonlinear function used for NNffnp¨q including ReLU, LeakyReLU, and Sigmoid. The maximum scaling factor rmax can be determined based on the range of latitude φ and longitude λ. For grid and theory, we use rmax “ 360 and for all Sphere2Vec, we use rmax “ 1. As for rbf and rff , we also tune their hyperparamaters including kernel size σ “ r0.5, 1, 2, 10s as well as the number of kernels M “ r100, 200, 500s. Based on hyperparameter tuning, we find out using 0.5 as the dropout rate and ReLU as the nonlinear activation function for NNffnp¨qworks best for every location encoder. Based on the hyperparameter tuning, the best hyperparameter combinations are selected for different models on different datasets. The best results are reported in Table 1. Note that each model has been running for 5 times and its mean Top1 score is reported. Due to the limit of space, the standard deviation of each model’s performance on each dataset is not included in Table 1. However, we report the standard deviations of all models’ performance on three datasets in Appendix 9.9.
Moreover, we find out ηsuper and rmin are the most important hyperparameters. Table 2 shows the best hyperparameter combinations of different Sphere2Vec models on different species image classification dataset. We use a smaller S for sphereDFS since it has OpS2q terms while the other models have OpSq terms. sphereDFS with S “ 8 yield a similar number of terms to the other models wth S “ 32 (See Table 3). Interestingly, all first four Sphere2Vec models (sphereC, sphereC`, sphereM , and sphereM`) shows the best performance on all five datasets with the same hyperparamter combinations. Note that compared with other datasets, iNat2017 and iNat2018 are more up-to-date datasets with more training samples and better geographic coverage. This indicates that the proposed 4 Sphere2Vec models show similar performance over different hyperparameter combinations.
We also find out that using a deeper MLP as NNffnp¨q, i.e., a larger h does not necessarily lead to better classification accuracy. In many cases, one hidden layer – h “ 1 achieves the best performance for many kinds of location encoders. We discuss this in detail in Appendix 9.9.

heading: 9.8.2 UNSUPERVISED TRAINING SETTING
text: In order to test the effectiveness of different unsupervised learning objectives, we also conduct another set of experiments by unsupervised training different location encoders before the supervised training phase. In addition to the hyperparameter we discussed in Section 9.8.1, we also fine tune the hyperparameters used in the unsupervised learning phase including the unsupervised training learning rate ηunsuper, the negative location loss weight α1, SimCSE loss weight α2, the number of negative locations we used |N s|, three temperature τ0, τ1, and τ2. As for unsupervised objective LBI, we also fine tune β1 and β2. The hyperparameters we have searched and the hyperparameter tuning results for the unsupervised learning setting are shown in Table 5, 6, 7 in Appedix 9.10.

heading: 9.9 MORE EXPERIMENTAL RESULTS ON GEO-AWARE IMAGE CLASSIFICATION TASKS
text: Table 4 is a complementary of Table 1 which compares the performance of sphereM` with all baseline models on the geo-aware image classification task. The results on three datasets are shown here including BirdSnap†, NABirds†, and iNat2018. For each model, we vary the depth of its NNffnpq, i.e., h “ r1, 2, 3, 4s. The best evaluation results with each h are reported. Moreover, we run each model with one specific h 5 times and report the standard deviation of the Top1 accuray, indicated in “()”. Several important observations can be made based on Table 4:
1. Although the absolute performance improvement between sphereM` and the best baseline model is not very large – 0.90%, 0.28%, and 0.71% for three datasets respectively, these performance improvements are statistically significant given the standard deviations of these Top1 scores.
2. These performance improvements are comparable to those from the previous studies on the same tasks. In other words, the small margin is due to the nature of these datasets. For example, Mai et al. (2020b) showed that grid or theory has 0.79%, 0.44% absolute Top1 accuracy improvement on BirdSnap† and NABirds† dataset respectively. Mac Aodha et al. (2019) showed that wrap has 0.09%, 0%, 0.04% absolute Top1 accuracy improvement on BirdSnap, BirdSnap† and NABirds† dataset. Here, we only consider the results of wrap that only uses location information, but not temporal information. Although Mac Aodha et al. (2019) showed that compared with tile and nearest neighbor methods, wrap has 3.19% and 3.71% performance improvement on iNat2017 and iNat2018 datatset, these large margins are mainly because the baselines they used are rather weak. When we consider the typical rbf and rff (Rahimi et al., 2007) used in our study, their performance improvement are down to -0.02% and 0.61%.

heading: 9.10 ABLATION STUDY ON DIFFERENT SELF-SUPERVISED LOSS
text: Both the BI (Equation 6) and MC (Equation 7) unsupervised loss have three loss component: inbatch, negative location, SimCSE loss component. How does each of them contribute to the overall unsupervised training? In order to answering this question, we do ablation studies on these two
BI indicates the full BI loss while BI´ LbatchBI ` LneglocBI deletes the
SimCSE component. BI´ LbatchBI additionally deletes the negative location loss component. Comparing among those three model settings can help us understand the effect of different loss components. Similarly, we do the same ablation study on MC loss and show in (b).
objectives by using iNat2018 dataset as an example. Figure 7a and 7b illustrate the ablation study results.
From Figure 7a, we see that as for BI, adding LneglocBI will significantly increase the model performance especially when Γ is large. Adding LsimcseBI also improves the model performance when Γ is small. As for the ablation study on MC loss as shown in Figure 7b, adding LneglocMC can significantly increase the model performance and adding LsimcseMC can also make the performance slightly better.
9.11 EFFECTIVENESS OF UNSUPERVISED PRETRAINING ON FMOW DATASET

heading: 9.12 IMPACT OF MRR BY THE NUMBER OF SAMPLES AT DIFFERENT LATITUDE BANDS
text: See Figure 9

heading: 9.13 PREDICTED DISTRIBUTIONS INAT2018
text: We plot the predicted species distributions from different models at different geographic regions, and compare them with the training sample locations of the corresponding species, see Figure 10. We can see that compared with wrap˚ and grid, in each geographic region with sparse training samples and the North Pole area, the spatial distributions produced by sphereC` are more compact while the other two have over-generalization issue.

heading: 9.14 EMBEDDING CLUSTERING
text: We use the location encoder trained on iNat2017 or iNat2018 dataset to produce a location embedding for the center of each small latitude-longitude cell. Then we do agglomerative clustering8 on all these embeddings to produce a clustering map. Figure 11 and 12 show the clustering results for different models with different hyperparameters on iNat2017 and iNat2018 dataset.
8https://scikit-learn.org/stable/modules/generated/sklearn.cluster. AgglomerativeClustering.html
a They share the same best hyperparameters: S “ 64, rmax “ 1, and 1 hidden ReLU layers of 512 neurons. bThey share the same best hyperparameters: S “ 32, rmax “ 1, and 1 hidden ReLU layers of 1024 neurons.

heading: 9.15 EXPERIMENTS WITH SYNTHETIC DATASET

heading: 9.15.1 SYNTHETIC DATASET GENERATION
text: To further investigate the effectiveness of our proposed spherical location encoder Sphere2Vec, we construct a set of synthetic datasets.
We utilize the von Mises–Fisher distribution (vMF ) (Izbicki et al., 2019), an analogy of the 2D Gaussian distribution on the spherical surface S2 to generate synthetic data points9. The probability density function of vMF is defined as
vMF px;µ, κq “ κ 2π sinhpκq exppκµ Tχpxqq
where χpxq “ rx, y, zs “ rcosφ cosλ, cosφ sinλ, sinφs (25)
Here, χpxq converts x into a coordinates in the 3D Euclidean space, and on the surface of a unit sphere. A vMF distribution is controlled by two parameters – the mean direction µ P R3 and concentration parameter κ P R`. µ indicates the center of a vMF distribution which is a 3D unit vector. κ is a positive real number which controls the concentration of vMF . A higher κ indicates more compact vMF distribution, while κ “ 1 correspond to a vMF distribution with standard deviation covering half of the unit sphere.
To simulate multi-modal distributions, we assume C classes with even prior, and each classes follows a vMF distribution. To create a dataset we first sample C sets of parameters tpµi, κiqu (C “ 50). Then we draw SP samples for each class (SP “ 100). So in total, each generated synthetic dataset has 5000 data points for 50 balanced classes.
9https://www.tensorflow.org/probability/api_docs/python/tfp/ distributions/VonMisesFisher#sample
a They share the same best hyperparameters: S “ 64, rmax “ 1, and 1 hidden ReLU layers of 512 neurons. bThey share the same best hyperparameters: S “ 32, rmax “ 1, and 1 hidden ReLU layers of 1024 neurons.
The concentration parameter κi is sampled by first drawing r from an uniform distribtuion Upκmin, κmaxq, and then take the square r2. The square helps to avoid sampling many large κi which yield very concentrated vMF distributions that are rather easy to be classified. We fix κmin “ 1 and vary κmax in r16, 32, 64, 128s. For the center parameter µi we adopt two sampling approaches:
1. Uniform Sampling: We uniformly sample C centers (µi) from the surface of a unit sphere. We generate four synthetic datasets (for different values of κmax) and indicate them as U1, U2, U3, U4. See Table 8 for the parameters we use to generate these datasets.
2. Stratified Sampling: We first evenly divide the latitude range r´π{2, π{2s intoNµ intervals. Then we uniformly sample Cµ centers (µi) from the spherical surface defined by each latitude interval. Since the latitude intervals in polar regions have smaller spherical surface area, this stratified sampling method has higher density in the polar regions. We keep Nµ ˆ Cµ “ C “ 50 fixed and varies Nµ in r5, 10, 25, 50s. Combined with the 4 κmax choices, this procedure yields 16 different synthetic datasets. We denote them as Si.j. See Table 8 for the parameters we use to generate these datasets.
Figure 13a-13d visualize the data point distributions of these four synthetic datasets based on the uniform sampling method in 2D space. Figure 13e visualized the U4 dataset in a 3D Euclidean space. We can see that when κmax is larger, the variation of point density among different vMF distributions becomes larger. Some vMF are very concentrated and the resulting data points are easier to be classified. Moreover, if we treat these datasets as 2D data points as grid and theory do, vMF distributions in the polar areas will be stretched to very extended shapes making model learning more difficult. However, this kind of systematic bias can be avoided if we use a spherical location encoder as Sphere2Vec.
Figure 14 visualizes the data distributions of four synthetic datasets with stratified sampling method. They have different Nµ but the same κmax. We can see that when Nµ increases, a more fine-grain stratified sampling is carried out. The resulting dataset has a larger data bias toward the polar areas.

heading: 9.15.2 SYNTHETIC DATASET EVALUATION RESULTS
text: We evaluate all baseline models as well as sphereM` on those generated 20 syhthetic datasets as described above. For each model, we do grid search on their hyperparameters for each dataset including supervised learning rate ηsuper, the number of scales S, the minimum scaling factor rmin, the number of hidden layers and number of neurons used in NNffnp¨q – h and k. The best performance of each model is reported in Table 8. Some observations can be made from Table 8:
1. sphereM` is able to outperform all baselines on all 20 synthetic datasets. The absolute Top1 improvement can go up to 2% and the error rate deduction can go up to 30.8%. This shows the robustness of sphereM`.
2. When the dataset is fairly easy to classify (i.e., all baseline models can produce 95+% Top1 accuracy), sphereM` is still able to further improve the performance and gives a very large error rate reduction (up to 30.8%). This indicates that sphereM` is very robust and reliable for datasets with different distribution characteristics.
3. Comparing the error rate of different stratified sampling generated datasets (S1.j - S4.j) we can see that when we keep κmax fixed and increase Nµ, the relative error reduction ER become larger. Increasing Nµ means we do a more fine-grain stratified sampling. The resulting datasets should sample more vMF distributions in the polar regions. This phenomenon shows that when the dataset has a larger data bias towards the polar area, we expect sphereM` will be more effective.

Review summaries:
The paper studies how to use location information (geographical location) of images to help with image classification tasks. The paper studies use of spherical based location over GPS lat long and argues that the euclidean space does not preserve distances and has distortions whereas a spherical coordinate approach would not have the inherent problem. The approach demonstrates a framework to train the location encoding in an unsupervised way to with losses based on representation learning approaches. The paper studies multiple ways of expanding the spherical coordinates and evaluates their performance and empirically demonstrate the strength in their performance. 

This paper proposes a strategy for learning a location embedding (i.e., translating lat/lon into a useful high-dimensional vector). The idea is to first encode the location into the double Fourier sphere basis (or a subset) and then pass this through a multilayer perceptron. Several alternatives to the first stage are evaluated.  Two stages of training are proposed: 1) Weakly Supervised Pre-Training: use a contrastive loss to train the embedding by comparing it to features extracted from geotagged images. The idea is that a good location embedding should be sufficient to recognize whether a given image was captured in that location. 2) Supervised Fine-Tuning: given a geotagged image, extract image and location features, combine them to predict the image classification label.  The approach is demonstrated on several geo-aware image classification applications. The idea with these applications is that combining image features with the location embedding is sometimes enough to resolve ambiguity from the image alone.

This paper explores developing location encodings for spherical surfaces and their application to tasks in geo-aware image classification. The authors propose a method for encoding spherical locations into a higher dimensional space (based on Double Fourier Sphere) that preserves distances. The proposed method, referred to as Sphere2Vec, is demonstrated for three tasks and evaluated using seven datasets. In addition, a self-supervised method is proposed for using unlabeled data to pretrain the location encoder.  This self-supervised pretraining stage is shown to improve performance, especially when available labeled data is limited.

This paper proposes a positional embedding method for GPS coordinates, which lie on a sphere.  Based on the Double Fourier Sphere, this paper proposes a family of encoding methods and show their effectiveness.  This paper also gives a family of loss functions to enable self-supervised training.