Sphere2Vec: Self-Supervised Location Representation Learning on Spherical Surfaces

1 INTRODUCTION . Spatial information has become an important component to many machine learning tasks that address the environmental , economic , and societal needs for sustainable development . Just to name a few , these tasks include geo-aware species fine-grained recognition ( Chu et al. , 2019 ; Mac Aodha et al. , 2019 ) , Point of Interest ( POI ) facade image classification ( Chu et al. , 2019 ; Yan et al. , 2018 ) , remote sensing image classification ( Christie et al. , 2018 ; Ayush et al. , 2020 ) , poverty prediction ( Jean et al. , 2016 ; 2019 ) , point cloud classification and semantic segmentation ( Qi et al. , 2017a ; b ) , traffic forecasting ( Li et al. , 2018 ; Cai et al. , 2020 ) , and geographic question answering ( Mai et al. , 2019 ; 2020a ) . Developing a general model for vector space representations of any point in space , termed a location encoder , would pave the way for many future applications that improves our well-being . Such a location encoder can operate on a 2D space or on other manifolds ; of particular interest is the spherical surface of the Earth . Here we use geo-aware image classification shown in Figure 1 and 2 as an example . During training we assume a dataset D “ tpIi , xi , yiq|i “ 1 , ... , Nu , where Ii is an image , yi P 1 , .... , C is the corresponding class label , xi represents the location ( longitude and latitude ) and optionally the time the image was taken1 . The image information is often processed by an state-of-the-art image classification model ( e.g. , InceptionV3 ) to predict the class labels as shown in Figure 1 . Although the image itself is the most critical for image classification , sometimes it can be insufficient – the image of two distinct classes may look very similar both for the central object and its surrounding environment . For example , Figure 2a and 2f shows example images of two different fox species : Arctic fox ( Vulpes lagopus ) and bat-eard fox ( Otocyon megalotis ) , for which the image model can be confused due to the high visual similarity of the two species . Fortunately , the geo-location information of an image may provide a clue to their real classes . Figure 2b , 2g show that these two species have distinct spatial distribution patterns , and the image locations provide complementary information to the images . This 1In this study we focus on the location information and leave the time aspect of modeling to future . insight motivates the development of hybrid classification models , which take both the images and their locations as input ( Mac Aodha et al. , 2019 ; Mai et al. , 2020b ) . In spite of their effectiveness , hybrid models have an issue when representing points in non-Euclidean spaces . The demand on representation and prediction learning at a global scale grows dramatically due to emerging global scale issues , such as the transition path of the latest pandemic ( Chinazzi et al. , 2020 ) , long lasting issue for malaria ( Caminade et al. , 2014 ) , under threaten global biodiversity ( Di Marco et al. , 2019 ; Ceballos et al. , 2020 ) , and numerous ecosystem and social system responses for climate change ( Hansen & Cramer , 2015 ) . However , when applying the state-of-the-art 2D space location encoders ( Gao et al. , 2019 ; Chu et al. , 2019 ; Mac Aodha et al. , 2019 ; Mai et al. , 2020b ; Zhong et al. , 2020 ) to large-scale real-world GPS coordinate datasets such as species images taken all over the world , a map projection distortion problem ( Williamson & Browning , 1973 ) emerges . These 2D location encoders are designed for preserving distance in 2D ( or 3D ) Euclidean space , while GPS coordinates are in fact on a spherical manifold , e.g. , point x “ pλ , φq on a sphere with longitude λ P r´π , πq and latitude φ P r´π { 2 , ´π { 2s ( See Figure 1 ) . Directly applying these 2D location encoders on spherical coordinates will yield a large distortion in the polar regions . Note that map projection distortion is unavoidable when projecting spherical coordinates into 2D space . This emphasizes the importance of calculating on a round planet ( Chrisman , 2017 ) . See Appendix 9.1 for more discussion . From a modeling perspective another key challenge is how to design the self-supervised training objective that effectively transfers the attribute information ( e.g. , image features ) at different locations into a location encoder . This is very important in practice since collecting and labeling geo-tagged images is particularly costly as annotations often require domain expertise . However , most existing location encoding approaches ( Chu et al. , 2019 ; Mac Aodha et al. , 2019 ; Mai et al. , 2020b ) are developed and trained in a supervised learning framework while massive unlabeled geographic data can not be used . Ayush et al . ( 2020 ) proposed a geographic self-supervised learning framework , but they mainly focus on training image encoders while the usage of geo-location information is limited to data augmentation in the pre-training stage . In contrast , in this work , we focus on training location encoders in a self-supervised manner so that geo-location information is explicitly use as part of the model input . In this work , we propose Sphere2Vec as shown in Figure 1 , which directly encodes point coordinates on a spherical surface while avoiding the map projection distortion . The multi-scale encoding method utilizes Double Fourier Sphere basis ( OpS2q terms ) or a subset ( OpSq terms ) of it while still being able to correctly measure the spherical distance . Furthermore , we develop a self-supervised learning framework which pre-trains deep location representations from unlabeled geo-tagged images by predicting image features or image identities base on their geo locations . We compare several self-supervised learning objectives including Mean Square Error loss ( MSE ) , contrastive binary classification loss ( BI ) , and contrastive multi-classification loss ( MC ) . The pre-trained Sphere2Vec model can be fine-tuned ( with an additional output layer ) to perform supervised geographic objects classification tasks . We demonstrate the effectiveness of Sphere2Vec on geo-aware image classification tasks including fine-grain species recognition ( Chu et al. , 2019 ; Mac Aodha et al. , 2019 ; Mai et al. , 2020b ) , POI fascade image classification ( Tang et al. , 2015 ; Mac Aodha et al. , 2019 ) , and remote sensing image recognition ( Christie et al. , 2018 ; Ayush et al. , 2020 ) . In summary , the contributions of our work are : . 1 . We develop a unified view of distant preserving location encoding methods on spheres based on Double Fourier Sphere ( DFS ) ( Merilees , 1973 ; Orszag , 1974 ) . 2 . We propose an effective pre-training strategy for location encoders from unlabeled geotagged images by predicting image features and image identities based on their geo locations . 3 . Sphere2Vec improves the performances of state-of-the-art results on four species recognition datasets ( BirdSnap , NABirds , iNat2017 , iNat2018 ) , one POI fascade image classification dataset ( YFCC ) , and one remote sensing image classification dataset ( fMoW ) . It also demonstrate significant advantage in few shot learning settings . 2 PROBLEM FORMULATION . Following previous work ( Mai et al. , 2020b ) , Euclidean location encoding can be formulated as follows . Given a set of points ( e.g. , POI ) P “ tpiu in L-D space ( L “ 2 , 3 ) , we define a function eP , θpxq : RL Ñ Rd ( L ! d ) , which is parameterized by θ and maps any coordinate x in space to a vector representation of d dimension . Each point ( e.g. , a restaurant ) pi “ pxi , viq is associated with a location xi and attributes vi ( i.e. , POI features such as types , names , capacity , images , etc. ) . The function eP , θpxq encodes the information about Point x which is useful for certain tasks such as predicting a label y . Attributes and coordinates of points can be seen as analogies to words and word positions in commonly used word embedding models . Similarly , we can formulate spherical location encoding as follows . Given a set of points P “ tpiu on the surface of a sphere S2 , e.g. , species occurrences all over the world , where pi “ pxi , viq and xi “ pλi , φiq P S2 indicates a point with longitude λi P r´π , πq and latitude φi P r´π { 2 , π { 2s . Define a function eP , θpxq : S2 Ñ Rd , which is parameterized by θ and maps any coordinate x in a spherical surface S2 to a vector representation of d dimension . See Appendix 9.1 for more discussions about the map projection distortion problem . 3 METHOD . The geo-aware image classification task ( Chu et al. , 2019 ; Mac Aodha et al. , 2019 ) can be formulated as follows . Given an image I taken from location/point x , we estimate which category y it belongs to by P py|I , xq . If we assume I and x are conditionally independent given y , then based on Bayes ’ theorem , we have P py|I , xq 9 P py|xqP py|Iq ( See Mac Aodha et al . ( 2019 ) ) . P py|Iq can be given by any state-of-the-art image classifier such as Inception V3 ( Szegedy et al. , 2016 ) for the species recognition task or MoCo-V2+TP ( Ayush et al. , 2020 ) for the RS image classification task . Figure 1 illustrates the whole training and inference process by using species recognition as the example task . P py|Iq and P py|xq are given by the image encoder ( the green box ) and location encoder ( the blue box ) respectively and are multiplied together for final prediction . In this work , we focus on estimating the geographic prior distribution of class y over the spherical surface P py|xq 9 σpepxqT : ,yq where σp9q is a sigmoid activation function . T P Rdˆc is a class embedding matrix where the yth column T : ,y P Rd indicates the class embedding for class y . In the following we introduce the two main contributions of our work – location encoding on spherical surfaces ( the blue box in Figure 1 ) , and a self-supervised training framework for location representation model ( the orange box in Figure 1 ) . In Section 3.1 , we will discuss the design of spherical distance-kept location encoder epxq , Sphere2Vec . We developed a unified view of distancereserving encoding on spheres based on Double Fourier Sphere ( DFS ) ( Merilees , 1973 ; Orszag , 1974 ) . In Section 3.2 we discuss training strategies for Sphere2Vec including self-supervised pre-training given unlabeled images , and supervised fine tuning . We use eP , θpxq to indicate the spherical location encoding neural network with parameters θ , and will refer it by epxq for simplicity . 3.1 Sphere2Vec ENCODER We want to find a function Gpsphereqpxq which does a one-to-one mapping from each point xi “ pλi , φiq P S2 to a multi-scale representation with S as the total number of scales such that it satisfies the following requirement : xGpsphereqpx1q , Gpsphereqpx2qy “ fp∆Dq , @ x1 , x2 P S2 , ( 1 ) where ∆D P r0 , πRs is the spherical surface distance between x1 , x2 , R is the radius of this sphere , and fpxq is a strictly monotonically decreasing function for x P r0 , πRs . In other words , we expect to find a function Gpsphereqpxq such that the resulting multi-scale representation of x preserves the spherical surface distance . Similar to the case for Euclidean space ( Mai et al. , 2020b ) , any point x “ pλ , φq P S2 with longitude λ P r´π , πq and latitude φ P r´π 2 , π 2 s , can be explicitly encoded in a multi-scale fashion in the form of epxqpxq “ NNffnpGp˚qpxqq where NNffnpq is a learnable multi-layer perceptron with h hidden layers and k neurons per layer . Gp˚qpxq is a concatenation of multi-scale spherical spatial features of S levels . p˚q indicates different types of spherical location encoders . In the following , we call epxqpxq location encoder and its component Gp˚qpxq position encoder . Let rmin , rmax be the minimum and maximum scaling factor , and g “ rmaxrmin . 2 sphereDFS Double Fourier Sphere ( DFS ) ( Merilees , 1973 ; Orszag , 1974 ) is a simple yet successful pseudospectral method , which has been applied to efficient analysis of large scale phenomenons such as weather ( Sun et al. , 2014 ) and blackholes ( Bartnik & Norton , 2000 ) . Our first intuition is to use the base functions of DFS to help decompose x “ pλ , φq into a high dimensional vector : GpsphereDFSqpxq “ S´1 ď n “ 0 rsinφn , cosφns Y S´1 ď m “ 0 rsinλm , cosλmsY S´1 ď n “ 0 S´1 ď m “ 0 rcosφn cosλm , cosφn sinλm , sinφn cosλm , sinφn sinλms . ( 2 ) where @ s “ 0 , 1 , ... , S ´ 1 , we define λs “ λfs , φs “ φ fs , fs “ rmin ¨ gs { pS´1q . Y means vector concatenation and ŤS´1 s “ 0 indicates vector concatenation through different scales . It basically lets all the scales of φ terms interact with all the scales of λ terms in the encoder . This would introduce a position encoder whose output has OpS2q dimensions and increase the memory burden in training and hurts generalization ( Figure 3a ) . 2In practice we fix rmax “ 1 meaning no scaling of λ , φ . An encoder might achieve better result by only using a subset of these terms . In comparison , the state of art encoder such as grid and wrap ( Mac Aodha et al. , 2019 ) define : 3 Gpgridqpxq “ S´1 ď s “ 0 rsinφs , cosφs , sinλs , cosλss . ( 3 ) We can see that grid employs a subset of terms from sphereDFS ( Figure 3b ) and performs poorly at a global scale due to its inability to preserve spherical distances ( See Appendix 9.3 for proof ) . We explore different subsets of DFS terms while achieving two goals : 1 ) efficient representation with OpSq dimensions 2 ) preserving distance measures on spheres . See Appendix 9.2 for the detail definition of sphereC , sphereC ` , sphereM , sphereM ` , and Figure 3 for a unified view . We hypothesize that encoding these terms in the multi-scale representation would make the training of the encoder easier and the order of output dimension is still OpSq . In location encoding , the uniqueness of encoding ( no two points on sphere having the same position encoding ) is very important , Gp˚qpxq in the five proposed methods are one-to-one mapping . Theorem 1 . @ ˚ P tsphereC , sphereC ` , sphereM , sphereM ` , sphereDFSu , Gp˚qpxq is an injective function . See the proof in Appendix 9.5 . 3.2 TRAINING Sphere2Vec Self-Supervized Pretraining ( the yellow box in Figure 1 ) . We consider two contrastive objectives . The first is a binary classification loss , or so called noise contrastive estimation ( Gutmann & Hyvärinen , 2010 ) , which avoids calculation of the partition function and has been successfully used in word embeddings ( Mikolov et al. , 2013 ) and language modeling ( Mnih & Teh , 2012 ) : LBIpP , N q “ ´Epa , bq „ P rlog σpspa , bqqs ´ Epa , b´q „ N rlogp1´ σpspa , b´qqqs ( 4 ) where BI stands for “ binary ” , P is a set of positive pairs ( denoted as pa , bq ) , N is a set of negative pairs ( denoted as pa , b´q ) , sp¨ , ¨q is a similarity function that assigns higher values when the inputs are more similar ( such as cosinepq ) , and σpvq “ ev { p1 ` evq is the sigmoid function . The second objective function is the multi-class classification loss with temperature , which has been successfully used in unsupervised learning for images ( He et al. , 2020 ) and text ( Gao et al. , 2021 ) : LMCpP , N , τq “ Epa , bq „ P « exppspa , bq { τq exppspa , bq { τq ` ř b´PN pb|aq exppspa , b´q { τq ff ( 5 ) where MC stands for “ multi-class ” , N pb|aq obtains a negative pair with first entry being a ; P and sp¨ , ¨q are defined as earlier . The temperature scaling parameter τ determines how soft the softmax is ( Hinton et al. , 2015 ) . In practice it help with the trade off between top ranked classes ( precision ) versus reset of the classes ( recall ) . In order to learn useful representations , we need to choose appropriate distributions for positive pairs P and negative pairs N . During the location encoder pretraining stage we use a geo-tagged but unlabeled image set Xunlabeled , each image is represented by a embedding generated from a pretrained image extract extractor F4 such as InceptionV3 or MoCo-V2+TP ( Ayush et al. , 2020 ) . Given an unlabeled image px , Iq P Xunlabeled , we use the following positive and negative instances : 3As a multi-scale encoder , grid degenerates to wrap when the number of scales S “ 1 . 4F should not see the current image labels during pretraining stage . • Geo-tagged positive Px “ tpepxq , WFpIqqu where W is a linear projection layer . • In-batch negatives N b “ tpepxq , WFpI´q|px´ , I´q P Bpx , Iqztpx , Iquu where Bpx , Iq represents the batch of examples for which px , Iq is part of during training . • Sampled negative locations N s “ tpepx´q , WFpIqqu of size N , where x´ is one of the N evenly sampled locations from the surface of the sphere for each example x . • Dropout positive Pd “ tpepxq , e1pxqqu , where we pass the same input x to the encoder epq twice and obtain two embeddings as “ positive pairs ” by applying independently sampled dropout masks . This is a data augmentation strategy ( so called SimCSE ) , which has been very successful in generating sentence embeddings ( Gao et al. , 2021 ) . We use e1pxq to denote the embedding from the second mask . • Dropout negative N d “ tpepxq , e1px´qq|px´ , I´q P Bpx , Iqztpx , Iqu , where e1p¨q has the same meaning as that defined in Pd and Bpx , Iq indicates the same as N b . We define two versions of contrastive losses , which both have three components : in-batch , negative location , and SimCSE . The self-supervized binary classification loss LBI is defined as LBIpXq “ LbatchBI pXq ` β1L negloc BI pXq ` β2L simcse BI pXq “ LBIpPx , N bq ` β1LBIpH , N sq ` β2LBIpPd , N dq ( 6 ) where β1 and β2 control the contribution of the last two loss components . The self-supervized multi-class classification loss LMC is defined as LMCpXq “ LbatchMC pXq ` α1L negloc MC pXq ` α2L simcse MC pXq “ LMCpPx , N b , τ0q ` α1LMCpPx , N s , τ1q ` α2LMCpPd , N d , τ2q ( 7 ) where α1 and α2 are hyper-parameters . Supervised Fine-Turning ( the blue box in Figure 1 ) . We directly use image labels in the training objective . Following Mac Aodha et al . ( 2019 ) , we used a presence-absence loss function which converts the multi-class labels into binary multi-labels . Given a set of training samples Xlabeled “ tpx , yqu , the loss function LsupervizedpXq is defined as : LsupervizedpXq “ βLBIpP l , Hq ` LBIpH , N lq ` LBIpH , N lsq ( 8 ) Here , β is a hyperparameter to enlarge the weight of positive samples , and the following positive and negative samples are used : • Labeled positives P l “ tpepxq , T : ,yq|px , yq P Xu . • Labeled negatives N l “ tpepxq , T : ,iq|px , yq P X , i P t1 .. cuztyuu . • Sampled negative locations N ls “ tpepx´q , T : ,iq|px , yq P X , i P t1 .. cuu , where x´ is one of the N evenly sampled locations from the surface of the sphere for each example x .

Summary: This paper proposes a strategy for learning a location embedding (i.e., translating lat/lon into a useful high-dimensional vector). The idea is to first encode the location into the double Fourier sphere basis (or a subset) and then pass this through a multilayer perceptron. Several alternatives to the first stage are evaluated.  Two stages of training are proposed: 1) Weakly Supervised Pre-Training: use a contrastive loss to train the embedding by comparing it to features extracted from geotagged images. The idea is that a good location embedding should be sufficient to recognize whether a given image was captured in that location. 2) Supervised Fine-Tuning: given a geotagged image, extract image and location features, combine them to predict the image classification label.  The approach is demonstrated on several geo-aware image classification applications. The idea with these applications is that combining image features with the location embedding is sometimes enough to resolve ambiguity from the image alone.