Sphere2Vec: Self-Supervised Location Representation Learning on Spherical Surfaces

1 INTRODUCTION . Spatial information has become an important component to many machine learning tasks that address the environmental , economic , and societal needs for sustainable development . Just to name a few , these tasks include geo-aware species fine-grained recognition ( Chu et al. , 2019 ; Mac Aodha et al. , 2019 ) , Point of Interest ( POI ) facade image classification ( Chu et al. , 2019 ; Yan et al. , 2018 ) , remote sensing image classification ( Christie et al. , 2018 ; Ayush et al. , 2020 ) , poverty prediction ( Jean et al. , 2016 ; 2019 ) , point cloud classification and semantic segmentation ( Qi et al. , 2017a ; b ) , traffic forecasting ( Li et al. , 2018 ; Cai et al. , 2020 ) , and geographic question answering ( Mai et al. , 2019 ; 2020a ) . Developing a general model for vector space representations of any point in space , termed a location encoder , would pave the way for many future applications that improves our well-being . Such a location encoder can operate on a 2D space or on other manifolds ; of particular interest is the spherical surface of the Earth . Here we use geo-aware image classification shown in Figure 1 and 2 as an example . During training we assume a dataset D “ tpIi , xi , yiq|i “ 1 , ... , Nu , where Ii is an image , yi P 1 , .... , C is the corresponding class label , xi represents the location ( longitude and latitude ) and optionally the time the image was taken1 . The image information is often processed by an state-of-the-art image classification model ( e.g. , InceptionV3 ) to predict the class labels as shown in Figure 1 . Although the image itself is the most critical for image classification , sometimes it can be insufficient – the image of two distinct classes may look very similar both for the central object and its surrounding environment . For example , Figure 2a and 2f shows example images of two different fox species : Arctic fox ( Vulpes lagopus ) and bat-eard fox ( Otocyon megalotis ) , for which the image model can be confused due to the high visual similarity of the two species . Fortunately , the geo-location information of an image may provide a clue to their real classes . Figure 2b , 2g show that these two species have distinct spatial distribution patterns , and the image locations provide complementary information to the images . This 1In this study we focus on the location information and leave the time aspect of modeling to future . insight motivates the development of hybrid classification models , which take both the images and their locations as input ( Mac Aodha et al. , 2019 ; Mai et al. , 2020b ) . In spite of their effectiveness , hybrid models have an issue when representing points in non-Euclidean spaces . The demand on representation and prediction learning at a global scale grows dramatically due to emerging global scale issues , such as the transition path of the latest pandemic ( Chinazzi et al. , 2020 ) , long lasting issue for malaria ( Caminade et al. , 2014 ) , under threaten global biodiversity ( Di Marco et al. , 2019 ; Ceballos et al. , 2020 ) , and numerous ecosystem and social system responses for climate change ( Hansen & Cramer , 2015 ) . However , when applying the state-of-the-art 2D space location encoders ( Gao et al. , 2019 ; Chu et al. , 2019 ; Mac Aodha et al. , 2019 ; Mai et al. , 2020b ; Zhong et al. , 2020 ) to large-scale real-world GPS coordinate datasets such as species images taken all over the world , a map projection distortion problem ( Williamson & Browning , 1973 ) emerges . These 2D location encoders are designed for preserving distance in 2D ( or 3D ) Euclidean space , while GPS coordinates are in fact on a spherical manifold , e.g. , point x “ pλ , φq on a sphere with longitude λ P r´π , πq and latitude φ P r´π { 2 , ´π { 2s ( See Figure 1 ) . Directly applying these 2D location encoders on spherical coordinates will yield a large distortion in the polar regions . Note that map projection distortion is unavoidable when projecting spherical coordinates into 2D space . This emphasizes the importance of calculating on a round planet ( Chrisman , 2017 ) . See Appendix 9.1 for more discussion . From a modeling perspective another key challenge is how to design the self-supervised training objective that effectively transfers the attribute information ( e.g. , image features ) at different locations into a location encoder . This is very important in practice since collecting and labeling geo-tagged images is particularly costly as annotations often require domain expertise . However , most existing location encoding approaches ( Chu et al. , 2019 ; Mac Aodha et al. , 2019 ; Mai et al. , 2020b ) are developed and trained in a supervised learning framework while massive unlabeled geographic data can not be used . Ayush et al . ( 2020 ) proposed a geographic self-supervised learning framework , but they mainly focus on training image encoders while the usage of geo-location information is limited to data augmentation in the pre-training stage . In contrast , in this work , we focus on training location encoders in a self-supervised manner so that geo-location information is explicitly use as part of the model input . In this work , we propose Sphere2Vec as shown in Figure 1 , which directly encodes point coordinates on a spherical surface while avoiding the map projection distortion . The multi-scale encoding method utilizes Double Fourier Sphere basis ( OpS2q terms ) or a subset ( OpSq terms ) of it while still being able to correctly measure the spherical distance . Furthermore , we develop a self-supervised learning framework which pre-trains deep location representations from unlabeled geo-tagged images by predicting image features or image identities base on their geo locations . We compare several self-supervised learning objectives including Mean Square Error loss ( MSE ) , contrastive binary classification loss ( BI ) , and contrastive multi-classification loss ( MC ) . The pre-trained Sphere2Vec model can be fine-tuned ( with an additional output layer ) to perform supervised geographic objects classification tasks . We demonstrate the effectiveness of Sphere2Vec on geo-aware image classification tasks including fine-grain species recognition ( Chu et al. , 2019 ; Mac Aodha et al. , 2019 ; Mai et al. , 2020b ) , POI fascade image classification ( Tang et al. , 2015 ; Mac Aodha et al. , 2019 ) , and remote sensing image recognition ( Christie et al. , 2018 ; Ayush et al. , 2020 ) . In summary , the contributions of our work are : . 1 . We develop a unified view of distant preserving location encoding methods on spheres based on Double Fourier Sphere ( DFS ) ( Merilees , 1973 ; Orszag , 1974 ) . 2 . We propose an effective pre-training strategy for location encoders from unlabeled geotagged images by predicting image features and image identities based on their geo locations . 3 . Sphere2Vec improves the performances of state-of-the-art results on four species recognition datasets ( BirdSnap , NABirds , iNat2017 , iNat2018 ) , one POI fascade image classification dataset ( YFCC ) , and one remote sensing image classification dataset ( fMoW ) . It also demonstrate significant advantage in few shot learning settings . 2 PROBLEM FORMULATION . Following previous work ( Mai et al. , 2020b ) , Euclidean location encoding can be formulated as follows . Given a set of points ( e.g. , POI ) P “ tpiu in L-D space ( L “ 2 , 3 ) , we define a function eP , θpxq : RL Ñ Rd ( L ! d ) , which is parameterized by θ and maps any coordinate x in space to a vector representation of d dimension . Each point ( e.g. , a restaurant ) pi “ pxi , viq is associated with a location xi and attributes vi ( i.e. , POI features such as types , names , capacity , images , etc. ) . The function eP , θpxq encodes the information about Point x which is useful for certain tasks such as predicting a label y . Attributes and coordinates of points can be seen as analogies to words and word positions in commonly used word embedding models . Similarly , we can formulate spherical location encoding as follows . Given a set of points P “ tpiu on the surface of a sphere S2 , e.g. , species occurrences all over the world , where pi “ pxi , viq and xi “ pλi , φiq P S2 indicates a point with longitude λi P r´π , πq and latitude φi P r´π { 2 , π { 2s . Define a function eP , θpxq : S2 Ñ Rd , which is parameterized by θ and maps any coordinate x in a spherical surface S2 to a vector representation of d dimension . See Appendix 9.1 for more discussions about the map projection distortion problem . 3 METHOD . The geo-aware image classification task ( Chu et al. , 2019 ; Mac Aodha et al. , 2019 ) can be formulated as follows . Given an image I taken from location/point x , we estimate which category y it belongs to by P py|I , xq . If we assume I and x are conditionally independent given y , then based on Bayes ’ theorem , we have P py|I , xq 9 P py|xqP py|Iq ( See Mac Aodha et al . ( 2019 ) ) . P py|Iq can be given by any state-of-the-art image classifier such as Inception V3 ( Szegedy et al. , 2016 ) for the species recognition task or MoCo-V2+TP ( Ayush et al. , 2020 ) for the RS image classification task . Figure 1 illustrates the whole training and inference process by using species recognition as the example task . P py|Iq and P py|xq are given by the image encoder ( the green box ) and location encoder ( the blue box ) respectively and are multiplied together for final prediction . In this work , we focus on estimating the geographic prior distribution of class y over the spherical surface P py|xq 9 σpepxqT : ,yq where σp9q is a sigmoid activation function . T P Rdˆc is a class embedding matrix where the yth column T : ,y P Rd indicates the class embedding for class y . In the following we introduce the two main contributions of our work – location encoding on spherical surfaces ( the blue box in Figure 1 ) , and a self-supervised training framework for location representation model ( the orange box in Figure 1 ) . In Section 3.1 , we will discuss the design of spherical distance-kept location encoder epxq , Sphere2Vec . We developed a unified view of distancereserving encoding on spheres based on Double Fourier Sphere ( DFS ) ( Merilees , 1973 ; Orszag , 1974 ) . In Section 3.2 we discuss training strategies for Sphere2Vec including self-supervised pre-training given unlabeled images , and supervised fine tuning . We use eP , θpxq to indicate the spherical location encoding neural network with parameters θ , and will refer it by epxq for simplicity . 3.1 Sphere2Vec ENCODER We want to find a function Gpsphereqpxq which does a one-to-one mapping from each point xi “ pλi , φiq P S2 to a multi-scale representation with S as the total number of scales such that it satisfies the following requirement : xGpsphereqpx1q , Gpsphereqpx2qy “ fp∆Dq , @ x1 , x2 P S2 , ( 1 ) where ∆D P r0 , πRs is the spherical surface distance between x1 , x2 , R is the radius of this sphere , and fpxq is a strictly monotonically decreasing function for x P r0 , πRs . In other words , we expect to find a function Gpsphereqpxq such that the resulting multi-scale representation of x preserves the spherical surface distance . Similar to the case for Euclidean space ( Mai et al. , 2020b ) , any point x “ pλ , φq P S2 with longitude λ P r´π , πq and latitude φ P r´π 2 , π 2 s , can be explicitly encoded in a multi-scale fashion in the form of epxqpxq “ NNffnpGp˚qpxqq where NNffnpq is a learnable multi-layer perceptron with h hidden layers and k neurons per layer . Gp˚qpxq is a concatenation of multi-scale spherical spatial features of S levels . p˚q indicates different types of spherical location encoders . In the following , we call epxqpxq location encoder and its component Gp˚qpxq position encoder . Let rmin , rmax be the minimum and maximum scaling factor , and g “ rmaxrmin . 2 sphereDFS Double Fourier Sphere ( DFS ) ( Merilees , 1973 ; Orszag , 1974 ) is a simple yet successful pseudospectral method , which has been applied to efficient analysis of large scale phenomenons such as weather ( Sun et al. , 2014 ) and blackholes ( Bartnik & Norton , 2000 ) . Our first intuition is to use the base functions of DFS to help decompose x “ pλ , φq into a high dimensional vector : GpsphereDFSqpxq “ S´1 ď n “ 0 rsinφn , cosφns Y S´1 ď m “ 0 rsinλm , cosλmsY S´1 ď n “ 0 S´1 ď m “ 0 rcosφn cosλm , cosφn sinλm , sinφn cosλm , sinφn sinλms . ( 2 ) where @ s “ 0 , 1 , ... , S ´ 1 , we define λs “ λfs , φs “ φ fs , fs “ rmin ¨ gs { pS´1q . Y means vector concatenation and ŤS´1 s “ 0 indicates vector concatenation through different scales . It basically lets all the scales of φ terms interact with all the scales of λ terms in the encoder . This would introduce a position encoder whose output has OpS2q dimensions and increase the memory burden in training and hurts generalization ( Figure 3a ) . 2In practice we fix rmax “ 1 meaning no scaling of λ , φ . An encoder might achieve better result by only using a subset of these terms . In comparison , the state of art encoder such as grid and wrap ( Mac Aodha et al. , 2019 ) define : 3 Gpgridqpxq “ S´1 ď s “ 0 rsinφs , cosφs , sinλs , cosλss . ( 3 ) We can see that grid employs a subset of terms from sphereDFS ( Figure 3b ) and performs poorly at a global scale due to its inability to preserve spherical distances ( See Appendix 9.3 for proof ) . We explore different subsets of DFS terms while achieving two goals : 1 ) efficient representation with OpSq dimensions 2 ) preserving distance measures on spheres . See Appendix 9.2 for the detail definition of sphereC , sphereC ` , sphereM , sphereM ` , and Figure 3 for a unified view . We hypothesize that encoding these terms in the multi-scale representation would make the training of the encoder easier and the order of output dimension is still OpSq . In location encoding , the uniqueness of encoding ( no two points on sphere having the same position encoding ) is very important , Gp˚qpxq in the five proposed methods are one-to-one mapping . Theorem 1 . @ ˚ P tsphereC , sphereC ` , sphereM , sphereM ` , sphereDFSu , Gp˚qpxq is an injective function . See the proof in Appendix 9.5 . 3.2 TRAINING Sphere2Vec Self-Supervized Pretraining ( the yellow box in Figure 1 ) . We consider two contrastive objectives . The first is a binary classification loss , or so called noise contrastive estimation ( Gutmann & Hyvärinen , 2010 ) , which avoids calculation of the partition function and has been successfully used in word embeddings ( Mikolov et al. , 2013 ) and language modeling ( Mnih & Teh , 2012 ) : LBIpP , N q “ ´Epa , bq „ P rlog σpspa , bqqs ´ Epa , b´q „ N rlogp1´ σpspa , b´qqqs ( 4 ) where BI stands for “ binary ” , P is a set of positive pairs ( denoted as pa , bq ) , N is a set of negative pairs ( denoted as pa , b´q ) , sp¨ , ¨q is a similarity function that assigns higher values when the inputs are more similar ( such as cosinepq ) , and σpvq “ ev { p1 ` evq is the sigmoid function . The second objective function is the multi-class classification loss with temperature , which has been successfully used in unsupervised learning for images ( He et al. , 2020 ) and text ( Gao et al. , 2021 ) : LMCpP , N , τq “ Epa , bq „ P « exppspa , bq { τq exppspa , bq { τq ` ř b´PN pb|aq exppspa , b´q { τq ff ( 5 ) where MC stands for “ multi-class ” , N pb|aq obtains a negative pair with first entry being a ; P and sp¨ , ¨q are defined as earlier . The temperature scaling parameter τ determines how soft the softmax is ( Hinton et al. , 2015 ) . In practice it help with the trade off between top ranked classes ( precision ) versus reset of the classes ( recall ) . In order to learn useful representations , we need to choose appropriate distributions for positive pairs P and negative pairs N . During the location encoder pretraining stage we use a geo-tagged but unlabeled image set Xunlabeled , each image is represented by a embedding generated from a pretrained image extract extractor F4 such as InceptionV3 or MoCo-V2+TP ( Ayush et al. , 2020 ) . Given an unlabeled image px , Iq P Xunlabeled , we use the following positive and negative instances : 3As a multi-scale encoder , grid degenerates to wrap when the number of scales S “ 1 . 4F should not see the current image labels during pretraining stage . • Geo-tagged positive Px “ tpepxq , WFpIqqu where W is a linear projection layer . • In-batch negatives N b “ tpepxq , WFpI´q|px´ , I´q P Bpx , Iqztpx , Iquu where Bpx , Iq represents the batch of examples for which px , Iq is part of during training . • Sampled negative locations N s “ tpepx´q , WFpIqqu of size N , where x´ is one of the N evenly sampled locations from the surface of the sphere for each example x . • Dropout positive Pd “ tpepxq , e1pxqqu , where we pass the same input x to the encoder epq twice and obtain two embeddings as “ positive pairs ” by applying independently sampled dropout masks . This is a data augmentation strategy ( so called SimCSE ) , which has been very successful in generating sentence embeddings ( Gao et al. , 2021 ) . We use e1pxq to denote the embedding from the second mask . • Dropout negative N d “ tpepxq , e1px´qq|px´ , I´q P Bpx , Iqztpx , Iqu , where e1p¨q has the same meaning as that defined in Pd and Bpx , Iq indicates the same as N b . We define two versions of contrastive losses , which both have three components : in-batch , negative location , and SimCSE . The self-supervized binary classification loss LBI is defined as LBIpXq “ LbatchBI pXq ` β1L negloc BI pXq ` β2L simcse BI pXq “ LBIpPx , N bq ` β1LBIpH , N sq ` β2LBIpPd , N dq ( 6 ) where β1 and β2 control the contribution of the last two loss components . The self-supervized multi-class classification loss LMC is defined as LMCpXq “ LbatchMC pXq ` α1L negloc MC pXq ` α2L simcse MC pXq “ LMCpPx , N b , τ0q ` α1LMCpPx , N s , τ1q ` α2LMCpPd , N d , τ2q ( 7 ) where α1 and α2 are hyper-parameters . Supervised Fine-Turning ( the blue box in Figure 1 ) . We directly use image labels in the training objective . Following Mac Aodha et al . ( 2019 ) , we used a presence-absence loss function which converts the multi-class labels into binary multi-labels . Given a set of training samples Xlabeled “ tpx , yqu , the loss function LsupervizedpXq is defined as : LsupervizedpXq “ βLBIpP l , Hq ` LBIpH , N lq ` LBIpH , N lsq ( 8 ) Here , β is a hyperparameter to enlarge the weight of positive samples , and the following positive and negative samples are used : • Labeled positives P l “ tpepxq , T : ,yq|px , yq P Xu . • Labeled negatives N l “ tpepxq , T : ,iq|px , yq P X , i P t1 .. cuztyuu . • Sampled negative locations N ls “ tpepx´q , T : ,iq|px , yq P X , i P t1 .. cuu , where x´ is one of the N evenly sampled locations from the surface of the sphere for each example x .

Summary: The paper studies how to use location information (geographical location) of images to help with image classification tasks. The paper studies use of spherical based location over GPS lat long and argues that the euclidean space does not preserve distances and has distortions whereas a spherical coordinate approach would not have the inherent problem. The approach demonstrates a framework to train the location encoding in an unsupervised way to with losses based on representation learning approaches. The paper studies multiple ways of expanding the spherical coordinates and evaluates their performance and empirically demonstrate the strength in their performance.