Sphere2Vec: Self-Supervised Location Representation Learning on Spherical Surfaces
Authors: Gengchen Mai
Yao Xuan
Wenyun Zuo
Yutong He
Stefano Ermon
Jiaming Song
Krzysztof Janowicz
Ni Lao
Venue: ICLR 2022 Submitted (2022)
Keywords: Self-Supervised Learning
Location Representation Learning
Double Fourier Sphere

Abstract: Location encoding is valuable for a multitude of tasks where both the absolute positions and local contexts (image, text, and other types of metadata) of spatial objects are needed for accurate predictions. However, most existing approaches do not leverage unlabeled data, which is crucial for use cases with limited labels. Furthermore, the availability of large-scale real-world GPS coordinate data demands representation and prediction at global scales. However, existing location encoding models assume that the input coordinates are in Euclidean space, which can lead to modeling errors due to distortions introduced when mapping coordinates from other manifolds (e.g., spherical surfaces) to Euclidean space. We introduceSphere2Vec, a location encoder, which can directly encode spherical coordinates while preserving spherical distances.Sphere2Vecis trained with a self-supervised learning framework which pre-trains deep location representations from unlabeled geo-tagged images with contrastive losses, and then fine-tunes to perform super-vised geographic object classification tasks.Sphere2Vecachieves the performances of state-of-the-art results on various image classification tasks ranging from species, Point of Interest (POI) facade, to remote sensing. The self-supervised pertaining significantly improves the performance ofSphere2Vecespecially when the labeled data is limited

Under review as a conference paper at ICLR 2022
Sphere2Vec: SELF-SUPERVISED LOCATION REPRESEN- TATION LEARNING ON SPHERICAL SURFACES
Anonymous authors Paper under double-blind review
ABSTRACT
Location encoding is valuable for multitude of tasks where both the absolute positions and local contexts (image, text and other types of metadata) of spatial objects are needed for accurate predictions. However, most existing approaches do not leverage unlabeled data, which is crucial for use cases with limited labels. Furthermore, the availability of large-scale real-world GPS coordinate data demand representation and prediction at global scales. Existing location encoding models assume that the input coordinates are in Euclidean space, which can lead to mod- eling errors due to distortions introduced when mapping coordinates from other manifolds (e.g., spherical surfaces) to Euclidean space. We introduce Sphere2 Vec, a location encoder, which can directly encode spherical coordinates while pre- serving spherical distances. Sphere2 Vec is trained with self-supervised learning framework which pre-trains deep location representations from unlabeled geo- tagged images with contrastive losses, and then fine-tunes perform supervised geographic object classification tasks. Sphere2Vec achieves the performances of sate-of-the-art results on various image classification tasks ranging from species, Point of Interest (POI) facade, to remote sensing. The self seff-supervieed pretraining significantly improves the performance of Sphere2 Vec especially when the labeled data is limited.
INTRODUCTION
Spatial information has become an important component to many machine learning tasks that address the environmental, economic, and societal needs for sustainable development. Just to name a few, these tasks include geo-aware species fine-grained recognition (Chu et al., 2019; Mac Aodha et al., 2019), Point of Interest (POI) facade image classification (Chu et al., 2019; Yan et al., 2018), remote sensing image classification (Christie al., 2018: Ayush et al., 2020), poverty prediction (Jean et al., 6; 2019), point cloud classification and semantic segmentation (Qi et al., 201 (a;b), traffic forecasting (Li et al., 2018; Cai et al., 2020), and geographic question answering (Mai al., 2019; 2020a). Developing general model for vector space representations of any point in space, termed a location encoder, would pave the way for many future applications that improves our well-being. Such location encoder can operate 2D space on other manifolds; of particular interest is the spherical surface of the Earth.
Here use geo-aware image classification shown in Figure and 2 as example. During training we assume a dataset D = {(4, X,, y;]li = L....,NI N). where I; is an image, Yt E 1,...., C' is the corresponding class label, x, represents the location (longitude and latitude) and optionally the time the image was taken'. The image information is often processed by an state-of-the-a image classification model (e.g.. Inception V3) predict the class labels as shown in Figure 1. Although the image itself is the most critical for image classification, sometimes it can be insufficient - the image of two distinct classes may look very similar both for the central object and its surrounding environment. For example, Figure 2a and 2f shows example images of two different fox species: Aretic fox (Vulpes lagopus) and bat-eard fox (Otocyon megalotis), for which the image model can be confused due to the high visual similarity of the two species. Fortunately, the geo-location information of an image may provide clue to their real classes. Figure 2b, 2g show that these two species have distinct spatial distribution patterns, and the image locations provide complementary information to the images. This
In this study focus the location information and leave the time aspect of modeling to future.

---

Under review as a conference paper at ICLR 2022
Image Supervised Training
Image1
Pretrered Clansher 07:0
P(y I
Unsupervised Training
Powisined Image Faatusn
Image Ernbedkingg
Exbrdor PL)
PSt
Profection
Magte
Infermncc
Contrnl
Loss
Prodictions
(A, ()
P())x)
d 2
Locatton Froonter
Incation Emitaciing Ersc
Locaton
Ene()
P-X
Location Supervised Training
Figure I: Applying Sphere2Vec to geo-aware image classification. The supervised image classification module (green) is InceptionV3 network similar to that of Mac Aodha ct al. (2019); the supervised location classification module (blue) can be Sphere2Vec or any other inductive location encoders (Chu et al., 2019; Mac Aodha et al., 2019; Mai et al., 2020b); the self-supervised training stage (orange) pre-trains the location encoder based on unlabeled image data. The dotted lines indicates that there prop-propergattin through these lines.
(a) Image
(b) Arctic Fox
(e) wrap
(d) grid
(c) sphereC'+
(i) Image
(g) Bat-Eared Fox
(h) wrap
grid
(i sphereeC++
Figure 2: Applying location encoders to differentiate two visually similar species; Arctic fox and bar-cared fox. (a) and (f) are their example images which look very similar. (b) and (g) show their distinct patterns in image locations. (c)-(): The predieted distributions of Vulpes lagopus from different location based models (without images input). (h)-(): The predicted distributions of Otocyon megalotis. While wrape (Mac Aodha al., 2019) produces a over-generalized species distribution, sphereC + (our model) produces more compact and fine-grain distribution on the polar region and in data sparse areas such as Africa (See Figure 2g-2j). grid (Mai et al., 2020b) is between the two. For more examples, please see Figure 10.
insight motivates the development of hybrid classification models, which take both the images and their locations as input (Mac Aodha et al., 2019; Mai et al., 2020b).
In spite of their effectiveness, hybrid models have an issue when representing points in non- non-Euclian Euclidean spaces. The demand on representation and prediction learning at a global scale grows dramatically due emerging global scale issues, such the transition path of the latest pandemic (Chinazzi et al., 2020), long lasting issue for malaria (Caminade et al., 2014), under threaten global biodi- versity(Di Marco al., 2019; Ceballos al., 2020), and numerous ecosystem and social system responses for climate change (Hansen & Cramer, 2015). However, when applying the state-of-the-art 2D space location encoders (Gao et al., 2019; Chu et al., 2019; Mac Aodha al., 2019; Mai et al., 2020b; Zhong et al., 2020) to large-scale real-world GPS coordinate datasets such as species images taken all over the world, map projection distortion problem (Williamson & Browning, 1973) emerges. These 2D location encoders are designed for preserving distance in 2D (or 3D) Euclidean space, while GPS coordinates are in fact on a spherical manifold, e.g., point x = (X, Φ) on sphere with longitude x [[T,, π)) and latitude @ [-T/2, - 2/2 (See Figure 1). Directly applying these 2D location encoders on spherical coordinates will yield a large distortion in the polar regions. Note that map projection distortion is unavoidable when projecting spherical coordinates into 2D space. This emphasizes the importance of calculating on a round planet (Chrisman, 2017). See Appendix 9.1 for more discussion.
2

---

Under review as a conference paper at ICLR 2022
From modeling perspective another key challenge is how to design the self-supervised training objective that effectively transfers the attribute information (c.g., image features) at different locations into a location encoder. This is very important in practice since collecting and labeling geo-tagged images particularly costly as annotations often require domain expertise. However, most existing location encoding approaches (Chu et al., 2019; Mac Aodha al., 2019; Mai et al., 2020b) are developed and trained in supervised learning framework while massive unlabeled geographic data cannot be used. Ayush al. (2020) proposed geographic self-supervised learning framework, but they mainly focus on training image encoders while the usage of geo-location information is limited data augmentation in the pre-training stage. In contrast, in this work, we focus on training location encoders in self-supervised manner so that geo-location information is explicitly use as part of the model input.
In this work, we propose Sphere2Vec as shown in Figure 1, which directly encodes point coordinates on a spherical surface while avoiding the map projection distortion. The multi-scale encoding method utilizes Double Fourier Sphere basis (O(S²) terms) or subset (O(S) terms) of it while still being able to correctly measure the spherical distance. Furthermore, we develop a self-supervised learn- ing framework which pre-trains deep location representations from unlabeled geo-tagged images by predicting image features or image identities base on their geo locations. We compare several self-supervised learning objectives including Mean Square Error loss (MSE), contrastive binary classification loss (BI), and contrastive multi-classiffcation loss (MC). The pre-trained Sphere Vec model can be fine-tuned (with an additional output layer) to perform supervised geographic objects classification tasks. We demonstrate the effectiveness of Sphere2 Vec on geo-aware image classifica- tion tasks including fine-grain species recognition (Chu al., 2019; Mac Aodha et al., 2019; Mai et al., 2020b), POI fascade image classification (Tang et al., 2015; Mac Aodha et al., 2019), and remote sensing image recognition (Christie et al., 2018; Ayush et al., 2020).
In summary, the contributions of our work are:
1. We develop a unified view of distant preserving location encoding methods on spheres based Double Fourier Sphere (DFS) (Merilees, 1973; Orszag, 1974).
2. We propose effecttive pre-training strategy for location encoders from unlabeled geo- tagged images by predicting image features and image identities based on their geo locations.
3. Sphere2 Vee improves the performances of state-of-the-art results on four species recognition datasets (BirdSnap, NABirds, iNar2017, iNar2018) 8), one POI fascade image classification dataset (YFCC), and one remote sensing image classification dataset (fMoW). It also demonstrate significant advantage in few shot learning settings.
2 PROBLEM FORMULATION
Following previous work (Mai et al., 2020b), Euclidean location encoding can be formulated as follows. Given set of points (e.g.. POI) P {p.) in L-D space (L = 2), define a function ex,o(x) : R - Rd (L & d), which is parameterized by 8 and maps any coordinate x in space to vector representation of d dimension. Each point (e.g., a restaurant) p; (x;, v;) is associated with a location x; and attributes v, (i.e., POI features such as types, names, capacity, images, etc.). The function ep, (xx) encodes the information about Point x which useful for certain tasks such as predicting a label Attributes and coordinates of points can be seen as analogies to words and word positions in commonly used word embedding models.
Similarly, we can formulate spherical location encoding as follows. Given a set of points P P= {p,} on the surface of a sphere $? e.g., species occurrences all over the world, where p; (x,, v.) and X; (As, (b:) $² indicates a point with longitude X; E [-T,, π) and latitude Φ; E [-m/2, π 7/2]. Define a function ex.)(x) : g - Rd, which is parameterized by 0 and maps any coordinate x in spherical surface 32
a vector representation of d dimension. See Appendix 9.1 for more discussions about the map projection distortion problem.
3 METHOD
The geo-aware image classification task (Chu et al., 2019; Mac Aodha et al., 2019) can be formulated as follows. Given an image taken from location/point x, we estimate which category y it belongs to by P[ x). If we assume I and conditionally independent given y, then based Bayes'
3

---

Under review as a conference paper at ICLR 2022
theorem, we have P x P(y|I) (See Mac Aodha al. (2019)). P(y can be given by any tfate-of-the-as image classifier such as Inception V3 (Szegedy et al., 2016) for the species recognition task or MoCo-V2+TP (Ayush et al., 2020) for the RS image classification task. Figure 1 illustrates the whole training and inference process by using species recognition as the example task. P(y(T) and P(y[x) are given by the image encoder (the green box) and location encoder (the blue box) respectively and are multiplied together for final prediction. In this work, we focus on estimating the geographic prior distribution of class y over the spherical surface P(y(x) (fe(x)TT..) where a is sigmoid activation function. T E Rdxe is class embedding matrix where the yea column T. Fd indicates the class embedding for class y.
In the following we introduce the two main contributions of our work - location encoding on spherical surfaces (the blue box in Figure 1), and self self-supervised training framework for location representation model (the orange box in Figure 1). In Section 3.1, we will discuss the design of spherical distance-kept location encoder e(x), Sphere2Vec. We developed a unified view distance- reserving encoding spheres based on Double Fourier Sphere (DFS) (Merilees, 1973; Orszag, 1974). In Section 3.2 we discuss training strategies for Sphere2Vec including self-supervised pre-training given unlabeled images, and supervised fine tuning. We use ep, (x) to indicate the spherical location encoding neural network with parameters 0, and will refer it by e(x) for simplicity.
3.1 Seeere2Vec ENCODER
We want to find a function ((aphere) (x) which does a one-to-one mapping from each point x; (A;, ob)) E S to multi-scale representation with S as the total number of scales such that it satisfies the following requirement:
(C(sphere) (x1), ((sphere) (x2)) S(AD), Vx1, X2 S
(1)
where ADe E [0. π RR] is the spherical surface distance between x X2, R is the radius of this sphere, and f(z) is a strictly monotonically decreasing function for x E [Ó, Tī.. In other words, we expect
to find a function (spherre (x) such that the resulting multi-scale representation x preserves the
spherical surface distance.
Similar to the case for Euclidean space (Mai et al., 2020b), any point x = (A, Φ) E S with longitude T 2. can be explicitly encoded in a multi-scale fashion in the
A E [-,,, π) and latitude @ E [- form of (x) = NN 5N (x)) where NN fn() is a learnable multi-layer perceptron with hidden layers and k neurons per layer. ((*((x) is a concatenation of multi-scale spherical spatial features of S levels. (+) indicates different types of spherical location encoders. In the following, we call e e(x)) (x) location encoder and its component ((*) position encoder. Let T'mnn Tmaz be the minimum and maximum scaling factor, and
Tmaz
sphereDFS Double Fourier Sphere (DFS) (Merilees, 1973; Orszag, 1974) is a simple yet successful
pseudospectral method, which has been applied to efficient analysis of large scale phenomenons such as weather (Sun et al., 2014) and blackholes (Bartnik & Norton, 2000). Our first intuition is to use the base functions of DFS to help decompose
(h, Φ) into a high dimensional vector:
S-1
S-1
GisphereD (x)
U [sin dn. cos Φr
U
[si Am, CoS Am
(2)
U [cos On cos Am, cos on sin Am sin on, cos Am, sin on, sin Amal-
Tmin (1) means vector concatenation and U$=1 indicates vector concatenation through different scales. It basically lets all the scales of á terms interact with all the scales of > terms in the encoder. This would introduce a position encoder whose output has (S2) dimensions and increase the memory burden in training and hurts generalization (Figure 3a).
where Vs = 0. O..., - 1, we define X -
II practice fix Tmaz meaning no scaling of X,
4

---

Under review as a conference paper at ICLR 2022
2 grad (c) sphereC
12 (d) sphereC+
22 A (e) sphereM
S
( sphere
Figure 3: Spectral representations of different encoders. Each point (A"", ; represent interaction terms of trioonometric functions the encoder. Points on the axis correspond to single terms with no interactions.
An encoder might achieve better result by only using subset of these terms. In comparison, the state of art encoder such as grid and wrap (Mac Aodha et al., 2019) define:
S-1 [sii o,. cos cos@, Φ.. sin X, cos Aa].
Glorid) (x)
We can see that grid employs subset of terms from sphereDFS (Figure 3b) and performs poorly at global scale due to its inability to preserve spherical distances (See Appendix 9.3 for proof).
We explore different subsets of DFS terms while achieving two goals: 1) efficient representation with (SS) dimensions 2) preserving distance measures on spheres. See Appendix 9.2 for the detail definition of sphereC, sphereC++, sphereM, sphere. +, and Figure for a unified view. We hypothesize that encoding these terms in the multi-scale representation would make the training of the encoder easier and the order of output dimension is still O(S).
In location encoding, the uniqueness of encoding (no two points on sphere having the same position encoding) is very important, (x*)(x) in the five proposed methods are one-to-one mapping.
(x)) is an Theorem 1. V (sphereC, sphereC+tsphereM, S,phreeM,,shhereee injective function. See the proof in Appendix 9.5.
3.2 TRAINING Sphere2Vec
Sell-Supervized Pretraining (the yellow box in Figure 1). We consider two contrastive objec- tives. The first is a binary classification loss, so called noise contrastive estimation (Gutmann & Hyvärinen, 2010), which avoids calculation of the partition function and has been successfully used in word embeddings (Mikolov al., 2013) and language modeling (Mnih & Teh, 2012):
(B)(P.N) - (aa.b)- a (s(a, bj)] Bla.b-)-w(oo((
((s(a,,b=))
(4)
where BT stands for "binary"" P is a set of positive pairs (denoted as b)), A a set of negative pairs (denoted as (a, b -). s(--)) is similarity function that assigns higher values when the inputs are more similar (such as cosine()), and o(v) = e²/(1 + e²) is the sigmoid function.
The second objective function is the multi-class classification loss with temperature, which has been successfully used in unsupervised learning for images (He et al., 2020) and text (Gao et al., 2021):
exp(s(a, b)/T)
(5)
exp(s(a, ())()()))/))))))))))))))))))))() + Eb-eN((ba) exp(s(a,
where MC stands for "multi-class", (b]a) obtains a negative pair with first entry being a: P and s(.,.) are defined as earlier. The temperature scaling parameter T determines how soft the softmax is (Hinton et al., 2015). In practice it help with the trade off between top ranked classes (precision) versus reset of the classes (recall).
In order to learn useful representationss we need to choose appropriate distributions for positive pairs P and negative pairs N. During the location encoder pretraining stage we use a geo-tagged but unlabeled image set Xanlabeled each image is represented by embedding generated from pretrained image extract extractor F² such as Inception V3 or MoCo- CCCTT (Ayush et al., 2020).
Given unlabeled image (x. Xunlabeled,
the following positive and negative instances:
As a multi-scale encoder, grid degenerates to wrop when the number of scales $ = 1.
F should see the current image labels during pretraining stage.
5

---

Under review as a conference paper at ICLR 2022
Geo-tagged positive p [(e(x). WF([))) where W is linear projection layer.
In-batch negatives N = {(e(x). WF([] x ; E B(x, Mx([x, 1) where (xx, 1) represents the batch of examples for which (x, I) is part of during training.
Sampled negative locations .\ {(e(x)), WF(I)) of size N, where x" is one of the evenly sampled locations from the surface of the sphere for each example x.
Dropout positive pd {(e(x)),''x))) (x))), where we pass the same input x to the encoder e() twice and obtain two embeddings as "positive pairs" by applying independently sampled dropout masks. This is data augmentation strategy (so called SimCSE), which has been very successful in generating sentence embeddings (Gao et al., 2021). We use e'(x) to denote the embedding from the second mask.
Dropout negative A/d {[e(x), e' (x (x") ]]([ same meaning as that defined in pd and B(x, I) indicates the same as No.
B(x, D(x,D)) Mxx 1),, where e'(+) has the
We define versions of contrastive losses, which both have three components: in-batch, negative location, and SimCSE. The self-supervized binary classification loss LBI is defined as
LBi (x) = Bi
Dbatch (X) + -1 Enegloc (X) + B1 Laimese (X)
(6)
LB " + 81 (BI + 32 (P
where and 32 control the contribution of the last loss components.
The supe-supe multi-class classification loss LMC is defined as
(xc(X)= = LMM
Chhtthh (X6) + 01 a1LMM
wwwgloo (x)) + a2 a2cMC
Caimese (x)
= LP ,N", TO) + a2LMc(("
+ azf (N',
where 01 and 0g are hyper-parameters.
Supervised Fine-Turning (the blue box in Figure 1). We directly use image labels in the training objective. Following Mac Aodha et (2019), we used a presence-absence loss function which converts the multi-class labels into binary multi-labels. Given set of training samples Xlabeled {(x, 3) the loss function [ Csupervized
(x) is defined as:
Csupervised
(x) (CB Ø) Crr(D,N") LBr(D,
(8)
Here, 3 is hyperparameter to enlarge the weight positive samples, and the following positive and negative samples are used:
Labeled positives pt = [(e(x), 9x, X}.
Labeled negatives A2 = ((x))),,)))(,,))) X,ie (I..))u))
Sampled negative locations A {(e(x"), T.,)) y) E X, {1..c)}, where x is one of the N evenly sampled locations from the surface of the sphere for each example x.
4 RELATED WORK
Multi-Scale Position/Location Encoding on Spheres Recent works various applications (Zhong al., 2020; Mai et al., 2020b; Mildenhall et al., 2020; Tancik et al., 2020) found that a simple sinusoidal mapping of input coordinates ("positional encoding") allows multilayer per- ceptrons (MLP) represent higher frequency content. More specifically fo is decomposed into fg NN E(x)). where PE is a deterministic function mapping x from R2d into higher dimen- sional space and NN; is simply MLP. This strategy is compelling since MLPs can be orders of magnitude more compact than grid-sample
representations in order to achieve similar order of details (Tancik et al., 2020). They show that passing input points through a simple Fourier feature mapping enables MLP to learn high- high-frequency functions in low-dimensional problem domains. This development is consistent with Rahaman al. (2019a) who show that deep networks are biased towards learning lower frequency functions, and mapping the inputs to higher dimensional space using high frequency functions enables better fitting of data that contains high frequency details. In this work we extend the success of this line of work to none Euclidean spaces such spherical surface. We leverage truncated Fourier transformation on spheres to achieve computation efficiency while avoiding the error caused by projection distortion.

---

Under review as a conference paper at ICLR 2022
Unsupervised Representation Learning Unsupervised text encoding models such trans- former (Vaswani et al., 2017: Devlin et al., 2019) has been effectively utilized in many Natural Language Processing (NLP) tasks. At its core, trained model encodes words into vector space representations based on their positions and context in the text. Following the success in NLP, there has been significen recent progress in unsupervised image pretraining (He et al., 2020; Caron et al., 2020; Ayush et al., 2020). Interestingly almost all of them are based on certain form of contrastive learning (Hadsell et al., 2006), which helps to construct unsupervised classification objectives from continuous inputs such as images. He et al. (2020) proposes Momentum Contrast (MoCo) for unsu- pervised visual representation learning. To increase the number of negative examples in contrastive training. they uses a queue of multiple mini-batches. Similar strategy has been adopted in NLP (Gao et al., 2021). To improve the encoding inconsistency between mini batches, they make the target image encoder parameterizes moving average of the query image encoder. In this work we are focusing on the pretraining of location encoder with a frozen image encoder. Our approach is very memory efficient (casily scaling up to 1024 batch size) and therefore avoid the need of multi-batch training. Ayush et al. (2020) combines a geo-location classification loss and a contrastive image loss similar MoCo during pre-training. For geo-location classification, deep image network is trained to predict a coarse geo-location of where in the world the image might come from. Since the geo-location is not part of the classifier's input, it is hard for the model to directly leverage in the prediction process. Zhai et al. (2018) learn location representation from unlabeled image-locatinn location pairs for image localization. They apply cross entropy loss to discretized location (or time), which cannot leverage the continuity of the approximated function. One main difference and also one of our main contributions our contrastive self-super loss, which avoids the need of discretization.
EXPERIMENT
5.1 THE EFFECTIVENESS OF SPHERICAL LOCATION ENCODING
To test the effectiveness of Sphere2Vec we first conduct geo-aware image classification experiments in full supervised setting (without the supe-supe pretraining step) seven large-scale real- world datasets including 5 (animal) species datasets, one POI dataset (YFCC), and one remote sensing dataset (fMoW). Please refer Mac Aodha al. (2019) and Christie et al. (2018) for the dataset descriptions. Table 1 compares the result of five variants of Sphere2Vec models against nine baseline models (see their detailed descriptions in Appendix 9.6. We can see that the Sphere2Vec models outperform baselines on all seven datasets, and the variants with linear number of DFS terms works as well as or even better than sphere Figure 2c-2e and 2h-2j show the predicted species distributions from different models for two fox species. We can see that wrap and grid have over- generalization issue in data sparse area (e.g., Africa) and the North Pole area. The distributions produced by sphereC'+ are more compact and generalizable.
To help understand the impact of Sphere2 we conduct detailed analysis on 7Nat2017 and iNat2018. Figure 4a shows the image locations in iNat2017 validation dataset. We split this dataset into different latitude bands and compare the AM RR between each model to grid. Figure 4b and 4c show the number of samples and AM in each band while Figure 4f shows that the contrast between these two variables for different models. We can see that Sphere2 models have larger AM R.R at the North Pole region (Figure 4c). Moreover, Sphere2Vec has advantages
bands with less data
samples, e.g. & E [-30%, - ). We also compare sphereC+ and grid at a quantized spatial resolution in latitude-longitude cells(Figure 4d and 4e). sphereC+ has more advantages in North Pole and data sparse cells. For more details please refer to Appendix 9.12, 9.13, and 9.14.
5.2 THE EFFECTIVENESS OF SELF-SUPERVISED PRETRAINING
Next, we investigate the impact of self-supervised contrastive pretraining (see Section 3.2). For species recognition and POI image classification task, we use the pretrained Inception-V V3 model provided by Py ToTchh as image feature extractor F in the yellow box of Figure 1. As for fMoW dataset, we use the unsupervised pretrained MoCo-V2+TP (Ayush et al., 2020) as F. Both models are supervised fine-tuned on the respective dataset to act as (FF()) in the green box of Figure 1. We consider three self-supervised training objectives mean square error loss MSE between the location embedding e(x) and the image feature IF (I), noise contrastive binary classification loss BI (see Equation 6), and contrastive multi-class classification loss MC (See Equation 7). In order to
7

---

Under review as a conference paper at ICLR 2022
Table I: Geo-aware image classification under fully supervised settings over three tasks: species recognition. POI image classification (YFCC), and remote sensing (RS) image classification (fMOW (Christic al., 2018)). tile indicates the results reported by Mac Aodha ct al. (2019). wraps indicates the original results reported by Mac Aodha al. (2019) while wrap is the best results we obtained by rerunning their code. See Appendix 9.6 for details about these baselines. Since the test sets for iNat2017 and iNat2018 not open-sourced, the baseline models and Sphere2Vec are highlighted as bold. All compared models use location only while ignoring time. The original result reported by Ayush el al. (2020) for No Prior on EMOW is 69.05. We obtain 69.8 69.84 by retraining their implementation. Here we report Topl accuracy. See Appendix 9.9 for more results. See Appendix 9.8.1 I for hyperparameter tuning details.
report results on validation sets. The best performance
Task
Species Recognition NABirdsf
POI
RS
Dataset
BirdSnap BirdSnapi
1Nar2017 INa22018
Avg
YFCC
IMOW
P(yl)) Prior Type
Test
Test
Test
Val
Val
Test
Val
No Prior (i.c. image model)
70.07
70.07
76.08
63.27
60.2
67.94
50.15
69.84
lile (Tang al., 2015)
70.16
72.33
77.34
66.15
65.61
70.32
50.43
ayz
71.85
78.97
81.2
69.39
71.75
74.63
50.75
70.18
wrape (Mac Aodha al., 2019)
71.66
78.65
81.15
69.34
72.41
74.64
50.70
wrap
71.87
79.06
81.62
69.22
72.92
74,94
50.90
70.29
wrap + ffn
71.99
79.21
81.36
69.4
71.95
74.78
50.76
70.28
rbf(Mai et al., 2020b)
71.78
79.4
81.32
68.52
71.35
74.47
51.09
70.65
f... et al., 2007)
71.92
79.16
81.3
69.36
71.8
74.71
50.67
70.27
grid (Mai et al., 2020b)
71.7
79.44
81.24
69.46
72.98
74.96
51.18
70.81
theory (Mai et al., 2020b)
71.88
79.24
81.59
69.47
73.01
75.04
51.16
70.82
sphereC
72.06
79.8
81.86
69.62
73.29
75.33
51.34
71.00
cphereC+
72.41
80.02
81.81
69.66
73.31
75.44
51.28
71.03
sphereM
72.02
79.75
81.69
69.69
73.25
75.28
51.35
70.98
sphereM+
72.24
80.34
81.9
69.67
73.72
75.57
51.24
71.10
sFhereDFS
71.75
79.18
81.39
69.65
73.24
75.04
51.15
71.46
(n) Validation Locations
(b) Samples per band
[c) AM RR per band
AM RR per cell
(e) AM RR per cell
LAMRR per & band
Figure 4: 7 dataset and model performance comparison: (a) Sample locations for validation set where the dashed and solid lines indicates latitudes; (b) The number of training and validation samples in different latitude intervals. (c) R.R difference between model and baseline grid the validation dataset. (d) AMRR MRR(sphereC++: positive and negative AM RR while darker color means high absolute value. number on each indicates the number of validation data points while "IK+" means there are more than IK points in a cell. (e) The number M (gigr per
M (Rrgidd)) for each longitd- cell. Red and blue color indicates
of validation samples v.s. AM RR = M RRR ((((sphereC+)) orange dots represent moving averages. (b) The number of validation samples v.s. AM per latitude hand.
cell. The
simulate the situations with limited labeled data we first perform self-supervised pretraining for e(x) with one of the three objectives using 100% of the unlabeled dataset X gantabeled (c.g.. iNatlist 2018). Then we perform stratified sampling obtain portion of the labeled data, denoted as Xr, while keeping its label distribution the same as the full training set. Finally we tne-tune the pretrained e(x) on Kr with the supervised learning objective (Equation 8).
8

---

Under review as a conference paper at ICLR 2022
Performance Comparison NNa2008
Model Performance Comparison iNa42018
1.4
Volsatorr
No Prix
Prizr
[sphoreM-1: Sip.
[rnoneg] Sup.
[rphareM-:: Sep MSE
[rneery]: Sip. MC Patt cast Capa
[sphareM=]: Sep. BI 2g Fi Di
bahonM J Stp.
(sshoMi J. Spp. MC- Ogt Cye
[ypheMM]]: Sip. & MC- Cal Cae
60
25
65. 06
5 Tokking Smpereed Trnning
Sacio Traiinn Labsh wod Sugerisaa Traning
(4) Comparing different loss functions
(b) Comparing different encoders e(x)
Figure 5: Comparing different self-supervised learning objectives C and different encoders e(x) in a label limited setting on iNar2018. e(x) is first trained on X xuntabeled with different self-supervised objectives, and then supervised trained with different amount training labels. Different curves indicates different self-supervised objectives L and different c(x) (denoted as "[I") indicated by the legend. The X axis indicates different training label ratio I and the Y axis indicates the Top 1 accuracy from the joint predict of image classifier and e(x). No Prior indicates standalone image classifier with no location encoder. Sup. indicates training location encoders with supervised learning loss. The shaded area along each curve indicates the standard deviations we get after running each setting for 5 times. (a) Comparison among five models with identical model architecture but different training objectives C on iNat2018 dataset. MSE, BI Chatch + + Enegloe Caimese and MC AMC Enegion + Liigese indicate training s with three different self-supervised objectives before supervised training. (b) Comparing the performance of sphereM and theory (the 2nd best model on iNat2018 dataset) in both supervised only (Sup.) and seff-supervieed plus supervised MCU settings. We can see that in both settings, sphereM + is able to outperform theory under different ratio r.
Figure 5a compares the model performance of sphere ++ on iNat2018 with different training objec- tives L. Each curve indicates the model performance of one training process in different supervised label ratios I' with identical e(x). From Figure 5a, we can see that compared with Sup. setting, all three self-supervised objectives can improve the model performances. The largest improvements hap- pen when I' is small (i.e., few shot learning setting), and we get a 3.9 % performance improvement when adding MC self-supervised training stage for = 0.05. BI and MC show competitive performance while MC is slightly better. MSE is less effective than the two. Furthermore, in order to show the impact of self-supervised training on different e(x), we compare the performance of sphereM and theory (the 2nd best model on iNat2018 under different C (See Figure 5b). We can see that MC can improve the performance of both exx) while sphereM+ + can outperform theory in either supervised or self-supervised + supervised settings.
We also perform a series of ablation studies on these self-supervised training objectives. Please refer to Appendix 9.10 for detail analysis. Moreover, we also compare MC and Sup. on the fMoW dataset and find that MC self self-supervised loss is also effective in the label limited learning setting on the fMoW dataset (more details Figure and Appendix 9.1
6 CONCLUSION
In this work, we propose a multi-scale spherical location encoder Sphere2Vec which can encode any location on the spherical surface into a high dimension vector. Moreover, we propose an self- supervised learning framework which pretrains location encoders based on unlabeled geo-tagged images. Experiment results on seven large-scal scale geo-aware image classification datasets shows that Sphere2 Vec can outperform multiple strong baselines. Moreover, we show that by adding the self-supervised pretraining stage, the performance of Sphere2Vec can be further improved, especially when labeled data is limited. In the future, we would like to explore the effectiveness of Sphere Vec on other machine learning tasks such as image geolocalization, geographic question answering, and so Moreover, only pretrain the location encoder while keeping the image encoder freeze during the self-supervised training stage. In the future, we would like to explore ways to pretrain dual encoders at the same time or early fusion of image and location data.
9

---

Under review as a conference paper at ICLR 2022
7 ETHICS STATEMENT
All datasets utilized in this work are publicly available. We only do some data preprocessing such as image feature extraction. No new information collected.
We find out that the geographic coverage of most datasets used in this work such as BirdSnap, NABird, iNat2017, iNat2018, YFCC, and fMoW are mostly concentrated on North American, Europe while they have poor geographic coverage on other regions and countries such Russia, China, Africa, and so on (See Figure 4a). This kind of geographic bias will also affect the machine learning models trained on them shown in Figure 4d. This paper shows the our Sphere2 Vec is more robust than the state-of-the-ar models for such bias. However, how to systematically handle such geographic bias should be an interesting research direction.
REPRODUCIBILITY STATEMENT
The proposed models as well as preprocessed datasets in this paper will be made public available through GitHub.
REFERENCES
Benjamin Adams, Grant McKenzie, and Mark Gahegan. Frankenplace: interactive thematic mapping for ad hoc exploratory search. Proceedings of the 24th international conference on world wide web, pp. 12-22. International World Wide Web Conferences Steering Committee, 2015.
Ivan Anokhin, Kirill Demochkin, Taras Khakhulin, Gleb Sterkin, Victor Lempitsky, and Denis Korzhenkov. Image generators with conditionally-indepeeeeee pixel synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14278-14287, 2021.
Kumar Ayush, Burak Uzkent, Chenlin Meng, Kumar Tanmay, Marshall Burke, David Lobell, and Stefano Ermon. Geography-aware self-supervised learning. arXiv preprint 2tt:::20/10099800 2020.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 1607.0600.06455 2016.
Robert Bartnik and Andrew Norton. Numerical methods for the einstein equations null quasi- spherical coordinates. SIAM Journal Scientific Computing, 22:917-950, 03 2000.
Ronen Basri, Meirav Galun, Amnon Geifman, David Jacobs, Yoni Kasten, and Shira Kritchman. Frequency bias in neural networks for input of non-uniform density. In International Conference on Machine Learning, pp. 685-694. PMLR, 2020.
Thomas Berg, Jiongxin Liu, Seung Woo Lee, Michelle L Alexander, David W Jacobs, and Peter N Belhumeur. BirdSnap: Large-scale fine-grained visual categorization of birds. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 201 2018. 2014.
Ling Cai, Krzysztof Janowicz, Gengchen Mai, Bo Yan, and Rui Zhu. Traffic transformer: Capturing the continuity and periodicity of time series for traffic forecasting. Transactions in GIS, 24(3): 736-755, 2020.
Cyril Caminade, Sari Kovats, Joacim Rocklov, Adrian M. Tompkins, Andrew P. Morse, Felipe J. Colón-González, Hans Stenlund, Pim Martens, and Simon J. Lloyd. Impact of climate change on global malaria distribution. Proceedings of the National Academy of Sciences, 111(9):3286 3291, 2014. ISSN 8424. doi: 10.1073/pnas. http:/paas..30089999 1302089111. URL htttps://wer pnas org/ content/lll. 3266.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Hugo Larochelle, Mare Aurelio Ranzato, Raia Hadsell, Maria-Florins Balcan, and Hsuan- Socan-Tien Lin (eds.), Advances in Neural Information Processing Systems, 2020.
10

---

Under review as a conference paper at ICLR 2022
Gerardo Ceballos, Paul R. Ehrlich, and Peter H. Raven. Vertebrates the brink indicators of biological annihilation and the sixth mass extinction. Proceedings of the National Academy of Sciences, 2020. ISSN 0027-8424. doi: 10. htt.000///aaa.99228...
URL https: //wp::///wwww////// pnas.
org/content//enn eerr wtteoooteeeeeaay////// httpl///222222222
922686117.
Matteo Chinazzi, Jessica T. Davis, Marco Ajelli, Corrado Gioannini, Maria Litvinova, Stefano Merler, Ana Pastore y Piontti, Kunpeng Mu, Luca Rossi, Kaiyuan Sun, Cécile Viboud, Xinyue Xiong, Hongjie Yu, M. Elizabeth Halloran, Ira M. Longiii, and Alessandro Vespignani. The effect of travel restrictions the spread of the 2019 novel coronavirus (covid-19) outbreak. URL Science, 368(6489):395-400 2020. ISSN 0036-8075. doi: 10.1126/science...e /scienceeeeeceeeeeeeeeeeee sciencemag httg://oneettt366/6///
Nicholas Chrisman. Calculating on round planet. International Journal of Geographical Information Science, 31(4):637-657,201 2017.
Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6172-6180, 2018.
Grace Chu, Brian Potetz, Weijun Wang, Andrew Howard, Yang Song, Fernando Brucher, Thomas Leung. and Hartwig Adam. Geo-aware networks for fine grained recognition. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 0-0, 2019.
Jacob Devlin, Ming- Beig-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL- pp- 4171 1171-4186, 2019.
Moreno Di Marco, Simon Ferrier, Tom D. Harwood, Andrew J. Hoskins, and James E. M. Watson. Wilderness areas halve the extinction risk terrestrial biodiversity. Nature, 573(7775),582-555 2019. ISSN 1476-4687. doi: 10. 101111111186--19-111111
URL https: btttps://do..oo /ttp:///ooooooo///////
1038/a41588 019- 1567-7.
Ruiqi Gao, Jianwen Xie, Song-Chus Zhu, and Ying Nian Wu. Learning grid cells as vector rep- resentation of self-position coupled with matrix representation of self-motion. In International Conference on Learning Representations, 2019.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings. In EMNLP 2021, 2021.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. volume 9 of Proceedings of Machine Learning Research, pp. 297-304, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning invariant mapping. CVPR '06, pp. 1742- USA, 2006. IEEE Computer Society.
Gerrit Hansen and Wolfgang Cramer. Global distribution of observed climate change impacts. Nature Climate Change, 5(3):18--188 185, 2015. doi: 10.10388mclimate2522, URL https::t///// /tttpp://doi.rrgg/ 10.1038 http///mmmma/2///
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 9726-97277779 7735,, 2020.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. CoRR, abs/1503.02531 2015. URL http: arxiv. oorg///ab... /100///03.... 02531.
Mike Izbicki, Evangelos E Papalexakis, and Vassilis J Tsotras. Exploiting the earth's spherical geometry to geolocate images. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 3-19. Springer, 2019.
Neal Jean, Marshall Burke. Michael Xie, W Matthew Davis, David B Lobell, and Stefano Ermon. Combining satellite imagery and machine learning to predict poverty. Science, 353(6301) 2016.
11

---

Under review as a conference paper at ICLR 2022
Neal Jean, Sherrie Wang, Anshul Samar, George Azzari, David Lobell, and Stefano Ermon. Tile2vec: Unsupervised representation learning for spatially distributed data. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 3967-3974, 2019.
Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusson convolutional recurrent neural network: Data-driven traffic forecasting. In International Conference Learning Representutions, 2018.
Oisin Mac Aodha, Elijah Cole, and Pietro Perona. Presence-only geographical priors for fine-grained image classification. In Proceedings of the IEEE International Conference on Computer Vision, pp. 9596-9606, 2019.
Gengchen Mai, Bo Yan, Krzysztof Janowicz, and Rui Zhu. Relaxing unanswerable geographic questions using a spatially explicit knowledge graph embedding model. In AGILE: The 22nd Annual International Conference on Geographic Information Science, pp. 21-39. Springer, 2019.
Gengchen Mai, Krzysztof Janowicz, Ling Cai, Rui Zhu, Blake Regalia, Bo Yan, Meilin Shi, and Ni Lao. SE-KGE: A location-awarr knowledge graph embedding model for geographic question answering and spatial semantic lifting. Transactions in GIS, 2020a. doi: 10.111 1/tgis.12629. 12629.
Gengchen Mai, Krzysztof Janowicz, Bo Yan, Rui Zhu, Ling Cai, and Ni Lao. Multi-scale repre- sentation learning for spatial feature distributions using grid cells. In The Eighth International Conference on Learning Representations. openreview, 2020b.
Philip E. Merilees. The pseudospectral approximation applied to the shallow water equations on a sphere. Atmosphere, 11(1):13-20, 1973. doi: 10. 10.1080000046973.11000
Tomas Mikolov, Πla Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pp. 3111 19, 2013.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020.
A Mnih and YW Teh. A fast and simple algorithm for training neural probabilistic language models. In Proceedings of the 29th International Conference on Machine Learning, ICMI. 20/2, volume 2, 2012.
Karen A Mulcahy and Keith C Clarke. Symbolization of map projection distortion: a review. Cartography and geographic information science, 1873)11671182, 2001.
Tu Dinh Nguyen, Trung Le, Hung Bui, and Dinh Phung. Large-scale online kernel learning with ran- dom feature reparameteriization. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, pp. 2543-2549, 2017. doi: 10.249633/ccaa.22117/33.
Steven A. Orszag. Fourier series on spheres. Mon. Wea. Rev, 102:56-75, 1974.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. PointNet: Deep learning point sets for 3D classification and segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 652-660, 2017a.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. PointNet++: Deep hierarchical feature learning on point sets in a metric space. In Advances in Neural Information Processing Systems, pp. 5099-5108, 2017b.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5301-5310. PMLR. 09-15 Jun 2019a
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference Machine Learning, pp. 5301-5310. PMLR, 2019b.
12

---

Under review as a conference paper at ICLR 2022
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In J. C. Platt, D. Koller, Y. Singer, and S. T. Roweis (eds.), Advances in Neural Information Processing Systems, pp. 1177-1184. Curran Associates, Inc., 2008.
Ali Rahimi, Benjamin Recht, et al. Random features for large-scale kernel machines. In NIPS, volume 3, pp. 5. Citeseer, 2007.
Cheng Sun, Jianping Li, Fei-Fei Jin, and Fei Xie. Contrasting meridional structures of stratospheric and tropospheric planetary wave variability in the northern hemisphere. Tellus A: Dynamic Meteorology and Oceanography, 66(1):25303, 2014.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-28888 1-26,,, 2016.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 7537-7547. Curran Associates, Inc., 2020.
Kevin Tang, Manohar Paluri, Li Fei-Fei, Rob Fergus, and Lubomir Bourdev. Improving image classification with location context. In Proceedings of the IEEE international conference computer vision, pp. 1016-1016, 2015.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pp. 5998 6008, 2017.
David Williamson and Gerald Browning. Comparison of grids and difference approximations for numerical weather prediction over a sphere. Journal of Applied Meteorology, 12:264-274, 02
1973.
Bo Yan, Krzysztof Janowicz, Gengchen Mai, and Rui Zhu. XNet+SC: Classifying places based on images by incorporating spatial contexts. In 10th International Conference on Geographic Information Science (GIScience 2018). Schloss Dagstubl-Lecibiiz-znnnn
fuer Informatik, 2018.
Menghua Zhai, Tawfiq Salem, Connor Greenwell. Scott Workman, Robert Pless, and Nathan Jacobs. Learning geo-temporal image features. In British Machine Vision Conference (BMVC), 2018.
Ellen D Zhong, Tristan Bepler, Joseph H Davis, and Bonnie Berger. Reconstructing continuous distributions 3d protein structure from cryo-em images. In International Conference on Learning Representations, 2020.
13

---

Under review as a conference paper at ICLR 2022
Ga) Mercator
(b) Miller
(c) Sinusoidal
(d) Equirectangular
Figure 6: An illustration for map projection distortion: (a)-(d): Tissot indicatrices for four projections. The equal area circles are putted in different locations show how the map distortion affect its shape.
APPENDIX
9.1
THE MAP PROJECTION DISTORTION PROBLEM
In fact there is map projection, which preserve distances in all directions. The so-called equidistant projection can only preserve distance on one direction, e.g., the longitude direction for the equirectangular projection (See Figure 6d), while the conformal map projections (See Figure 6a) can preserve directions while resulting in a large distance distortion. For a comprehensive overview of map projections and their distortions, see Mulcahy & Clarke (2001). This is a well recognized problem in Cartography which shows the importance of calculating on a round planet Chrisman (2017). Due to the limitations these 2D location encoders, there is an urgent need for a location encoding method which preserves the spherical distance (e.g., great circle distance") between two points. The multi-scale encoding method utilizes Double Fourier Sphere basis (O(S2) terms) or a subset (O(S) terms) of it while still being able to correctly measure the spherical distance. This inspires us to explore the most effective subset Fourier bases on spheres.
9.2
sphereC Inspired by the fact that any point (z, y, 2) in 3D Cartesian coordinate can be expressed by sin and cos basis of spherical coordinates (A, & plus radius) 6, define the basic form of Sphere2 namely sphereC encoder, for scale 8 as
Glsphere (x)
U s=0
ssin ds. cos o. cos As cos d, sin Xa].
can shown that when 1, G(ophereC) (x) directly satisfies our expectation Equation where f(x) cos( F) See more detailed analysis and comparison to the grid encoder in Appendix 9.3.
sphereM Considering the fact that many geographical features more sensitive to either latitude (e.g., temperature, sunshine duration) or longitude (c.g., timezones, geopolitical borderlines), we might want focus increasing the resolution of either ó or À while the other is hold relatively at large scale. Therefore, we introduce a multi-scale position encoder sphereM, where interaction terms between & and À always have one of them fixed at top scale:
S-1 Ussi dh, cos ds cos X, cos & cos As₁ cos dh sin X, cos øsin A,). (10)
Glsphere (x)
This new encoder ensures that the term interact with all the scales of À terms and À term interact with all the scales of o terms. Note that when S = 1, ((sphereM)) is equivalent to G(ephereC), Both sphereC and sphereM are multi-scale versions of a spherical distance-kept encoder (See Equation 12) and keep that the main term in their multi-scale representation.
hhttps:/ : wik httt:::://ewiixippiia.ooro.g /oce: Phttps://en wik www.wwwwwwwwiiiccmm hhttess//en...iiipppiii....
14

---

Under review as a conference paper at ICLR 2022
sphereC+ and sphereM+ From the analysis of the two proposed encoders and the state-of-the tf-tthe-rtt grid encoders (Appendix 9.3), we know that grid pays more attention to the sum of cos difference of latitudes and longitudes, while our proposed encoders pay In order to capture both information, we consider merging grid with each proposed encoders to get more powerful models that encode geographical information from different angles.
attention to the spherical distances.
GlaphereC+) (x) G(ophere (x) U Glarid) (x);
(11)
G(sphereM++) = G(ophereM) (xx) U Glarid) (x).
9.3 ANALYSIS OF sphereC AND ITS COMPARISON TO THE grid ENCODER
To illustrate that sphereC good at capturing spherical distance, we take a close look at its basic case S = 1 (define s = O and fa = 1). where the multi-scale encoder degenerates to
(12) These three terms are included in the multi-scale version (S > 1) and serve as the main terms the largest scale and also the lowest frequency (when 8 5 1). The high frequency terms are added to help the downstream neuron network to learn the point-feature more efficiently. Interestingly, G(sphere()) captures the spherical distance in a very explicit way:
(lsphere (x) = [sin(0), cos( cos( A), cos(() sin(
Theorem 2. Let X1, X2 be two points on the same sphere with radius R, then (x11), (G(sphereC) (x1), G(sphere (xx2) = cos(
AD
)
(13)
R
where AD is the great circle distance between X and X2. Under this metric,
(G(sphera (x1) ((spherol) (22)l 2sin(
AD
(14)
2R
Moreover, (S(sphere (x1) ((spherel)) (xx)]
AD when AD is small W.E.. R.
See the proof Appendix 9.4. Since the central angle Ad-
AD [0,m] and cos(x) is strictly R monotonically decrease for x E [0, T], Theorem 2 shows that ((sphereC)) (x) directly satisfies our
expectation Equation where J' f((x) = cos( #)) In comparison, when 1, the inner product in
the output space of grid encoder is
(Glorid) (x1), G(srid) (x²)) cos(ch )2) cos(A1 X2),
(15)
which models the latitude difference and longitude difference of x1 and x2 separately rather than spherical distance. This introduces problems in encoding. For instance, consider data pairs x,-
()1,6) Φ) and X2 (A2, Φ), the distance between them output space of grid, |[(lridd) (x1)
Glarid) (x²) = 2 2 cos(>1 22) stays as a constant in terms of Φ. However, when á varies from to 5. the actual spherical distance changes in wide range, c.g., the actual distance between the data pair at Φ = 2 (South Pole) is O while the distance between the data pair at Φ = 0 (Equator), gets the maximum value. This issue in measuring distances also has negative impact ability to model distributions in areas with sparse sample points because it is hard to learn the true spherical distance. We observe that grid reaches peak performance at much smaller Tmin than that of Sphere2 Vec encodings. Moreover, sphereC outperforms grid near polar regions where grid claims large distance though the spherical distance is small (A, B in Figure 1).
grid's
9.4 PROOF OF THEOREM
Proof. Since G(sphere (xx:) [sin():), cos(())) cos(A:), cos( sin(A:)) for i 1,2, the inner product
(Glephere (x)), ((epphere))xxa))
sin((1)) sin( 02) + cos( (01) cos(Ar) cos((z) cos(Az) + cos(Ø1) sin()1) cos( sin(>e) (16)
= =in(Ø1)) sin 02) + (01) cos( cos(Ar (11 - Az)
-co(AA) = cos( AD/R R))
15

---

Under review as a conference paper at ICLR 2022
where Ad is the central angle between x; xjand X2, and the spherical law of cosines is applied to derive the second last equality. So,
[((xx)) ( ((x)))
= =(G(xx) G(x2), G(xi) G(x2))
(17)
= coss AD/R)
So [[(x)) ((x2) 2sin(AD/2R) since AD/2R E [0, 2] By Taylor expansion, [|G(x))
G(x)) AD/R R when AD is small w.r.t. R.
9.5 PROOF OF THEOREM
V (sphereC, spheree++ sphereM, sphereM++), (x1) (x2) implies
sin(Ø1) sin(d2).
(18)
cos(Ø)) sin(A1) (1) = cos( sin(>2),
(19)
cos( cos A1) = cos cos( (22).
(20)
from terms. Equation 18 implies Φ1 = d2. If Φi = 62 7/2, then both points are at North
Pole, A A equal to whatever longitude defined North Pole. If Φ1 = d2 - it is similar
case at South Pole. When 61 = de (- 2: ) cos( = cos(Ø2) # O. Then from Equation 19 and 20,
sin A = sin() cos(A1) cos(A2),
(21)
which shows that A X2. In summary, X1 X2. so ((+) injective.
If * sphereDFS, (x,) g (x2) implies
sin(d)) sin( cos( cos(d2), (in(d)),si((A)) sin(A2), cos(A1) cos(A2),
(22)
which proves x1 X2 and (+) is injective directly.
9.6 BASELINES
In order understand the advantage of cathereicalddsttnneee location encoders, we compare different versions of Sphere2Vec Vec with multiple baselines:
No Prior indicates a image classifier without using any location information, i.e., predicting image labels purely based on image information (y)
tile divides the study area A (e.g.. the earth surface) into grids with equal intervals along the latitude and longitude direction. Each grid has an embedding to be used as the encoding for every location x fall into this grid. This is a common practice by many previous work when dealing with coordinate data (Berg et al., 2014; Adams et al., 2015; Tang et al., 2015).
wrap is location encoder model introduced by Mac Aodha al. (2019). Given location x (A, Φ), it uses coordinate wrap mechanism to convert each dimension of x into
2 numbers (wwrop) (x)
[sin(= 180 ), coss(T 180
X }² sin( π 90
), c,sos(T _
)- Then the
results are passed through multi-layered fully connected neural network NN () which consists of initial fully connected layer, followed by a series of h residual blocks, each consisting of two fully connected layers (k hidden neurons) with dropout layer in between. We adopt the official code Mae Aodha al. (2019)? for this implementation.
hhtt::/www.visiine wwt.:/www.iiii..oooo.o tech ht/e
16

---

Under review as a conference paper at ICLR 2022
wrap ffn is similar to wrap except that it replaces NN () with NN ffn O, simple learnable multi-layer perceptron with h hidden layers and & neurons per layer as that Sphere2Vec has. wrap + fn is used to exclude the effect of different NN(( on the performance of location encoders. In the following, all location encoder baselines use ffn( () as the learnable neural network component so that we can directly compare the effect of different position encoding G(-) on the model performance.
JN
zyz first converts x; = (As, de) E S into 3D Cartesian coordinates (r, y, =) centered at the sphere center by following Equation 23 before feeding into multilayer perceptron NN Here, we let (z. y, 2) to locate on a unit sphere with radius R = 1. zya can be treated as a special case of sphereC where S 1.
(x) = (2,0,, v] = [sin Φ cos & cos X, cos Φ sin X]
(23)
rbf randomly samples M points from the training dataset
RBF anchor points
[x [xanchor, m 1...M), and use gaussian kernels exp ( || x; xanchor
I|?
x
each
20
anchor points, where is the kernel size. Each point pt has -ddimension RBF feature vector which is fed into fn() to obtain the location embedding. This a strong baseline for representing floating number features in machine learning models.
rff. i.e., Random Fourier Features (Rahimi & Recht, 2008; Nguyen et al., 2017), first en- codes location into D dimension vector (x) (x()
v/2 v [cos (wff x + b:)]e=
i.j.d where wi ~ N(0, T) and b; is uniformly sampled from [0, 2m]. ]. I is an identity matrix. Each component of sx(x) first projects x into a random direction w; and makes a shift by b; Then it wraps this line onto the unit cirle in R²² with the cosine function. Rahimi & Recht (2008) show that xp(x) xx(') is an unbiased estimate of the Gaussian kernal K(x,x'). pp() is consist of D different estimates to produce a further lower variance approximation. To make f comparable to other baselines, we feed p(x) into (a(( to produce the final location embedding.
grid is a multi-scale location encoder on 2D Euclidean space proposed by Mai et al. (2020b). Here, we simply treat x = (A, Φ) as 2D coordinate. It first use G(arid((x) (x) shown in Equation 3 to encode location x into multi-scale representation and then feed it into NN ffx() to produce the final location embedding.
theory is another multi-scale location encoder on 2D Euclidean space proposed by Mai et al. (2020b). It use position encoder G (x) shown in Equation 24.. Here, x² [>a, c..] Ş 7 and a₁ = [1, O1. a2 = [-1/2, v 2] as [-1/2, -V322 R² are three unit vectors which orient 2//33 apart from each other. The encoding results are feeinto NN (n( to produce the final location embedding.
$ 8-1 3
((theory) (x) = Ulinki((x a,)),, cos((xx, a,))]]
(24)
==0 j-1
9.7 MODEL IMPLEMENTATION DETAIL
All types of Seh as well as all baseline models we compared share the same model set up (()) xx) NN (a(((((( (x)). Here. (-))() position encoder which preprocesses the input coordinates x = (X, Φ) E @ and outputs a position embedding (x). There are no learnable parameter ((+) (x). The position encoders (-+)(-) used for different Sphere2Vec are discussed in detail in Section 3.1 while (++)(.) of all baselines are discussed in Appendix 9.6. We can see that (.)(() (.) of grid, theory and different types of Sphere2Vec encode the input coordinates in a multi-scale fashion by using different sinusoidal functions with different frequencies. Many previous work call this practice "Fourier input mapping" (Rahaman et al., 2019b; Tancik et al., 2020; Basri et al., 2020; Anokhin et al., 2021). The difference is that grid and theory use the Fourier features from 2D Euclidean space while our Sphere2 Vee uses all or the subset of Double Fourier Sphere Features take into account the sphereical geometry and the distance distortion it brings.
17

---

Under review as a conference paper at ICLR 2022
Except for wrap, fa used by all the other location encoders is learnalle multi-layer percep- tron (MLP) with hidden layers and neurons per layer. We use the original MLP implementation provided by Mai al. (2020b). Here, each layer of NN ff. is linear layer followed by a dropout layer, a nonlinear activation function, a layer normalization layer (Ba et al., 2016), and a skip con- nection. Different from other location encoders, wrap uses a different implementation of the full connected layer NN detailed implementation of NN (.).
ffn (-), Please refer to Appendix 9.6 and Mac Aodha et al. (2019) for the
All models are implemented in PyTorch. We use the original implementation of wrop from Mac Aodha et al. (2019) and the implementation of grid and theory from Mai et al. (2020b). We train/evaluate each model on a Ubuntu machine with 2 GeForce GTX Nvidia GPU cores, each of which has 10GB memory.
9.8 MODEL TRAINING DETAIL AND HYPERPARAMETER TUNING
To train different types of Sphere2Vec as well as baseline models, we use Adam as the optimizer in both unsupervised and supervised training phase, Basically, we first do unsupervised training with an initial learning rate Thansuper by following either Equation 6 or 7 as the unsupervised training objective. In this setting, we predict the corresponding image embedding based on the input location. Only location encoder is trained during the unsupervised training phase while we keep the image encoder freezed as shown in Figure 1. Here, we just use a image encoder pretrained on other image datasets such ImageNet and use it image feature extractor F. After the model is convergence, we use a new learning rate Nsuper to do supervised training by following Equation 8.
9.8.1 FULL SUPERVISED TRAINING SETTING
In order test the effectiveness of our spherical encoding method Spheree we train all location encoders in a fully supervised setting. In order to find the best hyperparameter combinations for each model on each dataset, we use grid search do hyperparameter tuning including supervised training learning rate Nsuper = [0.01, 0.005, 0.002, 0.001, 0.0005, 0. 0.00005], the number of scales s [16, 32, 64]. the minimum scaling factor Timn [0.10.050 www.:0200000000.0000
the number of hidden layers and number of neurons used in NN f (-) - h = [1, 2, 4] and A
[256, 512, 1024], the dropout rate in NN fn(+)) - dropout [0.1,0.2,00 0.3, 0.4, 0.5, 0.7]. We also test multiple options for the nonlinear function used for NN (-)(() including ReLU, LeakyReLUU, and Sigmoid. The maximum scaling factor Tmaz can be determined based on the range of latitude e and longitude A. For grid and theory, we use Tmaz 360 and for all See,,e2Vec, we use Tman 1. As for rbf and fff. f. we also tune their hyperparamaters including kernel size o = [0.5, 1, 2, 10] as well as the number of kernels M [100, 200, 500].
Based on hyperparameter tuning. we find out using 0.5 as the dropout rate and ReLU as the nonlinear activation function for tuning, the best hyperparameter combinations are selected for different models on different datasets. The best results are reported in Table 1. Note that each model has been running for 5 times and its mean Top1 score is reported. Due to the limit of space, the standard deviation of each model's s performance on each dataset is not included in Table However, we report the standard deviations of all models' performance on three datasets in Appendix 9.9.
works best for every location encoder. Based on the hyperparameter
Moreover, we find out Nepрer and Tmán are the most important hyperparameters. Table 2 shows the best hyperparameter combinations of different Sphere2Vec Vec models on different species image classification dataset. We use a smaller S for sphereDFS since it has O(S2) terms while the other models have O(S) terms. sphereDFS with S = 8 yield a similar number of terms to the other models wth S = 32 (See Table 3). Interestingly, all first four Sphere2 Vec models (sphereC, sphereC'+, sphereM,, and sphereM+) shows the best performance on all five datasets with the same hyperparamter combinations. Note that compared with other datasets, iNat2017 and iNat2018 more up- to-te-d datasets with more training samples and better geographic coverage. This indi- cates that the proposed 4 Sphere Vec models show similar performance over different hyperparameter combinations.
18

---

Under review as a conference paper at ICLR 2022
The best hyperparameter combinations of Sphere2Vec models different image classification datasets. The learning rate Tepuer tends smaller for larger datasets; The best S 8 for sphereDFFS and 32 for all others; and we fix the maximum scale Tmaz as 1. Here, Tmin indicates the minimum scale. /t and & are the number of hidden layers and the number of neurons in NN() respectively.
Table 2:
Dataset
?super
Tmin
BirdSnap
0.001
10 6
512
BirdSnapt
0.001
10-1
1024
NABirdsf
0.001
10-4
1024
iNat2017
0.0001
10 -2
1024
iNat2018
0.0005
3
1024
Table 3: Dimension of position encoding for different models in terms of total scales $
Model
sphereC
sphereC+
sphereM
sphereM+
sphereDFS
Dimension
3S
65
58
85
45a + 4
We also find out that using a deeper MLP as NN (fn(), (.), ie. a larger h does not necessarily lead to better classification accuracy. In many cases, one hidden layer h = achieves the best performance for many kinds of location encoders. We discuss this detail in Appendix 9.9.
9.8.2 UNSUPERVISED TRAINING SETTING
In order to test the effectiveness of different unsupervised learning objectives, we also conduct another set experiments by unsupervised training different location encoders before the supervised training phase. In addition to the hyperparameter the hyperparameters used in the unsupervised learning phase including the unsupervised training learning rate Nunsuper, the negative location loss weight 01, SimCSE loss weight 02 the number of negative locations we used | three temperature To. Ti, and T2. As for unsupervised objective LBI. we also fine tune 31 and 32. The hyperparameters have searched and the hyperparameter tuning results for the unsupervised learning setting are shown in Table 5, 6, 7 in Appedix 9.10.
discussed in Section 9.8.1, 1, we also fine tune
9.9 MORE EXPERIMENTAL RESULTS GEO-AWARE IMAGE CLASSIFICATION TASKS
Table 4 is a complementary of Table 1 which compares the performance of sphereM+ with all baseline models on the geo-aware image classification task. The results on three datasets are shown here including BirdSnapt. NA NABirdsf, and iNat2018. For each model, we vary the depth of its NN /N/5n(). i.., = [1, 2,3, 3, 4]. The best evaluation results with each h are reported. Moreover, we each model with one specific h 5 times and report the standard deviation of the Top1 accuray, indicated in "0". Several important observations can be made based on Table 4:
Although the absolute performance improvement between sphereM + and the best baseline model is not very large- 0.90%, 0.28%, and 0.71% for three datasets respectively, these performance improvements are statistically significant given the standard deviations of these Topl scores.
2. These performance improvements are comparable those from the previous studies on the same tasks. In other words, the small margin is due to the nature of these datasets. For example, Mai al. (2020b) showed that grid or theory has 0.79%, 0.44% absolute Topl accuracy improvement on BirdSnapi and NABirdst dataset respectively. Mac Aodha et al. (2019) showed that wrap has 0.09%, 0%, 0.04% absolute Top1 accuracy improvement on BirdSnap, BirdSnapt and NABirdst dataset. Here, we only consider the results of wrop that only uses location information, but not temporal information. Although Mac Aodha et al. (2019) showed that compared with tile and nearest neighbor methods, wrop has 3.19% and 3.71% 1% performance improvement on iNat2017 and iNat2018 datatset, these large margins are mainly because the baselines they used are rather weak. When consider the typical rbf and rf f (Rahimi et al., 2007) used in our study, their performance improvement are down to -0.02% and 0.61 %.
19

---

3. By comparing the performances of the same model with different depth of its NN ffn O. i.e., h, we can see that the model performance is not sensitive to h. In fact, in most cases. one layer fx(() achieve the best result. This indicates that the depth of the Under review as a conference paper at ICLR 2022
Table 4: The impact of the depth / of multi-layer perceptrons 5N Topl accuracy. The numbers "0" indicates the standard deviations estimated from independent train/test runs. find that model performances are not very sensitive to NN INsr= sx()), and, in most cases, one layer ( achieve the best result. In other words, the larger performance gaps in fact comes from different GC*) we use. Moreover, given the performance's variance of each model, we can see that sphereM outperforms other baseline models on all these three datasets and the margins are statistically significant. The same conclusion can be drawn based on our experiments on other datasets, Here, we only show results on three datasets as an illustrative example.
Dataset
BirdSnapi Test
NABirdst
iNat2018 8
P(ylx) Prior Type
Test
Val
78,81 (0.10)
81.08 (0.05)
71.6 (0.08)
78.83 (0.10)
81.2 (0.09)
71.7 (0.02)
zy²
78.97 (0.06)
81.11 (0.06)
71.75 (0.04)
78.84 (0.09)
81.02 (0.03)
71.71 (0.03)
79.04 (0.13)
81.60 (0.04)
72.89 (0.08)
78.94 (0.13)
81.62 (0.04)
72.84 (0.07)
wrap (Mac Aodha et al., 2019)
79.08 (0.15)
81.53 (0.02)
72.92 (0.05)
79.06 (0.11)
81.51 (0.09)
72.77 (0.06)
78.97 (0.09)
81.23 (0.06)
71.90 (0.05)
79.02 (0.15)
81.36 (0.04)
71.95 (0.05)
wrap fn
79.21 (0.14)
81.35 (0.05)
71.94 (0.04)
79.06 (0.09)
81.27 (0.13)
71.93 (0.04)
(0.13)
81.32 (0.08)
71.02 (0.18)
79.38 (0.12)
81.22 (0.11)
71.29 (0.20)
rbf(Mai al., 2020b)
79.40 (0.04)
81.31 (0.07)
71.35 (0.21)
(0.05)
81.30 (0.07)
71.21 (0.19)
78.96 (0.18)
81.27 (0.07)
71.76 (0.06)
78.97 (0.04)
81.28 (0.05)
71.71 (0.09)
rf (Rahimi et al., 2007)
79.07 (0.12)
71.80 (0.04)
(0.13)
81.22 (0.11)
71.46 (0.05)
19.44(0.111
81.24 (0.06)
72.98 (0.05)
79.07 (0.05)
81.13 (0.04)
72.9 (0.03)
grid (Mai et al., 2020b)
79.27 (0.10)
81.04 (0.05)
72.72 (0.04)
78.88 (0.02)
80.78 (0.07)
72.49 (0.03)
79.24
81.23 (0.02)
73.01 (0.09)
79.16
81.31 (0.14)
72.70 (0.02)
theory (Mai al., 2020b)
78.89 (0.21)
81.07 (0.01)
72.57 (0.07)
79.06 (0.18)
80.72 (0.12)
72.33 (0.08)
80.34 (0.08)
81.86 (0.03)
73.72 (0.07)
79.91 (0.12)
81.84 (0.04)
73.46 (0.03)
S
80.06 (0.06)
81.90 (0.05)
73.73 (0.02)
79.92 (0.14)
81.84 (0.10)
73.21 (0.04)
MLP does not significantly affect the model performance and deeper MI.P does not necessarily lead a better performance. In other words, the systematic bias (i.e., distance distortion) introduced by grid and theory can not later be compensated by a deep MLP. shows the importance of designing a spherical-distance-aww location encoder.
9.10 ABLATION STUDY ON DIFFERENT SELF-SUPERVISED LOss
Both the BI (Equation 6) and MC (Equation 7) unsupervised loss have three loss component: in- batch, negative location, SimCSE loss component, How does each of them contribute to the overall unsupervised training? In order to answering this question, do ablation studies on these two
20

---

Under review as a conference paper at ICLR 2022
Model Perfomance Comparison INac200a
Model Performance Comparison iNa42018
1.4
the Volsatorr
No Prior
basharMMj:: Sxp.
[ssharuM-]: Stp..
[pphareM=]: Sep. B1- 43g
InphareM-]: Sep. & MC- og1
b Sep. B1 C3" 40
[spharMM Sup. & MC. Ogf 428"
204
[rphorrMe:: Sep B1- 22ra
[ephorM-]: Sip. 2 MC. Ag3 cye" 2Ag
05 04
Takkngg labels Sxpersedd Trocting
04
65 05
E 15 Tokning Stperoed Tia
(a) Ablation Study on BI
(b) Ablation Study on MC
Figure 7: Ablation study on different unsupervised learning setting on i dataset. (a) Alblation study on BI loss. BI- Chateh Carsloe + Lei Caimese indicates the full BI loss while BI Cbatch Cregloc deletes the SimCSE component. BI - Littch additionally deletes the negative location loss component. Comparing among those three model settings can help us understand the effect of different loss components. Similarly, we do the same ablation study MC loss and show in (b).
objectives by using iNat2018 dataset as an example. Figure 7a and 7b illustrate the ablation study results.
From Figure 7a, we see that as for BL, adding CBi Chegloc will significantly increase the model perfor- mance especially when r is large. Adding L Laimcse also improves the model performance when r is small. As for the ablation study on MC loss as shown in Figure 7b, adding LMC Laegloe can significantly
increase the model performance and adding Lac
Laimose can also make the performance slightly better.
9.11 EFFECTIVENESS OF UNSUPERVISED PRETRAINING ON FMOW DATASET
Prefoemance Companioon Fldele
No Prior
[jphoraM-j: Sip.
53
IpphoroM-] Sip. MC- Catl 1180
the Sadenie 8
TL1
in
me
8
300 10G
34
Trining
Traeiin
Figure 8: Comparison among two models with identical model architecture but different training objectives on fMoW dataset. Each point each curve indicates a specific training process. We repeat each of them for five times and show the standard deviations as shaded areas along the line. We can see that with statistically significance, MC unsupervised loss also effective the few shot learning setting on the IMoW dataset.
9.12
IMPACT OF MRR BY THE NUMBER OF SAMPLES AT DIFFERENT LATITUDE BANDS
See Figure 9
21

---

Under review as a conference paper at ICLR 2022
Table 5: Ablation Study unsupervised loss Luc iNat2018 dataset. We show the effect of different hyperparematers LmC on performance of location encoders. use sphereA + as an representative location encoder and use supervised training dataset ratio as 0.5. Ir indicates the learning rate used for unsupervised Luc training. dropout indicates the dropout rate used by location encoder which will affect the SimCSE as Gao (2021) shows. 01 and 02 are weights SImCSE loss LMG (X) component respectively. components. Equation 7 explanation.
negative location loss MC (X) and and the temperatures used three different
r
e(x)
C
Muneuper 01 W
02
70
71
dropout
Top
0.001
71.31
0.0005
71.86
0.0002
71.78
0.00005
0.5
71.76
0.00001
71.83
0.000005
71.7
0.000001
71.69
71,8
1
71.86
0.0005
0.5
0.5
71.71
0
71.42
2
71.71
1
71.86
0.0005
0.5
0.5
71.81
0
71.82
64
71.85
32
71.95
24
71.74
22
71,81
20
72.08
18
71.77
16
72.02
0.0005
0.5
0.5 sphereM+ LMc
12
71.87
8
71.93
4
71.69
2
71.89
71.86
0.1
71.67
0.01
71.63
71.85
0.0005
1
20
1
0.5
72.08
0.5
71.76
71.84
0.0005
1
20
0.5
72.08
0.5
71.69
72.08
0.0005
4
1
20
0.5
71.84
71.69
0.7
71.38
0.6
71.91
0.5
72.08
0.0005
1
1
20
0.4
71.55
0.3
71.08
0.2
69.96
0.1
68.17.
22

---

Under review as a conference paper at ICLR 2022
Table 6: Ablation Study unsupervised loss Lat iNa20018 dataset. We show the effect of different hyperparematers Lar on performance location encoders. We use sphereM + as an representative location encoder ad use supervised training dataset ratio I' as 0.5. Ir indicates the learning rate used for unsupervised LuC training.
e x
Munssppe
B
[
32
dropout
Top1
0.001
71.76
0,0008
71.87
0.0005
71.85
0.0002
72.02
0.5
0.0001
71.67
0.00001
71.62
0.000005
71.48
0.000002
71.56
71.98
1
72.02
0.0002
0.5
0.5
+
LBt
0.5
71.85
0
71.65
72.02
4
71.82
0.0002
0.5
8
71.94
16
71.84
0.00
72.02
0.10
72.06
0.0002
0.20
0.5
71.93
0.50
71.83
1.00
71.84.
Table 7: Ablation Study on unsupervised loss MSE over i dataset. We show the effect of different hyperparematers of MSE the performance of location encoders. We sphereM+ representative location encoder ad use supervised training dataset ratio as indicates learning rate used for unsupervised M MSE training.
r
e(x)
C
Munsuper
dropout
Top1
0.001
71.05
0.0005
71.41
0.0001
71.55
0.5
sphereM+
MSE
0.00001
0.5
71.58
0.000005
71.69
0,000002
71.71
0.000001
71.6
MRRIMode) MARIgnd
MRRR(Mode)) Millirr0)
-0.00
-0.00
wno
wapp r2f Dieory iphenec sphereCe
Bieory spherec
-030
2.5
sogpolssomppes
log10tisampics
iNar2017
(h) iNar2018
Figure 9: Impact of MRR by The Number Samples at Different Latitude Bands.
23

---

Under review as a conference paper at ICLR 2022
Feather duster
wrap*
(e)
(d) sphereC++
White-browed wagtail
wrap
grid
(b) sphereC+
Arctic fox
wrap
grid
① sphereC+
(m) Scarlet leafwing
(n) wrap
(o) grid
(p) sphereC++
(g) African pied hornbill
(r) wrap
(9 grid
i sphereC++
(w) False Tiger Moth
(v) wrap
(w) grid
(x) sphereC++
Compare the predicted distributions of example species from different models. The first figure of each marks the data points from iNat2018 training data.
Figure 10:
9.13 PREDICTED DISTRIBUTIONS INAT2018
We plot the predicted species distributions from different models at different geographic regions, and compare them with the training sample locations of the corresponding species, see Figure 10. We can that compared with wrape and grid, in each geographic region with sparse training samples and the North Pole area, the spatial distributions produced by sphereC+ are more compact while the other two have over-generalization issue
9.14 EMBEDDING CLUSTERING
We use the location encoder trained on iNat2017 or iNat2018 dataset to produce a location embedding for the center of each small latitude-longiudee cell. Then we agglomerative clustering" on these embeddings to produce clustering map. Figure 11 and 12 show the clustering results for different models with different hyperparameters on iNat2017 and iNat201 dataset.
$ http:///ccikii.tmmmmm..ottttaaba htm.or///taala/mmdiii. http://ooooooa//eneeeeeoeeeeeee es/generated/sk skts//klenmmm cf custer. Aggl Sggllooerativeecuserr wg..sttrriinng............. html
24

---

Under review as a conference paper at ICLR 2022
(a) wraps
(b) grid (rmen = 10 10
(o) grid (Tmen = 10-
d rbb (a 1, m 200)
(o) theory (rmin 10 1)
( theoryy (rmsn 10 9)
( sphereM (rmin 10 10-1)
(h) sphere M (Tmin 10-
@ sphere (rmin 10 10-3)
① sphereC+ (Fmsn 30
(k) sphereM+ (Fmin 10 10-3)
sphereDFS (rmmn 10
Figure 11: Embedding clusterings of iNat2017 models. (a) wraps with 4 hidden ReLU layers of 256 neurons; (d) rbf with the best kemeel size a = 1 and number of anchor points m 200; (b)())e((f) are 2pace2 models Mai al. (2020h) with different min scale T'mn {10- 0 10 models.
10-2)." j.a (g)-(1) are different Sphere Vec
They share the same best lyperparametees: 9 = 61, Pmmzz = 1, and hidden ReLU layens of 512 neurons. hidden ReLU layers of 1024 neurons.
"They shure same hyperparaneeter:: S 32, Tmaz
9.15
EXPERIMENTS WITH SYNTHETIC DATASET
9.15.1
SYNTHETIC DATASET GENERATION
To further investigate the effectiveness of our proposed spherical location encoder Sphere2Vec, we construct set of synthetic datasets.
We utilize the von Mises -isher distribution (vMF) (Izbicki al., 2019), an analogy of the 2D Gaussian distribution on the spherical surface S² to generate synthetic data points", The probability density function of vMM F is defined as
() 2x sinh(x)
exp(1pp T x(x))
(25)
where x(x) = [x, y, =] [cos ø cos X, cos sin X, sin ø]
Here, x(x) converts x into a coordinates in the 3D Euclidean space, and on the surface of a unit sphere. A vMF F distribution is controlled by two parameters - the mean direction μ IR3 and concentration parameter x e R+ f indicates the center of a vMF distribution which is a 3D unit vector. x is positive real number which controls the concentration of vMF. A higher x indicates more compact vMF distribution, while x = 1 correspond to a vM F distribution with standard deviation covering half of the unit sphere.
To simulate multi-modal distributions, we assume C' classes with even prior, and each classes follows vMF F distribution. To create dataset we first sample C sets of parameters {(H,, k:)} (C = 50). Then we draw SP samples for each class (SP = 100). So in total, each generated synthetic dataset has 5000 data points for 50 balanced classes.
9htpps: / http:/tttssso....ec tensor ff wttp:/rgg/bbbbbllttt//ooooo htt.d//occ//tttt///////// di distribon but i ons/VonMi sesP'i shert sample
25

---

Under review as a conference paper at ICLR 2022
(a) wraps
(b) grid (rmen = 10-9)
(o) grid (Tmen = 10
( rr (o L, 200)
(0) theory (rmin 10 3)
i theoryy (rmsn 10 6)
( sphereM (rmin 1
(h) sphereM M (Tmin 10-3)
@ sphereC (rmin 10 10-3)
① sphereC+ (Pmm = 3)
(k) sphereM+ (Fmin 10 10-3)
() sphereDFS (rmmn 10
Figure 12: Embedding clusterings of iNat2018 models. (a) wraps with 4 hidden ReLU layers of 256 neurons:
(d) rbf with the best kemel size a = 1 and number of anchor points m = 200; (b)()(e)() are Space2 models (Mai et al., 2020h) with different min scale Tmin [10 10 3). (g)-() are Sphere2Vec Vec models with different min scale." b
They share the same best hyperparamettes: 9 = 61, Pmmzz = 1, and hidden ReLU layens of 512 neurons. hidden ReLU layers of 1024 neurons.
"They shure same best hyperparanetters: S 32, Tmaz
The concentration parameter is sampled by first drawing from an uniform distribtuion U' (Kmmnn Kmaz), and then take the square r². The square helps to avoid sampling many large H2 which yield very concentrated FMF distributions that are rather easy to be classified. We fix Emin = and vary Kmaz in [16, 32, 64, 128].
For the center parameter μ4 adopt two sampling approaches:
1. Uniform Sampling: We uniformly sample C centers (1t) from the surface of unit sphere. We generate four synthetic datasets (for different values of Kmax) and indicate them as U1, U2. U3. U4. See Table 8 for the parameters we use to generate these datasets.
2. Stratified Sampling: We first evenly divide the latitude range [- T/2] into N₁, intervals. Then we uniformly sample Cp centers (n₂)) from the spherical surface defined by each latitude interval. Since the latitude intervals in polar regions have smaller spherical surface area, this stratified sampling method has higher density in the polar regions. We keep N₂ Cp. C 50 fixed and varies N₁. in [5, 10, 25, 50]. Combined with the 4 Kmaz choices, this procedure yields 16 different synthetic datasets. We denote them as Si.j. See Table for the parameters we use generate these datasets.
Figure 13a- 13d visualize the data point distributions of these four synthetic datasets based on the uniform sampling method in 2D space. Figure 13e visualized the U4 dataset in 3D Euclidean space. We can see that when Kmea is larger, the variation of point density among different vM F distributions becomes larger. Some vM F are very concentrated and the resulting data points are easier to be classified. Moreover, we treat these datasets as 2D data points as grid and theory do. eMF distributions in the polar areas will be stretched to very extended shapes making model learning more difficult. However, this kind of systematic bias can be avoided if we use a spherical location encoder as Sphere2Vec.
26

---

Under review as a conference paper at ICLR 2022
75
50
25
-25
-50
-50
-75
-75
(5)
U1 dataset in 2D degree Space (Rmaaze
16)
(b) U2 dataset in 2D degree Space (Nomone
32)
75
50
50
25
-25
-50
-50
-75
100
(c) U3 dataset in 2D degree Space (Kmaz
64)
(d) U4 dataset in 2D degree Space (Kmaa
128)
(e) U4 dataset in 3D space (Kiaax 128)
Figure 13: The data distributions of four synthetic datasets (U1, U2, U3, and U4) generated from the uniform sampling method. (c) shows the U4 dataset in 3D Fuclidean space. We 2D data points as grid theory, the vM distributions the polar areas will be stretched and look like aniostropic multivariate Gaussian distributions. However, this kind of systematic bias can avoided if we use spherical location encoder as Sphere Vec..
that
these datasets as
Figure 14 visualizes the data distributions of four synthetic datasets with stratified sampling method. They have different N2 but the same Kman- We can see that when N₂ increases, more fine- ine-grain stratified sampling is carried out. The resulting dataset has larger data bias toward the polar areas.
9.15.2 SYNTHETIC DATASET EVALUATION RESULTS
We evaluate all baseline models well as sphereM. those generated 20 syhthetic datasets as described above. For each model, we do grid search on their hyperparameters for each dataset including supervised learning rate Nsuper, the number of scales S, the minimum scaling factor Tmins the number of hidden layers and number of neurons used in NN (+) h and k. The best performance of each model is reported in Table 8. Some observations can be made from Table 8:
27

---

Under review as a conference paper at ICLR 2022
75
50
-25
-25
-50
-50
-75
(a) S1.3 dataset in 2D degree Space (N 5)
(b)
S2.3 dataset in 2D degree Space (N,, 10)
50
50
-55
-50
-50
(c) S3.3 dataset in 2D degree Space (N 25)
(d) S4.3 dataset in 2D degree Space (N, 50)
Figure 14: The data distributions of four synthetic datasets (S1.3, S2.3, S3.3, and S4.3) generated from the stratified sampling method with Kmaz 64. We can see that when N₂. increases, more tineggraie stratified sampling is carried out. The resulting dataset has a larger data bias toward the polar areas.
1. sphereM+ able to outperform all baselines on all 20 synthetic datasets. The absolute Topl improvement can go up to 2% and the error rate deduction can go up 30.8%. This shows the robustness of sphereM.
2. When the dataset is fairly easy to classify (i.e., all baseline models can produce 95+% Top1 accuracy), sphereM+ is still able to further improve the performance and gives a very large eтrоr rate reduction (up to 30.8%). This indicates that sphere. + is very robust and reliable for datasets with different distribution characteristics.
3. Comparing the error rate of different stratified sampling generated datasets (S1.j S4.j) we can see that when we keep Kmaz fixed and increase Nμ, the relative error reduction ER become larger. Increasing N₁, means do more fine-grain stratified sampling. The resulting datasets should sample more vM F distributions in the polar regions. This phenomenon shows that when the dataset has larger data bias towards the polar area, we expect sphereM + will be more effective.
28

---

Under review as a conference paper at ICLR 2022
Table 8: Compare sphereM + to baselines on synthetic datasets. U1 U4 indicate synthetic datasets generated based the uniform sampling approach (see Appendix 9.15.1). S1.1 S4.4 indicate 16 synthetic datasets generated based the stratified sampling apprach. For all datasets have C 50 and SP 100. For each model, we perform grid search on its hyperparameters for each dataset and report ATopl column shows the absolute performance improvement of sphereM + over the best baseline model (bolded) for each dataset. The ER column shows the relative reduction of error compared to the best baseline model (bolded). We synthetic datasets. The absolute Top1 accuracy improvement can be much 2.0% for datasets with lower precisions, and the error rate deduction can as much as 30.8% for datasets with high precisions.
Top1 accuracy. The
that sphere.M
outperform all other baseline models all of these 20
ID Method y Cp Nomin Nomoz
zyz wrap wraptffn rff f rbf grid theory sphereM
ATopl
ER
67.2
66.9
66.8 46.6 68.6
69.2
0.6
-1.9
U2
32
73.1 75.1
73.9
72.3 58.4 76.2
76.5
77.4
0.9
-3.8
uniform
U3
64
86.1 90.1
88.3
89.0
92.3
92.7
93.3
0.6
-8.2
U4
128
91.8
94.9
92.3
92.5 97.5
97.7
98.0
0.3
-1300
S1.1
68.7
69.7
68.8
68.6
69.5
72.3
1.8
-6.1
$1.2
32
76.7 79.1
78.1
78.4 81.1 81.2
79.2
82.9
1.7
-9.0
10
$1.3
64
91.2 92.5
92.9
92.6 94.7 94.8
94.9
95.4
0.5
-9.8
$1.4
128
86.5
91.6
88.3
92.4 95.2
94.9
96.1
0.9
-8.7
S2.T
70.5
70.7
76.4 72.0
74.0
2.0
-7.1
S2.2
32
76.1 79.7
78.2
68.6 80.9
80.5
82.3
1.4
-7.3
10 5
$2.3
64
88.0 89.9
88.2
88.5 80.0 92.5
91.9
93.3
0.8
100.7
S2.4
128
94.4 96.6
96.7
95.5 94.0 97.6
97.6
98.1
0.5
-0.88
$3.1 stratified
16
66.2 66.3
64.7
65.6 67.1 66.7
66.7
68.3
1.2
-3.6
80 82.5
80.7
81.6 84.5
82.1
1.4
-9.0
25 2
S3.3
64
85.4
86.0
85.7
86.2 89.1 89.6
88.6
91.0
1.4
-13.5
$3.4
128
93.2 96.0
94.8
95.7 97.2 97.3
97.4
98.0
0.6
23.1
$4.1
16
64.8 67.4
66.0
66.3 66.9 67.1
64.5
68.4
-3.1
S4.2
75.6
77.4
77.4 78.4 80.1
0.9
-4.5
50 1
$4.3
91.3 93.9
93.7
93.8 95.0 95.2
94.0
0.9
-18.7
$4.4
128
94.3 95.5
94.4
94.7 95.4 97.4
96.5
98.2
0.8
330.8
29


Reviews:
[{"review_id": "mvHRm7ZAmdY", "reviewer": "Reviewer_6cCW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positives:\n\n- I think the problem setting, geo-aware image classification, is interesting and I haven't seen a lot of work exploring the issue of encoding geographic location, which has some clear challenges.\n- There are quite a few datasets and baselines used in the comparison.\n\nNegatives:\n\n- Table 1 shows that the effect size is relatively small. The proposed approach isn't clearly worse than others, but it doesn't seem to make a major difference. \n\n- The core method is composed of several components (the initial encoding, the MLP, and self-supervised pre-training). However, the only thing that is varied is the initial encoding. This leaves open the question of how changes to the other components would impact the metrics. It could be that a slightly deeper or shallower MLP would change which method is the best. The paper claims that many aspects were fixed to provide a fair evaluation, but this often hides the fact that the proposed contribution is only superior given the particular settings, and not that significant overall.\n\n- It is claimed that \"grid ... performs poorly at a global scale due to its inability to preserve spherical distances\" and provides a proof. However, the quantitative metrics show that grid performs about as well as the proposed method. \n\n- The presentation needs significant improvement. Some figures are too small (Fig 4) or have labels that are too small (Fig 5). Also, see below for grammar issues. There are important details that aren't described well, for example, what exactly is the MLP? How sensitive are the results to the hyperparameters of the MLP?\n\n- Section 5.1: it is argued that \"wrap and grid have over generalization issue in data sparse area\" and that the proposed method provides better results. It's unclear why this is the case. Much of the theoretical justification relies on the fact that the proposed approach preserves spherical distances, but the connection between that preservation and the differences between the methods isn't made strongly enough.\n\nMinor Comments:\n\n- I would have liked to have seen a comparison to the conversion of lat/lon into x,y,z coordinates using something like earth-centered, earth fixed coordinates.\n\n- There are numerous small grammatical errors. They didn't distract from clarity, but it needs significant improvement. Here are a few from the paper (but there are others and quite a few in the supplemental as well):\n   - \"may provide clue to their real classes\" -> \"may provide a clue to their real classes\"\n   - \"data parse areas\" -> \"data sparse areas\"\n   - \"Despite of its effectiveness\" -> \"Despite its effectiveness\"\n   - \"distance reserving\" -> \"distance preserving\"\n   - \"identities base on their\" -> \"identities based on their\"\n   - \"are public available\" -> \"are publicly available\"\n   - There are many more in the supplemental.\n\n- It seems like there are too many important details left to the appendix, including defining a key loss component \"simcse\", which isn't defined until 9.11.\n\n- It might be worth including a reference to http://bmvc2018.org/contents/papers/0586.pdf (Section 3.1), which proposes an approach to learning a location embedding in conjunction with geo-tagged ground-level images. This is like the first stage of training in the proposed method but requires a known timestamp.\n", "summary_of_the_paper": "This paper proposes a strategy for learning a location embedding (i.e., translating lat/lon into a useful high-dimensional vector). The idea is to first encode the location into the double Fourier sphere basis (or a subset) and then pass this through a multilayer perceptron. Several alternatives to the first stage are evaluated.\n\nTwo stages of training are proposed:\n1) Weakly Supervised Pre-Training: use a contrastive loss to train the embedding by comparing it to features extracted from geotagged images. The idea is that a good location embedding should be sufficient to recognize whether a given image was captured in that location.\n2) Supervised Fine-Tuning: given a geotagged image, extract image and location features, combine them to predict the image classification label.\n\nThe approach is demonstrated on several geo-aware image classification applications. The idea with these applications is that combining image features with the location embedding is sometimes enough to resolve ambiguity from the image alone.", "summary_of_the_review": "I don't think this paper is suitable in its current state. The experimental results show a small effect size and it's unclear how stable the results are to changes in the MLP, and other components. There is also a weak connection between some of the stated benefits (preserving spherical distances) and the actual benefits. It's entirely possible that the proposed methods are better in some settings for some other reason.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DGAN2QxSStZ", "reviewer": "Reviewer_toM9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nIt's interesting to have a positional encoding method for spherical coordinates, which could have potentials for many other tasks.\n\nWeaknesses:\n\n1) Though the idea is interesting,  I would propose an alternative for embedding GPS coordinates. Would you please discuss it or add it as a baseline?\n\nGiven GPS coordinates, we can first transform them to an Earth-centered, Earth-fixed coordinate system, which is a Cartesian coordinate system. Using a simple MLP or (sin/cos) to encode these Cartesian coordinates to the network and check the performance.\n\n2) From Table 1, it seems the proposed method marginally improves previous methods. \n\n3) From the introduction section and Figure 2, it seems the assumption of \"assume I and x are conditionally independent given y\" does not hold since I and x co-exist.\n\n4) Equation 1. Why do we need to enforce that \"multi-scale representation of x preserves the spherical surface distance\"? Would you please add some motivations? From the tasks (Figure 2) listed in this paper, I would assume the GPS coordinates are distributed sparsely. Pairwise spherical surface distance may vary on a large scale and is noisy. Do we really need this constraint?", "summary_of_the_paper": "This paper proposes a positional embedding method for GPS coordinates, which lie on a sphere.\n\nBased on the Double Fourier Sphere, this paper proposes a family of encoding methods and show their effectiveness.\n\nThis paper also gives a family of loss functions to enable self-supervised training.", "summary_of_the_review": "While I like this work and it proposes an interesting positional encoding method for spherical coordinates, I would like to see the authors' reply for the weaknesses to make further decisions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9tGUcZJqIVa", "reviewer": "Reviewer_vjnw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positives\n\n  - how to integrate contextual information, e.g., location, as part\n    of vision pipelines is an interesting and challenging problem, with\n    demonstrated performance improvements on various tasks (e.g., fine-grained\n    species classification)\n\n  - the authors propose a novel multi-scale method for encoding\n    spherical coordinates (several variants) based on Double Fourier\n    Sphere, including an analysis of the distance preserving quality in\n    the supplemental\n\n  - results show consistent improvements over baselines on seven different\n    datasets\n\n  - extensive experiments, including ablation studies of various\n    components in the supplemental material\n\n  - the self-supervised component, for pretraining the location\n    encoder using unlabeled data, is shown to be useful in scenarios\n    where there is limited labeled data\n\n\nNegatives\n\n  -  grammatical errors throughout\n\n    For example, I believe the first contribution in Section 1 should be \"distance preserving\" instead of \"distant reserving\".\n\n  - missing details\n\n    Section 3.2 has some elements whose description could be expanded\n    on. For example, the notation for omitting terms of the loss\n    function. Another example would be the linear projection layer\n    that is mentioned only once in passing (and shown in Figure 1).\n    Further, implementation details (optimizer, learning rate schedule,\n    hardware, etc.) seem to be missing or relegated to the supplemental.\n\n  - though the results are impressive, it is not clear from the\n    results which version of Sphere2Vec (from the proposed variants)\n    should be used in which scenario (in other words, there is no\n    consistent winner)\n\n  - the self-supervised component of the proposed approach\n    appears to be limited to a pretraining strategy, with results\n    showing it useful only when there is limited labeled data", "summary_of_the_paper": "This paper explores developing location encodings for spherical\nsurfaces and their application to tasks in geo-aware image\nclassification. The authors propose a method for encoding spherical\nlocations into a higher dimensional space (based on Double Fourier\nSphere) that preserves distances. The proposed method, referred to as\nSphere2Vec, is demonstrated for three tasks and evaluated using seven\ndatasets. In addition, a self-supervised method is proposed for using\nunlabeled data to pretrain the location encoder.  This self-supervised\npretraining stage is shown to improve performance, especially when\navailable labeled data is limited.", "summary_of_the_review": "  This paper explores two important problems: 1) given a geotagged\n  image, how to encode location information and incorporate it as\n  context in existing image processing pipelines, and 2) how to take\n  advantage of a large corpus of images with known geolocation, that\n  are not annotated (unlabeled) for the primary task. Specifically,\n  this paper explores how to encode geospatial information in the form\n  of spherical coordinates.\n\n  For several tasks in geo-aware image classification, the\n  proposed multi-scale spherical location encoder outperforms\n  baselines. However, the proposed self-supervised method for\n  incorporating unlabeled geotagged imagery is limited to a pretraining\n  strategy when there is limited labeled data. Additionally, several\n  variants of the method are proposed, and it is not clear what variant\n  to use when.\n\n  Ultimately, I think this paper tackles an important problem and\n  demonstrates the general applicability of the method. While the\n  manuscript has minor errors throughout and some missing details, I\n  lean towards accept for my initial rating.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6M1XXcKpAD_", "reviewer": "Reviewer_6Te4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1) The approach is demonstrated to be better than the previous method both qualitatively and quantitatively. \n2) Description of the losses and algorithms were fairly complete with minor errors.\n3) Good references on topics beyond CV/DL papers.\n\n\nWeaknesses:\n1) Paper was slightly harder read and some useful information such as the definition of the studied approaches were left in the appendices. Reordering the information in the text would help here.\n2) Most of the gains come from the framework over the spherical coordinate approach. This seems reasonable as using spherical coordinates over euclidean coordinates doesn't seem to be something a few layers of networks can't learn easily. 3\n3) Some errors in the text:    \n    a) In practice it help with --> helps with\n    b) Eq. 6: second term in the second line seems to have errors\n    c) Supervised Fine-Turning --> Fine-tuning\n    d) \\delta MRR was not defined in the text\n \n", "summary_of_the_paper": "The paper studies how to use location information (geographical location) of images to help with image classification tasks. The paper studies use of spherical based location over GPS lat long and argues that the euclidean space does not preserve distances and has distortions whereas a spherical coordinate approach would not have the inherent problem. The approach demonstrates a framework to train the location encoding in an unsupervised way to with losses based on representation learning approaches. The paper studies multiple ways of expanding the spherical coordinates and evaluates their performance and empirically demonstrate the strength in their performance. ", "summary_of_the_review": "Paper proposes a framework and uses spherical coordinates over encoding method to help with geo-aware image classification. The paper propose the title of sphere2vec for novelty but the main improvements to the method is not with the encoding but with the framework itself. For readers, it would be a slightly disappointing results if the reader is reviewing spherical coordinate as an approach. As a paper, the results does show improvements through the framework, in the form of representation learning. Thus, the recommendation is a marginally below acceptance threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: In spite of some slightly mixed scores (with one borderline positive review), scores are ultimately lukewarm and tend toward negative (and furthermore, reviews are broadly in agreement as to the issues they raise). Main issues center around low significance of the results, and issues with the presentation that need to be addressed.
Decision: Reject