Multi-Scale Representation Learning  for Spatial Feature Distributions using Grid Cells
Authors: Gengchen Mai
Krzysztof Janowicz
Bo Yan
Rui Zhu
Ling Cai
Ni Lao
Venue:  (2020)
Keywords: Grid cell
space encoding
spatially explicit model
multi-scale periodic representation
unsupervised learning

Abstract: Unsupervised text encoding models have recently fueled substantial progress in NLP. The key idea is to use neural networks to convert words in texts to vector space representations (embeddings) based on word positions in a sentence and their contexts, which are suitable for end-to-end training of downstream tasks. We see a strikingly similar situation in spatial analysis, which focuses on incorporating both absolute positions and spatial contexts of geographic objects such as POIs into models. A general-purpose representation model for space is valuable for a multitude of tasks. However, no such general model exists to date beyond simply applying discretization or feed-forward nets to coordinates, and little effort has been put into jointly modeling distributions with vastly different characteristics, which commonly emerges from GIS data. Meanwhile, Nobel Prize-winning Neuroscience research shows that grid cells in mammals provide a multi-scale periodic representation that functions as a metric for location encoding and is critical for recognizing places and for path-integration. Therefore, we propose a representation learning model called Space2Vec to encode the absolute positions and spatial relationships of places. We conduct experiments on two real-world geographic data for two different tasks: 1) predicting types of POIs given their positions and context, 2) image classification leveraging their geo-locations. Results show that because of its multi-scale representations, Space2Vec outperforms well-established ML approaches such as RBF kernels, multi-layer feed-forward nets, and tile embedding approaches for location modeling and image classification tasks. Detailed analysis shows that all baselines can at most well handle distribution at one scale but show poor performances in other scales. In contrast, Space2Vec ’s multi-scale representation can handle distributions at different scales.

Published as a conference paper at ICLR 2020
MULTI-SCALE REPRESENTATION LEARNING FOR SPA- TIAL FEATURE DISTRIBUTIONS USING GRID CELLS
Gengchen Mai', Krzysztof Janowicz', Bo Yan?, Rui Zhu', Ling Cai' & Ni Lao STOO Lab, University of California, Santa Barbara, CA, USA, 93106 syengchen_maiis janowicz, ruizhu, lingcail ) @uucss. edu LLinkedIn Corporation, Mountain View, CA, USA, 94043 boyan @ @linkedin... com MoyyMosaic Inc., Palo Alto, CA, USA, 94303 n lao@mosaix.at
ABSTRACT
Unsupervised text encoding models have recently fueled substantial progress in Natural Language Processing (NLP). The key idea is use neural networks convert words in texts to vector space representations (embeddings) based on word positions a sentence and their contexts, which are suitable for end-to-end training of downstream tasks. We see a strikingly similar situation in spatial analysis, which focuses on incorporating both absolute positions and spatial contexts geographic objects such as Points of Interest (POIs) into models. A general-purposs representation model for space is valuable for a multitude of tasks. However, no such general model exists to date beyond simply applying discretization feed- forward nets to coordinates, and little effort has been put into jointly modeling distributions with vastly different characteristics, which commonly emerges from GIS data. Meanwhile, Nobel Prize-winning Neuroscience research shows that grid cells in mammals provide multi-scale periodic representation that functions as metric for location encoding and is critical for recognizing places and for path-integration. Therefore, we propose a representation learning model called Space2 Vec to encode the absolute positions and spatial relationships of places. We conduct experiments on two real-world geographic data for two different tasks: 1) predicting types of POIs given their positions and context, 2) image classification leveraging their geo-locations. Results show that because of its multi- scale representations, Space2 outperforms well-established ML approaches such as RBF kernels, multi-layer feed-forward nets, and tile embedding approaches for location modeling and image classification tasks. Detailed analysis shows that all baselines can at most well handle distribution at one scale but show poor performances in other scales. In contrast, Space2 's multi-scale representation can handle distributions at different scales.
INTRODUCTION
Unsupervised text encoding models such Word2Vec (Mikolov al. 2013). Glove (Pennington et al. 2014), ELMo (Peters et al. 2018), and BERT (Devlin et al., 2018) have been effectively utilized in many Natural Language Processing (NLP) tasks. At their core they train models which encode words into vector space representations based on their positions in the text and their context. A similar situation can be encountered in the field of Geographic Information Science (GIScience). For example, spatial interpolation aims at predicting an attribute value, e.g., elevation, at an unsampled location based on the known attribute values of nearby samples. Geographic information has become an important component to many tasks such as fine-grained image classification (Mac Aodha et al. 2019) point cloud classification and semantic segmentation (Qi et al., 2017); reasoning about Point of Interest (POI) type similarity (Yan et al. 2017). land cover classification (Kussul et al.) (017). and geographic question answering (Mai et al. 2019b). Developing a general model for vector space representation of any point in space would pave the way for many future applications.
Link to project repository: http::///ggitibbboo///gggggg/eee

---

Published as a conference paper at ICLR 2020
fores
Clation
Office
Ex
Fea
SatSenberr
1658
1859
(a) Women's ' Cloth
(b) Education
(c) Ripley's 's K
(d) Renormalized Ripley's K
Figure I:
The challenge of joint modeling distributions with very different characteristics. (a)(b) The POI locations (red dots) in Las Vegas and Space2Voe predicted conditional likelihood of Women's 's Clothing (with a clustered distribution) and Education (with even distribution). The dark area in (b) indicates that the downtown area has more POIs of other types than education. (c) Ripley's 's K curves of POI types for which
Space2 has the largest and smallest improvement over wrap (Mac Aodha et al.) 2019). Each curve represents the number of POIs of a certain type inside certain radios centered at every POI of that type; (d) Ripley's 's K curves renormalized by POI densities and shown in log-scale. To efficiently achieve multi-scale representation Space2 Vee concatenates the grid cell encoding of 64 scales (with wave lengths ranging from 50 meters to 40k: meters) as the first layer a deep model, and trains with POI data in an unsupervised fashion.
However, existing models often utilize specific methods to deal with geographic information and often disregards geographic coordinates. For example, Place2Vec (Yan et al., 2017) converts the coordinates of POIs into spatially collocated POI pairs within certain distance bins, and does not preserve information about the (cardinal) direction between points. [Li et al. (2017) propose DCRNN for traffic forecasting in which the traffic sensor network is converted to a distance weighted graph which necessarily forfeits information about the spatial layout of sensors. There is, however, no general representation model beyond simply applying discretization (Berg et al. 2014 Tang et al.
2015) or feed-forward nets (Chu et al. 2019 Mac Aodha et al. 2019) coordinates.
A key challenge in developing a ceneral-purpont representation model for space is how to deal with mixtures of distributions with very different characteristics (see an example in Figure D. which often emerges in spatial datasets (McKenzie et a 2015). For example, there are POI types with clustered distributions such as women's clothing, while there are other POI types with regular distributions such as education. These feature distributions co-exist in the same space, and yet we want a single representation to accommodate all of them in a task such as location-awan image classification (Mac Aodha al. 2019). Ripley's 's K is spatial analysis method used to describe point patterns over a given area of interest. Figure Ic shows the K plot of several POI types in Las Vegas. One can see that the radius grows the numbers of POIs increase at different rates for different POI types. In order to see the relative change of density at different scales, we renormalize the curves by each POI type's 's density and show it in log scale in Figure Td. One can see two distinct POI type groups with different distribution patterns with clustered and even distributions. If we want to model the distribution of these POIs by discretizing the study area into tiles, we have use small grid sizes for women's 's clothing while using larger grid sizes for educations because smaller grid sizes lead to over- parameterization of the model and overfitting. In order jointly describe these distributions and their patterns, we need an encoding method which supports multi-scale representations.
Nobel Prize winning Neuroscience research (Abbott & Callaway| 2014) has demonstrated that grid cells in mammals provide multi-scale periodic representation that functions encoding, which is critical for integrating self-motion. Moreover, Blair et al. (2007) show that the multi-scale periodic representation of grid cells can be simulated by summing three cosine grating functions oriented 60% apart, which may be regarded as a simple Fourier model of the hexagonal lattice. This research inspired us encode locations with multi-scale periodic representations. Our assumption is that decomposed geographic coordinates helps machine learning models, such as deep neural nets, and multi-scale representations deal with the inefficiency of intrinsically single-scale methods such as RFB kernels or discretization (tile embeddings). To validate this intuition, we propose an encoder-decoder framework to encode the distribution of point-featured in space and
metric for location
II GTS and spatial analysis, 'features' are representations of real-world entities. A tree can, for instance, be modeled by point-fcature, while street would he represented as line string feature.
2

---

Published as a conference paper at ICLR 2020
train such model in an unsupervised manner. This idea of using sinusoid functions with different frequencies to encode positions is similar to the position encoding proposed in the Transformer model /Vaswani et al., 2017p. However, the position encoding model of Transformer deals with a discrete ID space the positions of words in a sentence while our model works on higher dimensional continuous spaces such as the surface of earth.
In summary, the contributions of our work are as follows:
We propose an encoder-decoder encoding framework called Space2 using sinusoid functions with different frequencies to model absolute positions and spatial contexts. We also propose a multi-head attention mechanism based on context points. To the best of our knowledge, this is the first attention model that explicitly considers the spatial relationships between the query point and context points.
2. We conduct experiments on two real world geographic data for two different tasks: 1) predicting types of POIs given their positions and context, 2) image classification leveraging their geo-locations. Space2 Vec outperforms well-established encoding methods such as RBF kernels, multi-layer feed-forward nets, and tile embedding approaches for location modeling and image classification.
3. To understand the advantages of Space2Vec we visualize the firing patterns (response maps) of location models' encoding layer neurons and show how they handle spatial structures at different scales by integrating multi-scale representations. Furthermore the firing patterns for the spatial context models neurons give insight into how the grid-like cells capture the decreasing distance effect with multi-scale representations.
PROBLEM FORMULATION
Distributed representation of point-features in space can be formulated as follows. Given a set of points P = {p:}. i.e., Points of Interests (POIs), in L-D space (L = 2) define function fr.o(x) : R- - Rd (L & d)), which is parameterized by 0 and maps any coordinate vector representation of d dimension. Each point (e.g., restaurant) p; (x;, v;) is associated with a location x, and attributes v. (i.c., POI features such as type, name, capacity, etc.). The function (x)) encodes the probability distribution of point features over space and give representation of any point in the space. Attributes (e.g. place types such as Museum) and coordinate of point can be seen analogies words and word positions in commonly used word embedding models.
space to a
RELATED WORK
There has been theoretical research on neural network based path integration/spatial localization models and their relationships with grid cells. Both Cueva & Weil (2018) and Banino et al. (2018)
showed that grid-like spatial response patterns emerge in trained networks for navigation tasks which demonstrate that grid cells are critical for vector-based navigation. Moreover, Gao al. (2019) propose a representational model for grid cells in navigation tasks which has good quality such as magnified local isometry. All these research focusing on understanding the relationship between the grid-like spatial response patterns and navigation tasks from a theoretical perspective. In contrast, our goal focuses on utilizing these theoretical results on real world data in geoinformatics.
Radial Basis Function (RBF) kernel is a well-established approach to generating learning friendly representation from points in space for machine learning algorithms such as SVM classification (Bau- dat & Anouar 2001) and regression (Bierens) 1994). However, the representation is example based i.e., the resultant model uses the positions of training examples as the centers of Gaussian kernel functions (Mazyy & Schmidi 1996). In comparison, the grid cell based location encoding relies on sine and cosine functions, and the resultant model inductive and does not store training examples.
Recently the computer vision community shows increasing interests in incorporating geographic information (e.g. coordinate encoding) into neural network architectures for multiple tasks such as image classification (Tang et al.] 2015) and fine grained recognition (Berg et al.] 2014; Chu et al. 2019: Mac Aodha et al. 2019). Both Berg et al. (2014) and Tang et al. (2015) proposed to discretize the study area into regular grids. To model the geographical prior distribution of the image categories, the grid id is used for GPS encoding instead of the raw coordinates. However, choosing the correct discretization is challenging (Openshaw 1984 Fotheringham & Wong. 1991).
3

---

Published as a conference paper at ICLR 2020
and incorrect choices significantly affect the final performance (Moat et al. 2018: Lechner et al. 2012). In addition, discretization does not scale well in terms of memory use. To overcome these difficulties, both Chu et al. (2019) and [Mc Aodha et al. (2019) advocated the idea of inductive location encoders which directly encode coordinates into a location emhedding. However, both of
them directly feed the coordinates into feed-forward neural network (Chu et al. (0T9) or residual
blocks (Mac Aodha et al., (019) without any feature decomposition strategy. Our experiments show that this direct encoding approach insufficient to capture the spatial feature distribution and
Space2Vec Vec significantly outperforms them by integrating spatial representations of different scales.
4 METHOD
We solve distributed representation of point-features in space (defined in Section 2 with an encoder- decoder architecture:
1. Given point p; = (x,, v:) point space encoder Enc(*)() encodes location x; into a location embedding e[x.] R2d0 and point feature encoder Enc(") encodes its feature into a feature embedding e[v,] E Rdd"" c = [c[x:]:[vv.]] e R is the full representation of point E P, where d = d(x) + d(w)) [: represents vector concatenation. In contrast, geographic entities not in P within the studied space can be represented by their location embedding e[x]] since its v, is unknown.
2. We developed two types of decoders which can be used independently or jointly. location decoder (ec,() reconstructs point feature embedding e[vi] given location embedding e[x.]. and a spatial context decoder Dec,() reconstructs the feature embedding e[v.] of point ps based on the space and feature embeddings (e......,e,, eij. ..., em of nearest neighboring points (Psi,... Pij...., Pin), where n is a hyper-parameter
4.1 ENCODER
Point Feature Encoder Each point p: = (x,, Vi) in a point set P is often associated with features such as the air pollution station data associate with some air quality measures, set of POIs with POI types and names, a set of points from survey and mapping with elevation values, a set of points from geological survey with mineral content measure, and so on. The point feature encoder (nc())
encodes such features V; into a feature embedding e[v;] E Rd"") The implementation of Enc (*) () depends on the nature of these features. For example, if each point represents a POI with multiple POI types (as in this study), the feature embedding el e[v:] can simply be the mean of each POI types'
الا tn where tf()) indicates the fith POI type embedding of a POI pa with H POI types. We apply L2 normalization the POI type embedding matrix.
embeddings e[ve]
Point Space Encoder A part of the novelty of this paper is from the point space encoder (). We first introduce Theorem I which provide an analytical solution ((x) as the base of encoding any location x E IR22 in 2D space to distributed representation:
Theorem 1. Let (x) (e*a,ii)), .j 1, 3) E C3 where e0 cos O + isn 0 is the Euler notation of complex values; (a,.x) the inner product of a, and a1. a2, ag E R are 2D vectors such that the angle between aę and a; is 2 /3. Vj, ||aı|| 2 Les C 33x3 be random complex matrix such as C*C II. Then Φ(x) (x(()) M Ax) = Cdiag( T (Ax)) C satisfies
o(x + Ax) = M (Ax)((x))
(I)
and
(Ø(x + Ax), o(x)) d(1 a Ax||?
(2)
where d ==3 3 the dimension of 6(x) and Ax is small displacement from
The proof of Theorem I can be seen in Gao et al. (2019). o(x) = CP(() E C amounts to a
6-dimension real value vector and each dimension shows hexagon firing pattern which models the
grid cell behavior. Because of the periodicity of sin() and cos(), this single scale representation o(x)
does not form a global codebook of 2D positions, i.c. there can be x + y. but Φ(x) @(y).
Inspired by Theorem I and the multi-scale periodic representation of grid cells in mammals (Ab
bott & Callaway 2014) we set up our point space encoder e[x]
Enceheory
Fen(f) (w)
(x) to sine
4

---

Published as a conference paper at ICLR 2020
and cosine functions of different frequencies to encode positions in space. Given any point x in
the studied 2D space, the space encoder Enchheory xx NN(PE((())) where PE (x)
(a)
(x): (x)) (x); ...;;PP PE.
(x)] is a concatenation of multi-scale representations of d(z)
65 dimensions. Here is the total number of grid scales and 8 = 0, 1, S NN)) represents fully connected ReLU layers. Let a1 = [1, 0J", a2 = [-1/2, v v3/2]" a3 [-1/2, - -V3222 e R² be three unit vectors and the angle between any of them is 2m/33. Amin: Amar are the minimum and maximum grid scale and = is a concatenation of three components, where
Amuz, At each scale s, PE. (x) (x):
(x); PE PE (x)]
Aminn
(x, ax)
(x, a
PE (x) = [cos(
Amin g*/(5-1) ( sin(
Aminn ge/(s-1) 1,,,,
1,2,3
(3)
NN() and (x( are analogies of C and T (x) in Theorem
Similarly we can define another space encoder Enc( (x)d(()) NN(PE(()x)) inspired by
the position encoding model of Transformer Vaswani et al. 2017). where PE())(x
[PE (x); P.!.) (x); PE (x)] is still concatenation its multi-scale representations,
while PE!") (x) [PE] (x): P P [x)] handles each component of x separately:
x
x!
PF( 0.1 (x)
[cos(
sin
g 1AAl 1,2
Amin (*/(s-1)
Amin
4.2 DECODER
Two types of decoders are designed for two major types of GIS problems: location modeling and spatial context modeling (See Section55.1)
Location Decoder Dec,0) directly reconstructs point feature embedding e[v.] given its space embedding c[x;]. We use one layer feed-forward neural network NNo
e[v.J' = Dec, (x,i Osec,) = (exx)(((x.)
(5)
For training we use inner product to compare the reconstructed feature embedding elv; against the real feature embeddings of e(v.] and other negative points (see training detail in 4.3).
Spatial Context Decoder Dec,() reconstructs the feature embedding e[v.] of the center point p4 based the space and feature embeddings {e:1, Biy, ..., emn of nearby points {Pa ..., Pijy.... }. Note that the feed-in order of context points should not affect the predic- tion results, which can be achieved by permutation invariant neural network architectures (Zaheer et al. 2017) like PointNet (Qi et al. 2017).
1
K
elv.J'= = Deco(xi, (ea eij. ein); Odcc.)
= g(
(6)
K
Here g is activation function such sigmoid. Origk jth neighbor through the kth attention head, and
ezp(assk) Dem1 exp(atoo)
is the attention of pa with its
Jisk LeakyReLU (af e/y - x;,Il)
where E IR Id +d is the attention parameter in the kth attention head. The multi-head attention
mechanism is inspired by Graph Attention Network (Veličković et al. 2018) and Mai et al. (2019a).
To represent the spatial relationship (distance and direction) between each context point Ptj (Xij. Vij) and the center point p; (X;; Vi), we use the space encoder Enc(#)( encode the displacement between them Ax,y = X; Xij. Note that we are modeling the spatial interactions between the center point and n context points simultaneously.
In Eq. e[va]init indicates the initial guess of the feature embedding e[v.] of point p: which is com- exp(o(st)
puted by using another multi-head attention layer as Eq. 6 where the weight d'y En ezp(o Here, ofk is computed Eq. g where the query embedding e[v:] is excluded.
o lgk LeakyReLU (elv (alkksRReeL((TTTll))))ellele x1,ll)
(8)
5

---

Published as a conference paper at ICLR 2020
4.3 UNSUPERVISED TRAINING
The unsupervised learning task can simply be maximizing the log likelihood of observing the true point pj position x; among all the points in P
exp(efv.1'eevv.1
(0 - Σ log P(pilpa, -.. Pij..... Pin)
Σ log
_. exp(e[v.]"eevvj]
P.PP
P.EP
Here only the feature embedding of p; is used (without location embedding) to prevent revealing the identities of the point candidates, and 0 [Oenc; Odec]
Negative sampling by Mikolov et al. 2013 can be used to improve the efficiency of training
(0)
Σ
(oogo(e[v:]'e(v.]] +
log logo<(e(vo))eev))
(10)
[M
PIEP
PoEN.
Here NG S P is set of sampled negative points for p: (ps $ M2) and (x) = 1/(1 + e-a).
5 EXPERIMENT
In this section we compare Space2Ves Vec with commonly used position encoding methods, and analyze them both quantitatively and qualitatively.
Baselines Our baselines include 1) direct directly applying feed-forward nets (Chu et al. 2019);
2) tile discretization (Berg al. 2014 AAdms et al. 2013 Tang et al.
2015); 3) wrap forward Radial Basis Function (RBF)
nets with coordinate wrapping (Mae Aodha et al. 2019); and 4) rbf
kernels (Baudat & Anouar 2001 Bierens] 1994). See Appendix A. for details of the baselines.
5.1 POI TYPE CLASSIFICATION TASKS
Dataset and Tasks To test the proposed model, conduct experiments on geographic datasets with POI position and type information. We utilize the open-source dataset published by Yelp Data Challenge and select all POIs within the Las Vegas downtown area There are 21,830 POIs with 1,191 different POI types in this dataset. Note that each POI may be associated with one more types, and do not any other meta-data such business names, reviews for this study. We project geographic coordinates into projection coordinates using the NAD83/Conus Albers projection coordinate system The POIs are split into training, validation, and test dataset with ratios 80%: 10%:10%. We create two tasks setups which represent different types of modeling need in Geographic Information Science:
Location Modeling predicts the feature information associated with a POI based on its location x, represented by the location decoder Dec,(). This represents large number of location prediction problems such as image fine grained recognition with geographic prior (Chu al. 2019, and species potential distribution prediction (Zuo al. 2008).
Spatial Context Modeling predicts the feature information associated with a POI based on its context (e...... eij. ..., ein) represented by the spatial context decoder Deco(). This represents collections of spatial context prediction problem such spatial context based facade image classification (Yan et al. 2018). and all spatial interpolation problems.
We use POI prediction metrics to evaluate these models. Given the real point feature embedding c[v;] and negative feature embeddings MG {c[v,]=}. we compare the predicted c[v,]' with them by cosine distance. The cosine scores are used to rank e[v.] and N negative samples. The negative feature embeddings are the feature embeddings points py randomly sampled from P and p. Pj. We evaluate each model using Negative Log-Likelihood (NLL), Mean Reciprocal Rank (MRR) and HIT@5 (the chance the true POI being ranked to top 5. We train and test each model 10 times estimate standard deviations. See Appendix A.2 for hyper-parameter selection details.
5.1.1 LOCATION MODELING EVALUATION
We first study location modeling with the location decoder Dec, in Section 4.2 sample size of N 100. Table shows the average metrics of different models with their best hyper-
We use negative
'The geographic range is (35. 989438, 36.270897) for latitude and (-1 (-115.047977, -1.5.3290609) for longitude. htttss:/epsg... 1

---

Published as a conference paper at ICLR 2020
(a) direct
(b) tile
(c) wrap
(d) rbf (o=lk) (c) Amon=lk (1) Amaon 5500 (g) Amiin =50
Figure 2:
Embeddiigg clustering of (a) direct; (b) tile with the best cell size c - 500; (c) wrap (h- 3, 512); (d) rbʃ with the best (Ik) and 200 anchor points (red) and (e)D(h) theory models with different Amin. but fixed Amez = 40k and S = 64. All models use hidden ReLU layers of 512 neurons except wrap.
Table 1: The evaluation results of different location models on the validation and test dataset.
Train NLL
Validation MRR
Testing
NLL
HIT@5
MRR
HIT@5
random
0.052 (0.002)
4.8 (0.5)
0.051 (0.002)
5.0 (0.5)
direct
1.285
1.332
0.089 (0.001)
10.6 (0.2)
0.090 (0.001)
11.3 (0.2)
tile (c=500)
1.118
1.261
0.123 (0.001)
16.8 (0.2)
0.120 (0.001)
17.1 (0.3)
wrap(h==,0=511)
1.222
1.288
0.112 (0.001)
14.6 (0.1)
0.119 (0.001)
(0.2)
rbf (a=1k)
1,209
1.279
0.115 (0.001)
15.2
0.123 (0.001)
16.8 (0.3)
grid (mmon=50))
1.156
I,258
0.128 (0.001)
(0.3)
0.139 (0.001)
20.0)
hexa (Amin=50)
1.230
1.297
0.107 (0.001)
14.0 (0.2)
0.105 (0.001)
14.5
theoryding AAin==5)
1.277
1.324
0.094 (0.001)
12.3 (0.3)
0.094 (0.002)
11.2 (0.3)
theory (mmi==1k)
1.207
1.281
0.123 (0.002)
16.3 (0.5)
0.121 (0.001)
(0.1)
theory (Amin.5000
1.188
1.269
0.132 (0.001)
17.6 (0.3)
0.129 (0.001)
17.7 (0.2)
theory OAmin 550)
1.098
1.249
0.137 (0.002)
19.4 (0.1)
0.144 (0.001)
(0.2)
parameter setting on the validation set. We can that direct and theorydiag are less competitive, only beating the random selection baseline. Other methods with single scale representations - including tile, wrap, and rbf - perform better. The best results come from various version of the grid cell models, which are capable of dealing with multi-scale representations.
In order to understand the reason for the superiority of grid cell models we provide qualitative analysis of their representations. We apply hierarchical clustering to the location embeddings produced by studied models using cosine distance as the distance metric (See Fig. 2 we can see that when restricted to large grid sizes (Amin = 1k,, theory has similar representation (Fig. 2d 2e, and Fig. 4d 4e) and performance compared to rbf (a = 1k). However it is able to significantly outperform rbf (a 1k) (and tile and wrap) when small grid sizes (Amin 500, 50) are available. The relative improvements over rbf (a 1k) are -0.2%, +0.6%, +2.1% 1% MRR for Alin= 500, 50 respectively.
5.1.2 MULTI-SCALE ANALYSIS OF LOCATION MODELING
In order to show how our multi-scale location representation model will affect the prediction of POI types with different distribution patterns, we classify all 1,191 POI types into three groups based on radius r, which is derived from each POI types' renormalized Ripley's 's K curve (See Figure Id for examples). It indicates the x axis value of the intersection between the curve and the line of y = 3.0. A lower indicates more clustered distribution patterns. These three groups are listed below:
1. Clustered (r & 100m):: POI types with clustered distribution patterns;
2. Middle (100m < r < 200m): POI types with less extreme scales;
3. Even (r >/ 200m): POI types with even distribution patterns.
Table Ishoe the performance (M RR) of direct, tile, wrap, rbf, and our theory model on the test dataset of the location modeling task with respect to these three different POI distribution groups. The numbers in () indicate the MRR difference betweeb baseline and theory. # POI refers to total number of POI belong to each grou We can see that 1) The two neural net approaches (direct and wrap) have no scale related parameter and are not performing ideally across all scales, with direct
'The reason why the sum of # POI of these three groups does not equal to the total number of POI is because one POT can have multiple types and they may belonging different groups
7

---

Published as a conference paper at ICLR 2020
Table 2: Comparing performances in different POI groups. We classify all 1,191 POI types into three groups based on the radius of their root types, where their renormalized Ripley's K curve (See Figure [d] reach 3.0: 1) Clustered (r & 100m): POI types with clustered distribution patterns; 2) Middle (100m < r < 200m)): POI types with unclear distribution patterns; 3) Even (r 200m): POI types with even distribution patterns. The MRR of wrap and theory on those three groups are shown. The numbers in () indicate the difference between the MRR of baseline model and the MRR of theory with respect to a specific group. #POI refers to the total number of POIs belonging to each group. Root Types indicates the root categories of those POI types belong to each group.
POI Groups
Clustered
Middle
Even
(r v 100m)
(100m 200m)
(r e 200m)
direct
0.080 (-0.047)
0.108 (-0.030)
0.084 (-0.047)
wrap
0.106 (-0.021)
0.126 (-0.012)
0.122 (-0.009)
tile
0.108 (-0.019)
0.135 (-0,003)
0.111 (-0.020)
rbf
0.1 12 (-0.015)
0.136 (-0.002)
0.119 (-0.012)
theory
0.127 (-)
0.138 (-)
0.131 (-)
# POI
16,016
7,443
3,915
Restaurants; Shopping: Food;
Beauty & Spas; Health & Medical;
Home Services; Event Planning & Services; Pets; Education
Nightlife; Automotive; Active
Local Services; Hotels & Travel;
Root Types
Life; Arts & Entertainment;
Professional Services;
Financial Services
Public Services & Govermment
performs worse because of its simple single layer network. 2) The two approaches with built-in scale parameter (tile and rbf) have to trade off the performance of different scales. Their best parameter settings lead to close performances to that of Space2 Vec at the middle scale, while performing poorly both clustered and regular groups. These observation clearly shows that all baselines can at most well handle distribution at one scale but show poor performances in other scales. In contrast, Space2 Vec's multi-scale representation can handle distributions at different scales.
5.1.3 SPATIAL CONTEXT MODELING EVALUATION
We use the same evaluation set up as location modeling. The context points are obtained by querying the n-th nearest points using PostGIS (n 10). As for validation and test datasets, we make sure the center points are all unknown during the training phase. Table shows the evaluation results of different models for spatial context modeling. The baseline approaches (direct, tile, wrop, rbf) generally perform poorly context modeling. We designed specialized version of these approaches (polar, polar _fle, scaled rbf) with polar coordinates, which lead to significantly improvements. Note that these are models proposed by us specialized for context modeling and therefore are less general than the grid cell approaches.
Next, we evaluate the spatial context decoder Deca() Sec. 4.2
Table 3: The evaluation results of different spatial context models on the validation and test dataset. All encoders contains hidden layer FFN. All grid cell encoders set Amin Amaz=1a =10k.
Train NLL
Validation MRR
Testing
Space2Vec
NLL
HIT@5
MRR
HIT@5
none
1.163
1.297
0.159 (0.002)
22.4
0.167 (0.006)
23.4 (0.7)
direct
1.151
1.282
0.170 (0.002)
24.6 (0.4)
0.175 (0.003)
24.7 (0.5)
polar
1.157
1.283
0.176 (0.004)
25.4
0.178 (0.006)
24.9 (0.1)
life (c 50)
1.163
1.298
0.173 (0.004)
(0.6)
0.173 (0.001)
23.4 (0.1)
polar_tilet (ilar_tieeeSee 64)
1.161
1.282
0.173 (0.003)
(0.1)
0.177 (0.001)
24.5 (0.3)
wrop (h=2,0=512)
1.167
1.291
0.159 (0.001)
(0.1
0.170 (0.001)
23,9 (0.2)
rbf (a 50)
1.160
1.281
0.179 (0.002)
25.2
0.172 (0.001)
(0.00
acaled_rbf (o=40,8=0.1)
1.150
1.272
0.177 (0.002)
0.181 (0.001)
25.3 (0.1)
grid(Amiin=100
I.172
1,285
0.178 (0.004)
(0.9
0.181 (0.001)
(0.3)
hera (Amss==0)
1.156
1.289
0.173 (0.002)
(0.2)
0.183 (0.002)
25.3 (0.2)
theorydiag (Ammn = 10)
1.156
1.287
0.168 (0.001)
24.1 (0.4)
0.174 (0.005)
24.9 (0.1)
theory(Xmin=20)
1.168
1.295
0.159 (0.001)
23.1
0.170 (0.001)
23.2 (0.2)
theory(Amannn500
1.157
1.275
0.171 (0.001)
24.3)
0.173 (0.001)
24.8 (0.4)
theory(Amsn=1100
1.158
1.280
0.177 (0.003)
25.2 (0.3)
0.185 (0.002)
25.7 (0.3)
8

---

Published as a conference paper at ICLR 2020
direct
polar
(c) wrop
(d) polar_tile
(c) scaled rbf
theory
(g) direct
(h) polar
(i) wrap
( polar_tice
(k) scaled
(l) theory
Figure 3: Embedding clustering in the original space of (a) direct; (b) polar; (c) wrop, h=2,o=512; (d) polar __le, S 64, (e) scaled_rbf,
40, 3=0.1; and (f) theory, Amin 10, Amar 10k,
S 64. (())))))((( are the clustering results of the same models in the polar-distance space using log( Ax₁ +1). All models use 1 hidden ReLU (except wrap) layers of 512 neurons. Most models except wrap can capture a shift when distance is around e - 150 meters.
Nevertheless the grid cell approaches are able to perform better than the specialized approaches on the test dataset while have competitive performance on validation dataset. See Appendix ?? for the visualization of context models. Actually the gains are small for all baseline approaches also. The reason that we expect location encoding be less important when context information accessible. Similarly as discussed in (Gao et al. 2019), it is when there is a lack of visual clues that the grid cells of animals are the most helpful for their navigation.
Figure 6 shows the location embedding clustering results in both Cartesian and polar coordinate systems. We can see that direct (Fig.
3a
3g) only captures the distance information when the context 5) while in the farther spatial context it purely models the
POI is very close (log(I) Ax.j +1
&
3h has the similar behaviors but captures the distance infor- mation in more fine- fine-grained manner. wrap (Fig. 3c| 3i mainly focuses on differentiating relative positions in farther spatial context cont which might explain its lower performance" polar_ter file (Fig. 3d) mostly responds to distance information. Interestingly, scaled_rbh rbf and theory have similar representations in the polar coordinate system (Fig. (I) and similar performance (Table While scaled_rbi captures the gradually decreased distance effect with scaled kernel size which becomes larger in farther distance, theory achieves this by integrating representations of different scales.
direction information. polar (Fig. 3b
5.2 FINE-GRAINED IMAGE CLASSIFICATION TASKS
To demonstrate the generalizability of Space2Vec for space representation utilized the proposed point space encoder Enc(")() model in well-known computer vision task: fine-grained image classification. As we discussed in Section 3 many studies (Berg et al.) 2014; Chu et al. 2019) Mac Aodha et al. 2019) have shown that geographic prior information where (and when) the image taken is very important additional information for the fine-grained image classification task and can substantially improve the model performance. For example, the appearance information usually not sufficient to differentiate two visually similar species. In this case, the geographic prior becomes much more important because these two species may have very different spatial prior distributions such as the example of European Toads and Spiny Toads in Figure of Mac Aodha et al. (2019).
We adopt the task setup of Mac Aodha al. (2019). During training have set of tuples {[I,, X;, Vi, Pt) = I,.., Nj where 1; indicates an image, y/ E {1,2, .... C} is the corresponding class label (species category), x; [longitude, latitude;] the geographic coordinates where the image was taken, and is the id of the photographer who took this image. At training time, location encoder is trained to capture the spatial prior information P(y x). At inference time, p; information is not available and the final image classification prediction is calculated based on the
'Note that wrap is original proposed by Mac Aodha et al. modelling. This results indicates wrap not good this task.
2019 for location modelling, not spatial context

---

Published as a conference paper at ICLR 2020
Table 4: Fine-grained image classification results on two datasets: BirdSnapt and NABirdst. The classification accuracy is calculated by combining image classification predictions P(y 7) with different spatial priors P(y x). The grid and theory model use 1 hidden ReLU layers of 512 neurons. The evaluation results of the baseline models are from Table I of Mac Aodha et al. (2019).
BirdSnap²
NABirdsf
No Prior (i.e. uniform)
70.07
76.08
Nearest Neighbor (num)
77.76
79.99
Nearest Neighbor (spatial)
77.98
80.79
Adaptive Kernel (Berg al.
2014
78.65
81.11
tile (Tang et al.
2013) (location only)
77.19
79.58
wrap (Mac Aodha al.
2019 (location only)
78.65
81.15
rbf (o=1k)
78.56
81.13
grid (Amin=0.0001, Amax S = 64)
79.44
81.28
theory (Amim=000000 3300, S = 64)
79.35
81.59
combination of two models: 1) the trained location encoder which captures the spatial priors P(y x) and 2) the pretrained image classification model, Inception network (Szegedy et al. 2016), which captures P(y I). Bayesian theory has been used to derive the joint distribution P x.x). .. See
Mac Aodha et al. (219)) for detail explanation well as the loss function. Note that while Space2Vec outperforms specialized density estimation methods such as Adaptive Kernel (Berg et al. 2014), would be interesting explore early fusion Space2 's representations with the image module.
We use two versions of our point space encoder Enc(*)() model (grid, theory) as the location encoder to capture the spatial prior information P(y x). The evaluation results of our models as well We can that both grid, theory outperform previous
multiple baselines models as well as that of Mac Aodha et al. (2019) on two fine-grained image classification datasets with significant sizes: BirdSnap', N. NABirdst. theory shows superiority over grid on NABirdst while fail to outperform grid on BirdSnapt. Note that we only pick baseline models which capture spatial-only prior and drop models which additionally consider time information. Both grid and theory use 1 hidden ReLU layers of 512 neurons for NN() and they have the same hyperparameters: Ammr=000001, Am62=360, S = 64. Like Mac Aodha et al. (2019), the location embedding size d(x) is 1024 and we train the location encoder for 30 epochs. Our implementation is based on the original code of Mac Aodha et al. (2019) for both model training and evaluation phase.
shown in Table 4
CONCLUSION
We introduced an encoder-decoder framework as general- general-purpose representation model for space inspired by biological grid cells' multi-scale periodic representations. The model is an inductive learning model and can be trained in an unsupervised manner. We conduct two experiments on POI type prediction based on 1) POI locations and 2) nearby POIs. The evaluation results demonstrate the effectiveness of our model. Our analysis reveals that it is the ability to integrate representations of different scales that makes the grid cell models outperform other baselines on these two tasks. In the future, we hope to incorporate the presented framework to more complex GIS tasks such as social network analysis, and sea surface temperature prediction.
ACKNOWLEDGMENTS
The presented work is partially funded by the NSF award 1936677 C-Accel Pilot Track A1 (Open Knowledge Network): Spatially-Fxplici Models, Methods, And Services For Open Knowledge Networks, Esri Inc., and Microsoft Al for Earth Grant: Deep Species Spatio-tempord Distribution Modeling for Biodiversity Hotspot Prediction. We thank Dr. Ruiqi Gao for discussions about grid cells, Dr. Wenyun Zuo for discussion about species potential distribution prediction and Dr. Yingjie Hu for his suggestions about the introduction section.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Alison Abbott and Ewen Callaway. Nobel prize for decoding brain's ' sense of place. Nature News, 51437521):153, 2014.
Benjamin Adams, Grant McKenzie, and Mark Gahegan. Frankenplace: interactive thematic mapping for ad hoc exploratory search. In Proceedings of the 24th international conference on world wide web, pp. 12-22. International World Wide Web Conferences Steering Committee, 2015.
Andrea Banino, Caswell Barry, Benigno Uria, Charles Blundell, Timothy Lillicrap, Piotr Mirowski, Alexander Pritzel, Martin Chadwick, Thomas Degris, Joseph Modayil, et al. Vector-based navigation using grid-like representations in artificial agents. Nature, 557(7705):429, 2018.
G Baudat and F Anouar. Kernel-based methods and function approximation. volume 2, pp. 1244 1249 vol.2, 02 2001. ISBN 0-7803-7044-9, doi: 10.1 101/11CNNN.20011999333
Thomas Berg, Jiongxin Liu, Seung Woo Lee, Michelle L Alexander, David W Jacobs, and Peter N Belhumeur. Birdsnap: Large-scale fine-grained visual categorization of birds. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 201 1-2018, 2014.
Herman J. Bierens. The nadaraya- aattaray--wattnn watson kernel regression function estimator. Topics in Advanced Econometrics, 16:212-244 247, 1994.
Hugh T Blair, Adam C Welday, and Kechen Zhang. Scale-invariam memory representations emerge from moire interference between grid fields that produce theta oscillations: a computational model. Journal of Neuroscience, 27(12):3211-3222 2007.
Grace Chu, Brian Potetz, Weijun Wang, Andrew Howard, Yang Song, Fernando Brucher, Thomas Leung, and Hartwig Adam. Geo-aware networks for fine grained recognition. arXiv preprint 1XX6:1196001737 2019.
Christopher J Cueva and Xue-Xin Wei. Emergence of grid-like representations by training recurrent neural networks to perform spatial localization. arXiv preprint arXiv: 1rXiv:1803.07770 2018.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint / 10.0000, 2018.
The modifiable areal unit problem in multivariate statistical analysis. Environment and planning A, 23(7):1025-1044 1991.
Stewart Fotheringham and David WS Wong.
Ruiqi Gao, Jianwen Xie, Song-Chun Zhu, and Ying Nian Wu. Learning grid cells as vector represen- tation of self- self-position position coupled with matrix representation of self-motion In Proceedings of ICLR 2019, 2019.
Nataliia Kussul, Mykola Lavreniuk, Sergii Skakun, and Andrii Shelestov. Deep learning classification of land cover and crop types using remote sensing data. IEEE Geoscience and Remote Sensing Letters, 14(5):778-78) 2017.
Alex M Lechner, William T Langford, Simon D Jones, Sarah A Bekessy, and Ascelin Gordon. Inves- tigating species-environee environment relationships at multiple scales: Diffcrentiating between intrinsic scale and the modifiable arcal unit problem. Ecological Complexity, 1 11:91-102, 2012.
Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffussonn convolutional recurrent neural network:
Data-driven traffic forecasting. arXiv preprint arXiv: 2017.
Oisin Mac Aodha, Elijah Cole, and Pietro Perona. Presence-only geographical priors for fine-grained image classification. arXiv preprint 1xxiv:1906.05272 2019.
Gengchen Mai, Krzysztof Janowiez, Bo Yan, Rui Zhu, Ling Cai, and Ni Lao. Contextual graph attention for answering logical queries over incomplete knowledge graphs. In Proceedings of the 10th International Conference Knowledge Capture, pp. 171-178, 2019a.
Gengchen Mai, Bo Yan, Krzysztof Janowiez, and Rui Zhu. Relaxing unanswerable geographic questions using a spatially explicit knowledge graph embedding model. In AGILE: The 22nd Annual International Conference on Geographic Information Science, pp. 21-39. Springer, 2019b.
11

---

Published as a conference paper at ICLR 2020
V Maz' ya and G Schmidt. On approximate approximations using gaussian kernels. IMA Journal of Numerical Analysis, 16:13-29 29, 01 1996.
Grant McKenzie, Krzysztof Janowicz, Song Gao, Jiue-An Yang. and Yingjie Hu. Poi pulse: A multi-granular, semantic signature-based information observatory for the interactive visualization of big geosocial data. Carlographica: The International Journal for Geographic Information and Geovisualization, 50(2):71-85 2015.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg $ Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality, In Advances in neural information processing systems, pp. 3111-3119, 2013.
Justin Moat, Steven Bachman, Richard Field, and Doreen S Boyd. Refining area of occupancy to address the modifiable areal unit problem in ecology and conservation. Conservation biology, 32 (6): (6):1277-2289, 2018.
Stan Openshaw. The modifiable areal unit problem. Concepts and techniques in modern geography, 1984.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings the 2014 conference empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Matthew E Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint arXiv: 1802.05365, 2018.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 652-660, 2017.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Kevin Tang, Manohar Paluri, Li Fei-Fei, Rob Fergus, and Lubomir Bourdev. Improving image classification with location context. In Proceedings of the IEEE international conference on computer vision, pp. 1008-1016.2 2015.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems. pp. 5998-6008, 6008, 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. In ICLR 2018, 2018
Bo Yan, Krzysztof Janowiez, Gengehen Mai, and Song Gao. From itdl to place2vec: Reasoning about place type similarity and relatedness by learning embeddings from augmented spatial contexts. In Proceedings of the 25th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems, pp. 35, ACM, 2017.
Bo Yan, Krzysztof Janowicz, Gengchen Mai, and Rui Zhu. xnet+ Classifying places based on images by incorporating spatial contexts. In 10th International Conference on Geographic Information Science (GIScience 2018). Schloss Dagstuhl-ecbniiz-ZZntrum fuer Informatik, 2018.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan R Salakhutdinov. and Alexander Smola. Deep sets. In Advances in neural information processing systems, pp. 3391-3401, 2017.
Wenyun Zuo, Ni Lao, Yuying Geng, and Keping Ma. Geosvm: an efficient and effective tool to predict species' potential distributions. Journal of Plant Ecology. 1(2):143-145, 2008
12

---

Published as a conference paper at ICLR 2020
A
APPENDIX
A. BASELINES
To help understand the mechanism of distributed space representation we compare multiple ways of encoding spatial information. Different models use different point space encoder (nc(=)( encode either location x, (for location modeling loc) or the displacement between the center point and one context point Ax, = x, Xij (for spatial context modeling cont
random shuffles the order of the correct POI and N negative samples randomly as the predicted ranking. This shows the lower bound of each metrics.
direct directly encode location x; (or Ax., for conl) into a location embedding e[x,] (or ef e[Ax,,) 1) using feed-forward neural networks (FFNs denoted Enc (x) without decomposing coordinates into a multi-scale periodic representation. This is essentially the GPS encoding method used by Chu et al. (2019). Note that al. (2019) is not open sourced and we end up implementing the model architecture ourselves.
tile divides the study area Atoc (for loc) or the range of spatial context defined by Amazz Acant, (for cond) into grids with equal grid sizes c. Each grid has an embedding be used as the encoding for every location x; or displacement Ax.y fall into this grid. This is a common practice by many previous work when dealing with coordinate data (Berg et al.
2014 Adams et al. 2015 Tang et al. 2015).
wrap a location encoder model recently introduced by Mac Aodha et al. (2019). first normalizes (or Ax) into the range [-1, 1] and uses coordinate wrap mechanism [sin( c convert each dimension of into 2 numbers. This is then passed through an initial fully connected layer, followed by a series of h residual blocks, each consisting of two fully connected lavers (o hidden neurons) with a dropout layer in between. We adopt the official code Mac Aodha et al. (2019 10 for this implementation.
rbf randomly samples M points from the training dataset as RBF anchor points [x [xanchor m 1...M) } (or samples M Axanchor from Acont for cont) [II and use gaus-
x; -x xatcchor
112) (or exp Ax,, Axanchor
1| for conl)
X-m
sian kernels exp(
2o²
20
each anchor points, where a is the kernel size. Each point p; has M-dimension RBF feature vector which is fed into a FNN to obtain the spatial embedding. This is a strong baseline for representing floating number features in machine learning models.
grid as described in Section 4. inspired by the position encoding in Transformer (Vaswani et al. 2017).
heza Same as grid but use sin(O), sin(0 0 + 2T/3),, and sin + 4m/3) in PE( (x).
theory as described in Section uses the theoretical models (Gao et al. 2019) as the first
layer of Enccheory (x) or Fhecyh (Axiz).
thcorydiag further constrains NN() block diagonal matrix, with each scale block.
We also have the following baselines which are specific to the spatial context modeling task.
none the decoder Dec=() does not consider the spatial relationship between the center point and context points but only the co-locate patterns such as Place2Vec (Yan et al. 2017). That
means we drop the e[Ax;j] from the attention mechanism in Equ. 7 |aand 8
polar first converts the displacement Ax;y into polar coordinates (r, 8) centered at the center point where log(Il Axij || +1). Then it uses [r. 0r as the input for a FFN to obtain the spatial relationship embedding in Equ. 7 We find out that it has a significant performance improvement over the variation with Axig ||.
We will use meter as the unit of Amsn, Amaz, JT, we first normalizes (or Ax) into the range [-1,1]
"htpp:/ / /ww..w vi s oioon ca tech. techhedu///eeeeeeeeeeeeeeee http://chhe/rrgjjccs////ooppoeee
these anchor points fixed in hoth loc and cont.
13

---

Published as a conference paper at ICLR 2020
polar tile is modified version of tile but the grids are extracted from polar coordinates (r, 0) centered at the center point where r = log( Ax,; || +1). Instead of using grid size c, the number of grids along (or r) axis, F. as the only hyperparameter. Similarly, We find that r = log(I) Ax,j II [11 outperform r =1 Ax;y || significantly.
scaled_rby rbf / modified version of rbf for cont whose kernel size is proportional to the distance between the current anchor point and the origin, ||
Axauchor ||. That is
exp (- Ax,; Axanchor
12
B Axarchor where is the basic kernel size and 3 is kernel rescale factor, constant. We developed this mechanism help RFB to deal with relations different scale, and we observe that produces significantly better result than vanilla RBFs.
Here dscaled
20 soated
A.2 HYPER-PARAMETEE SELECTION
We perform grid search for all methods based on their performance on the validation sets.
Location Modeling The hyper-parameters of theory models are based on grid search with d(v) (32, 64, 128, 256), d(x) (32, 64, 128, 256).
(4, 8, 16, 32, 64, 128), and Amin (1,5, 10, 50, 100, 200, 500, 1k) while Amaz 40k is decided based on the total size of the study area. We find the best performances of different grid cell based models are obtained when d(r) 64, d(²) 64, S 64, and Aman 50. In terms of tile, the hyper- syper-parameters
se- lected from c = (10, 50, 100, 200, 500, 1000) while = 500 gives us the best performance.As the hyper-parameters: M (10,50, 100, 200, 400, 800) and a = (102, 10 100,100 10 10 100 107). The best performance of rbf is obtain when M 200 and 10 As for wrap, grid search is performed on: h (1, 2, 3, 4) and while = 3 and o 512 gives the best result. All models use FFNs in their Enc except wrap. The number of layers f and the number of hidden state neurons of the FFN are selected from (1 and (128, 256, 512). We find out f 1 and 512 give the best performance for direct, tile, rbJ, and theory. So we use them for every model for a fair comparison.
for rbf, we do grid search
(64, 128, 256, 512)
Spatial Context Modeling Grid search is used for hyperparameter tuning and the best performance
of different grid cell models is obtain when (() 64, ddx) = 64, S = 64, and Amin 10. We set Amaz 10k based on the maximum displacement between context points and center points make
the location encoding unique. As for multiple baseline models, grid search used again to obtain the best model. The best model hyperparameters shown in () besides the model names in Table 3 Note that both rbf and scaled_rbl rbf achieve the best performance with 100.
A.3 FIRING PATTERN FOR THE NEURONS
(a) direct
(b) tále (c=500)
(c) wrap (h=3,o=512)
(d) rbf (a=1k)
(c) theory (mmn=11k)
(f theory (mmn=550)
Figure 4: The firing pattern for the first neurons (out 64) given different encoders location modeling.
14

---

Published as a conference paper at ICLR 2020
A.4
EMBEDDING CLUSTERING OF RBF AND THEORY MODELS
1000
200
3500
1004
-200
E346
1008
100
990
5000
2000
1109
-5204
2000
3900
16X
S099
1000
0008
1293
100
(a) Ammn= 200
(b) Amiin 100
(c) Amin 550
(d) Amin= =10
Ditarce
(c) Amin 200
(f) Amann = 100
Amin=100
(g) Amin=50
(h) Ama==10
original space (a)()) Lheory with different Amins but same Amar 10k and S - 64. (e)()())hh are the embedding clustering results of the same models in the polar-distance space. All models use hidden ReLU layers of 512 ncurons.
Figure Embedding clustering
30000
30302
1000
3000
-000
5500
-00X
(a) 8=0.0
(b) 8=0.1
(c) 8-0.2
(d) 9=0.3
A99A
(e) 3=0.0
(l) 8=0.1
8=0.2
(h) 3=0.3
Figure 6: Embedding clustering of RBF models with different kernel rescalar factor 3 (a) in the original space; (e) in the distane space. Here 8=0.0 indicates the original RBF model. All models use 0=10m as the basic kernel size and hidden ReLU layers 512 neurons.
15


Reviews:
[{"review_id": "SyxmYOC6KB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces Space2Vec, a space representation learning model. The work is motivated by the biological grid cell’s multi-scale periodic representations and the success of representation learning of NLP. So, the key idea behind the model is two-fold. On one hand, utilize the position information and the context associated with the position. On the other hand, the authors build a multiscale point space encoder based on Theorem 1 (in the paper), which was previously proved by Gao et al. (2019).\nThe multi-scale point feature encoder is novel. The experimental results turn out that the whole model is good at predicting features using only location information but does not outperform the RBF kernel (on validation) in terms of using spatial context modeling.\nOne core selling point of this paper is dealing with location distributions with very different characteristics. This is very well motivated at the beginning. More analysis/statistics would help better understand how the model \"theory\" wins Table one. See comments on experiments.  \n\nI have some comments/questions about model architecture and also experimental results/analysis.\n\n1. Regarding Contextual Embedding\n-- The encoder of this paper is not doing much “contextual embedding”.  The encoder typically encodes features of each position independently thus lead to very local embedding. \n-- The location decoder would reconstruct the same (or similar) type of point features given embeddings of locations of the same type. As the distributions of different types of locations are very different, an encoder capable to deal with multi-scale data is crucial here. \n-- The Spatial context decoder, like a context-dependent language model, would reconstruct the current position features, given the neighboring information.\n-- Overall, unlike many existing pre-training models in NLP with deep encoders, the full model of this paper is with very local encoder, while the decoder does the most work of “gathering contextual information\". \nAs you claim your model to be \"a general-purpose space representation model\", can you describe/specify how you would use your model for other tasks? Will you take some intermediate output of decoders to be representations? Or will you fine-tune the whole encoder-decoder?\n\n2. For experiments:\na.\tI do not prefer saying your method outperforms RBF in the “spatial context modeling” task when you getting worse validation set performance. It is interesting that RBF is stronger in terms of validation.\nb.\tIs it possible for the authors to do some statistics on the different types of locations? For your first task “location modeling”, should we expect to see that your model does not have very bad performance on certain types of locations while other non-multi-scale approaches do? This is trying to provide better support to one of your core contributions.\nc.    Again, to claim the model can be widely applied, try more tasks?\n\n\nTo clarify my \"experience assessment\", I mean I read many related representation learning papers rather than specific papers related to GIS data.\nMy actual rating for the paper is between weak reject to weak accept (but the system does not have intermediate choices). I would like to hear the author's feedback to further revise the rating.\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Hye56-ChYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a new method called \"Space2Vec\" to compute spatial embeddings of a pixel in a spatial data. The primary motivation of Space2Vec is to integrate representations of different spatial scales which could potentially make the spatial representations more informative and meaningful as features. Space2Vec is trained as a part of an encoder-decoder framework, where Space2Vec encodes the spatial features of all the points that are fed as input to the framework. \n\nThey conducted experiments on real world geographic data where they predict types of point of interests (POIs) at given positions based on their 1) locations (location-modeling) and 2) spatial neighborhood (spatial context modeling). They evaluated Space2Vec against other ML approaches for encoding spatial information including RBF kernels, multi-layer feed forward nets, and tile embedding approaches. Their results indicate that that Space2Vec approach performs  better (albeit marginally) than other ML methods. \n\nI am giving this paper a weak reject rating mainly because of weak results and lack of motivation for location modeling problem (where their approach performs significantly better than baselines).   I explain my concerns below under detailed comments.\n\nDetailed Comments: \n1) Motivation of location modeling problem does not sound compelling enough to me, especially in the context of Point of Interest(POI) classification approach. I could not imagine any scenario where access to information from spatial neighborhood will be denied. If authors could present strong motivating examples for this problem and demonstrate the utility of their proposed approach in that setting, that will make the paper much stronger.  \n2) In spatial context modeling problem, the improvements in the results (Table 2) appear to be marginal(0.185 against 0.181, 25.7 against 25.3). Authors should try out more datasets to convincingly justify the superiority of their approach over other methods.\n\nEDIT: AFTER RECEIVING AUTHOR'S RESPONSE\nI am satisfied with author's response to my comments. I am updating my rating to Weak Accept. \n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1lSQ12vcS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a model that learns an embedding/representation for spatial points (POI's). There are two specific things the representations are trying to encode - location modeling and spatial context modeling and the model tries to do it in multi-scale manner to increase the information/granularity of the learnt representations.\n\nThe experiments are performed on Yelp Data challenge which has 21,830 POI's with 1191 POI types. In the location context experiments authors show that by going after a smaller grid size we can get much better results compared to other methods while other methods like tile, wrap and rbf have more parameters causing overfitting. Similarly, on spatial context modeling we see better results.\n\nOverall, the problem of learning vector representations for spatial points is interesting and useful and this paper has valuable contributions on how to do it. \n\nOne thing I would like to have seen to strengthen the paper further is the application of these representations in other tasks like image classification or recommendation systems or retrieval. The paper currently misses that. \n\nFor ex - https://arxiv.org/abs/1505.03873 uses location information to improve image classification, similarly can we use the representation learned through this method instead of positional coordinates and show that it helps the final task.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to follow inspiration from NLP method that use position embeddings and adapt them to spatial analysis  that also makes use of both absolute and contextual information, and presents a representation learning approach called space2vec to capture absolute positions and spatial relationships of places. Experiments show promising results on real data compared to a number of existing approaches.
Reviewers recognize the promise of this approach and suggested a few additional experiments such as using this spatial encoding as part of other tasks such as image classification, as well as clarification and further explanations on many important points. Authors performed these experiments and incorporated the results in their revisions, further strengthening the submission. They also provided more analyses and explanations about the granularity of locality and motivation for their approach, which answered the main concerns of reviewers.
Overall, the revised paper is solid and we recommend acceptance.
Decision: Accept (Spotlight)