TIAL FEATURE DISTRIBUTIONS USING GRID CELLS
Year: 2020

Abstract:
Unsupervised text encoding models have recently fueled substantial progress in Natural Language Processing (NLP). The key idea is to use neural networks to convert words in texts to vector space representations (embeddings) based on word positions in a sentence and their contexts, which are suitable for end-to-end training of downstream tasks. We see a strikingly similar situation in spatial analysis, which focuses on incorporating both absolute positions and spatial contexts of geographic objects such as Points of Interest (POIs) into models. A general-purpose representation model for space is valuable for a multitude of tasks. However, no such general model exists to date beyond simply applying discretization or feedforward nets to coordinates, and little effort has been put into jointly modeling distributions with vastly different characteristics, which commonly emerges from GIS data. Meanwhile, Nobel Prize-winning Neuroscience research shows that grid cells in mammals provide a multi-scale periodic representation that functions as a metric for location encoding and is critical for recognizing places and for path-integration. Therefore, we propose a representation learning model called Space2Vec to encode the absolute positions and spatial relationships of places. We conduct experiments on two real-world geographic data for two different tasks: 1) predicting types of POIs given their positions and context, 2) image classification leveraging their geo-locations. Results show that because of its multiscale representations, Space2Vec outperforms well-established ML approaches such as RBF kernels, multi-layer feed-forward nets, and tile embedding approaches for location modeling and image classification tasks. Detailed analysis shows that all baselines can at most well handle distribution at one scale but show poor performances in other scales. In contrast, Space2Vec ’s multi-scale representation can handle distributions at different scales. 1

heading: 1 INTRODUCTION
text: Unsupervised text encoding models such as Word2Vec (Mikolov et al., 2013), Glove (Pennington et al., 2014), ELMo (Peters et al., 2018), and BERT (Devlin et al., 2018) have been effectively utilized in many Natural Language Processing (NLP) tasks. At their core they train models which encode words into vector space representations based on their positions in the text and their context. A similar situation can be encountered in the field of Geographic Information Science (GIScience). For example, spatial interpolation aims at predicting an attribute value, e.g., elevation, at an unsampled location based on the known attribute values of nearby samples. Geographic information has become an important component to many tasks such as fine-grained image classification (Mac Aodha et al., 2019), point cloud classification and semantic segmentation (Qi et al., 2017), reasoning about Point of Interest (POI) type similarity (Yan et al., 2017), land cover classification (Kussul et al., 2017), and geographic question answering (Mai et al., 2019b). Developing a general model for vector space representation of any point in space would pave the way for many future applications.
1Link to project repository: https://github.com/gengchenmai/space2vec
However, existing models often utilize specific methods to deal with geographic information and often disregards geographic coordinates. For example, Place2Vec (Yan et al., 2017) converts the coordinates of POIs into spatially collocated POI pairs within certain distance bins, and does not preserve information about the (cardinal) direction between points. Li et al. (2017) propose DCRNN for traffic forecasting in which the traffic sensor network is converted to a distance weighted graph which necessarily forfeits information about the spatial layout of sensors. There is, however, no general representation model beyond simply applying discretization (Berg et al., 2014; Tang et al., 2015) or feed-forward nets (Chu et al., 2019; Mac Aodha et al., 2019) to coordinates.
A key challenge in developing a general-purpose representation model for space is how to deal with mixtures of distributions with very different characteristics (see an example in Figure 1), which often emerges in spatial datasets (McKenzie et al., 2015). For example, there are POI types with clustered distributions such as women’s clothing, while there are other POI types with regular distributions such as education. These feature distributions co-exist in the same space, and yet we want a single representation to accommodate all of them in a task such as location-aware image classification (Mac Aodha et al., 2019). Ripley’s K is a spatial analysis method used to describe point patterns over a given area of interest. Figure 1c shows the K plot of several POI types in Las Vegas. One can see that as the radius grows the numbers of POIs increase at different rates for different POI types. In order to see the relative change of density at different scales, we renormalize the curves by each POI type’s density and show it in log scale in Figure 1d. One can see two distinct POI type groups with different distribution patterns with clustered and even distributions. If we want to model the distribution of these POIs by discretizing the study area into tiles, we have to use small grid sizes for women’s clothing while using larger grid sizes for educations because smaller grid sizes lead to over- parameterization of the model and overfitting. In order to jointly describe these distributions and their patterns, we need an encoding method which supports multi-scale representations.
Nobel Prize winning Neuroscience research (Abbott & Callaway, 2014) has demonstrated that grid cells in mammals provide a multi-scale periodic representation that functions as a metric for location encoding, which is critical for integrating self-motion. Moreover, Blair et al. (2007) show that the multi-scale periodic representation of grid cells can be simulated by summing three cosine grating functions oriented 60˝ apart, which may be regarded as a simple Fourier model of the hexagonal lattice. This research inspired us to encode locations with multi-scale periodic representations. Our assumption is that decomposed geographic coordinates helps machine learning models, such as deep neural nets, and multi-scale representations deal with the inefficiency of intrinsically single-scale methods such as RFB kernels or discretization (tile embeddings). To validate this intuition, we propose an encoder-decoder framework to encode the distribution of point-features2 in space and
2In GIS and spatial analysis, ‘features’ are representations of real-world entities. A tree can, for instance, be modeled by a point-feature, while a street would be represented as a line string feature.
train such a model in an unsupervised manner. This idea of using sinusoid functions with different frequencies to encode positions is similar to the position encoding proposed in the Transformer model (Vaswani et al., 2017). However, the position encoding model of Transformer deals with a discrete 1D space – the positions of words in a sentence – while our model works on higher dimensional continuous spaces such as the surface of earth.

heading: In summary, the contributions of our work are as follows:
text: 1. We propose an encoder-decoder encoding framework called Space2Vec using sinusoid functions with different frequencies to model absolute positions and spatial contexts. We also propose a multi-head attention mechanism based on context points. To the best of our knowledge, this is the first attention model that explicitly considers the spatial relationships between the query point and context points.
2. We conduct experiments on two real world geographic data for two different tasks: 1) predicting types of POIs given their positions and context, 2) image classification leveraging their geo-locations. Space2Vec outperforms well-established encoding methods such as RBF kernels, multi-layer feed-forward nets, and tile embedding approaches for location modeling and image classification.
3. To understand the advantages of Space2Vec we visualize the firing patterns (response maps) of location models’ encoding layer neurons and show how they handle spatial structures at different scales by integrating multi-scale representations. Furthermore the firing patterns for the spatial context models neurons give insight into how the grid-like cells capture the decreasing distance effect with multi-scale representations.

heading: 2 PROBLEM FORMULATION
text: Distributed representation of point-features in space can be formulated as follows. Given a set of points P “ tpiu, i.e., Points of Interests (POIs), in L-D space (L “ 2, 3) define a function fP,θpxq : RL Ñ Rd (L ! d), which is parameterized by θ and maps any coordinate x in space to a vector representation of d dimension. Each point (e.g., a restaurant) pi “ pxi,viq is associated with a location xi and attributes vi (i.e., POI features such as type, name, capacity, etc.). The function fP,θpxq encodes the probability distribution of point features over space and can give a representation of any point in the space. Attributes (e.g. place types such as Museum) and coordinate of point can be seen as analogies to words and word positions in commonly used word embedding models.

heading: 3 RELATED WORK
text: There has been theoretical research on neural network based path integration/spatial localization models and their relationships with grid cells. Both Cueva & Wei (2018) and Banino et al. (2018) showed that grid-like spatial response patterns emerge in trained networks for navigation tasks which demonstrate that grid cells are critical for vector-based navigation. Moreover, Gao et al. (2019) propose a representational model for grid cells in navigation tasks which has good quality such as magnified local isometry. All these research is focusing on understanding the relationship between the grid-like spatial response patterns and navigation tasks from a theoretical perspective. In contrast, our goal focuses on utilizing these theoretical results on real world data in geoinformatics.
Radial Basis Function (RBF) kernel is a well-established approach to generating learning friendly representation from points in space for machine learning algorithms such as SVM classification (Baudat & Anouar, 2001) and regression (Bierens, 1994). However, the representation is example based – i.e., the resultant model uses the positions of training examples as the centers of Gaussian kernel functions (Maz’ya & Schmidt, 1996). In comparison, the grid cell based location encoding relies on sine and cosine functions, and the resultant model is inductive and does not store training examples.
Recently the computer vision community shows increasing interests in incorporating geographic information (e.g. coordinate encoding) into neural network architectures for multiple tasks such as image classification (Tang et al., 2015) and fine grained recognition (Berg et al., 2014; Chu et al., 2019; Mac Aodha et al., 2019). Both Berg et al. (2014) and Tang et al. (2015) proposed to discretize the study area into regular grids. To model the geographical prior distribution of the image categories, the grid id is used for GPS encoding instead of the raw coordinates. However, choosing the correct discretization is challenging (Openshaw, 1984; Fotheringham & Wong, 1991),
and incorrect choices can significantly affect the final performance (Moat et al., 2018; Lechner et al., 2012). In addition, discretization does not scale well in terms of memory use. To overcome these difficulties, both Chu et al. (2019) and Mac Aodha et al. (2019) advocated the idea of inductive location encoders which directly encode coordinates into a location embedding. However, both of them directly feed the coordinates into a feed-forward neural network (Chu et al., 2019) or residual blocks (Mac Aodha et al., 2019) without any feature decomposition strategy. Our experiments show that this direct encoding approach is insufficient to capture the spatial feature distribution and Space2Vec significantly outperforms them by integrating spatial representations of different scales.

heading: 4 METHOD
text: We solve distributed representation of point-features in space (defined in Section 2) with an encoderdecoder architecture:
1. Given a point pi “ pxi,viq a point space encoder Encpxqpq encodes location xi into a location embedding erxis P Rd pxq and a point feature encoderEncpvqpq encodes its feature
into a feature embedding ervis P Rd pvq
. e “ rerxis; erviss P Rd is the full representation of point pi P P , where d “ dpxq ` dpvq. r; s represents vector concatenation. In contrast, geographic entities not in P within the studied space can be represented by their location embedding erxjs since its vi is unknown.
2. We developed two types of decoders which can be used independently or jointly. A location decoder Decspq reconstructs point feature embedding ervis given location embedding erxis, and a spatial context decoder Deccpq reconstructs the feature embedding ervis of point pi based on the space and feature embeddings tei1, ..., eij , ..., einu of nearest neighboring points tpi1, ..., pij , ..., pinu, where n is a hyper-parameter.

heading: 4.1 ENCODER
text: Point Feature Encoder Each point pi “ pxi,viq in a point set P is often associated with features such as the air pollution station data associate with some air quality measures, a set of POIs with POI types and names, a set of points from survey and mapping with elevation values, a set of points from geological survey with mineral content measure, and so on. The point feature encoder Encpvqpq encodes such features vi into a feature embedding ervis P Rd pvq . The implementation of Encpvqpq depends on the nature of these features. For example, if each point represents a POI with multiple POI types (as in this study), the feature embedding ervis can simply be the mean of each POI types’ embeddings ervis “ 1
H
řH h“1 t pγq h , where t pγq h indicates the hth POI type embedding of a POI pi
with H POI types. We apply L2 normalization to the POI type embedding matrix.
Point Space Encoder A part of the novelty of this paper is from the point space encoder Encpxqpq. We first introduce Theorem 1 which provide an analytical solution φpxq as the base of encoding any location x P R2 in 2D space to a distributed representation: Theorem 1. Let Ψpxq “ peixaj ,xy, j “ 1, 2, 3qT P C3 where eiθ “ cos θ ` i sin θ is the Euler notation of complex values; xaj ,xy is the inner product of aj and x. a1,a2,a3 P R2 are 2D vectors such that the angle between ak and al is 2π{3, @j, }aj} “ 2 ? α. Let C P C3ˆ3 be a random complex matrix such as C˚C “ I. Then φpxq “ CΨpxq, Mp∆xq “ CdiagpΨp∆xqqC˚ satisfies φpx`∆xq “Mp∆xqφpxq (1)
and xφpx`∆xq, φpxqy “ dp1´ α}∆x}2q (2)
where d “ 3 is the dimension of φpxq and ∆x is a small displacement from x.
The proof of Theorem 1 can be seen in Gao et al. (2019). φpxq “ CΨpxq P C3 amounts to a 6-dimension real value vector and each dimension shows a hexagon firing pattern which models the grid cell behavior. Because of the periodicity of sinpq and cospq, this single scale representation φpxq does not form a global codebook of 2D positions, i.e. there can be x ‰ y, but φpxq “ φpyq. Inspired by Theorem 1 and the multi-scale periodic representation of grid cells in mammals (Abbott & Callaway, 2014) we set up our point space encoder erxs “ Encpxqtheorypxq to use sine
and cosine functions of different frequencies to encode positions in space. Given any point x in the studied 2D space, the space encoder Encpxqtheorypxq “ NNpPEptqpxqq where PEptqpxq “ rPEptq0 pxq; ...;PE ptq s pxq; ...;PEptqS´1pxqs is a concatenation of multi-scale representations of dpxq “ 6S dimensions. Here S is the total number of grid scales and s “ 0, 1, 2, ..., S ´ 1. NNpq represents fully connected ReLU layers. Let a1 “ r1, 0sT ,a2 “ r´1{2, ? 3{2sT ,a3 “ r´1{2,´ ? 3{2sT P R2 be three unit vectors and the angle between any of them is 2π{3. λmin, λmax are the minimum and maximum grid scale and g “ λmaxλmin . At each scale s, PE ptq s pxq “ rPEptqs,1pxq;PE ptq s,2pxq;PE ptq s,3pxqs is a concatenation of three components, where
PE ptq s,jpxq “ rcosp xx,ajy λmin ¨ gs{pS´1q q; sinp xx,ajy λmin ¨ gs{pS´1q qs@j “ 1, 2, 3; (3)
NNpq and PEptqpxq are analogies of C and Ψpxq in Theorem 1.
Similarly we can define another space encoder Encpxqgridpxq “ NNpPEpgqpxqq inspired by the position encoding model of Transformer (Vaswani et al., 2017), where PEpgqpxq “ rPEpgq0 pxq; ...;PE pgq s pxq; ...;PEpgqS´1pxqs is still a concatenation of its multi-scale representations, while PEpgqs pxq “ rPEpgqs,1 pxq;PE pgq s,2 pxqs handles each component l of x separately:
PE pgq s,l pxq “ rcosp
xrls
λmin ¨ gs{pS´1q q; sinp x
rls
λmin ¨ gs{pS´1q qs@l “ 1, 2 (4)

heading: 4.2 DECODER
text: Two types of decoders are designed for two major types of GIS problems: location modeling and spatial context modeling (See Section 5.1).
Location Decoder Decspq directly reconstructs point feature embedding ervis given its space embedding erxis. We use one layer feed-forward neural network NNdecpq ervis1 “ Decspxi; θdecsq “ NNdecperxisq (5) For training we use inner product to compare the reconstructed feature embedding ervis1 against the real feature embeddings of ervis and other negative points (see training detail in Sec 4.3).
Spatial Context Decoder Deccpq reconstructs the feature embedding ervis of the center point pi based on the space and feature embeddings tei1, ..., eij , ..., einu of n nearby points tpi1, ..., pij , ..., pinu. Note that the feed-in order of context points should not affect the prediction results, which can be achieved by permutation invariant neural network architectures (Zaheer et al., 2017) like PointNet (Qi et al., 2017).
ervis1 “ Deccpxi, tei1, ..., eij , ..., einu; θdeccq “ gp 1
K
K ÿ
k“1
n ÿ j“1 αijkervijsq (6)
Here g is an activation function such as sigmoid. αijk “ exppσijkqřn o“1 exppσiokq
is the attention of pi with its jth neighbor through the kth attention head, and
σijk “ LeakyReLUpaTk rervisinit; ervijs; erxi ´ xijssq (7)
where ak P R2d pvq`dpxq is the attention parameter in the kth attention head. The multi-head attention mechanism is inspired by Graph Attention Network (Veličković et al., 2018) and Mai et al. (2019a).
To represent the spatial relationship (distance and direction) between each context point pij “ pxij ,vijq and the center point pi “ pxi,viq, we use the space encoder Encpxqpq to encode the displacement between them ∆xij “ xi ´ xij . Note that we are modeling the spatial interactions between the center point and n context points simultaneously.
In Eq. 7, ervisinit indicates the initial guess of the feature embedding ervis of point pi which is computed by using another multi-head attention layer as Eq. 6 where the weight α1ijk “ exppσ1ijkq řn
o“1 exppσ1iokq .
Here, σ1ijk is computed as Eq. 8 where the query embedding ervis is excluded.
σ1ijk “ LeakyReLUpa1Tk rervijs; erxi ´ xijssq (8)

heading: 4.3 UNSUPERVISED TRAINING
text: The unsupervised learning task can simply be maximizing the log likelihood of observing the true point pi at position xi among all the points in P
LPpθq “ ´ ÿ
piPP logP ppi|pi1, ..., pij , ..., pinq “ ´
ÿ
piPP log
exppervisTervis1q ř
poPP exppervosTervis1q (9)
Here only the feature embedding of pi is used (without location embedding) to prevent revealing the identities of the point candidates, and θ “ rθenc; θdecs Negative sampling by Mikolov et al. (2013) can be used to improve the efficiency of training
L1Ppθq “ ´ ÿ
piPP
´ log σpervisTervis1q ` 1 |Ni| ÿ
poPNi
log σp´ervosTervis1q ¯
(10)
Here Ni Ď P is a set of sampled negative points for pi (pi R Ni) and σpxq “ 1{p1` e´xq.

heading: 5 EXPERIMENT
text: In this section we compare Space2Vec with commonly used position encoding methods, and analyze them both quantitatively and qualitatively.
Baselines Our baselines include 1) direct directly applying feed-forward nets (Chu et al., 2019); 2) tile discretization (Berg et al., 2014; Adams et al., 2015; Tang et al., 2015); 3) wrap feed-forward nets with coordinate wrapping (Mac Aodha et al., 2019); and 4) rbf Radial Basis Function (RBF) kernels (Baudat & Anouar, 2001; Bierens, 1994). See Appendix A.1 for details of the baselines.

heading: 5.1 POI TYPE CLASSIFICATION TASKS
text: Dataset and Tasks To test the proposed model, we conduct experiments on geographic datasets with POI position and type information. We utilize the open-source dataset published by Yelp Data Challenge and select all POIs within the Las Vegas downtown area3. There are 21,830 POIs with 1,191 different POI types in this dataset. Note that each POI may be associated with one or more types, and we do not use any other meta-data such as business names, reviews for this study. We project geographic coordinates into projection coordinates using the NAD83/Conus Albers projection coordinate system4. The POIs are split into training, validation, and test dataset with ratios 80%:10%:10%. We create two tasks setups which represent different types of modeling need in Geographic Information Science:
• Location Modeling predicts the feature information associated with a POI based on its location xi represented by the location decoder Decspq. This represents a large number of location prediction problems such as image fine grained recognition with geographic prior (Chu et al., 2019), and species potential distribution prediction (Zuo et al., 2008). • Spatial Context Modeling predicts the feature information associated with a POI based
on its context tei1, ..., eij , ..., einu represented by the spatial context decoder Deccpq. This represents a collections of spatial context prediction problem such as spatial context based facade image classification (Yan et al., 2018), and all spatial interpolation problems.
We use POI prediction metrics to evaluate these models. Given the real point feature embedding ervis and N negative feature embeddings Ni “ tervis´u, we compare the predicted ervis1 with them by cosine distance. The cosine scores are used to rank ervis and N negative samples. The negative feature embeddings are the feature embeddings of points pj randomly sampled from P and pi ‰ pj . We evaluate each model using Negative Log-Likelihood (NLL), Mean Reciprocal Rank (MRR) and HIT@5 (the chance of the true POI being ranked to top 5. We train and test each model 10 times to estimate standard deviations. See Appendix A.2 for hyper-parameter selection details.

heading: 5.1.1 LOCATION MODELING EVALUATION
text: We first study location modeling with the location decoder Decspq in Section 4.2. We use a negative sample size of N “ 100. Table 1 shows the average metrics of different models with their best hyper-
3The geographic range is (35.989438, 36.270897) for latitude and (-115.047977, -115.3290609) for longitude. 4https://epsg.io/5070-1252
parameter setting on the validation set. We can see that direct and theorydiag are less competitive, only beating the random selection baseline. Other methods with single scale representations – including tile, wrap, and rbf – perform better. The best results come from various version of the grid cell models, which are capable of dealing with multi-scale representations.
In order to understand the reason for the superiority of grid cell models we provide qualitative analysis of their representations. We apply hierarchical clustering to the location embeddings produced by studied models using cosine distance as the distance metric (See Fig. 2). we can see that when restricted to large grid sizes (λmin “ 1k), theory has similar representation (Fig. 2d, 2e, and Fig. 4d, 4e) and performance compared to rbf (σ “ 1k). However it is able to significantly outperform rbf (σ “ 1k) (and tile and wrap) when small grid sizes (λmin “ 500, 50) are available. The relative improvements over rbf (σ “ 1k) are -0.2%, +0.6%, +2.1% MRR for λmin=1k, 500, 50 respectively.

heading: 5.1.2 MULTI-SCALE ANALYSIS OF LOCATION MODELING
text: In order to show how our multi-scale location representation model will affect the prediction of POI types with different distribution patterns, we classify all 1,191 POI types into three groups based on radius r, which is derived from each POI types’ renormalized Ripley’s K curve (See Figure 1d for examples). It indicates the x axis value of the intersection between the curve and the line of y “ 3.0. A lower r indicates a more clustered distribution patterns. These three groups are listed below:
1. Clustered (r ď 100m): POI types with clustered distribution patterns; 2. Middle (100m ă r ă 200m): POI types with less extreme scales; 3. Even (r ě 200m): POI types with even distribution patterns.
Table 2 shows the performance (MRR) of direct, tile, wrap, rbf , and our theory model on the test dataset of the location modeling task with respect to these three different POI distribution groups. The numbers in pq indicate the MRR difference betweeb a baseline and theory. # POI refers to total number of POI belong to each group5. We can see that 1) The two neural net approaches (direct and wrap) have no scale related parameter and are not performing ideally across all scales, with direct
5The reason why the sum of # POI of these three groups does not equal to the total number of POI is because one POI can have multiple types and they may belonging to different groups.
performs worse because of its simple single layer network. 2) The two approaches with built-in scale parameter (tile and rbf ) have to trade off the performance of different scales. Their best parameter settings lead to close performances to that of Space2Vec at the middle scale, while performing poorly in both clustered and regular groups. These observation clearly shows that all baselines can at most well handle distribution at one scale but show poor performances in other scales. In contrast, Space2Vec’s multi-scale representation can handle distributions at different scales.

heading: 5.1.3 SPATIAL CONTEXT MODELING EVALUATION
text: Next, we evaluate the spatial context decoder Deccpq in Sec. 4.2. We use the same evaluation set up as location modeling. The context points are obtained by querying the n-th nearest points using PostGIS (n “ 10). As for validation and test datasets, we make sure the center points are all unknown during the training phase. Table 3 shows the evaluation results of different models for spatial context modeling. The baseline approaches (direct, tile, wrap, rbf ) generally perform poorly in context modeling. We designed specialized version of these approaches (polar, polar_tile, scaled_rbf ) with polar coordinates, which lead to significantly improvements. Note that these are models proposed by us specialized for context modeling and therefore are less general than the grid cell approaches.
Nevertheless the grid cell approaches are able to perform better than the specialized approaches on the test dataset while have competitive performance on validation dataset. See Appendix ?? for the visualization of context models. Actually the gains are small for all baseline approaches also. The reason is that we expect location encoding to be less important when context information is accessible. Similarly as discussed in (Gao et al., 2019), it is when there is a lack of visual clues that the grid cells of animals are the most helpful for their navigation.
Figure 6 shows the location embedding clustering results in both Cartesian and polar coordinate systems. We can see that direct (Fig. 3a, 3g) only captures the distance information when the context POI is very close (logp‖ ∆xij ‖ `1q ď 5) while in the farther spatial context it purely models the direction information. polar (Fig. 3b, 3h) has the similar behaviors but captures the distance information in a more fine-grained manner. wrap (Fig. 3c, 3i) mainly focuses on differentiating relative positions in farther spatial context cont which might explain its lower performance6. polar_tile (Fig. 3d) mostly responds to distance information. Interestingly, scaled_rbf and theory have similar representations in the polar coordinate system (Fig. 3k, 3l) and similar performance (Table 3). While scaled_rbf captures the gradually decreased distance effect with a scaled kernel size which becomes larger in farther distance, theory achieves this by integrating representations of different scales.

heading: 5.2 FINE-GRAINED IMAGE CLASSIFICATION TASKS
text: To demonstrate the generalizability of Space2Vec for space representation we utilized the proposed point space encoder Encpxqpq model in a well-known computer vision task: fine-grained image classification. As we discussed in Section 3, many studies (Berg et al., 2014; Chu et al., 2019; Mac Aodha et al., 2019) have shown that geographic prior information - where (and when) the image is taken - is very important additional information for the fine-grained image classification task and can substantially improve the model performance. For example, the appearance information is usually not sufficient to differentiate two visually similar species. In this case, the geographic prior becomes much more important because these two species may have very different spatial prior distributions such as the example of European Toads and Spiny Toads in Figure 1 of Mac Aodha et al. (2019).
We adopt the task setup of Mac Aodha et al. (2019). During training we have a set of tuples D “ tpIi,xi, yi, piq | i “ 1, ..., Nu where Ii indicates an image, yi P t1, 2, ..., Cu is the corresponding class label (species category), xi “ rlongitudei, latitudeis is the geographic coordinates where the image was taken, and pi is the id of the photographer who took this image. At training time, a location encoder is trained to capture the spatial prior information P py | xq. At inference time, pi information is not available and the final image classification prediction is calculated based on the
6Note that wrap is original proposed by Mac Aodha et al. (2019) for location modelling, not spatial context modelling. This results indicates wrap is not good at this task.
combination of two models: 1) the trained location encoder which captures the spatial priors P py | xq and 2) the pretrained image classification model, InceptionV3 network (Szegedy et al., 2016), which captures P py | Iq. Bayesian theory has been used to derive the joint distribution P py | I,xq. See Mac Aodha et al. (2019) for detail explanation as well as the loss function. Note that while Space2Vec outperforms specialized density estimation methods such as Adaptive Kernel (Berg et al., 2014), it would be interesting to explore early fusion Space2Vec ’s representations with the image module.
We use two versions of our point space encoder Encpxqpq model (grid, theory) as the location encoder to capture the spatial prior information P py | xq. The evaluation results of our models as well as multiple baselines are shown in Table 4. We can see that both grid, theory outperform previous models as well as that of Mac Aodha et al. (2019) on two fine-grained image classification datasets with significant sizes: BirdSnap:, NABirds:. theory shows superiority over grid on NABirds: while fail to outperform grid on BirdSnap:. Note that we only pick baseline models which capture spatial-only prior and drop models which additionally consider time information. Both grid and theory use 1 hidden ReLU layers of 512 neurons for NNpq and they have the same hyperparameters: λmin=0.0001, λmax=360, S = 64. Like Mac Aodha et al. (2019), the location embedding size dpxq is 1024 and we train the location encoder for 30 epochs. Our implementation is based on the original code7 of Mac Aodha et al. (2019) for both model training and evaluation phase.

heading: 6 CONCLUSION
text: We introduced an encoder-decoder framework as a general-purpose representation model for space inspired by biological grid cells’ multi-scale periodic representations. The model is an inductive learning model and can be trained in an unsupervised manner. We conduct two experiments on POI type prediction based on 1) POI locations and 2) nearby POIs. The evaluation results demonstrate the effectiveness of our model. Our analysis reveals that it is the ability to integrate representations of different scales that makes the grid cell models outperform other baselines on these two tasks. In the future, we hope to incorporate the presented framework to more complex GIS tasks such as social network analysis, and sea surface temperature prediction.

heading: ACKNOWLEDGMENTS
text: The presented work is partially funded by the NSF award 1936677 C-Accel Pilot - Track A1 (Open Knowledge Network): Spatially-Explicit Models, Methods, And Services For Open Knowledge Networks, Esri Inc., and Microsoft AI for Earth Grant: Deep Species Spatio-temporal Distribution Modeling for Biodiversity Hotspot Prediction. We thank Dr. Ruiqi Gao for discussions about grid cells, Dr. Wenyun Zuo for discussion about species potential distribution prediction and Dr. Yingjie Hu for his suggestions about the introduction section.
7https://github.com/macaodha/geo_prior/

heading: A APPENDIX
text: A.1 BASELINES
To help understand the mechanism of distributed space representation we compare multiple ways of encoding spatial information. Different models use different point space encoder Encpxqpq to encode either location xi (for location modeling loc) or the displacement between the center point and one context point ∆xij “ xi ´ xij (for spatial context modeling cont)8.
• random shuffles the order of the correct POI and N negative samples randomly as the predicted ranking. This shows the lower bound of each metrics.
• direct directly encode location xi (or ∆xij for cont) into a location embedding erxis (or er∆xijs) using a feed-forward neural networks (FFNs)9, denoted as Encpxqdirectpxq without decomposing coordinates into a multi-scale periodic representation. This is essentially the GPS encoding method used by Chu et al. (2019). Note that Chu et al. (2019) is not open sourced and we end up implementing the model architecture ourselves.
• tile divides the study area Aloc (for loc) or the range of spatial context defined by λmax, Acont, (for cont) into grids with equal grid sizes c. Each grid has an embedding to be used as the encoding for every location xi or displacement ∆xij fall into this grid. This is a common practice by many previous work when dealing with coordinate data (Berg et al., 2014; Adams et al., 2015; Tang et al., 2015).
• wrap is a location encoder model recently introduced by Mac Aodha et al. (2019). It first normalizes x (or ∆x) into the range r´1, 1s and uses a coordinate wrap mechanism rsinpπxrlsq; cospπxrlsqs to convert each dimension of x into 2 numbers. This is then passed through an initial fully connected layer, followed by a series of h residual blocks, each consisting of two fully connected layers (o hidden neurons) with a dropout layer in between. We adopt the official code of Mac Aodha et al. (2019)10 for this implementation.
• rbf randomly samples M points from the training dataset as RBF anchor points {xanchorm ,m “ 1...M} (or samples M ∆xanchorm from Acont for cont) 11, and use gaus-
sian kernels exp ` ´ ‖ xi ´ x anchor m ‖2
2σ2 ˘ (or exp `
´ ‖ ∆xij ´∆x anchor m ‖2
2σ2 ˘ for cont) on each anchor points, where σ is the kernel size. Each point pi has a M -dimension RBF feature vector which is fed into a FNN to obtain the spatial embedding. This is a strong baseline for representing floating number features in machine learning models.
• grid as described in Section 4.1 inspired by the position encoding in Transformer (Vaswani et al., 2017).
• hexa Same as grid but use sinpθq, sinpθ ` 2π{3q, and sinpθ ` 4π{3q in PEpgqs,l pxq. • theory as described in Section 4.1, uses the theoretical models (Gao et al., 2019) as the first
layer of Encpxqtheorypxq or Enc pxq theoryp∆xijq.
• theorydiag further constrains NNpq as a block diagonal matrix, with each scale as a block.
We also have the following baselines which are specific to the spatial context modeling task.
• none the decoder Deccpq does not consider the spatial relationship between the center point and context points but only the co-locate patterns such as Place2Vec (Yan et al., 2017). That means we drop the er∆xijs from the attention mechanism in Equ. 7 and 8. • polar first converts the displacement ∆xij into polar coordinates pr, θq centered at the
center point where r “ logp‖ ∆xij ‖ `1q. Then it uses rr, θs as the input for a FFN to obtain the spatial relationship embedding in Equ. 7. We find out that it has a significant performance improvement over the variation with r “‖ ∆xij ‖.
8We will use meter as the unit of λmin, λmax, σ, c. 9we first normalizes x (or ∆x) into the range r´1, 1s
10http://www.vision.caltech.edu/~macaodha/projects/geopriors/ 11these anchor points are fixed in both loc and cont.
• polar_tile is a modified version of tile but the grids are extracted from polar coordinates pr, θq centered at the center point where r “ logp‖ ∆xij ‖ `1q. Instead of using grid size c, we use the number of grids along θ (or r) axis, F , as the only hyperparameter. Similarly, We find that r “ logp‖ ∆xij ‖ `1q outperform r “‖ ∆xij ‖ significantly. • scaled_rbf is a modified version of rbf for cont whose kernel size is proportional to
the distance between the current anchor point and the origin, ‖ ∆xanchorm ‖. That is
exp ` ´ ‖ ∆xij ´∆x anchor m ‖2
2σ2scaled
˘
. Here σscaled “ σ ` β ‖ ∆xanchorm ‖ where σ is the
basic kernel size and β is kernel rescale factor, a constant. We developed this mechanism to help RFB to deal with relations at different scale, and we observe that it produces significantly better result than vanilla RBFs.
A.2 HYPER-PARAMETER SELECTION
We perform grid search for all methods based on their performance on the validation sets.
Location Modeling The hyper-parameters of theory models are based on grid search with dpvq “ p32, 64, 128, 256q, dpxq “ p32, 64, 128, 256q, S “ p4, 8, 16, 32, 64, 128q, and λmin “ p1, 5, 10, 50, 100, 200, 500, 1kq while λmax “ 40k is decided based on the total size of the study area. We find out the best performances of different grid cell based models are obtained when dpvq “ 64, dpxq “ 64, S “ 64, and λmin “ 50. In terms of tile, the hyper-parameters are selected from c “ p10, 50, 100, 200, 500, 1000q while c “ 500 gives us the best performance.As for rbf , we do grid search on the hyper-parameters: M “ p10, 50, 100, 200, 400, 800q and σ “ p102, 103, 104, 105, 106, 107q. The best performance of rbf is obtain when M “ 200 and σ “ 103. As for wrap, grid search is performed on: h “ p1, 2, 3, 4q and o “ p64, 128, 256, 512q while h “ 3 and o “ 512 gives us the best result. All models use FFNs in their Encpxqpq except wrap. The number of layers f and the number of hidden state neurons u of the FFN are selected from f “ p1, 2, 3q and u “ p128, 256, 512q. We find out f “ 1 and u “ 512 give the best performance for direct, tile, rbf , and theory. So we use them for every model for a fair comparison.
Spatial Context Modeling Grid search is used for hyperparameter tuning and the best performance of different grid cell models is obtain when dpvq “ 64, dpxq “ 64, S “ 64, and λmin “ 10. We set λmax “ 10k based on the maximum displacement between context points and center points to make the location encoding unique. As for multiple baseline models, grid search is used again to obtain the best model. The best model hyperparameters are shown in () besides the model names in Table 3. Note that both rbf and scaled_rbf achieve the best performance with M “ 100.
A.3 FIRING PATTERN FOR THE NEURONS
A.4 EMBEDDING CLUSTERING OF RBF AND THEORY MODELS

Review summaries:
The paper introduces Space2Vec, a space representation learning model. The work is motivated by the biological grid cell’s multi-scale periodic representations and the success of representation learning of NLP. So, the key idea behind the model is two-fold. On one hand, utilize the position information and the context associated with the position. On the other hand, the authors build a multiscale point space encoder based on Theorem 1 (in the paper), which was previously proved by Gao et al. (2019).

This paper presents a new method called "Space2Vec" to compute spatial embeddings of a pixel in a spatial data. The primary motivation of Space2Vec is to integrate representations of different spatial scales which could potentially make the spatial representations more informative and meaningful as features. Space2Vec is trained as a part of an encoder-decoder framework, where Space2Vec encodes the spatial features of all the points that are fed as input to the framework.