Multi-Scale Representation Learning  for Spatial Feature Distributions using Grid Cells

1 INTRODUCTION . Unsupervised text encoding models such as Word2Vec ( Mikolov et al. , 2013 ) , Glove ( Pennington et al. , 2014 ) , ELMo ( Peters et al. , 2018 ) , and BERT ( Devlin et al. , 2018 ) have been effectively utilized in many Natural Language Processing ( NLP ) tasks . At their core they train models which encode words into vector space representations based on their positions in the text and their context . A similar situation can be encountered in the field of Geographic Information Science ( GIScience ) . For example , spatial interpolation aims at predicting an attribute value , e.g. , elevation , at an unsampled location based on the known attribute values of nearby samples . Geographic information has become an important component to many tasks such as fine-grained image classification ( Mac Aodha et al. , 2019 ) , point cloud classification and semantic segmentation ( Qi et al. , 2017 ) , reasoning about Point of Interest ( POI ) type similarity ( Yan et al. , 2017 ) , land cover classification ( Kussul et al. , 2017 ) , and geographic question answering ( Mai et al. , 2019b ) . Developing a general model for vector space representation of any point in space would pave the way for many future applications . 1Link to project repository : https : //github.com/gengchenmai/space2vec However , existing models often utilize specific methods to deal with geographic information and often disregards geographic coordinates . For example , Place2Vec ( Yan et al. , 2017 ) converts the coordinates of POIs into spatially collocated POI pairs within certain distance bins , and does not preserve information about the ( cardinal ) direction between points . Li et al . ( 2017 ) propose DCRNN for traffic forecasting in which the traffic sensor network is converted to a distance weighted graph which necessarily forfeits information about the spatial layout of sensors . There is , however , no general representation model beyond simply applying discretization ( Berg et al. , 2014 ; Tang et al. , 2015 ) or feed-forward nets ( Chu et al. , 2019 ; Mac Aodha et al. , 2019 ) to coordinates . A key challenge in developing a general-purpose representation model for space is how to deal with mixtures of distributions with very different characteristics ( see an example in Figure 1 ) , which often emerges in spatial datasets ( McKenzie et al. , 2015 ) . For example , there are POI types with clustered distributions such as women ’ s clothing , while there are other POI types with regular distributions such as education . These feature distributions co-exist in the same space , and yet we want a single representation to accommodate all of them in a task such as location-aware image classification ( Mac Aodha et al. , 2019 ) . Ripley ’ s K is a spatial analysis method used to describe point patterns over a given area of interest . Figure 1c shows the K plot of several POI types in Las Vegas . One can see that as the radius grows the numbers of POIs increase at different rates for different POI types . In order to see the relative change of density at different scales , we renormalize the curves by each POI type ’ s density and show it in log scale in Figure 1d . One can see two distinct POI type groups with different distribution patterns with clustered and even distributions . If we want to model the distribution of these POIs by discretizing the study area into tiles , we have to use small grid sizes for women ’ s clothing while using larger grid sizes for educations because smaller grid sizes lead to over- parameterization of the model and overfitting . In order to jointly describe these distributions and their patterns , we need an encoding method which supports multi-scale representations . Nobel Prize winning Neuroscience research ( Abbott & Callaway , 2014 ) has demonstrated that grid cells in mammals provide a multi-scale periodic representation that functions as a metric for location encoding , which is critical for integrating self-motion . Moreover , Blair et al . ( 2007 ) show that the multi-scale periodic representation of grid cells can be simulated by summing three cosine grating functions oriented 60˝ apart , which may be regarded as a simple Fourier model of the hexagonal lattice . This research inspired us to encode locations with multi-scale periodic representations . Our assumption is that decomposed geographic coordinates helps machine learning models , such as deep neural nets , and multi-scale representations deal with the inefficiency of intrinsically single-scale methods such as RFB kernels or discretization ( tile embeddings ) . To validate this intuition , we propose an encoder-decoder framework to encode the distribution of point-features2 in space and 2In GIS and spatial analysis , ‘ features ’ are representations of real-world entities . A tree can , for instance , be modeled by a point-feature , while a street would be represented as a line string feature . train such a model in an unsupervised manner . This idea of using sinusoid functions with different frequencies to encode positions is similar to the position encoding proposed in the Transformer model ( Vaswani et al. , 2017 ) . However , the position encoding model of Transformer deals with a discrete 1D space – the positions of words in a sentence – while our model works on higher dimensional continuous spaces such as the surface of earth . In summary , the contributions of our work are as follows : . 1 . We propose an encoder-decoder encoding framework called Space2Vec using sinusoid functions with different frequencies to model absolute positions and spatial contexts . We also propose a multi-head attention mechanism based on context points . To the best of our knowledge , this is the first attention model that explicitly considers the spatial relationships between the query point and context points . 2 . We conduct experiments on two real world geographic data for two different tasks : 1 ) predicting types of POIs given their positions and context , 2 ) image classification leveraging their geo-locations . Space2Vec outperforms well-established encoding methods such as RBF kernels , multi-layer feed-forward nets , and tile embedding approaches for location modeling and image classification . 3 . To understand the advantages of Space2Vec we visualize the firing patterns ( response maps ) of location models ’ encoding layer neurons and show how they handle spatial structures at different scales by integrating multi-scale representations . Furthermore the firing patterns for the spatial context models neurons give insight into how the grid-like cells capture the decreasing distance effect with multi-scale representations . 2 PROBLEM FORMULATION . Distributed representation of point-features in space can be formulated as follows . Given a set of points P “ tpiu , i.e. , Points of Interests ( POIs ) , in L-D space ( L “ 2 , 3 ) define a function fP , θpxq : RL Ñ Rd ( L ! d ) , which is parameterized by θ and maps any coordinate x in space to a vector representation of d dimension . Each point ( e.g. , a restaurant ) pi “ pxi , viq is associated with a location xi and attributes vi ( i.e. , POI features such as type , name , capacity , etc. ) . The function fP , θpxq encodes the probability distribution of point features over space and can give a representation of any point in the space . Attributes ( e.g . place types such as Museum ) and coordinate of point can be seen as analogies to words and word positions in commonly used word embedding models . 3 RELATED WORK . There has been theoretical research on neural network based path integration/spatial localization models and their relationships with grid cells . Both Cueva & Wei ( 2018 ) and Banino et al . ( 2018 ) showed that grid-like spatial response patterns emerge in trained networks for navigation tasks which demonstrate that grid cells are critical for vector-based navigation . Moreover , Gao et al . ( 2019 ) propose a representational model for grid cells in navigation tasks which has good quality such as magnified local isometry . All these research is focusing on understanding the relationship between the grid-like spatial response patterns and navigation tasks from a theoretical perspective . In contrast , our goal focuses on utilizing these theoretical results on real world data in geoinformatics . Radial Basis Function ( RBF ) kernel is a well-established approach to generating learning friendly representation from points in space for machine learning algorithms such as SVM classification ( Baudat & Anouar , 2001 ) and regression ( Bierens , 1994 ) . However , the representation is example based – i.e. , the resultant model uses the positions of training examples as the centers of Gaussian kernel functions ( Maz ’ ya & Schmidt , 1996 ) . In comparison , the grid cell based location encoding relies on sine and cosine functions , and the resultant model is inductive and does not store training examples . Recently the computer vision community shows increasing interests in incorporating geographic information ( e.g . coordinate encoding ) into neural network architectures for multiple tasks such as image classification ( Tang et al. , 2015 ) and fine grained recognition ( Berg et al. , 2014 ; Chu et al. , 2019 ; Mac Aodha et al. , 2019 ) . Both Berg et al . ( 2014 ) and Tang et al . ( 2015 ) proposed to discretize the study area into regular grids . To model the geographical prior distribution of the image categories , the grid id is used for GPS encoding instead of the raw coordinates . However , choosing the correct discretization is challenging ( Openshaw , 1984 ; Fotheringham & Wong , 1991 ) , and incorrect choices can significantly affect the final performance ( Moat et al. , 2018 ; Lechner et al. , 2012 ) . In addition , discretization does not scale well in terms of memory use . To overcome these difficulties , both Chu et al . ( 2019 ) and Mac Aodha et al . ( 2019 ) advocated the idea of inductive location encoders which directly encode coordinates into a location embedding . However , both of them directly feed the coordinates into a feed-forward neural network ( Chu et al. , 2019 ) or residual blocks ( Mac Aodha et al. , 2019 ) without any feature decomposition strategy . Our experiments show that this direct encoding approach is insufficient to capture the spatial feature distribution and Space2Vec significantly outperforms them by integrating spatial representations of different scales . 4 METHOD . We solve distributed representation of point-features in space ( defined in Section 2 ) with an encoderdecoder architecture : 1 . Given a point pi “ pxi , viq a point space encoder Encpxqpq encodes location xi into a location embedding erxis P Rd pxq and a point feature encoderEncpvqpq encodes its feature into a feature embedding ervis P Rd pvq . e “ rerxis ; erviss P Rd is the full representation of point pi P P , where d “ dpxq ` dpvq . r ; s represents vector concatenation . In contrast , geographic entities not in P within the studied space can be represented by their location embedding erxjs since its vi is unknown . 2 . We developed two types of decoders which can be used independently or jointly . A location decoder Decspq reconstructs point feature embedding ervis given location embedding erxis , and a spatial context decoder Deccpq reconstructs the feature embedding ervis of point pi based on the space and feature embeddings tei1 , ... , eij , ... , einu of nearest neighboring points tpi1 , ... , pij , ... , pinu , where n is a hyper-parameter .

Summary: The paper introduces Space2Vec, a space representation learning model. The work is motivated by the biological grid cell’s multi-scale periodic representations and the success of representation learning of NLP. So, the key idea behind the model is two-fold. On one hand, utilize the position information and the context associated with the position. On the other hand, the authors build a multiscale point space encoder based on Theorem 1 (in the paper), which was previously proved by Gao et al. (2019).