Multi-Scale Representation Learning  for Spatial Feature Distributions using Grid Cells

1 INTRODUCTION . Unsupervised text encoding models such as Word2Vec ( Mikolov et al. , 2013 ) , Glove ( Pennington et al. , 2014 ) , ELMo ( Peters et al. , 2018 ) , and BERT ( Devlin et al. , 2018 ) have been effectively utilized in many Natural Language Processing ( NLP ) tasks . At their core they train models which encode words into vector space representations based on their positions in the text and their context . A similar situation can be encountered in the field of Geographic Information Science ( GIScience ) . For example , spatial interpolation aims at predicting an attribute value , e.g. , elevation , at an unsampled location based on the known attribute values of nearby samples . Geographic information has become an important component to many tasks such as fine-grained image classification ( Mac Aodha et al. , 2019 ) , point cloud classification and semantic segmentation ( Qi et al. , 2017 ) , reasoning about Point of Interest ( POI ) type similarity ( Yan et al. , 2017 ) , land cover classification ( Kussul et al. , 2017 ) , and geographic question answering ( Mai et al. , 2019b ) . Developing a general model for vector space representation of any point in space would pave the way for many future applications . 1Link to project repository : https : //github.com/gengchenmai/space2vec However , existing models often utilize specific methods to deal with geographic information and often disregards geographic coordinates . For example , Place2Vec ( Yan et al. , 2017 ) converts the coordinates of POIs into spatially collocated POI pairs within certain distance bins , and does not preserve information about the ( cardinal ) direction between points . Li et al . ( 2017 ) propose DCRNN for traffic forecasting in which the traffic sensor network is converted to a distance weighted graph which necessarily forfeits information about the spatial layout of sensors . There is , however , no general representation model beyond simply applying discretization ( Berg et al. , 2014 ; Tang et al. , 2015 ) or feed-forward nets ( Chu et al. , 2019 ; Mac Aodha et al. , 2019 ) to coordinates . A key challenge in developing a general-purpose representation model for space is how to deal with mixtures of distributions with very different characteristics ( see an example in Figure 1 ) , which often emerges in spatial datasets ( McKenzie et al. , 2015 ) . For example , there are POI types with clustered distributions such as women ’ s clothing , while there are other POI types with regular distributions such as education . These feature distributions co-exist in the same space , and yet we want a single representation to accommodate all of them in a task such as location-aware image classification ( Mac Aodha et al. , 2019 ) . Ripley ’ s K is a spatial analysis method used to describe point patterns over a given area of interest . Figure 1c shows the K plot of several POI types in Las Vegas . One can see that as the radius grows the numbers of POIs increase at different rates for different POI types . In order to see the relative change of density at different scales , we renormalize the curves by each POI type ’ s density and show it in log scale in Figure 1d . One can see two distinct POI type groups with different distribution patterns with clustered and even distributions . If we want to model the distribution of these POIs by discretizing the study area into tiles , we have to use small grid sizes for women ’ s clothing while using larger grid sizes for educations because smaller grid sizes lead to over- parameterization of the model and overfitting . In order to jointly describe these distributions and their patterns , we need an encoding method which supports multi-scale representations . Nobel Prize winning Neuroscience research ( Abbott & Callaway , 2014 ) has demonstrated that grid cells in mammals provide a multi-scale periodic representation that functions as a metric for location encoding , which is critical for integrating self-motion . Moreover , Blair et al . ( 2007 ) show that the multi-scale periodic representation of grid cells can be simulated by summing three cosine grating functions oriented 60˝ apart , which may be regarded as a simple Fourier model of the hexagonal lattice . This research inspired us to encode locations with multi-scale periodic representations . Our assumption is that decomposed geographic coordinates helps machine learning models , such as deep neural nets , and multi-scale representations deal with the inefficiency of intrinsically single-scale methods such as RFB kernels or discretization ( tile embeddings ) . To validate this intuition , we propose an encoder-decoder framework to encode the distribution of point-features2 in space and 2In GIS and spatial analysis , ‘ features ’ are representations of real-world entities . A tree can , for instance , be modeled by a point-feature , while a street would be represented as a line string feature . train such a model in an unsupervised manner . This idea of using sinusoid functions with different frequencies to encode positions is similar to the position encoding proposed in the Transformer model ( Vaswani et al. , 2017 ) . However , the position encoding model of Transformer deals with a discrete 1D space – the positions of words in a sentence – while our model works on higher dimensional continuous spaces such as the surface of earth . In summary , the contributions of our work are as follows : . 1 . We propose an encoder-decoder encoding framework called Space2Vec using sinusoid functions with different frequencies to model absolute positions and spatial contexts . We also propose a multi-head attention mechanism based on context points . To the best of our knowledge , this is the first attention model that explicitly considers the spatial relationships between the query point and context points . 2 . We conduct experiments on two real world geographic data for two different tasks : 1 ) predicting types of POIs given their positions and context , 2 ) image classification leveraging their geo-locations . Space2Vec outperforms well-established encoding methods such as RBF kernels , multi-layer feed-forward nets , and tile embedding approaches for location modeling and image classification . 3 . To understand the advantages of Space2Vec we visualize the firing patterns ( response maps ) of location models ’ encoding layer neurons and show how they handle spatial structures at different scales by integrating multi-scale representations . Furthermore the firing patterns for the spatial context models neurons give insight into how the grid-like cells capture the decreasing distance effect with multi-scale representations . 2 PROBLEM FORMULATION . Distributed representation of point-features in space can be formulated as follows . Given a set of points P “ tpiu , i.e. , Points of Interests ( POIs ) , in L-D space ( L “ 2 , 3 ) define a function fP , θpxq : RL Ñ Rd ( L ! d ) , which is parameterized by θ and maps any coordinate x in space to a vector representation of d dimension . Each point ( e.g. , a restaurant ) pi “ pxi , viq is associated with a location xi and attributes vi ( i.e. , POI features such as type , name , capacity , etc. ) . The function fP , θpxq encodes the probability distribution of point features over space and can give a representation of any point in the space . Attributes ( e.g . place types such as Museum ) and coordinate of point can be seen as analogies to words and word positions in commonly used word embedding models . 3 RELATED WORK . There has been theoretical research on neural network based path integration/spatial localization models and their relationships with grid cells . Both Cueva & Wei ( 2018 ) and Banino et al . ( 2018 ) showed that grid-like spatial response patterns emerge in trained networks for navigation tasks which demonstrate that grid cells are critical for vector-based navigation . Moreover , Gao et al . ( 2019 ) propose a representational model for grid cells in navigation tasks which has good quality such as magnified local isometry . All these research is focusing on understanding the relationship between the grid-like spatial response patterns and navigation tasks from a theoretical perspective . In contrast , our goal focuses on utilizing these theoretical results on real world data in geoinformatics . Radial Basis Function ( RBF ) kernel is a well-established approach to generating learning friendly representation from points in space for machine learning algorithms such as SVM classification ( Baudat & Anouar , 2001 ) and regression ( Bierens , 1994 ) . However , the representation is example based – i.e. , the resultant model uses the positions of training examples as the centers of Gaussian kernel functions ( Maz ’ ya & Schmidt , 1996 ) . In comparison , the grid cell based location encoding relies on sine and cosine functions , and the resultant model is inductive and does not store training examples . Recently the computer vision community shows increasing interests in incorporating geographic information ( e.g . coordinate encoding ) into neural network architectures for multiple tasks such as image classification ( Tang et al. , 2015 ) and fine grained recognition ( Berg et al. , 2014 ; Chu et al. , 2019 ; Mac Aodha et al. , 2019 ) . Both Berg et al . ( 2014 ) and Tang et al . ( 2015 ) proposed to discretize the study area into regular grids . To model the geographical prior distribution of the image categories , the grid id is used for GPS encoding instead of the raw coordinates . However , choosing the correct discretization is challenging ( Openshaw , 1984 ; Fotheringham & Wong , 1991 ) , and incorrect choices can significantly affect the final performance ( Moat et al. , 2018 ; Lechner et al. , 2012 ) . In addition , discretization does not scale well in terms of memory use . To overcome these difficulties , both Chu et al . ( 2019 ) and Mac Aodha et al . ( 2019 ) advocated the idea of inductive location encoders which directly encode coordinates into a location embedding . However , both of them directly feed the coordinates into a feed-forward neural network ( Chu et al. , 2019 ) or residual blocks ( Mac Aodha et al. , 2019 ) without any feature decomposition strategy . Our experiments show that this direct encoding approach is insufficient to capture the spatial feature distribution and Space2Vec significantly outperforms them by integrating spatial representations of different scales . 4 METHOD . We solve distributed representation of point-features in space ( defined in Section 2 ) with an encoderdecoder architecture : 1 . Given a point pi “ pxi , viq a point space encoder Encpxqpq encodes location xi into a location embedding erxis P Rd pxq and a point feature encoderEncpvqpq encodes its feature into a feature embedding ervis P Rd pvq . e “ rerxis ; erviss P Rd is the full representation of point pi P P , where d “ dpxq ` dpvq . r ; s represents vector concatenation . In contrast , geographic entities not in P within the studied space can be represented by their location embedding erxjs since its vi is unknown . 2 . We developed two types of decoders which can be used independently or jointly . A location decoder Decspq reconstructs point feature embedding ervis given location embedding erxis , and a spatial context decoder Deccpq reconstructs the feature embedding ervis of point pi based on the space and feature embeddings tei1 , ... , eij , ... , einu of nearest neighboring points tpi1 , ... , pij , ... , pinu , where n is a hyper-parameter .

Summary: This paper presents a new method called "Space2Vec" to compute spatial embeddings of a pixel in a spatial data. The primary motivation of Space2Vec is to integrate representations of different spatial scales which could potentially make the spatial representations more informative and meaningful as features. Space2Vec is trained as a part of an encoder-decoder framework, where Space2Vec encodes the spatial features of all the points that are fed as input to the framework.