Representation Learning on Spatial Networks

1 Introduction . Spatial data and network data are both popular types of data in modern big data era . The study of spatial data focuses on the properties of continuous spatial entities under specific geometry , while analysis of network data investigates the properties of discrete objects and their pairwise relationship . Spanning these two data types , spatial networks is a crucial type of data structure that nodes occupy positions in a Euclidean space , where spatial patterns and constraints may have a strong effect on their connectivity patterns [ 6 ] . Understanding the mechanism of organizing spatial networks has significant importance for a broad range of fields [ 29 ] , ranging from micro-scale ( e.g. , molecule structure [ 94 ] ) , to middle-scale ( e.g. , biological neural network [ 31 ] ) , to macro-scale ( e.g. , mobility networks [ 18 ] ) . Effectively learning the representations of spatial networks is extremely challenging due to the close interactions between network and spatial topology , the incompatibility between the treatments for discrete and continuous data , and particular properties 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . such as permutation invariant and rotation-translation invariant . Spatial networks have long been researched in the domains such as physics and mathematics , which usually extend complex networks and graph theory into spatial networks [ 80 , 7 ] . They typically rely on network generation principles predefined by human heuristics and prior knowledge . Such methods usually characterize well on the aspects of the data that have been covered by the predefined principles , but not on those have not been covered [ 6 ] . However , the underlying network process in complex networks is largely unknown and extremely difficult to be predefined in simple rules , especially in crucial and open domains such as brain network modeling [ 81 ] , network catastrophic failure [ 72 ] , and protein folding [ 27 ] . Remarkable progress has been made towards generalizing deep representation learning approaches in spatial data and network data [ 96 , 13 , 46 , 43 , 28 ] , respectively , in recent years . For spatial data , deep learning achieved significant progress in different commonly used formats such as images [ 53 , 75 , 61 , 21 ] , point clouds [ 34 , 71 , 57 ] , meshes [ 85 , 90 ] , and volumetric grids [ 95 , 66 ] . On the other hand , deep learning has also boosted the research of encoding graph structure on network data [ 46 , 51 , 45 ] , and downstream applications such as recommender systems [ 99 , 60 ] , drug discovery [ 39 , 22 , 41 , 42 ] , FinTech [ 91 ] , customer care [ 93 ] , and natural language processing [ 62 , 8 , 92 ] . Despite the respective progress in representation learning on spatial data and network data in parallel , the representation learning for spatial networks have been largely underexplored and has just started to attract fast-increasing attention . Merely combining spatial and graph representations separately can not handle that for spatial networks where spatial and network process are deeply coupled together [ 6 , 68 , 33 ] . For example , Fig . 2 shows a simple example with a pair of spatial networks , in which there are different formation rules on the edges relied on the spatial distance , that is non-distinguishable for spatial and network embedding methods , respectively . Few recent attempts have been proposed to handle representation learning on spatial networks but still suffer from key challenges : Spatial network representation learning is a problem extremely difficult to address due to several unique challenges : 1 ) Difficult in distinguishing the patterns that require joint spatial and graph consideration . Examples like Figure . 2 that share the same spatial and network topology , respectively , but with significantly different interaction mechanisms , are non-distinguishable to either spatial or graph methods . 2 ) Difficult in jointly maintaining that the learned representation is invariant to node permutation , and rotation and translation transformations . Notice that spatial and graph information confine each other which neutralizes conventional methods to have either of them . For example , although point clouds representation learning can easily preserve rotation- and translation-invariant by using spatial nearest neighbors , here in spatial networks the neighbor is confined also by graph neighbors . Such additional confinement largely harden our task . 3 ) High efficiency and scalability in the graph size . The confinement between spatial and graph information inevitably leads to taking into account more entities simultaneously to maintain sufficient information . The requirement to handle incremental information increases the demand for model efficiency and scalability . In order to address all the aforementioned challenges , this paper proposes a new spatial graph message passing neural network ( SGMP ) for learning the representations of generic spatial networks , with theoretical guarantees on discriminative power and various spatial and network properties , and an accelerating algorithm which adjusts to our theoretical framework . Specifically , to capture and model the intrinsic coupled spatial and graph properties , we propose a novel message passing neural network to organically aggregate the spatial and graph information . To ensure the invariance of learned representation under rotation and translation transformations , a novel way to represent the node spatial information by characterizing geometric invariant features with lossless information is proposed . To alleviate the efficiency issue , we propose a new accelerating algorithm for learning on graph-structured data . The proposed accelerating algorithm effectively reduces the time and memory complexity from O ( N3 ) to O ( N ) , and maintains the theoretical guarantees for spatial networks . Finally , we demonstrate the strength of our theoretical findings through extensive experiments on both synthetic datasets and real-world datasets . 2 Related Work . Spatial Networks There has been a long time of research efforts on the subjects of spatial networks [ 6 ] . In the area of quantitative geography , Haggett and Chorley discussed the relevance of space in the formation and evolution of networks , and developed models to characterize spatial networks at least fourty years ago [ 44 , 17 ] . New insights leading to modern quantitative solutions are gained due to the advance in complex networks [ 32 , 89 , 1 , 4 , 3 , 20 ] , and appears in more practical fields such as transportation networks [ 2 , 54 , 55 ] , mobility networks [ 18 , 24 ] , biological networks [ 31 , 76 ] , and computational chemistry [ 38 , 73 , 52 ] . Geometric Deep Learning . This is a more recent domain which handles non-Euclidean structured data such as graphs and manifolds [ 13 ] . • Geometric Deep Learning on Manifolds . There is a large body of research efforts of generalizing deep learning models to 3D shapes as manifolds in the computer graphics community . Many works have been conducted to find a better approach to generalize convolution-like operations to the nonEuclidean domain [ 65 , 10 , 74 , 98 , 64 , 59 ] . J. Masci et at . proposed the framework of generalizing convolution neural network paradigm to manifolds by applying filters to extract local patches in polar coordinates [ 65 ] . Litany et at . [ 59 ] proposed FMNet to learn the dense correspondence between deformable 3D shapes . • Geometric Deep Learning on Graphs . The earliest attempts we are aware of to generalize neural networks to graphs are attributed to M. Gori et at . [ 40 ] . More recently , a number of approaches encouraged by the success of convolutional neural networks [ 53 ] have attempted to generalize the notion of convolution to graphs . One important stream of convolution graph neural networks is spectral-based , where emerges after the pioneering work of Bruna et at . [ 11 ] which based on the spectral graph theory . There have been many following works [ 48 , 25 , 51 , 56 ] . Another stream of work define graph convolutions as extracting locally connected regions from the graph [ 30 , 58 , 69 , 45 , 97 , 67 ] . Many of these works were formulated in the family of message passing neural networks [ 38 ] which apply parametric functions to a node and its proximities , and use pooling operations to generate features for the node . Efficiency and scalability for deep graph learning is very important especially for large graphs and higher-order operations , which triggers research on accelerating GNNs [ 45 , 16 , 15 ] . Hamilton et at . [ 45 ] first introduced sampling scheme on neighborhood nodes to restrict the size . Chen et at . [ 16 ] proposed a method which samples vertices rather than neighbors . However , none of these works can guarantee the sampled graph is connected . Deep Learning on Spatial Data . Deep learning has also boosted the study on spatial data . Significant progress has been achieved on deep learning on images since AlexNet [ 47 , 75 ] . For 3D point clouds , PointNet [ 71 ] is a pioneering work which addressed the permutation invariance by a symmetric function . PointCNN [ 57 ] transforms the input points into a latent and potentially canonical order by a χ-conv transformations . Volumetric-based methods usually apply a 3D Convolution Neural Network ( CNN ) to 3D grids [ 95 , 66 ] . Wang et at . [ 85 ] first performed shape segmentation on 3D meshes by taking three low-level geometric features as its input . Despite the success of generalizing deep learning to network and spatial data separately , there has been relatively little work that simultaneously characterize both of them and their interaction . Previous models such as [ 38 , 73 , 52 ] are domain-specific , [ 73 , 52 ] treat spatial networks as point clouds which ignores the influence of network structure , and [ 87 , 86 ] consider POI ( Point of Interest ) categories , hence such concept graphs are not physically embedded in a geometric space . In addition , existing works [ 88 , 23 ] typically utilize the off-the-shelf deep neural networks with Cartesian coordinates as inputs and a large amount of rotation-and translation-augmented data , which is computationally expensive and lacks theoretical guarantee of rotation- and translation-invariant on the representation . To the best of our knowledge , our proposed method is the first generic framework of spatial network representation learning that handles substantial properties of rotation- and translation-invariant and the interplay between spatial and graph patterns with a theoretical guarantee . 3 Preliminaries . In this section , we first formalize spatial networks , and the problem of representation learning on spatial networks , then we introduce the challenges in order to solve this problem . Spatial graphs ( also known as spatial networks [ 6 ] ) are networks for which the nodes and edges are embedded in a geometric space . Spatial networks is ubiquitous in real world , such as molecular graphs [ 94 ] , biological neural networks [ 31 ] , and mobility networks [ 18 ] , where the spatial and network properties are usually coupled together tightly . For example , chemical bonds are derived from spatially close atoms , and fiber nerves tend to connect neurons close to each other . A spatial network is typically defined as S = ( G , P ) , where a graph G = ( V , E ) denotes the graph topology such that V is the set of N nodes and E ⊆ V × V is the set of M edges . eij ∈ E is an edge connecting nodes vi and vj ∈ V . P denotes the spatial information that is expressed as a set of points P = { ( xi , yi , zi ) |xi , yi , zi ∈ R } in Cartesian coordinate system , such that for a node vi ∈ V , its coordinate is denoted as ( xi , yi , zi ) ∈ P . Permutation invariance are crucial to graph structured data [ 96 ] . The collections of permutation-invariant functions on graph-structured data is defined so that f ( π†Sπ ) = f ( S ) , for all π ∈ Sn , where Sn is the permutation group of n elements . Rotation and translation invariance are in natural and common requirements for spatial data [ 36 , 52 ] . The collections of rotation- and translation-invariant functions on spatial networks is defined so that f ( G , T ( P ) ) = f ( G , P ) , for all T ∈ SE ( 3 ) , where SE ( 3 ) is the continuous Lie group of rotation and translation transformations in R3 . The main goal of this paper is to learn the representation f ( S ) of spatial network S = ( G , P ) , with the simultaneous satisfaction of strong discriminative power and the aforementioned significant symmetry properties , which is a problem extremely difficult to address due to several unique challenges : 1 ) Difficult in distinguishing the patterns that require joint spatial and graph consideration . 2 ) Difficult in jointly maintaining that the representation is invariant to node permutations , rotation and translation transformations . 3 ) High efficiency and scalability according to the requirement of handling incremental information .

Summary: The paper proposed a new graph representation learning method specifically for spatial networks. This method considered both spatial and network information with a theoretical guarantee with node permutation, rotation and translation invariance by introducing length three path. Additionally, an approach was developed to improve time and memory efficiency. Its ability of network representation has been demonstrated in multiple synthetic and real-world datasets. Overall, this paper is easy to follow and well-written.