Representation Learning on Spatial Networks

1 Introduction . Spatial data and network data are both popular types of data in modern big data era . The study of spatial data focuses on the properties of continuous spatial entities under specific geometry , while analysis of network data investigates the properties of discrete objects and their pairwise relationship . Spanning these two data types , spatial networks is a crucial type of data structure that nodes occupy positions in a Euclidean space , where spatial patterns and constraints may have a strong effect on their connectivity patterns [ 6 ] . Understanding the mechanism of organizing spatial networks has significant importance for a broad range of fields [ 29 ] , ranging from micro-scale ( e.g. , molecule structure [ 94 ] ) , to middle-scale ( e.g. , biological neural network [ 31 ] ) , to macro-scale ( e.g. , mobility networks [ 18 ] ) . Effectively learning the representations of spatial networks is extremely challenging due to the close interactions between network and spatial topology , the incompatibility between the treatments for discrete and continuous data , and particular properties 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . such as permutation invariant and rotation-translation invariant . Spatial networks have long been researched in the domains such as physics and mathematics , which usually extend complex networks and graph theory into spatial networks [ 80 , 7 ] . They typically rely on network generation principles predefined by human heuristics and prior knowledge . Such methods usually characterize well on the aspects of the data that have been covered by the predefined principles , but not on those have not been covered [ 6 ] . However , the underlying network process in complex networks is largely unknown and extremely difficult to be predefined in simple rules , especially in crucial and open domains such as brain network modeling [ 81 ] , network catastrophic failure [ 72 ] , and protein folding [ 27 ] . Remarkable progress has been made towards generalizing deep representation learning approaches in spatial data and network data [ 96 , 13 , 46 , 43 , 28 ] , respectively , in recent years . For spatial data , deep learning achieved significant progress in different commonly used formats such as images [ 53 , 75 , 61 , 21 ] , point clouds [ 34 , 71 , 57 ] , meshes [ 85 , 90 ] , and volumetric grids [ 95 , 66 ] . On the other hand , deep learning has also boosted the research of encoding graph structure on network data [ 46 , 51 , 45 ] , and downstream applications such as recommender systems [ 99 , 60 ] , drug discovery [ 39 , 22 , 41 , 42 ] , FinTech [ 91 ] , customer care [ 93 ] , and natural language processing [ 62 , 8 , 92 ] . Despite the respective progress in representation learning on spatial data and network data in parallel , the representation learning for spatial networks have been largely underexplored and has just started to attract fast-increasing attention . Merely combining spatial and graph representations separately can not handle that for spatial networks where spatial and network process are deeply coupled together [ 6 , 68 , 33 ] . For example , Fig . 2 shows a simple example with a pair of spatial networks , in which there are different formation rules on the edges relied on the spatial distance , that is non-distinguishable for spatial and network embedding methods , respectively . Few recent attempts have been proposed to handle representation learning on spatial networks but still suffer from key challenges : Spatial network representation learning is a problem extremely difficult to address due to several unique challenges : 1 ) Difficult in distinguishing the patterns that require joint spatial and graph consideration . Examples like Figure . 2 that share the same spatial and network topology , respectively , but with significantly different interaction mechanisms , are non-distinguishable to either spatial or graph methods . 2 ) Difficult in jointly maintaining that the learned representation is invariant to node permutation , and rotation and translation transformations . Notice that spatial and graph information confine each other which neutralizes conventional methods to have either of them . For example , although point clouds representation learning can easily preserve rotation- and translation-invariant by using spatial nearest neighbors , here in spatial networks the neighbor is confined also by graph neighbors . Such additional confinement largely harden our task . 3 ) High efficiency and scalability in the graph size . The confinement between spatial and graph information inevitably leads to taking into account more entities simultaneously to maintain sufficient information . The requirement to handle incremental information increases the demand for model efficiency and scalability . In order to address all the aforementioned challenges , this paper proposes a new spatial graph message passing neural network ( SGMP ) for learning the representations of generic spatial networks , with theoretical guarantees on discriminative power and various spatial and network properties , and an accelerating algorithm which adjusts to our theoretical framework . Specifically , to capture and model the intrinsic coupled spatial and graph properties , we propose a novel message passing neural network to organically aggregate the spatial and graph information . To ensure the invariance of learned representation under rotation and translation transformations , a novel way to represent the node spatial information by characterizing geometric invariant features with lossless information is proposed . To alleviate the efficiency issue , we propose a new accelerating algorithm for learning on graph-structured data . The proposed accelerating algorithm effectively reduces the time and memory complexity from O ( N3 ) to O ( N ) , and maintains the theoretical guarantees for spatial networks . Finally , we demonstrate the strength of our theoretical findings through extensive experiments on both synthetic datasets and real-world datasets . 2 Related Work . Spatial Networks There has been a long time of research efforts on the subjects of spatial networks [ 6 ] . In the area of quantitative geography , Haggett and Chorley discussed the relevance of space in the formation and evolution of networks , and developed models to characterize spatial networks at least fourty years ago [ 44 , 17 ] . New insights leading to modern quantitative solutions are gained due to the advance in complex networks [ 32 , 89 , 1 , 4 , 3 , 20 ] , and appears in more practical fields such as transportation networks [ 2 , 54 , 55 ] , mobility networks [ 18 , 24 ] , biological networks [ 31 , 76 ] , and computational chemistry [ 38 , 73 , 52 ] . Geometric Deep Learning . This is a more recent domain which handles non-Euclidean structured data such as graphs and manifolds [ 13 ] . • Geometric Deep Learning on Manifolds . There is a large body of research efforts of generalizing deep learning models to 3D shapes as manifolds in the computer graphics community . Many works have been conducted to find a better approach to generalize convolution-like operations to the nonEuclidean domain [ 65 , 10 , 74 , 98 , 64 , 59 ] . J. Masci et at . proposed the framework of generalizing convolution neural network paradigm to manifolds by applying filters to extract local patches in polar coordinates [ 65 ] . Litany et at . [ 59 ] proposed FMNet to learn the dense correspondence between deformable 3D shapes . • Geometric Deep Learning on Graphs . The earliest attempts we are aware of to generalize neural networks to graphs are attributed to M. Gori et at . [ 40 ] . More recently , a number of approaches encouraged by the success of convolutional neural networks [ 53 ] have attempted to generalize the notion of convolution to graphs . One important stream of convolution graph neural networks is spectral-based , where emerges after the pioneering work of Bruna et at . [ 11 ] which based on the spectral graph theory . There have been many following works [ 48 , 25 , 51 , 56 ] . Another stream of work define graph convolutions as extracting locally connected regions from the graph [ 30 , 58 , 69 , 45 , 97 , 67 ] . Many of these works were formulated in the family of message passing neural networks [ 38 ] which apply parametric functions to a node and its proximities , and use pooling operations to generate features for the node . Efficiency and scalability for deep graph learning is very important especially for large graphs and higher-order operations , which triggers research on accelerating GNNs [ 45 , 16 , 15 ] . Hamilton et at . [ 45 ] first introduced sampling scheme on neighborhood nodes to restrict the size . Chen et at . [ 16 ] proposed a method which samples vertices rather than neighbors . However , none of these works can guarantee the sampled graph is connected . Deep Learning on Spatial Data . Deep learning has also boosted the study on spatial data . Significant progress has been achieved on deep learning on images since AlexNet [ 47 , 75 ] . For 3D point clouds , PointNet [ 71 ] is a pioneering work which addressed the permutation invariance by a symmetric function . PointCNN [ 57 ] transforms the input points into a latent and potentially canonical order by a χ-conv transformations . Volumetric-based methods usually apply a 3D Convolution Neural Network ( CNN ) to 3D grids [ 95 , 66 ] . Wang et at . [ 85 ] first performed shape segmentation on 3D meshes by taking three low-level geometric features as its input . Despite the success of generalizing deep learning to network and spatial data separately , there has been relatively little work that simultaneously characterize both of them and their interaction . Previous models such as [ 38 , 73 , 52 ] are domain-specific , [ 73 , 52 ] treat spatial networks as point clouds which ignores the influence of network structure , and [ 87 , 86 ] consider POI ( Point of Interest ) categories , hence such concept graphs are not physically embedded in a geometric space . In addition , existing works [ 88 , 23 ] typically utilize the off-the-shelf deep neural networks with Cartesian coordinates as inputs and a large amount of rotation-and translation-augmented data , which is computationally expensive and lacks theoretical guarantee of rotation- and translation-invariant on the representation . To the best of our knowledge , our proposed method is the first generic framework of spatial network representation learning that handles substantial properties of rotation- and translation-invariant and the interplay between spatial and graph patterns with a theoretical guarantee . 3 Preliminaries . In this section , we first formalize spatial networks , and the problem of representation learning on spatial networks , then we introduce the challenges in order to solve this problem . Spatial graphs ( also known as spatial networks [ 6 ] ) are networks for which the nodes and edges are embedded in a geometric space . Spatial networks is ubiquitous in real world , such as molecular graphs [ 94 ] , biological neural networks [ 31 ] , and mobility networks [ 18 ] , where the spatial and network properties are usually coupled together tightly . For example , chemical bonds are derived from spatially close atoms , and fiber nerves tend to connect neurons close to each other . A spatial network is typically defined as S = ( G , P ) , where a graph G = ( V , E ) denotes the graph topology such that V is the set of N nodes and E ⊆ V × V is the set of M edges . eij ∈ E is an edge connecting nodes vi and vj ∈ V . P denotes the spatial information that is expressed as a set of points P = { ( xi , yi , zi ) |xi , yi , zi ∈ R } in Cartesian coordinate system , such that for a node vi ∈ V , its coordinate is denoted as ( xi , yi , zi ) ∈ P . Permutation invariance are crucial to graph structured data [ 96 ] . The collections of permutation-invariant functions on graph-structured data is defined so that f ( π†Sπ ) = f ( S ) , for all π ∈ Sn , where Sn is the permutation group of n elements . Rotation and translation invariance are in natural and common requirements for spatial data [ 36 , 52 ] . The collections of rotation- and translation-invariant functions on spatial networks is defined so that f ( G , T ( P ) ) = f ( G , P ) , for all T ∈ SE ( 3 ) , where SE ( 3 ) is the continuous Lie group of rotation and translation transformations in R3 . The main goal of this paper is to learn the representation f ( S ) of spatial network S = ( G , P ) , with the simultaneous satisfaction of strong discriminative power and the aforementioned significant symmetry properties , which is a problem extremely difficult to address due to several unique challenges : 1 ) Difficult in distinguishing the patterns that require joint spatial and graph consideration . 2 ) Difficult in jointly maintaining that the representation is invariant to node permutations , rotation and translation transformations . 3 ) High efficiency and scalability according to the requirement of handling incremental information .

Summary: In this paper, the authors studied the problem of representation learning on spatial graphs. To achieve the goal, the authors proposed a generic framework by aggregating message passing mechanisms and spatial information. The authors also evaluated the proposed method on both synthetic data and real-world data.