Topological Graph Neural Networks

1 INTRODUCTION . Graphs are a natural description of structured data sets in many domains , including bioinformatics , image processing , and social network analysis . Numerous methods address the two dominant graph learning tasks of graph classification or node classification . In particular , graph neural networks ( GNNs ) describe a flexible set of architectures for such tasks and have seen many successful applications over recent years ( Wu et al. , 2021 ) . At their core , many GNNs are based on iterative message passing schemes ( see Sanchez-Lengeling et al . ( 2021 ) for an introduction to GNNs ) . Since these schemes are collating information over the neighbours of every node , GNNs can not necessarily capture certain topological structures in graphs , such as cycles ( Bouritsas et al. , 2021 ) . These structures , however , are relevant for certain applications , such as the analysis of molecular graphs , whose classification necessitates knowledge about connectivity information ( Hofer et al. , 2020 ; Swenson et al. , 2020 ) . In this paper , we address this issue by proposing a Topological Graph Layer ( TOGL ) that can be easily integrated into any GNN to make it ‘ topology-aware ’ . Our method is rooted in the nascent field of topological data analysis ( TDA ) , which focuses on describing coarse structures that can be used to describe the shape of complex structured and unstructured data sets at multiple scales . We thus obtain a generic way to augment existing GNNs and increase their expressivity in graph learning tasks . Section 1 provides a motivational example that showcases the potential benefits of using topological information : ( i ) high predictive performance is reached earlier for a smaller number of layers , and ( ii ) learnable topological representations outperform fixed ones if more complex topological structures are present in a data set . Our contributions . We propose TOGL , a novel layer based on TDA concepts that can be integrated into any GNN . Our layer is differentiable and capable of learning contrasting topological representations of a graph . We prove that TOGL enhances expressivity of a GNN since it incorporates the ability to work with multi-scale topological information in a graph . Moreover , we show that TOGL improves predictive performance of several GNN architectures when topological information is relevant for the respective task . 2 BACKGROUND : COMPUTATIONAL TOPOLOGY . We consider undirected graphs of the form G = ( V , E ) with a set of vertices V and a set of edges E ⊆ V × V . The basic topological features of such a graph G are the number of connected components β0 and the number of cycles β1 . These counts are also known as the 0-dimensional and 1-dimensional Betti numbers , respectively , and can be computed efficiently . Betti numbers are invariant under graph isomorphism ( Hatcher , 2002 , pp . 103–133 ) . The expressivity of Betti numbers can be increased by assuming the existence of a graph filtration , i.e . a sequence of nested subgraphs of G such that ∅ = G ( 0 ) ⊆ G ( 1 ) ⊆ G ( 2 ) ⊆ · · · ⊆ G ( n−1 ) ⊆ G ( n ) = G. A filtration makes it possible to obtain more insights into the graph by ‘ monitoring ’ topological features of each G ( i ) and calculating their topological relevance , also referred to as their persistence . If a topological feature appears for the first time in G ( i ) and disappears in G ( j ) , we assign this feature a persistence of j − i. Equivalently , we can represent the feature as a tuple ( i , j ) , which we collect in a persistence diagram D. If a feature never disappears , we represent it by a tuple ( i , ∞ ) ; such features are the ones that are counted for the respective Betti numbers . This process was formalised and extended to a wider class of structured data sets , namely simplicial complexes , and is known under the name of persistent homology . One of its core concepts is the use of a filtration function f : V → Rd , with d = 1 typically , to accentuate certain structural features of a graph . This replaces the aforementioned tuples of the form ( i , j ) by tuples based on the values of f , i.e . ( fi , fj ) . Persistent homology has shown excellent promise in different areas of machine learning research ( see Hensel et al . ( 2021 ) for a recent survey ) , with existing work stressing that choosing or learning an appropriate filtration function f is crucial for high predictive performance ( Hofer et al. , 2020 ; Zhao and Wang , 2019 ) . Notation . We denote the calculation of persistence diagrams of a graph G under some filtration f by ph ( G , f ) . This will result in two persistence diagrams D ( 0 ) , D ( 1 ) , containing information about topological features in dimension 0 ( connected components ) and dimension 1 ( cycles ) . The cardinality of D ( 0 ) is equal to the number of nodes n in the graphs and each tuple in the 0-dimensional diagram is naturally associated with the vertex that created it . The cardinality of D ( 1 ) is the number of cycles . However , from a computational perspective , we pair each tuple in D ( 1 ) with the edge that created it . Unpaired edges—edges that are not used to create a cycle—are given a ‘ dummy ’ tuple value , such as ( 0 , 0 ) . All other edges will be paired with the maximum value of the filtration , following previous work by Hofer et al . ( 2017 ) . Please refer to a Appendix A for a more technical description of persistent homology . 3 RELATED WORK . Graph representation learning has received a large amount of attention by the machine learning community . Graph kernel methods address graph classification via ( implicit or explicit ) embeddings in Reproducing Kernel Hilbert Spaces ( Borgwardt et al. , 2020 ; Kriege et al. , 2020 ; Nikolentzos et al. , 2019 ) . While powerful and expressive , they can not capture partial similarities between neighbourhoods . This can be achieved by graph neural networks , which typically employ message passing on graphs to learn hidden representations of graph structures ( Kipf and Welling , 2017 ; Wu et al. , 2021 ) . Recent work in this domain is abundant and includes attempts to utilise additional substructures ( Bouritsas et al. , 2021 ) as well as defining higher-order message passing schemes ( Morris et al. , 2019 ) or algorithms that generalise message passing to more complex domains ( Bodnar et al. , 2021 ) . Our approach falls into the realm of topological data analysis ( Edelsbrunner and Harer , 2010 ) and employs persistent homology , a technique for calculating topological features—such as connected components and cycles—of structured data sets . These features are known to be highly characteristic , leading to successful topology-driven graph machine learning approaches ( Hofer et al. , 2017 ; 2020 ; Rieck et al. , 2019 ; Zhao and Wang , 2019 ) . At their core is the notion of a filtration , i.e . a sequence of nested subgraphs ( or simplicial complexes in a higher-dimensional setting ) . Choosing the right filtration is known to be crucial for obtaining good performance ( Zhao and Wang , 2019 ) . This used to be a daunting task because persistent homology calculations are inherently discrete . Recent advances in proving differentiability enable proper end-to-end training of persistent homology ( Carrière et al. , 2021 ) , thus opening the door for hybrid methods of increased expressivity by integrating the somewhat complementary view of topological features . Our method TOGL builds upon the theoretical framework by Hofer et al . ( 2020 ) , who ( i ) demonstrated that the output of a GNN can be used to ‘ learn ’ one task-specific filtration function , and ( ii ) described the conditions under which a filtration function f is differentiable . This work culminated in GFL , a topological readout function that exhibited improved predictive performance for graph classification tasks . Our work significantly extends the utility of topological features by making existing GNNs ‘ topology-aware ’ through the development of a generic layer that makes topological information available to all downstream GNN layers . Thus , TOGL can be integrated into any GNN architecture , enabling the creation of hybrid models whose expressivity is provably more powerful than that of a GNN alone . Moreover , while GFL only uses the output of a GNN to drive the calculation of topological features by means of a single filtration ( thus limiting the applicability of the approach , as the topological features can not inform the remainder of a network ) , TOGL learns multiple filtrations of a graph in an end-to-end manner . More precisely , TOGL includes topological information in the hidden representations of nodes , enabling networks to change the importance of the topological signal . Closest to the scope of TOGL is Zhao et al . ( 2020 ) , who presented a way to enhance GNNs using topological information for node classification . In their framework , however , topology is only used to provide additional scalar-valued weights for the message passing scheme , and topological features are only calculated over small neighbourhoods , making use of a static vectorisation technique of persistence diagrams . Similarly , Wong and Vong ( 2021 ) use static , i.e . non-learnable , topological features for 3D shape segmentation . By contrast , TOGL , being end-to-end differentiable , is more general and permits the calculation of topological features at all scales—including graph-level features—as well as an integration into arbitrary GNN architectures . 4 TOGL : A TOPOLOGICAL GRAPH LAYER . TOGL is a new type of graph neural network layer that is capable of utilising multi-scale topological information of input graphs . In this section , we give a brief overview of the components of this layer before discussing algorithmic details , theoretical expressivity , computational complexity , and limitations . Figure 2 presents an overview of our method ( we show only a single graph being encoded , but in practice , the layer operates on batches of graphs ) . The layer takes as input a graph G = ( V , E ) equipped with a set of n vertices V and a set of edges E , along with a set of d-dimensional node attribute vectors x ( v ) ∈ Rd for v ∈ V . These node attributes can either be node features of a data set or hidden representations learnt by some GNN . We employ a family of k vertex filtration functions of the form fi : Rd → R for i = 1 , . . . , k. Each filtration function fi can focus on different properties of the graph . The image of fi is finite and results in a set of node values a ( 1 ) i < · · · < a ( n ) i such that the graph G is filtered according to ∅ = G ( 0 ) i ⊆ G ( 1 ) i ⊆ · · · ⊆ G ( n ) i = G , where G ( j ) i = ( V ( j ) i , E ( j ) i ) , with V ( j ) i : = { v ∈ V | fi ( x ( v ) ) ≤ a ( j ) i } , and E ( j ) i : = { v , w ∈ E | max { fi ( x ( v ) ) , fi ( x ( w ) ) } ≤ a ( j ) i } . Given this filtration , we calculate a set of persistence diagrams , i.e . ph ( G , fi ) = { D ( 0 ) i , . . . , D ( l ) i } . We fix l = 1 ( i.e . we are capturing connected components and cycles ) to simplify our current implementation , but our layer can be extended to arbitrary values of l ( see Appendix C for a discussion ) . In order to benefit from representations that are trainable end-to-end , we use an embedding function Ψ ( l ) : { D ( l ) 1 , . . . , D ( l ) k } → Rn′×d for embedding persistence diagrams into a high-dimensional space that will be used to obtain the vertex representations , where n′ is the number of vertices n if l = 0 and the number of edges if l = 1 . This step is crucial as it enables us to use the resulting topological features as node features , making TOGL a layer that can be integrated into arbitrary GNNs . We later explain the precise mapping of Ψ ( l ) from a set of diagrams to the elements of a graph . Details on filtration computation and output generation . We compute our family of k vertex-based filtrations using Φ : Rd → Rk , an MLP with a single hidden layer , such that fi : = πi ◦ Φ , i.e . the projection of Φ to the ith dimension . We apply Φ to the hidden representations x ( v ) of all vertices in the graph . Moreover , we treat the resulting persistence diagrams in dimension 0 and 1 differently . For dimension 0 , we have a bijective mapping of tuples in the persistence diagram to the vertices of the graph , which was previously exploited in topological representation learning ( Moor et al. , 2020 ) . Therefore , we aggregate Ψ ( 0 ) with the original node attribute vector x ( v ) of the graph in a residual fashion , i.e . x̃ ( v ) = x ( v ) + Ψ ( 0 ) ( D ( 0 ) 1 , . . . , D ( 0 ) k ) [ v ] , where Ψ ( 0 ) [ v ] denotes taking vth row of Ψ ( 0 ) ( i.e the topological embedding of vertex v ) . The output of our layer for dimension 0 therefore results in a new representation x̃ ( v ) ∈ Rd for each vertex v , making it compatible with any subsequent ( GNN ) layers . By contrast , Ψ ( 1 ) is pooled into a graph-level representation , to be used in the final classification layer of a GNN . This is necessary because there is no bijective mapping to the vertices , but rather to edges . For stability reasons ( Bendich et al. , 2020 ) , we consider it more useful to have this information available only on the graph level . For further details on the computational aspects , please refer to Section A.4 . Complexity and limitations . Persistent homology can be calculated efficiently for dimensions 0 and 1 , having a worst-case complexity of O ( mα ( m ) ) for a graph with m sorted edges , where α ( · ) is the extremely slow-growing inverse Ackermann function , which can be treated as constant for all intents and purposes . The calculation of ph is therefore dominated by the complexity of sorting all edges , i.e . O ( m logm ) , making our approach efficient and scalable . Higher-dimensional persistent homology calculations unfortunately do not scale well , having a worst-case complexity of O ( md ) for calculating d-dimensional topological features , which is why we restrict ourselves to l = 1 here . At the moment , our approach is therefore limited to connected components and cycles . Moreover , our filtrations are incapable of assessing the impact of feature interactions on the topology ; for this , we would have to learn multifiltrations ( Carlsson et al. , 2009 ) , but multifiltrations do not afford a concise , efficient representation as the scalar-valued filtrations discussed in this paper . We therefore leave their treatment for future work .

Summary: The authors present a topology analysis improvement to GCN, using persistent homology, to capture global information regarding the topology of the graph.  The authors conduct several experiments, from graph to node classification, and also introduce two novel data sets to exemplify the importance of topology.  In most cases, the proposed method outperforms other baseline methods, as well as other topology aware methods.