Topological Graph Neural Networks

1 INTRODUCTION . Graphs are a natural description of structured data sets in many domains , including bioinformatics , image processing , and social network analysis . Numerous methods address the two dominant graph learning tasks of graph classification or node classification . In particular , graph neural networks ( GNNs ) describe a flexible set of architectures for such tasks and have seen many successful applications over recent years ( Wu et al. , 2021 ) . At their core , many GNNs are based on iterative message passing schemes ( see Sanchez-Lengeling et al . ( 2021 ) for an introduction to GNNs ) . Since these schemes are collating information over the neighbours of every node , GNNs can not necessarily capture certain topological structures in graphs , such as cycles ( Bouritsas et al. , 2021 ) . These structures , however , are relevant for certain applications , such as the analysis of molecular graphs , whose classification necessitates knowledge about connectivity information ( Hofer et al. , 2020 ; Swenson et al. , 2020 ) . In this paper , we address this issue by proposing a Topological Graph Layer ( TOGL ) that can be easily integrated into any GNN to make it ‘ topology-aware ’ . Our method is rooted in the nascent field of topological data analysis ( TDA ) , which focuses on describing coarse structures that can be used to describe the shape of complex structured and unstructured data sets at multiple scales . We thus obtain a generic way to augment existing GNNs and increase their expressivity in graph learning tasks . Section 1 provides a motivational example that showcases the potential benefits of using topological information : ( i ) high predictive performance is reached earlier for a smaller number of layers , and ( ii ) learnable topological representations outperform fixed ones if more complex topological structures are present in a data set . Our contributions . We propose TOGL , a novel layer based on TDA concepts that can be integrated into any GNN . Our layer is differentiable and capable of learning contrasting topological representations of a graph . We prove that TOGL enhances expressivity of a GNN since it incorporates the ability to work with multi-scale topological information in a graph . Moreover , we show that TOGL improves predictive performance of several GNN architectures when topological information is relevant for the respective task . 2 BACKGROUND : COMPUTATIONAL TOPOLOGY . We consider undirected graphs of the form G = ( V , E ) with a set of vertices V and a set of edges E ⊆ V × V . The basic topological features of such a graph G are the number of connected components β0 and the number of cycles β1 . These counts are also known as the 0-dimensional and 1-dimensional Betti numbers , respectively , and can be computed efficiently . Betti numbers are invariant under graph isomorphism ( Hatcher , 2002 , pp . 103–133 ) . The expressivity of Betti numbers can be increased by assuming the existence of a graph filtration , i.e . a sequence of nested subgraphs of G such that ∅ = G ( 0 ) ⊆ G ( 1 ) ⊆ G ( 2 ) ⊆ · · · ⊆ G ( n−1 ) ⊆ G ( n ) = G. A filtration makes it possible to obtain more insights into the graph by ‘ monitoring ’ topological features of each G ( i ) and calculating their topological relevance , also referred to as their persistence . If a topological feature appears for the first time in G ( i ) and disappears in G ( j ) , we assign this feature a persistence of j − i. Equivalently , we can represent the feature as a tuple ( i , j ) , which we collect in a persistence diagram D. If a feature never disappears , we represent it by a tuple ( i , ∞ ) ; such features are the ones that are counted for the respective Betti numbers . This process was formalised and extended to a wider class of structured data sets , namely simplicial complexes , and is known under the name of persistent homology . One of its core concepts is the use of a filtration function f : V → Rd , with d = 1 typically , to accentuate certain structural features of a graph . This replaces the aforementioned tuples of the form ( i , j ) by tuples based on the values of f , i.e . ( fi , fj ) . Persistent homology has shown excellent promise in different areas of machine learning research ( see Hensel et al . ( 2021 ) for a recent survey ) , with existing work stressing that choosing or learning an appropriate filtration function f is crucial for high predictive performance ( Hofer et al. , 2020 ; Zhao and Wang , 2019 ) . Notation . We denote the calculation of persistence diagrams of a graph G under some filtration f by ph ( G , f ) . This will result in two persistence diagrams D ( 0 ) , D ( 1 ) , containing information about topological features in dimension 0 ( connected components ) and dimension 1 ( cycles ) . The cardinality of D ( 0 ) is equal to the number of nodes n in the graphs and each tuple in the 0-dimensional diagram is naturally associated with the vertex that created it . The cardinality of D ( 1 ) is the number of cycles . However , from a computational perspective , we pair each tuple in D ( 1 ) with the edge that created it . Unpaired edges—edges that are not used to create a cycle—are given a ‘ dummy ’ tuple value , such as ( 0 , 0 ) . All other edges will be paired with the maximum value of the filtration , following previous work by Hofer et al . ( 2017 ) . Please refer to a Appendix A for a more technical description of persistent homology . 3 RELATED WORK . Graph representation learning has received a large amount of attention by the machine learning community . Graph kernel methods address graph classification via ( implicit or explicit ) embeddings in Reproducing Kernel Hilbert Spaces ( Borgwardt et al. , 2020 ; Kriege et al. , 2020 ; Nikolentzos et al. , 2019 ) . While powerful and expressive , they can not capture partial similarities between neighbourhoods . This can be achieved by graph neural networks , which typically employ message passing on graphs to learn hidden representations of graph structures ( Kipf and Welling , 2017 ; Wu et al. , 2021 ) . Recent work in this domain is abundant and includes attempts to utilise additional substructures ( Bouritsas et al. , 2021 ) as well as defining higher-order message passing schemes ( Morris et al. , 2019 ) or algorithms that generalise message passing to more complex domains ( Bodnar et al. , 2021 ) . Our approach falls into the realm of topological data analysis ( Edelsbrunner and Harer , 2010 ) and employs persistent homology , a technique for calculating topological features—such as connected components and cycles—of structured data sets . These features are known to be highly characteristic , leading to successful topology-driven graph machine learning approaches ( Hofer et al. , 2017 ; 2020 ; Rieck et al. , 2019 ; Zhao and Wang , 2019 ) . At their core is the notion of a filtration , i.e . a sequence of nested subgraphs ( or simplicial complexes in a higher-dimensional setting ) . Choosing the right filtration is known to be crucial for obtaining good performance ( Zhao and Wang , 2019 ) . This used to be a daunting task because persistent homology calculations are inherently discrete . Recent advances in proving differentiability enable proper end-to-end training of persistent homology ( Carrière et al. , 2021 ) , thus opening the door for hybrid methods of increased expressivity by integrating the somewhat complementary view of topological features . Our method TOGL builds upon the theoretical framework by Hofer et al . ( 2020 ) , who ( i ) demonstrated that the output of a GNN can be used to ‘ learn ’ one task-specific filtration function , and ( ii ) described the conditions under which a filtration function f is differentiable . This work culminated in GFL , a topological readout function that exhibited improved predictive performance for graph classification tasks . Our work significantly extends the utility of topological features by making existing GNNs ‘ topology-aware ’ through the development of a generic layer that makes topological information available to all downstream GNN layers . Thus , TOGL can be integrated into any GNN architecture , enabling the creation of hybrid models whose expressivity is provably more powerful than that of a GNN alone . Moreover , while GFL only uses the output of a GNN to drive the calculation of topological features by means of a single filtration ( thus limiting the applicability of the approach , as the topological features can not inform the remainder of a network ) , TOGL learns multiple filtrations of a graph in an end-to-end manner . More precisely , TOGL includes topological information in the hidden representations of nodes , enabling networks to change the importance of the topological signal . Closest to the scope of TOGL is Zhao et al . ( 2020 ) , who presented a way to enhance GNNs using topological information for node classification . In their framework , however , topology is only used to provide additional scalar-valued weights for the message passing scheme , and topological features are only calculated over small neighbourhoods , making use of a static vectorisation technique of persistence diagrams . Similarly , Wong and Vong ( 2021 ) use static , i.e . non-learnable , topological features for 3D shape segmentation . By contrast , TOGL , being end-to-end differentiable , is more general and permits the calculation of topological features at all scales—including graph-level features—as well as an integration into arbitrary GNN architectures . 4 TOGL : A TOPOLOGICAL GRAPH LAYER . TOGL is a new type of graph neural network layer that is capable of utilising multi-scale topological information of input graphs . In this section , we give a brief overview of the components of this layer before discussing algorithmic details , theoretical expressivity , computational complexity , and limitations . Figure 2 presents an overview of our method ( we show only a single graph being encoded , but in practice , the layer operates on batches of graphs ) . The layer takes as input a graph G = ( V , E ) equipped with a set of n vertices V and a set of edges E , along with a set of d-dimensional node attribute vectors x ( v ) ∈ Rd for v ∈ V . These node attributes can either be node features of a data set or hidden representations learnt by some GNN . We employ a family of k vertex filtration functions of the form fi : Rd → R for i = 1 , . . . , k. Each filtration function fi can focus on different properties of the graph . The image of fi is finite and results in a set of node values a ( 1 ) i < · · · < a ( n ) i such that the graph G is filtered according to ∅ = G ( 0 ) i ⊆ G ( 1 ) i ⊆ · · · ⊆ G ( n ) i = G , where G ( j ) i = ( V ( j ) i , E ( j ) i ) , with V ( j ) i : = { v ∈ V | fi ( x ( v ) ) ≤ a ( j ) i } , and E ( j ) i : = { v , w ∈ E | max { fi ( x ( v ) ) , fi ( x ( w ) ) } ≤ a ( j ) i } . Given this filtration , we calculate a set of persistence diagrams , i.e . ph ( G , fi ) = { D ( 0 ) i , . . . , D ( l ) i } . We fix l = 1 ( i.e . we are capturing connected components and cycles ) to simplify our current implementation , but our layer can be extended to arbitrary values of l ( see Appendix C for a discussion ) . In order to benefit from representations that are trainable end-to-end , we use an embedding function Ψ ( l ) : { D ( l ) 1 , . . . , D ( l ) k } → Rn′×d for embedding persistence diagrams into a high-dimensional space that will be used to obtain the vertex representations , where n′ is the number of vertices n if l = 0 and the number of edges if l = 1 . This step is crucial as it enables us to use the resulting topological features as node features , making TOGL a layer that can be integrated into arbitrary GNNs . We later explain the precise mapping of Ψ ( l ) from a set of diagrams to the elements of a graph . Details on filtration computation and output generation . We compute our family of k vertex-based filtrations using Φ : Rd → Rk , an MLP with a single hidden layer , such that fi : = πi ◦ Φ , i.e . the projection of Φ to the ith dimension . We apply Φ to the hidden representations x ( v ) of all vertices in the graph . Moreover , we treat the resulting persistence diagrams in dimension 0 and 1 differently . For dimension 0 , we have a bijective mapping of tuples in the persistence diagram to the vertices of the graph , which was previously exploited in topological representation learning ( Moor et al. , 2020 ) . Therefore , we aggregate Ψ ( 0 ) with the original node attribute vector x ( v ) of the graph in a residual fashion , i.e . x̃ ( v ) = x ( v ) + Ψ ( 0 ) ( D ( 0 ) 1 , . . . , D ( 0 ) k ) [ v ] , where Ψ ( 0 ) [ v ] denotes taking vth row of Ψ ( 0 ) ( i.e the topological embedding of vertex v ) . The output of our layer for dimension 0 therefore results in a new representation x̃ ( v ) ∈ Rd for each vertex v , making it compatible with any subsequent ( GNN ) layers . By contrast , Ψ ( 1 ) is pooled into a graph-level representation , to be used in the final classification layer of a GNN . This is necessary because there is no bijective mapping to the vertices , but rather to edges . For stability reasons ( Bendich et al. , 2020 ) , we consider it more useful to have this information available only on the graph level . For further details on the computational aspects , please refer to Section A.4 . Complexity and limitations . Persistent homology can be calculated efficiently for dimensions 0 and 1 , having a worst-case complexity of O ( mα ( m ) ) for a graph with m sorted edges , where α ( · ) is the extremely slow-growing inverse Ackermann function , which can be treated as constant for all intents and purposes . The calculation of ph is therefore dominated by the complexity of sorting all edges , i.e . O ( m logm ) , making our approach efficient and scalable . Higher-dimensional persistent homology calculations unfortunately do not scale well , having a worst-case complexity of O ( md ) for calculating d-dimensional topological features , which is why we restrict ourselves to l = 1 here . At the moment , our approach is therefore limited to connected components and cycles . Moreover , our filtrations are incapable of assessing the impact of feature interactions on the topology ; for this , we would have to learn multifiltrations ( Carlsson et al. , 2009 ) , but multifiltrations do not afford a concise , efficient representation as the scalar-valued filtrations discussed in this paper . We therefore leave their treatment for future work .

Summary: This paper introduces TOGL, a new layer for Graph Neural Networks (GNN), making the GNN "aware" of topological information during this training phase. It differs from the closely related work *Graph Filtration Learning (GFL)* (although taking inspiration from it) as GFL is mostly a readout function (roughly, final layer in a GNN) while TOGL is a more general type of GNN-layer. Numerical experiments show how, when topological information is relevant, TOGL helps to leverage it.