Weisfeiler and Lehman Go Cellular: CW Networks

1 Introduction . The operations performed by message passing Graph Neural Networks ( GNNs ) emulate the structure of the input graph . While this property has clear computational advantages , it brings with it a series of fundamental limitations . As observed by Xu et al . [ 74 ] and Morris et al . [ 55 ] the local neighbourhood aggregations used by GNNs are at most as powerful as the Weisfeiler-Lehman ( WL ) test [ 71 ] in distinguishing non-isomorphic graphs . Therefore , GNNs fail to detect certain higer-order meso-scale structures such as cliques or ( induced ) cycles [ 2 , 15 ] , which are particularly important in applications dealing with social and biological networks or molecular graphs . At the same time , many such layers have to be stacked to make long-range interactions in the graph possible . Besides the computational burden this incurs , deep GNNs typically come with additional problems such as over-smoothing [ 51 ] and over-squashing [ 1 ] of the node representations . ⇤Authors contributed equally . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . To address these problems , we propose a novel message passing procedure based on ( regular ) cell complexes , also known as CW complexes2 , topological objects that form the building block of algebraic topology [ 38 ] . When paired with a theoretically-justified “ lifting ” transformation augmenting the graph with higher-dimensional constructs called “ cells ” , our method results in a multi-dimensional and hierarchical message passing procedure over the input graph . Our approach generalises and subsumes the recently proposed Message Passing Simplicial Networks ( MPSNs ) [ 8 ] , which operate on simplicial complexes ( SCs ) , topological generalisations of graphs . However , SCs have a rigid combinatorial structure that significantly limits the range of lifting transformations one could use to meaningfully modulate the message passing procedure . In contrast , we show that cell complexes , which in turn generalise simplicial complexes and come with additional flexibility , allow one to construct new and better ways of decoupling the input and computational graphs . Main Contributions To summarise , we propose a message passing scheme operating on regular cell complexes . We call this family of models CW Networks ( CWNs ) and study their expressive power using a cellular version of the WL test . We show that for an entire class of “ lifting ” transformations CWNs are at least as powerful as the WL test . Furthermore , we prove that for some of the maps in this class , CWNs can be strictly more powerful than WL , Simplicial WL ( SWL ) and also not less powerful than 3-WL . We also express the fundamental symmetries of these models and show how they can be seen as generalised convolutional operators on cell complexes . Experimentally , we focus our attention on a particular “ lifting ” map based on induced cycles . When applied to molecular graphs , it leads to an intuitive hierarchical message passing procedure involving the atoms , the bonds between them and the chemical rings of the molecules . We demonstrate that this provably powerful approach obtains state-of-the-art results on popular large-scale molecular graph datasets and other related tasks . To the best of our knowledge , this is the first work proposing a cell complex representation for molecules . Our code is available at https : //github.com/twitter-research/cwn . 2 Background . Definition 1 ( Hansen and Ghrist [ 36 ] ) . A regular cell complex ( Figure 1 ) is a topological space X together with a partition { X } 2PX of subspaces X of X called cells , and such that 1 . For each x 2 X there exists an open neighbordhood of x that intersects finitely many cells . 2 . For all , ⌧ we have that X⌧ \X 6= ; iff X⌧ ✓ X , where X is the closure of a cell . 3 . Every cell is homeomorphic to Rn for some n. 4 . ( Regularity ) For every 2 PX there is a homeomorphism of a closed ball in Rn to X such that the restriction of to the interior of the ball is a homeomorphism onto X . Figure 1 : A cell complex X and the corresponding homeomorphisms to the closed balls for three cells of different dimensions in the complex . We note that by condition ( 2 ) the indexing set PX has a poset structure ⌧ , X⌧ ✓ X , while condition ( 4 ) guarantees that this poset structure encodes all the topological information about X . Thus , we can identify a regular cell complex X with this poset , called face poset of X . We also use ⌧ < for the strict version of this partial order . Intuitively , one constructs a cell complex through a hierarchical gluing procedure . One starts with a set of vertices ( 0-cells ) . Then edges ( 1-cells ) are attached to these by gluing the endpoints of closed line segments to them . We have now only described a ( multi ) graph . However , one can generalise this even further by taking a two-dimensional closed disk and glue its boundary ( i.e . a circle ) to any simple cycle in the ( multi ) graph previously built as in Figure 2 . While we are generally not concerned with dimensions above two , this can be further generalised by gluing the boundary of n-dimensional balls to certain ( n 1 ) cells in the complex . Consider the examples in Figure 3 . The shown sphere is a cell complex obtained from two 0-cells ( i.e . vertices ) , to which two 1-cells ( i.e . edges ) , which form the equator , were attached . The boundary 2We use these terms interchangeably . For the latter , the C stands for “ closure-finite ” , and the W for “ weak ” topology . The term was coined by Whitehead [ 72 ] . of two 2-dimensional disks ( i.e . the two hemispheres ) were glued to the equator to form a sphere . The second example is a tetrahedron with empty interior . It is a particular type of cell complex called a simplicial complex ( SC ) . The only 2-cells it allows are triangle-shaped . More generally , the n-dimensional cells of SCs are n-simplices , which makes them slightly more rigid structures . Definition 2 . The k-skeleton of a cell complex X , denoted X ( k ) , is the subcomplex of X consisting of cells of dimension at most k. This definition is useful for referring for certain parts of the complex . For instance , X ( 0 ) contains the vertices in the complex , while X ( 1 ) contains the vertices and the edges ( i.e . the underlying graph ) . The combinatorial structure of the complex can be more compactly described by an incidence relation we call the boundary relation , whose reflexive and transitive closure gives the partial order defined above . The boundary relation describes what cells are on the boundary of other cells . For instance , the edges of the sphere in Figure 3 are on the boundary of the 2-cells forming the two hemispheres . Definition 3 . We have the boundary relation ⌧ iff < ⌧ and there is no cell such that < < ⌧ . We can use this to define the four types of ( local ) adjacencies present in cell complexes . These adjacencies will be the fundamental building block of our message passing procedure . To explain these in more familiar terms , for each adjacency , we exemplify how it shows up in graphs . Definition 4 ( Cell complex adjacencies ) . For a cell complex X and a cell 2 PX , we define : 1 . The boundary adjacent cells B ( ) = { ⌧ | ⌧ } . These are the lower-dimensional cells on the boundary of . For instance , the boundary cells of an edge are its vertices . 2 . The co-boundary adjacent cell C ( ) = { ⌧ | ⌧ } . These are the higher-dimensional cells with on their boundary . For instance , the co-boundary cells of a vertex are the edges it is part of . 3 . The lower adjacent cells N # ( ) = { ⌧ | 9 such that and ⌧ } . These are the cells of the same dimension as that share a lower dimensional cell on their boundary . The line graph adjacencies between the edges are a classic example of this . 4 . The upper adjacent cells N '' ( ) = { ⌧ | 9 such that and ⌧ } . These are the cells of the same dimension as that are on the boundary of the same higher-dimensional cell as . The typical graph adjacencies between vertices are the canonical example here . 3 Cellular Weisfeiler Lehman . Overview The results in this section show how one can transform graphs into higher-dimensional cell complexes in such a way that performing colour refinement on the resulting cell complexes makes it easier to test their isomorphism . The message passing model from Section 4 will take advantage of these theoretical results . All proofs can be found in Appendix A . Definition 5 . Let c be a colouring of the cells in a complex X with c denoting the colour assigned to cell 2 PX . Define B ( , ⌧ ) : = B ( ) \B ( ⌧ ) and C ( , ⌧ ) : = C ( ) \C ( ⌧ ) . We define the following multi-sets of colours : 1 . The colours of the boundary cells of : cB ( ) = { { c⌧ | ⌧ 2 B ( ) } } . 2 . The colours of the co-boundary cells of : cC ( ) = { { c⌧ | ⌧ 2 C ( ) } } . 3 . The lower adjacent colours of : c # ( ) = { { ( c⌧ , c ) | ⌧ 2 N # ( ) and 2 B ( , ⌧ ) } } . 4 . The upper adjacent colours of : c '' ( ) = { { ( c⌧ , c ) | ⌧ 2 N '' ( ) and 2 C ( , ⌧ ) } } . Note that unlike in graphs and simplicial complexes , the sets B ( , ⌧ ) and C ( , ⌧ ) can have more than one element . For instance , two ( closed ) 2-cells might intersect in more than one edge ( e.g . the two hemispheres in Figure 3 ) , and conversely , two edges might be on the boundary of the same two 2-cells . This illustrates the more flexible combinatorial structure of cell complexes . Cellular WL ( CWL ) We consider CWL , a colour refinement scheme for cell complexes that generalises the Simplicial WL [ 8 ] and WL [ 71 ] tests . We use ct to refer to the colour assigned by CWL to cell at iteration t of the algorithm . When the input is a simplicial complex , this recovers the SWL algorithm . A step of the algorithm is graphically depicted in Figure 4 for a single cell . 1 . Given a regular cell complex X , all the cells are initialised with the same colour . 2 . Given the colour ct of cell at iteration t , we compute the colour of cell at the next iteration c t+1 by injectively mapping the multi-sets of colours belonging to the adjacent cells of using a perfect HASH function : ct+1 = HASH c t , c t B ( ) , c t C ( ) , c t # ( ) , c t `` ( ) . 3 . The algorithm stops when a stable colouring is reached . Two cell complexes are considered non-isomorphic if their colour histograms are different . Otherwise , the test is inconclusive . First , we state the following theorem from Bodnar et al . [ 8 ] involving SWL and simplicial complexes . This theorem shows that on simplicial complexes , certain adjacencies can be pruned without affecting the non-isomorphic SCs that can be distinguished . This has important computational implications . Theorem 6 . SWL without coboundary and lower-adjacencies has the same expressive power in distinguishing non-isomorphic simplicial complexes as SWL with the complete set of adjacencies . It is not immediately clear whether an equivalent theorem would also hold for cell complexes . This is because cells , unlike simplices , can have widely different shapes and , as described above , the adjacencies between them take more complicated forms . Nevertheless , we show that a positive result can be obtained . Theorem 7 . CWL without coboundary and lower-adjacencies has the same expressive power in distinguishing non-isomorphic cell complexes as CWL with the complete set of adjacencies . We note this does not mean that the removed adjacencies are completely redundant in practice . Even if they are not needed from a ( theoretical ) colour refinment perspective , they might still include important inductive biases that make them suitable for certain tasks . We are now interested in examining various procedures for mapping , or “ lifting ” , graphs into the space of regular cell complexes . Such a procedure can be used to test the isomorphism of two graphs by performing colour refinement on the cell complexes they are mapped to . The hope is that CWL applied to these cell complexes is more powerful than WL applied to the initial graphs . We will later show that for a wide range of transformations , this is indeed the case . We start by rigorously defining what we mean by a “ lifting ” . Definition 8 . A cellular lifting map is a function f : G ! X from the space of graphs G to the space of regular cell complexes X with the property that two graphs G1 , G2 are isomorphic iff the cell complexes f ( G1 ) , f ( G2 ) are isomorphic . This property ensures that testing the isomorphism of the two cell complexes is equivalent to testing the isomorphism in the input graphs . This would not be the case if two non-isomoprhic graphs were mapped to the same cell complex . Example 9 . It can be verified that the function mapping each graph to its clique complex ( i.e . every ( k + 1 ) -clique in the graph becomes a k-simplex ) is a cellular lifting map . The clique complex lifting map from Example 9 has been used by Bodnar et al . [ 8 ] to show that SWL is strictly more powerful than WL . We restate this result : Theorem 10 . SWL with clique complex lifting is strictly more powerful than WL . A natural question is what other lifting transformations make CWL strictly more powerful than WL ? We first describe a space of lifting transformations that make CWL at least as powerful as WL . Definition 11 . A lifting map is skeleton-preserving if for any graph G , the 1-skeleton of f ( G ) and G are isomorphic as ( multi ) graphs . Intuitively , skeleton-preserving liftings ensure that the additional structure added by the lifting map comes from attaching cells of dimension at least two to the graph . These mappings keep the 0-cells and 1-cells intact and are , therefore , restricted from making modifications to the input graph structure . An important remark is that for simplicial complexes , attaching simplices based on cliques present in the graph is the only possible skeleton preserving transformation . Once again , this illustrates the limitations of simplicial complexes for adding useful higher-dimensional structures to the graph . Example 12 . The function from Example 9 is also skeleton-preserving because the 1-skeleton of the clique complex of a graph is trivially isomorphic to the graph . A lifting function mapping each graph to a multi-graph where each edge is doubled by a parallel edge is not skeleton-preserving ( Figure 5 ) . We now show that all the maps in the skeleton-preserving class have the following desirable property : Theorem 13 . Let f be a skeleton-preserving lifting map . Then CWL ( f ) ( i.e . CWL using lifting f ) is at least as powerful as WL in distinguishing non-isomorphic graphs . To prove that some of these make CWL strictly more powerful than WL , it is sufficient to find a pair of graphs that can not be distinguished by WL , but can be distinguished by CWL . The following result gives examples of such maps . Definition 14 . Let k-CL , k-IC , k-C be the lifting maps attaching cells to all the cliques , induced cycles and simple cycles , respectively , of size at most k. Corollary 15 . For all k 3 , CWL ( k-CL ) , CWL ( k-IC ) and CWL ( k-C ) are strictly more powerful than WL . We note that this is not a complete list . For instance , the result can also be extended to combinations of the above or other transformations . We can also relate CWL to the higher-order 3-WL test . Theorem 16 . There exists a pair of graphs indistinguishable by 3-WL but distinguishable by CWL ( kCL ) with k 4 , CWL ( k-IC ) with k 4 and CWL ( k-C ) with k 8 . Finally , we conclude this section by showing how CWL can achieve a superior expressive power compared to SWL . This result is proven by Corollary 31 in the Appendix . Theorem 17 . Let k-CL [ k-IC and k-CL [ k-C denote combined liftings attaching cells to the union of the specified substructures . CWL ( k1-CL [ k2-IC ) and CWL ( k1-CL [ k2-C ) are strictly more powerful than SWL ( k1-CL ) for all k2 5 .

Summary: In this work, the authors propose a new member of graph neural networks, called CWN, that considers the higher-order structure of graphs. The proposed model leverages the topological properties of cell complexes and establishes a novel hierarchical message passing procedure. The authors demonstrate that the proposed CWN is more powerful than the WL test and can be comparable to the 3-WL test. The proposed model achieves encouraging performance on graphs with complicated structures, e.g., molecules, while it requires a preprocessing step with the tolerable time cost.