Weisfeiler and Lehman Go Cellular: CW Networks

1 Introduction . The operations performed by message passing Graph Neural Networks ( GNNs ) emulate the structure of the input graph . While this property has clear computational advantages , it brings with it a series of fundamental limitations . As observed by Xu et al . [ 74 ] and Morris et al . [ 55 ] the local neighbourhood aggregations used by GNNs are at most as powerful as the Weisfeiler-Lehman ( WL ) test [ 71 ] in distinguishing non-isomorphic graphs . Therefore , GNNs fail to detect certain higer-order meso-scale structures such as cliques or ( induced ) cycles [ 2 , 15 ] , which are particularly important in applications dealing with social and biological networks or molecular graphs . At the same time , many such layers have to be stacked to make long-range interactions in the graph possible . Besides the computational burden this incurs , deep GNNs typically come with additional problems such as over-smoothing [ 51 ] and over-squashing [ 1 ] of the node representations . ⇤Authors contributed equally . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . To address these problems , we propose a novel message passing procedure based on ( regular ) cell complexes , also known as CW complexes2 , topological objects that form the building block of algebraic topology [ 38 ] . When paired with a theoretically-justified “ lifting ” transformation augmenting the graph with higher-dimensional constructs called “ cells ” , our method results in a multi-dimensional and hierarchical message passing procedure over the input graph . Our approach generalises and subsumes the recently proposed Message Passing Simplicial Networks ( MPSNs ) [ 8 ] , which operate on simplicial complexes ( SCs ) , topological generalisations of graphs . However , SCs have a rigid combinatorial structure that significantly limits the range of lifting transformations one could use to meaningfully modulate the message passing procedure . In contrast , we show that cell complexes , which in turn generalise simplicial complexes and come with additional flexibility , allow one to construct new and better ways of decoupling the input and computational graphs . Main Contributions To summarise , we propose a message passing scheme operating on regular cell complexes . We call this family of models CW Networks ( CWNs ) and study their expressive power using a cellular version of the WL test . We show that for an entire class of “ lifting ” transformations CWNs are at least as powerful as the WL test . Furthermore , we prove that for some of the maps in this class , CWNs can be strictly more powerful than WL , Simplicial WL ( SWL ) and also not less powerful than 3-WL . We also express the fundamental symmetries of these models and show how they can be seen as generalised convolutional operators on cell complexes . Experimentally , we focus our attention on a particular “ lifting ” map based on induced cycles . When applied to molecular graphs , it leads to an intuitive hierarchical message passing procedure involving the atoms , the bonds between them and the chemical rings of the molecules . We demonstrate that this provably powerful approach obtains state-of-the-art results on popular large-scale molecular graph datasets and other related tasks . To the best of our knowledge , this is the first work proposing a cell complex representation for molecules . Our code is available at https : //github.com/twitter-research/cwn . 2 Background . Definition 1 ( Hansen and Ghrist [ 36 ] ) . A regular cell complex ( Figure 1 ) is a topological space X together with a partition { X } 2PX of subspaces X of X called cells , and such that 1 . For each x 2 X there exists an open neighbordhood of x that intersects finitely many cells . 2 . For all , ⌧ we have that X⌧ \X 6= ; iff X⌧ ✓ X , where X is the closure of a cell . 3 . Every cell is homeomorphic to Rn for some n. 4 . ( Regularity ) For every 2 PX there is a homeomorphism of a closed ball in Rn to X such that the restriction of to the interior of the ball is a homeomorphism onto X . Figure 1 : A cell complex X and the corresponding homeomorphisms to the closed balls for three cells of different dimensions in the complex . We note that by condition ( 2 ) the indexing set PX has a poset structure ⌧ , X⌧ ✓ X , while condition ( 4 ) guarantees that this poset structure encodes all the topological information about X . Thus , we can identify a regular cell complex X with this poset , called face poset of X . We also use ⌧ < for the strict version of this partial order . Intuitively , one constructs a cell complex through a hierarchical gluing procedure . One starts with a set of vertices ( 0-cells ) . Then edges ( 1-cells ) are attached to these by gluing the endpoints of closed line segments to them . We have now only described a ( multi ) graph . However , one can generalise this even further by taking a two-dimensional closed disk and glue its boundary ( i.e . a circle ) to any simple cycle in the ( multi ) graph previously built as in Figure 2 . While we are generally not concerned with dimensions above two , this can be further generalised by gluing the boundary of n-dimensional balls to certain ( n 1 ) cells in the complex . Consider the examples in Figure 3 . The shown sphere is a cell complex obtained from two 0-cells ( i.e . vertices ) , to which two 1-cells ( i.e . edges ) , which form the equator , were attached . The boundary 2We use these terms interchangeably . For the latter , the C stands for “ closure-finite ” , and the W for “ weak ” topology . The term was coined by Whitehead [ 72 ] . of two 2-dimensional disks ( i.e . the two hemispheres ) were glued to the equator to form a sphere . The second example is a tetrahedron with empty interior . It is a particular type of cell complex called a simplicial complex ( SC ) . The only 2-cells it allows are triangle-shaped . More generally , the n-dimensional cells of SCs are n-simplices , which makes them slightly more rigid structures . Definition 2 . The k-skeleton of a cell complex X , denoted X ( k ) , is the subcomplex of X consisting of cells of dimension at most k. This definition is useful for referring for certain parts of the complex . For instance , X ( 0 ) contains the vertices in the complex , while X ( 1 ) contains the vertices and the edges ( i.e . the underlying graph ) . The combinatorial structure of the complex can be more compactly described by an incidence relation we call the boundary relation , whose reflexive and transitive closure gives the partial order defined above . The boundary relation describes what cells are on the boundary of other cells . For instance , the edges of the sphere in Figure 3 are on the boundary of the 2-cells forming the two hemispheres . Definition 3 . We have the boundary relation ⌧ iff < ⌧ and there is no cell such that < < ⌧ . We can use this to define the four types of ( local ) adjacencies present in cell complexes . These adjacencies will be the fundamental building block of our message passing procedure . To explain these in more familiar terms , for each adjacency , we exemplify how it shows up in graphs . Definition 4 ( Cell complex adjacencies ) . For a cell complex X and a cell 2 PX , we define : 1 . The boundary adjacent cells B ( ) = { ⌧ | ⌧ } . These are the lower-dimensional cells on the boundary of . For instance , the boundary cells of an edge are its vertices . 2 . The co-boundary adjacent cell C ( ) = { ⌧ | ⌧ } . These are the higher-dimensional cells with on their boundary . For instance , the co-boundary cells of a vertex are the edges it is part of . 3 . The lower adjacent cells N # ( ) = { ⌧ | 9 such that and ⌧ } . These are the cells of the same dimension as that share a lower dimensional cell on their boundary . The line graph adjacencies between the edges are a classic example of this . 4 . The upper adjacent cells N '' ( ) = { ⌧ | 9 such that and ⌧ } . These are the cells of the same dimension as that are on the boundary of the same higher-dimensional cell as . The typical graph adjacencies between vertices are the canonical example here . 3 Cellular Weisfeiler Lehman . Overview The results in this section show how one can transform graphs into higher-dimensional cell complexes in such a way that performing colour refinement on the resulting cell complexes makes it easier to test their isomorphism . The message passing model from Section 4 will take advantage of these theoretical results . All proofs can be found in Appendix A . Definition 5 . Let c be a colouring of the cells in a complex X with c denoting the colour assigned to cell 2 PX . Define B ( , ⌧ ) : = B ( ) \B ( ⌧ ) and C ( , ⌧ ) : = C ( ) \C ( ⌧ ) . We define the following multi-sets of colours : 1 . The colours of the boundary cells of : cB ( ) = { { c⌧ | ⌧ 2 B ( ) } } . 2 . The colours of the co-boundary cells of : cC ( ) = { { c⌧ | ⌧ 2 C ( ) } } . 3 . The lower adjacent colours of : c # ( ) = { { ( c⌧ , c ) | ⌧ 2 N # ( ) and 2 B ( , ⌧ ) } } . 4 . The upper adjacent colours of : c '' ( ) = { { ( c⌧ , c ) | ⌧ 2 N '' ( ) and 2 C ( , ⌧ ) } } . Note that unlike in graphs and simplicial complexes , the sets B ( , ⌧ ) and C ( , ⌧ ) can have more than one element . For instance , two ( closed ) 2-cells might intersect in more than one edge ( e.g . the two hemispheres in Figure 3 ) , and conversely , two edges might be on the boundary of the same two 2-cells . This illustrates the more flexible combinatorial structure of cell complexes . Cellular WL ( CWL ) We consider CWL , a colour refinement scheme for cell complexes that generalises the Simplicial WL [ 8 ] and WL [ 71 ] tests . We use ct to refer to the colour assigned by CWL to cell at iteration t of the algorithm . When the input is a simplicial complex , this recovers the SWL algorithm . A step of the algorithm is graphically depicted in Figure 4 for a single cell . 1 . Given a regular cell complex X , all the cells are initialised with the same colour . 2 . Given the colour ct of cell at iteration t , we compute the colour of cell at the next iteration c t+1 by injectively mapping the multi-sets of colours belonging to the adjacent cells of using a perfect HASH function : ct+1 = HASH c t , c t B ( ) , c t C ( ) , c t # ( ) , c t `` ( ) . 3 . The algorithm stops when a stable colouring is reached . Two cell complexes are considered non-isomorphic if their colour histograms are different . Otherwise , the test is inconclusive . First , we state the following theorem from Bodnar et al . [ 8 ] involving SWL and simplicial complexes . This theorem shows that on simplicial complexes , certain adjacencies can be pruned without affecting the non-isomorphic SCs that can be distinguished . This has important computational implications . Theorem 6 . SWL without coboundary and lower-adjacencies has the same expressive power in distinguishing non-isomorphic simplicial complexes as SWL with the complete set of adjacencies . It is not immediately clear whether an equivalent theorem would also hold for cell complexes . This is because cells , unlike simplices , can have widely different shapes and , as described above , the adjacencies between them take more complicated forms . Nevertheless , we show that a positive result can be obtained . Theorem 7 . CWL without coboundary and lower-adjacencies has the same expressive power in distinguishing non-isomorphic cell complexes as CWL with the complete set of adjacencies . We note this does not mean that the removed adjacencies are completely redundant in practice . Even if they are not needed from a ( theoretical ) colour refinment perspective , they might still include important inductive biases that make them suitable for certain tasks . We are now interested in examining various procedures for mapping , or “ lifting ” , graphs into the space of regular cell complexes . Such a procedure can be used to test the isomorphism of two graphs by performing colour refinement on the cell complexes they are mapped to . The hope is that CWL applied to these cell complexes is more powerful than WL applied to the initial graphs . We will later show that for a wide range of transformations , this is indeed the case . We start by rigorously defining what we mean by a “ lifting ” . Definition 8 . A cellular lifting map is a function f : G ! X from the space of graphs G to the space of regular cell complexes X with the property that two graphs G1 , G2 are isomorphic iff the cell complexes f ( G1 ) , f ( G2 ) are isomorphic . This property ensures that testing the isomorphism of the two cell complexes is equivalent to testing the isomorphism in the input graphs . This would not be the case if two non-isomoprhic graphs were mapped to the same cell complex . Example 9 . It can be verified that the function mapping each graph to its clique complex ( i.e . every ( k + 1 ) -clique in the graph becomes a k-simplex ) is a cellular lifting map . The clique complex lifting map from Example 9 has been used by Bodnar et al . [ 8 ] to show that SWL is strictly more powerful than WL . We restate this result : Theorem 10 . SWL with clique complex lifting is strictly more powerful than WL . A natural question is what other lifting transformations make CWL strictly more powerful than WL ? We first describe a space of lifting transformations that make CWL at least as powerful as WL . Definition 11 . A lifting map is skeleton-preserving if for any graph G , the 1-skeleton of f ( G ) and G are isomorphic as ( multi ) graphs . Intuitively , skeleton-preserving liftings ensure that the additional structure added by the lifting map comes from attaching cells of dimension at least two to the graph . These mappings keep the 0-cells and 1-cells intact and are , therefore , restricted from making modifications to the input graph structure . An important remark is that for simplicial complexes , attaching simplices based on cliques present in the graph is the only possible skeleton preserving transformation . Once again , this illustrates the limitations of simplicial complexes for adding useful higher-dimensional structures to the graph . Example 12 . The function from Example 9 is also skeleton-preserving because the 1-skeleton of the clique complex of a graph is trivially isomorphic to the graph . A lifting function mapping each graph to a multi-graph where each edge is doubled by a parallel edge is not skeleton-preserving ( Figure 5 ) . We now show that all the maps in the skeleton-preserving class have the following desirable property : Theorem 13 . Let f be a skeleton-preserving lifting map . Then CWL ( f ) ( i.e . CWL using lifting f ) is at least as powerful as WL in distinguishing non-isomorphic graphs . To prove that some of these make CWL strictly more powerful than WL , it is sufficient to find a pair of graphs that can not be distinguished by WL , but can be distinguished by CWL . The following result gives examples of such maps . Definition 14 . Let k-CL , k-IC , k-C be the lifting maps attaching cells to all the cliques , induced cycles and simple cycles , respectively , of size at most k. Corollary 15 . For all k 3 , CWL ( k-CL ) , CWL ( k-IC ) and CWL ( k-C ) are strictly more powerful than WL . We note that this is not a complete list . For instance , the result can also be extended to combinations of the above or other transformations . We can also relate CWL to the higher-order 3-WL test . Theorem 16 . There exists a pair of graphs indistinguishable by 3-WL but distinguishable by CWL ( kCL ) with k 4 , CWL ( k-IC ) with k 4 and CWL ( k-C ) with k 8 . Finally , we conclude this section by showing how CWL can achieve a superior expressive power compared to SWL . This result is proven by Corollary 31 in the Appendix . Theorem 17 . Let k-CL [ k-IC and k-CL [ k-C denote combined liftings attaching cells to the union of the specified substructures . CWL ( k1-CL [ k2-IC ) and CWL ( k1-CL [ k2-C ) are strictly more powerful than SWL ( k1-CL ) for all k2 5 .

Summary: Recently, there have been several attempts to put higher-order structures on graphs and perform message passing over these structures. One example is the Message Passing Simplicial Networks proposed in e.g, ref[4]. The present paper extends the higher-order structures from simplicial complexes (SCs) to regular cell complexes (also known as CW complexes), or more precisely, generalizes the Simplicial WL of ref[4] to a Cellular WL (CWL). Both theoretical understanding, and experimental studies of CWL are presented. Specific contributions include:   (1) To obtain a CW complex from a graph, the paper introduces a ``cellular lifting map", and shows a skeleton-preserving lifting map gives rise to a CWL that is at least as powerful as standard WL (Weisfeiler-Lehman) in graph isomorphism testing. It also gives examples of specific natural lifting maps (which includes the previous SWL in [4] as a special case) that is at least as powerful as the so-called 3-WL.   (2) It describes and implements a CW network using a lifting map that attaches 2-cells to all induced cycles in the input graphs. This architecture (CWN) makes sense especially for molecular graphs applications (in capturing both the covalent bonds and chemical rings).   (3) It presents various experimental results on synthetic and real datasets, which show that a specific instantiation of CWN, called CIN, can both differentiate certain hard cases for graph isomorphism testing (e.g, 4-regular graphs), comparable or better performance on TUDatasets, and better performance in molecular benchmark datasets.