Coloring graph neural networks for node disambiguation

1 INTRODUCTION . Learning good representations is seen by many machine learning researchers as the main reason behind the tremendous successes of the field in recent years ( Bengio et al. , 2013 ) . In image analysis ( Krizhevsky et al. , 2012 ) , natural language processing ( Vaswani et al. , 2017 ) or reinforcement learning ( Mnih et al. , 2015 ) , groundbreaking results rely on efficient and flexible deep learning architectures that are capable of transforming a complex input into a simple vector while retaining most of its valuable features . The universal approximation theorem ( Cybenko , 1989 ; Hornik et al. , 1989 ; Hornik , 1991 ; Pinkus , 1999 ) provides a theoretical framework to analyze the expressive power of such architectures by proving that , under mild hypotheses , multi-layer perceptrons ( MLPs ) can uniformly approximate any continuous function on a compact set . This result provided a first theoretical justification of the strong approximation capabilities of neural networks , and was the starting point of more refined analyses providing valuable insights into the generalization capabilities of these architectures ( Baum and Haussler , 1989 ; Geman et al. , 1992 ; Saxe et al. , 2014 ; Bartlett et al. , 2018 ) . Despite a large literature and state-of-the-art performance on benchmark graph classification datasets , graph neural networks yet lack a similar theoretical foundation ( Xu et al. , 2019 ) . Universality for these architectures is either hinted at via equivalence with approximate graph isomorphism tests ( k-WL tests in Xu et al . 2019 ; Maron et al . 2019a ) , or proved under restrictive assumptions ( finite node attribute space in Murphy et al . 2019 ) . In this paper , we introduce Colored Local Iterative Procedure1 ( CLIP ) , which tackles the limitations of current Message Passing Neural Networks ( MPNNs ) by showing , both theoretically and experimentally , that adding a simple coloring scheme can improve the flexibility and power of these graph representations . More specifically , our contributions are : 1 ) we provide a precise mathematical definition for universal graph representations , 2 ) we present a general mechanism to design universal neural networks using separability , 3 ) we propose a novel node coloring scheme leading to CLIP , the first provably universal extension of MPNNs , 4 ) we show that CLIP achieves state of the art results on benchmark datasets while significantly outperforming traditional MPNNs as well as recent methods on graph property testing . The rest of the paper is organized as follows : Section 2 gives an overview of the graph representation literature and related works . Section 3 provides a precise definition for universal representations , as well as a generic method to design them using separable neural networks . In Section 4 , we show that most state-of-the-art representations are not sufficiently expressive to be universal . Then , using the analysis of Section 3 , Section 5 provides CLIP , a provably universal extension of MPNNs . Finally , 1Code will be available at https : //github.com/ after the review process . Section 6 shows that CLIP achieves state-of-the-art accuracies on benchmark graph classification taks , as well as outperforming its competitors on graph property testing problems . 2 RELATED WORKS . The first works investigating the use of neural networks for graphs used recurrent neural networks to represent directed acyclic graphs ( Sperduti and Starita , 1997 ; Frasconi et al. , 1998 ) . More generic graph neural networks were later introduced by Gori et al . ( 2005 ) ; Scarselli et al . ( 2009 ) , and may be divided into two categories . 1 ) Spectral methods ( Bruna et al. , 2014 ; Henaff et al. , 2015 ; Defferrard et al. , 2016 ; Kipf and Welling , 2017 ) that perform convolution on the Fourier domain of the graph through the spectral decomposition of the graph Laplacian . 2 ) Message passing neural networks ( Gilmer et al. , 2017 ) , sometimes simply referred to as graph neural networks , that are based on the aggregation of neighborhood information through a local iterative process . This category contains most state-of-the-art graph representation methods such as ( Duvenaud et al. , 2015 ; Grover and Leskovec , 2016 ; Lei et al. , 2017 ; Ying et al. , 2018 ; Verma and Zhang , 2019 ) , DeepWalk ( Perozzi et al. , 2014 ) , graph attention networks ( Velickovic et al. , 2018 ) , graphSAGE ( Hamilton et al. , 2017 ) or GIN ( Xu et al. , 2019 ) . Recently , ( Xu et al. , 2019 ) showed that MPNNs were , at most , as expressive as the WeisfeilerLehman ( WL ) test for graph isomorphism ( Weisfeiler and Lehman , 1968 ) . This suprising result led to several works proposing MPNN extensions to improve their expressivity , and ultimately tend towards universality ( Maron et al. , 2019a ; b ; c ; Murphy et al. , 2019 ; Chen et al. , 2019 ) . However , these graph representations are either as powerful as the k-WL test ( Maron et al. , 2019a ) , or provide universal graph representations under the restrictive assumption of finite node attribute space ( Murphy et al. , 2019 ) . Other recent approaches ( Maron et al. , 2019c ) implies quadratic order of tensors in the size of the considered graphs . Some more powerfull GNNs are studied and benchmarked on real classical datasets and on graph property testing ( Kriege et al. , 2018 ; Murphy et al. , 2019 ; Chen et al. , 2019 ) : a set of problems that classical MPNNs can not handle . Our work thus provides a more general and powerful result of universality , matching the original definition of ( Cybenko , 1989 ) for MLPs . 3 UNIVERSAL REPRESENTATIONS VIA SEPARABILITY . In this section we present the theoretical tools used to design our universal graph representation . More specifically , we show that separable representations are sufficiently flexible to capture all relevant information about a given object , and may be extended into universal representations . 3.1 NOTATIONS AND BASIC ASSUMPTIONS . Let X , Y be two topological spaces , then F ( X , Y ) ( resp . C ( X , Y ) ) denotes the space of all functions ( resp . continuous functions ) from X to Y . Moreover , for any group G acting on a set X , X/G denotes the set of orbits of X under the action of G ( see Appendix B for more details ) . Finally , ‖ · ‖ is a norm on Rd , and Pn is the set of all permutation matrices of size n. In what follows , we assume that all the considered topological spaces are Hausdorff ( see e.g . ( Bourbaki , 1998 ) for an in-depth review ) : each pair of distinct points can be separated by two disjoint open sets . This assumption is rather weak ( e.g . all metric spaces are Hausdorff ) and is verified by most topological spaces commonly encountered in the field of machine learning . 3.2 UNIVERSAL REPRESENTATIONS . Let X be a set of objects ( e.g . vectors , images , graphs , or temporal data ) to be used as input information for a machine learning task ( e.g . classification , regression or clustering ) . In what follows , we denote as vector representation of X a function f : X → Rd that maps each element x ∈ X to a d-dimensional vector f ( x ) ∈ Rd . A standard setting for supervised representation learning is to define a class of vector representations Fd ⊂ F ( X , Rd ) ( e.g . convolutional neural networks for images ) and use the target values ( e.g . image classes ) to learn a good vector representation in light of the supervised learning task ( i.e . one vector representation f ∈ Fd that leads to a good accuracy on the learning task ) . In order to present more general results , we will consider neural network architectures that can output vectors of any size , i.e . F ⊂ ∪d∈N∗F ( X , Rd ) , and will denote Fd = F ∩ F ( X , Rd ) the set of d-dimensional vector representations of F. A natural characteristic to ask from the class F is to be generic enough to approximate any vector representation , a notion that we will denote as universal representation ( Hornik et al. , 1989 ) . Definition 1 . A class of vector representations F ⊂ ∪d∈N∗F ( X , Rd ) is called a universal representation of X if for any compact subset K ⊂ X and d ∈ N∗ , F is uniformly dense in C ( K , Rd ) . In other words , F is a universal representation of a normed space X if and only if , for any continuous function φ : X → Rd , any compact K ⊂ X and any ε > 0 , there exists f ∈ F such that ∀x ∈ K , ‖φ ( x ) − f ( x ) ‖ ≤ ε . ( 1 ) One of the most fundamental theorems of neural network theory states that one hidden layer MLPs are universal representations of the m-dimensional vector space Rm . Theorem 1 ( Pinkus , 1999 , Theorem 3.1 ) . Let ϕ : R→ R be a continuous non polynomial activation function . For any compact K ⊂ Rm and d ∈ N∗ , two layers neural networks with activation ϕ are uniformly dense in the set C ( K , Rd ) . However , for graphs and structured objects , universal representations are hard to obtain due to their complex structure and invariance to a group of transformations ( e.g . permutations of the node labels ) . We show in this paper that a key topological property , separability , may lead to universal representations of those structures . 3.3 SEPARABILITY IS ( ALMOST ) ALL YOU NEED . Loosely speaking , universal representations can approximate any vector-valued function . It is thus natural to require that these representations are expressive enough to separate each pair of dissimilar elements of X . Definition 2 ( Separability ) . A set of functions F ⊂ F ( X , Y ) is said to separate points of X if for every pair of distinct points x and y , there exists f ∈ F such that f ( x ) 6= f ( y ) . For a class of vector representations F ⊂ ∪d∈N∗F ( X , Rd ) , we will say that F is separable if its 1-dimensional representations F1 separates points of X . Separability is rather weak , as we only require the existence of different outputs for every pair of inputs . Unsurprisingly , we now show that it is a necessary condition for universality ( see Appendix A for all the detailed proofs ) . Proposition 1 . Let F be a universal representation of X , then F1 separates points of X . While separability is necessary for universal representations , it is also key to designing neural network architectures that can be extended into universal representations . More specifically , under technical assumptions , separable representations can be composed with a universal representation of Rd ( such as MLPs ) to become universal . Theorem 2 . For all d ≥ 0 , letMd be a universal approximation of Rd . Let F be a class of vector representations of X such that : ( i ) Continuity : every f ∈ F is continuous , ( ii ) Stability by concatenation : for all f , g ∈ F , x 7→ ( f ( x ) , g ( x ) ) ∈ F , ( iii ) Separability : F1 separates points of X . Then { ψ ◦ f : ∃d ≥ 1 s.t . ψ ∈Md , f ∈ F } is a universal representation of X . Stability by concatenation is verified by most neural networks architectures , as illustrated for MLPs in Figure 1 . The proof of Theorem 2 relies on the Stone-Weierstrass theorem ( see e.g . Rudin , 1987 , Theorem 7.32 ) whose assumptions are continuity , separability , and the fact that the class of functions is an algebra . Fortunately , composing a separable and concatenable representation with a universal representation automatically leads to an algebra , and thus the applicability of the StoneWeierstrass theorem and the desired result . A complete derivation is available in Appendix A . Since MLPs are universal representations of Rd , Theorem 2 implies a convenient way to design universal representations of more complex object spaces : create a separable representation and compose it with a simple MLP ( see Figure 2 ) . Corollary 1 . A continuous , concatenable and separable representation of X composed with an MLP is universal . Note that many neural networks of the deep learning literature have this two steps structure , including classical image CNNs such as AlexNet ( Krizhevsky et al. , 2012 ) or Inception ( Szegedy et al. , 2016 ) . In this paper , we use Corollary 1 to design universal graph and neighborhood representations , although the method is much more generic and may be applied to other objects .

Summary: This paper proposes an extension of MPNN which leverages the random color augmentation to improve the representation power of MPNN. Authors also prove that two variants of the proposed method have universal representation power (one is exact and the other holds in expectation) from the separability perspective. Experiments on some small graph benchmark datasets and structural property tests are reported.