On the Equivalence between Positional Node Embeddings and Structural Graph Representations

1 INTRODUCTION . The theory of structural graph representations is a recently emerging field . It creates a link between relational learning and invariant theory . Interestingly , or rather unfortunately , there is no unified theory connecting node embeddings —low-rank matrix approximations , factor analysis , latent semantic analysis , etc.— with structural graph representations . Instead , conflicting interpretations have manifested over the last few years , that further confound practitioners and researchers alike . For instance , consider the direction , word embeddings→ structural representations , where the structural equivalence between men→ king and women→ queen is described as being obtained by just adding or subtracting their node embeddings ( positions in the embedding space ) ( Arora et al. , 2016 ; Mikolov et al. , 2013 ) . Hence , can all ( positional ) node embeddings provide structural relationships akin to word analogies ? We provide a visual example in Appendix ( Section 7 ) using the food web of Figure 1 . In the opposite direction , structural representations→ node embeddings , graph neural networks ( GNNs ) are often optimized to predict edges even though their structural node representations are provably incapable of performing the task . For instance , the node representations of the lynx and the orca in Figure 1 are indistinguishable due to an isomorphic equivalence between the nodes , making any edge prediction task that distinguishes the edges of lynx and orca a seemly futile exercise ( see Appendix ( Section 7 ) for more details ) . Hence , are structural representations in general —and GNNs in particular— fundamentally incapable of performing link ( dyadic ) and multi-ary ( polyadic ) predictions tasks ? GNNs , however , can perform node classification tasks , which is a task not associated with positional node embeddings ( see Appendix ( Section 7 ) for a concrete visual interpretation of the differences between positional node embeddings and structural representations over node classification and link prediction tasks ) . Confirmation bias has seemingly appeared to thwart recent efforts to bring node embeddings and structural representations into a single overarching framework . Preconceived notions of the two being fundamentally different ( see Appendix ( Section 7 ) ) have been reinforced in the existing literature , arguing they belong in different applications : ( Positional ) Node embeddings would find applications in multi-ary relationships such as link prediction , clustering , and natural language processing and knowledge acquisition through word and entity embeddings . Structural representations would find applications in node classification , graph classification , and role discovery . A unified theory is required if we wish to eliminate these artificial boundaries , and better cross-pollinate , node embeddings and structural representations in novel techniques . Contributions : In this work we use invariant theory and axiomatic counterfactuals ( causality ) to develop a unified theoretical framework that clarifies the differences between node embeddings and structural representations and emphasizes their correspondence . More specifically , ( a ) we show that structural representations and node embeddings have the same relationship as distributions and their samples ; ( b ) we prove that all tasks that can be performed by node embeddings can also be performed by structural representations and vice-versa . Moreover , ( c ) we introduce new guidelines to creating and using node embeddings , which we hope will replace the less-than-optimal standard operating procedures used today . Finally , ( d ) we show that the concepts of transductive and inductive learning —commonly used to describe relational methods— are unrelated to node embeddings and structural representations.s 2 PRELIMINARIES . This section introduces some basic definitions , attempting to keep the mathematical jargon in check as much as we can , sometimes even sacrificing generality for clarity . We recommend Bloem-Reddy & Teh ( 2019 ) for a more formal description of some of the definitions in this section . Definition 1 ( Graph ) . We consider either a directed or an undirected attributed graph , denoted by G = ( V , E , X , E ) , where V is a set of n = |V | vertices , E is the set of edges in V × V , with matrix X ∈ Rn×k , k > 0 and 3-mode tensor E ∈ Rn×n×k′ , k′ > 0 representing the node and edge features , respectively . The edge set has an associated adjacency matrix A ∈ { 0 , 1 } n×n . In order to simplify notation , we will compress E and A into a single tensor A ∈ Rn×n× ( k′+1 ) . When explicit vertex and edge features and weights are unavailable , we will consider X = 11T and A = A , where 1 is a n × 1 vector of ones . We will abuse notation and denote the graph as G = ( A , X ) . Without loss of generality , we define number the nodes in V = { 1 , . . . , n } following the same ordering as the adjacency tensor A and the rows in X . We denote ~S and a vector of the elements in S ∈ P ? ( V ) sorted in ascending order , where P ? ( V ) is the power set of V without the empty set . One of the most important operators in our mathematical toolkit will be that of a permutation action , orbits , G-invariance , and G-equivariance : Definition 2 ( Permutation action π ) . A permutation action π is a function that acts on any vector , matrix , or tensor defined over the nodes V , e.g. , ( Zi ) i∈V , and outputs an equivalent vector , matrix , or tensor with the order of the nodes permuted . We define Πn as the set of all n ! such permutation actions . Definition 3 ( Orbits ) . An orbit is the result of a group action Πn acting on elements of a group correspond to bijective transformations of the space that preserve some structure of the space . The orbit of an element is the set of equivalent elements under action Πn , i.e. , Πn ( x ) = { π ( x ) | π ∈ Πn } . Definition 4 ( G-equivariant and G-invariant functions ) . Let Σn be the set of all possible attributed graphs G of size n ≥ 1 . More formally , Σn is the set of all tuples ( A , X ) with adjacency tensors A and corresponding node attributes X for n nodes . A function g : Σn → Rn×· is G-equivariant w.r.t . valid permutations of the nodes V , whenever any permutation action π ∈ Πn in the Σn space associated with the same permutation action of the nodes in the Rn×· space . A function g : Σn → R· is G-invariant whenever it is invariant to any permutation action π ∈ Πn in Σn . Definition 5 ( Graph orbits & graph isomorphism ) . Let G = ( A , X ) be a graph with n nodes , and let Πn ( G ) = { ( A′ , X ′ ) : ( A′ , X ′ ) = ( π ( A ) , π ( X ) ) , ∀π ∈ Πn } be the set of all equivalent ( isomorphic ) graphs under the permutation action π . Two graphs G1 = ( A1 , X1 ) and G2 = ( A2 , X2 ) are said isomorphic iff Πn ( G1 ) = Πn ( G2 ) . Definition 6 ( Node orbits & node isomorphism ) . The equivalence classes of the vertices of a graph G under the action of automorphisms are called vertex orbits . If two nodes are in the same node orbit , we say that they are isomorphic . In Figure 1 , the lynx and the orca are isomorphic ( they have the same node orbits ) . We now generalize Definition 6 to subsets of nodes S ∈ P ? ( V ) , where P ? ( V ) is the power set of V without the empty set . Definition 7 ( Vertex subset orbits and joint isomorphism ) . The equivalence classes of k-sized subsets of vertices S ∈ P ? ( V ) of a graph G under the action of automorphisms between the subsets are called vertex subset orbits , k ≥ 2 . If two proper subsets S1 , S2 ∈ P ? ( V ) \V are in the same vertex subset orbit , we say they are jointly isomorphic . Next we define the relationship between structural representations and node embeddings . 3 A UNIFYING THEORETICAL FRAMEWORK OF NODE EMBEDDINGS AND STRUCTURAL REPRESENTATIONS . How are node embeddings and structural representations related ? This section starts with a familiar , albeit naı̈ve , view of the differences between node embeddings and structural representations , preparing the groundwork to later broadening and rectifying these into precise model-free mathematical statements using invariant theory . This broadening is needed since model-free node embeddings need not be related to node closeness in the graph ( or to lower dimensional projections for that matter ) , as it is impossible to have a model-free definition of closeness . A familiar interpretation of node embeddings : Node embeddings are often seen as a lowerdimensional projection of the rows and columns of the adjacency matrix A from Rn to Rd , d < n , that preserves relative positions of the nodes in a graph ( Graham & Winkler , 1985 ; Linial et al. , 1995 ) ; for instance , in Figure 1 , the lynx and the coyote would have close node embeddings , while the node embeddings of the lynx and the orca would be significantly different . Node embeddings are often seen as encoding the fact that the lynx and the coyote are part of a tightly-knit community , while the lynx and orca belong to distinct communities . The structural representation of a node , on the other hand , shows which nodes have similar roles ( structural similarities ) on a graph ; for instance , the lynx and the orca in Figure 1 must have the same structural representation , while the lynx and the coyote likely have different structural representations . The lynx , like the orca , is a top predator in the food web while the coyote is not a top predator . The incompatibility of the familiar interpretation with the theory of structural graph representations : The above interpretation of node embeddings must be tied to a model that defines closeness . Structural graph representations are model-free . Hence , we need a model-free definition of node embedding to connect it with structural representations . Unfortunately , one can not define closeness without a model . Hence , in the remainder of this paper , we abandon this familiar interpretation in favor of a model-free definition . Roadmap : In what follows , we restate some existing model-free definitions of structural graph representation and introduce some new ones . Then , we introduce a model-free definition of node embeddings . We will retain the terminology node embedding for historical reasons , even though our node embedding need not be an embedding ( a projection into lower dimensional space ) . 3.1 ON STRUCTURAL REPRESENTATIONS . In what follows we use the terms link and edge interchangeably . Proofs are left to the Appendix . Definition 8 ( Structural node representations ) . The structural representation of node v ∈ V in a graph G = ( A , X ) is the G-invariant representation Γ ( v , A , X ) , where Γ : V × Σn → Rd , d ≥ 1 , such that ∀u ∈ V , Γ ( u , A , X ) = Γ ( π ( u ) , π ( A ) , π ( X ) ) for all permutation actions ∀π ∈ Πn . Moreover , for any two isomorphic nodes u , v ∈ V , Γ ( u , A , X ) = Γ ( v , A , X ) . Definition 9 ( Most-expressive structural node representations Γ ? ) . A structural representation of a node v ∈ V , Γ ? ( v , A , X ) , is most-expressive iff , ∀u ∈ V , there exists a bijective measurable map between Γ ? ( u , A , X ) and the orbit of node u in G = ( A , X ) ( Definition 7 ) . Trivially , by Definitions 6 and 9 , two graphs G1 = ( A1 , X1 ) and G2 = ( A2 , X2 ) are isomorphic ( Definition 5 ) iff the most-expressive structural node representations ( Γ ? ( u , A1 , X1 ) ) u∈V and ( Γ ? ( v , A2 , X2 ) ) v∈V are the same up to a valid permutation π ∈ Πn of the nodes . In what follows P ? is the power set excluding the empty set . We now describe the relationship between structural node representations and node isomorphism . Lemma 1 . Two nodes v , u ∈ V , have the same most-expressive structural representations Γ ? ( v , A , X ) = Γ ? ( u , A , X ) iff u and v are isomorphic nodes in G = ( A , X ) . Having described representation of nodes , we now generalize these representations to subsets of V . Definition 10 ( Joint structural representation Γ ) . A joint structural representation of a graph with node set V is defined as Γ : P ? ( V ) ×Σn → Rd , d ≥ 1 . Furthermore , Γ is G-invariant over all node subsets , i.e. , ∀S ∈ P ? ( V ) and ∀ ( A , X ) ∈ Σn , it must be that Γ ( ~S , A , X ) = Γ ( π ( ~S ) , π ( A ) , π ( X ) ) for all permutation actions ∀π ∈ Πn . Moreover , for any two isomorphic subsets S , S′ ∈ P ? ( V ) , Γ ( ~S , A , X ) = Γ ( ~S′ , A , X ) . We now mirror Definition 9 in our generalization of Γ : Definition 11 ( Most-expressive joint structural representations Γ ? ) . A structural representation Γ ? ( ~S , A , X ) of a non-empty subset S ∈ P ? ( V ) , of a graph ( A , X ) ∈ Σn , is most-expressive iff , there exists a bijective measurable map between Γ ? ( ~U , A , X ) and the orbit of U inG ( Definition 7 ) , ∀U ∈ P ? ( V ) and ∀ ( A , X ) ∈ Σn . Note , however , the failure to represent the link ( lynx , coyote ) in Figure 1 using the mostexpressive node representations of the lynx and the coyote . A link needs to be represented by a joint representation of two nodes . For instance , we can easily verify from Definition 11 that Γ ? ( ( lynx , coyote ) , A , X ) 6= Γ ? ( ( orca , coyote ) , A , X ) , even though Γ ? ( lynx , A , X ) = Γ ? ( orca , A , X ) . Next we show that joint prediction tasks only require joint structural representations . But first we need to show that any causal model defined through axiomatic counterfactuals ( Galles & Pearl , 1998 ) can be equivalently defined through noise outsourcing ( Austin , 2008 ) , a straightforward result that we were unable to find in the literature . Lemma 2 ( Causal modeling through noise outsourcing ) . Definition 1 of Galles & Pearl ( 1998 ) gives a causal model as a triplet M = 〈U , V ′ , F 〉 , where U is a set of exogenous variables , V ′ is a set of endogenous variables , and F is a set of functions , such that in the causal model , v′i = f ( ~pai , u ) is the realization of random variable V ′ i ∈ V ′ and a sequence of random variables ~PAi with PAi ⊆ V \V ′i as the endogenous variable parents of variable V ′i as given by a directed acyclic graph . Then , there exists a pure random noise and a set of ( measurable ) functions { gu } u∈U such that for Vi ∈ V ′ , V ′i can be equivalently defined as v′i a.s. = f ( ~pai , gu ( u ) ) , where u has joint distribution ( u ) ∀u∈U a.s. = g′ ( ) for some Borel measurable function g′ and a random variable ∼ Uniform ( 0 , 1 ) . The latter defines M via noise outsourcing ( Austin , 2008 ) . The proof of Lemma 2 is given in the Appendix . Lemma 2 defines the causal model entirely via endogenous variables , deterministic functions , and a pure random noise random variable . We are now ready for our theorem showing that joint prediction tasks only require ( most-expressive ) joint structural representations . Theorem 1 . Let S ⊆ P ? ( V ) be a set of non-empty subsets of the vertices V . Let Y ( S , A , X ) = ( Y ( ~S , A , X ) ) S∈S be a sequence of random variables defined over the sets S ∈ S of a graph G = ( A , X ) , that are invariant to the ordering of ~S , S ∈ S , such that Y ( ~S1 , A , X ) d = Y ( ~S2 , A , X ) for any two jointly isomorphic subsets S1 , S2 ∈ S ( Definition 7 ) , where d = means equality in their marginal distributions . Then , there exists a measurable function ϕ such that , Y ( S , A , X ) a.s.= ( ϕ ( Γ ? ( ~S , A , X ) , S ) ) S∈S , where S is the random noise that defines the exogenous variables of Lemma 2 , with joint distribution p ( ( S′ ) ∀S′∈S ) independent of A and X . Theorem 1 extends Theorem 12 of Bloem-Reddy & Teh ( 2019 ) in multiple ways : ( a ) to all subsets of nodes , S ∈ P ? ( V ) , ( b ) to include causal language , and , most importantly , ( c ) to showing that any prediction task that can be defined over S , requires only a most-expressive joint structural representation over S. For instance , any task with |S| = 2 predicting a missing link ( u , v ) on a graph G = ( A , X ) , requires only the most-expressive structural representation Γ ? ( ( u , v ) , A , X ) . Note that , in order to predict directed edges , we must use Γ ? ( ( u , v ) , A , X ) to also predict the edge ’ s direction : → , ← , ↔ , but a detailed procedure showing how to predict directed edges is relegated to a future journal version of this paper . Theorem 1 also includes node tasks for |S| = 1 , hyperedge tasks for 2 < |S| < n , and graph-wide tasks for S = V . Remark 1 ( GNNs and link prediction ) . Even though structural node representations of GNNs are not able to predict edges , GNNs are often still optimized to predict edges ( e.g. , ( Hamilton et al. , 2017a ; Xu et al. , 2018 ) ) in transfer learning tasks . This optimization objective guarantees that any small topological differences between two nearly-isomorphic nodes without an edge will be amplified , while differences between nodes with an edge will be minimized . Hence , the topological differences in a close-knit community will be minimized in the representation . This procedure is an interesting way to introduce homophily in structural representations and should work well for node classification tasks in homophilic networks ( where node classes tend to be clustered ) . We now turn our attention to node embeddings and their relationship with joint representations .

Summary: This paper tries to clarify some confounding concepts around graph/node representation learning by providing a unifying theoretical framework based on invariant theory. The authors define node embedding and structural graph representation based on a minimal number of requirements, which is, in turn, used to derive some useful properties and unify two seemly different concepts. Based on these, a new graph neural network framework is proposed. The experiments on multiple tasks with multiple datasets validate the main claim that a single node embedding is insufficient to capture the structural representation of a graph.