Equivariant Heterogeneous Graph Networks

1 INTRODUCTION . Many real-world datasets and problems can be modelled as sets of objects with different relationships between them , and so graphs are a natural choice for representing these problems . Common examples include modelling interactions between users in a social network , properties of molecules , or modelling connections between entities in a knowledge base . Graph neural networks ( GNNs ) have become a popular technique for node and graph-level property predictions . These models have mostly focused on standard homogeneous graphs , wherein all nodes and edges are treated the same , with any differences encoded as feature vectors . However , in practical application settings , data is often complex and multi-typed , necessitating the use of heterogeneous graphs , where nodes and edges can be of different types , with potentially completely different semantics . Typical ways to apply GNNs to heterogeneous networks involve preprocessing techniques such as encoding node and edge types into feature vectors or collapsing heterogeneous networks into homogeneous ones by replacing paths along multiple different edge types with single edges . These techniques reduce the structural information available for any network to learn from and often require domain knowledge and hand-engineered features . In this paper , we design a neural network that can operate directly on entire heterogeneous graphs while fully respecting the independence and relationships between different node and edge types . We model a heterogeneous graph as a collection of node-node adjacency matrices , one for each edge type , and create mappings from each edge type to every other edge type . For example , in a heterogeneous network that includes PUBLICATIONS , AUTHORS , and VENUES , and the relationships between these entities , our model can learn how PUBLICATION-published-at-VENUE relationships may influence AUTHOR-associated-with-VENUE relationships by constructing a linear mapping between their adjacency matrices ; see Fig . 1 . A key property of any neural network that operates on graphs is that they must be invariant or equivariant to permutations of nodes . That is to say , if a graph is represented by an adjacency matrix A ∈ Rn×n , for any permutation matrix π ∈ { 0,1 } n×n , a neural network f ∶ Rn×n → Rn×n must have the property that f ( πAπ⊺ ) = πf ( A ) π⊺ ) ( equivariance ) , or when making graph-level predictions with f ∶ Rn×n → R , we require invariance f ( πAπ⊺ ) = f ( A ) . For heterogeneous graphs , this invariance or equivariance constraint is to permutations within each node type . We identify all linear operations that map one adjacency matrix to another while maintaining permutation equivariance within each separate node type . By combining these operations , we are able to construct maximally expressive linear equivariant layers that can then be stacked together to produce a heterogeneous graph neural network . We create two different architectures , and apply them to two common heterogeneous graph tasks : node classification , and link prediction . Finally , we extend our treatment to the general case of relationships involving hyperedges between more than two node types , providing a general prescription of how to efficiently implement linear layers that act on heterogeneous hypergraphs . 2 RELATED WORK . 2.1 HETEROGENEOUS GRAPH NETWORKS . The ubiquity of complex multi-typed data in real-world problems has caused heterogeneous graph learning to attract a lot of attention in applied settings . Heterogeneous graph networks have been applied to such diverse tasks as text classification ( Linmei et al. , 2019 ) , disease diagnosis ( Wang et al. , 2021 ) , and malicious account detection ( Liu et al. , 2018 ) . The majority of heterogeneous graph learning techniques rely on meta-paths : sequences of different node and edge types ( Sun & Han , 2012 ; Shi et al. , 2017 ) . For example , in a citation network with AUTHORS , PAPERS , AND VENUES , the “ path ” AUTHOR – PUBLICATION – VENUE – PUBLICATION – AUTHOR represents one meta-path between two AUTHORS that have published at the same venue . These meta-paths are usually hand-designed , requiring domain knowledge . Heterogeneous graph learning techniques can be broadly classified into either “ shallow ” embedding models , or “ deep ” neural models ( Dong et al. , 2020 ; Yang et al. , 2020 ) . Shallow methods ( such as Dong et al . ( 2017 ) ; Tang et al . ( 2015 ) ; Fu et al . ( 2017 ) ) aggregate node attributes using techniques such as random walks over different edge types , in order to obtain structure-preserving embeddings for each node , which are then passed on to other machine learning models for downstream tasks . These are limited to transductive settings . Deep methods extend conventional GNNs , but learn parameters or embeddings specific to each node or edge type ; see Wu et al . ( 2020 ) for a survey of homogeneous GNNs . Examples include R-GCN ( Schlichtkrull et al. , 2018 ) which extends GCN by learning edge-specific weight matrices , Heterogeneous Graph Attention Network ( HAN , Wang et al. , 2019 ) and Metapath Aggregated Graph Neural Network ( MAGNN ; Fu et al. , 2020 ) , which extend graph attention to attend over different meta-paths . Some methods , such as Heterogeneous Graph Transformer ( HGT , Hu et al. , 2020 ) and Graph Transformer Network ( GTN , Yun et al. , 2019 ) can automatically discover what metapaths are worth using , but even then they are not able to capture as much information as if they were to directly use the full heterogeneous graph . For two recent surveys of heterogeneous graph representation learning techniques , see Yang et al . ( 2020 ) and Dong et al . ( 2020 ) . Lv et al . ( 2021 ) recently called into question whether most heterogeneous graphs neural networks are able to properly exploit the information provided by node and edge types . They show that under fair comparisons , they are often outperformed by conventional graph neural networks that simply ignore node and edge type information , such as GCN ( Kipf & Welling , 2016 ) and GAT ( Veličković et al. , 2017 ) . 2.2 EQUIVARIANT AND INVARIANT LEARNING . A fundamental property that all graph neural networks share is an equivariance or invariance to node permutations . The general requirement that neural networks must be equivariant and invariant to certain symmetries of their input data has been a very active area of research and has motivated models for a diverse set of data types . Examples include models equivariant to the translational symmetries of images ( LeCun et al. , 1989 ) , the permutation symmetry of sets ( Zaheer et al. , 2017 ) , and symmetry to 3D rotations on a sphere ( Cohen et al. , 2018 ) . As an inductive bias , equivariance of the network to a set of transformations is intuitively equivalent to having seen all such transformations of each training data-point . Since the set of such permutation is exponentially large for graph data , equivariance is essential for any graph neural network . Several works directly seek the set of operations with this property to use them as building blocks in graph neural networks . Of particular relevance is the work of Kondor et al . ( 2018 ) , which introduces permutation equivariant operations that can be applied to tensor representations of graphs , and Maron et al . ( 2018 ) , which characterizes a basis for equivariant linear operations on tensor representations of graphs and hypergraphs . Our work extends this work to the more general case of heterogeneous graphs , where we are presented with a set of different edge types . Furthermore , while they characterise the set of equivariant linear bases , their analysis does not give a practical algorithm since such large matrices that form the linear bases are too large to store in memory for any large graph . While Maron et al . ( 2018 ) give an efficient implementation based on pooling and broadcasting for standard homogeneous graphs , they do not provide a general implementation for arbitrary node and edge types . Albooyeh et al . ( 2019 ) give a pooling and broadcasting view of operations for hyper-graphs and incidence structures of other geometric entities . However , all such structures have a single node type that is assumed exchangeable . Some other related works that have a symmetry-based approach to GNNs include Maron et al . ( 2019 ) ; de Haan et al . ( 2020 ) ; Azizian & Lelarge ( 2020 ) . 3 OUR MODEL . 3.1 NOTATION . A heterogeneous graph G = ⟨D , R , V , X⟩ is a four tuple , where D = { 1 , . . . , d } is the set of node types . For each node type i ∈ D we have a set of ni nodes Vi = { v1 , . . . , vni } . We use V = ⋃iVi for the set of all n = ∑i∈D ni nodes . R = { r1 , . . . , r ` } is the set of edge types , where r = ⟨r , r⟩ for r , r ∈ D is a pair of node types that appear in the edge type r. X is a set of node adjacency matrices X = { X r ∈ Rnr×nr ∣ r ∈ R } , one for each edge type r ∈ R. Such matrices can represent both node and edge attributes using diagonal and off-diagonal elements respectively . For simplicity , we initially assume scalar node and edge attributes , later we show generalization of this to vectors using multiple channels . In the definition above , D , R , and V contain the blueprint of the heterogeneous graph , while X contains the actual data . 3.2 EQUIVARIANCE FOR HETEROGENEOUS GRAPHS . Given the heterogeneous graph G our goal is to identify all equivariant linear operators that map the set of matrices X = { X1 , . . . , X ` } to another set of matrices Y = { Y1 , . . . , Y ` } of the same form . For this , it is sufficient to identify all such maps from one edge type to another Lr→r ′ ∶ Rnr×nr → Rnr′×nr′ . The overall equivariant map LR→R can be built from the collection Lr1→r1 , Lr1→r2 , . . . , Lr1→r ` , . . . , Lr ` →r ` . The equivariance condition on the linear operator Lr→r ′ ensures that any permutation of the input nodes of the same type leads to the same permutation of the nodes in the output for that node type . Let πi ∈ S ( ni ) be a permutation matrix acting on ni nodes of type i. Equivariance constraint requires Lr→r ′ ( πrX rπ⊺r ) = πr′ Lr→r ′ ( X r ) π⊺r′ ∀πr , πr , πr′ , πr ∈ S ( nr ) × S ( nr ) × S ( n′r ) × S ( n′r ) ( 1 ) where the permutation matrices correspond to two pairs of node types that appear in the input ( r ) and output edge types ( r′ ) . As also observed in related contexts ( Kondor et al. , 2018 ; Albooyeh et al. , 2019 ) such linear operators often involve pooling and broadcasting over input and output matrices . Our plan is to enumerate all such operations and prove that these are indeed the only linear operations with the desired equivariance property Eq . ( 1 ) . Example 1 . To build an intuition for these operations , consider two relations between ⟨AUTHOR , VENUE⟩ and ⟨PUBLICATION , AUTHOR⟩ . Let r = ⟨1,2⟩ denote the former and r′ = ⟨3,1⟩ be the latter , noting that these two edge types have a node type in common . The desired linear map Lr→r ′ should be equivariant to independent permutation of AUTHOR nodes , PUBLICATION nodes and VENUE nodes in our graph . The results that follow this example show that any equivariant Lr→r ′ has the following form : Lr→r ′ ( X ) = w1 ( X1nr1n′r ) ⊺ +w21n′ r ( 1⊺nrX1nr ) 1 ⊺ n′r ( 2 ) where w1 , w2 ∈ R are arbitrary weights and 1n is the identity vector of length n. Here , following Zaheer et al . ( 2017 ) we are performing pooling and broadcasting operations using multiplication by identity vectors . The first operation ( X1nr1n′r ) ⊺ pools over the columns of X ( i.e. , VENUES ) , and broadcasts the resulting column vector to create a nr′ × nr′ matrix which is then transposed to match the dimensions of the target edge type . We can think of the pooling operation above as collecting edge attributes from all the VENUES that are adjacent to each AUTHOR . Similarly , the broadcasting operation disperses this pooled information over all the PUBLICATION nodes adjacent to each AUTHOR . This example shows that an equivariant linear map is able to propagate relevant information across different edge types ( within a single layer of a deep network ) .

Summary: The focus of this paper is in the design of GNNs that are amenable to operate on heterogeneous graphs --- that is, networks where relationships can be of different types. Current methods do not account for these different types of edges, or reduce everything to the same graph --- thereby loosing valuable information.   In this case, the heterogeneous graph is modelled as a collection of node-node adjacency matrices, one for each edge type. The additional desiderata that they add here is that  that the GNNs that they seek must be invariant or equivariant to permutations of nodes (depending on whether we're doing node level or graph level predictions). For heterogeneous graphs, this invariance or equivariance constraint is to permutations within each node type (eg, within authors, or venues). To build a repository of linear operations that verify these properties, the authors simply enumerate them.