On the Expressiveness and Learning of Relational Neural Networks on Hypergraphs

1 INTRODUCTION . Reasoning over graph-structured data is an important task in various applications , including molecule analysis , social network modeling , and knowledge graph reasoning ( Gilmer et al. , 2017 ; Schlichtkrull et al. , 2018 ; Liu et al. , 2017 ) . While we have seen great success of various relational neural networks , such as Graph Neural Networks ( GNNs ; Scarselli et al. , 2008 ) , Neural Logical Machines ( NLM ; Dong et al. , 2019 ) , and Transformers ( Vaswani et al. , 2017 ) in a variety of applications ( Battaglia et al. , 2018 ; Merkwirth & Lengauer , 2005 ; Veličković et al. , 2020 ) , we do not yet have a full understanding of how different design parameters , such as the depth of the neural network , affects the expressiveness of these models , or how effectively these models generalize from data . In this paper , we develop a general framework , generalized relational neural networks ( RelNNs ) , that unifies a large class of relational neural networks including GNNs , NLMs , and Transformers . Unlike previous work that analyzes the expressiveness of graph neural networks ( Xu et al. , 2019 ; 2020 ; Barceló et al. , 2020 ) , our framework applies to hypergraphs , i.e. , graphs that have representations that directly relate groups of more than two nodes . Next , we quantify the expressiveness of generalized relational neural networks in terms of their structural parameters ; in particular , the maximum arity of hyperedges and the depth of the network . We formally proved the “ if and only if ” conditions for the expressive power w.r.t . the arity . That is , k-ary hyper-graph relational neural networks are sufficient and necessary for realizing FOCk , a fragment of first-order logic which involves at most k variables . This is a helpful result because now we can determine whether a RelNN can solve a problem by thinking whether a logic formula can represent the solution to this problem . Next , we formally described the relationship between expressiveness and non-constant-depth RelNNs . We proposed a conjecture about the ” depth hierarchy , ” and we connect the potential proof of this conjecture to distributed computing literature . We also develope an upper bound for RelNNs . This is a very interesting result : applying a k-ary hyper-graph RelNN for more than O ( nk−1 ) iterations has no effect on the expressiveness . Finally , we study the learning capabilities of RelNNs . We prove , under certain realistic assumptions , it is possible to train a RelNN on a finite set of graphs , and it will generalize to arbitrarily large graphs . This is an outcome due to the weight-sharing nature of RelNNs . Next , we proved a finer-grained sample-complexity bound for RelNNs . This result theoretically suggests that using max aggregation function is provably better than sum aggregation function when realizing a first-order logic formula . We hope our work can serve as a foundation for designing relational neural networks : to solve a specific problem , what arity do you need ? What depth do you need ? What aggregation function should you use ? Will my model have structural generalization ( i.e. , to larger graphs ) ? Our theoretical results on learning are further supported by experiments , for empirical demonstration of the theorems and for exploring the tightness of the sample-complexity bounds and effectiveness of different aggregation functions . Specifically , we show that RelNNs with sufficient expressiveness power can be trained using gradient descent to realize the desired function . Meanwhile , when the training data distribution is uniformly distributed , RelNNs do demonstrate strong generalization to larger graphs than those seen during training . Finally , we also empirically show the generalization of RelNNs with different aggregation functions . 2 GRAPH REASONING AND RELATIONAL NEURAL NETWORKS . In this section , we define a very general class of relational networks and show that it can represent several important classes of relational neural networks , including graph neural networks , neural logic machines , and transformers . 2.1 GRAPH REASONING PROBLEM . A graph reasoning problem is a prediction task on a hypergraph G. It can predict a discrete or real-valued property of the whole graph , of individual nodes , or of tuples of nodes . Much of the work on graph-structured neural-network models has focused on cases with only binary edges , but many important problems are best modeled using hypergraphs , in which a hyperedge can connect multiple nodes . A further generalization is to labeled hypergraphs , in which each hyperedge can have one of a discrete set of labels . We will make use of a still more general structure , in which hyperedges can have real and/or vector-valued labels . A hypergraph representation G is a tuple ( V , X ) , where V is a set of entities ( nodes ) , and X is a set of hypergraph representation functions . Specifically , X = { X0 , X1 , X2 , · · · , Xk } , where Xj : ( v1 , v2 , · · · , vj ) → S is a function mapping every tuple of j nodes to a value . We call j the arity of the hyperedge and k is the max arity of input hyperedges . The range S can be any set of discrete labels that describes relation type , or a scalar number ( e.g. , the length of an edge ) , or a vector . In general , we will use the arity 0 representation function X0 ( ∅ ) → S to represent any global properties of the graph as a whole . A graph reasoning function f is a mapping from a hypergraph representation G = ( V , X ) to another hyperedge representation function Y on V . There are several important types : • graph classification function : computes a global label , Y = { Y0 } ; Y0 ( ∅ ) → S ′ = { 0 , 1 } for the whole graph . • graph regression function : computes a global real output , Y = { Y0 } , where Y0 ( ∅ ) → R. • node classification/regression function : computes Y = { Y1 } , where Y1 ( v0 ) → { 0 , 1 } is a binary label for each individual node . • edge classification function : extends previous prediction notions to higher-arity hyperedges . For example , asking whether a graph is fully connected is a graph classification problem ; computing its diameter is a graph regression problem ; finding the set of disconnected subgraphs of size k is a k-ary hyperedge classification problem . Example ( undirected graph connectivity ) . The input is G = ( V , X ) , where X = { X0 , X1 , X2 } , in which X2 is mapping from each pair of nodes v1 , v2 ∈ V to { 0 , 1 } , indicating whether ( v1 , v2 ) are connected . The input is an undirected graph ; thus , X2 ( v1 , v2 ) = X2 ( v2 , v1 ) for every v1 , v2 . Note that X0 and X1 are constant functions and not used for describing the undirected graph . The output is Y = { Y0 } , where Y0 ( ∅ ) → { 0 , 1 } is a binary classification label . The desired label is 1 when the graph is connected , and 0 otherwise . 2.2 RELATIONAL NEURAL NETWORKS . We now introduce introduce generalized relational neural networks ( RelNNs ) , a general framework that unifies a set of important relational neural network models that can be trained to solve graph reasoning problems . In particular , we will describe relational neural networks following the notation and computation of neural logic machines ( NLMs ; Dong et al. , 2019 ) . We choose NLM as a base model because it naturally generalizes to hyperedges with arities greater than 2 . As we will see later , other relational neural networks such as graph neural networks ( GNNs ; Scarselli et al. , 2008 ) and transformers ( Vaswani et al. , 2017 ) are equivalent to NLMs and each other in terms of expressiveness . Showing this equivalence allows us to focus the rest of the paper on analyzing a single model type , with the understanding that the conclusions generalize to a broader class of relational neural networks . A RelNN is a multi-layer neural network that operates on hypergraph representations , in which the hypergraph representation functions are represented as tensors . The input is a hypergraph representation ( V , X ) . There are then several computational layers , each of which produces a hypergraph representation with nodes V and a new set of representation functions . Specifically , a B-ary RelNN produces hypergraph representation functions with arities from 0 up to a maximum hyperedge arity of B . We let Ti , j denote the tensor representation for the output at layer i and arity j . Each entry in the tensor is a mapping from a set of node indices ( v1 , v2 , · · · , vj ) to a vector in a latent space RW . Thus , Ti , j is a tensor of j+1 dimensions , with the first j dimensions corresponding to j-tuple of nodes , and the last feature dimension . For convenience , we write h0 , · for the input hypergraph representation and hD , · for the output of the RelNN . Fig . 1a shows the overall architecture of RelNNs . It has D×B computation blocks , namely relational reasoning layers ( RRLs ) . Each block RRLi , j , illustrated in Fig . 1b , takes the output from neighboring arities in the previous layer , Ti−1 , j−1 , Ti−1 , j and Ti−1 , j+1 , and produces Ti , j . Below we show the computation of each primitive operation in an RRL . The expand operation takes tensor Ti−1 , j−1 ( arity j − 1 ) and produces a new tensor TEi−1 , j−1 of arity j . The reduce operation takes tensor Ti−1 , j+1 ( arity j + 1 ) and produces a new tensor TRi−1 , j+1 of arity j + 1 . Mathematically , TEi−1 , j−1 [ v1 , v2 , · · · , vj ] = Ti−1 , j−1 [ v1 , v2 , · · · , vj−1 ] ; TRi−1 , j+1 [ v1 , v2 , · · · , vj ] = Aggvj+1 { Ti−1 , j+1 [ v1 , v2 , · · · , vj , vj+1 ] } . Here , Agg is called the aggregation function of a RelNN . For example , a sum aggregation function takes the summation along the dimension j + 1 of the tensor , and a max aggregation function takes the max along that dimension . The concat ( concatenate ) operation ⊕ is applied at the “ vector representation ” dimension . The permute operation generates a new tensor of the same arity , but it fuses the representations of hyperedges that share the same set of entities but in different order , such as ( v1 , v2 ) and ( v2 , v1 ) . Mathematically , for tensor X of arity j , if Y = permute ( X ) then Y [ v1 , v2 , · · · , vj ] = Concat σ∈Sj { X [ vσ1 , vσ2 , · · · , vσj ] } , where σ ∈ Sj iterates over all permuations of { 1 , 2 , · · · j } . NNj is a multi-layer perceptron ( MLP ) applied to each entry in the tensor produced after permutation , with nonlinearity σ ( e.g. , ReLU ) . It is important to note that we intentionally name the MLPs NNj instead of NNi , j . In generalized relational neural networks , for a given arity j , all MLPs across all layers i are shared . It is straightforward to see that this “ weight-shared ” model can realize a “ non-weight-shared ” RelNN that uses different weights for MLPs at different layers when the number of layers is a constant . With a sufficiently large length of the representation vector , we can simulate the computation of applying different transformations by constructing block matrix weights . ( A more formal proof is in Appendix B.1 ) The advantage of this weight sharing is that the network can be easily extended to a “ recurrent ” model . For example , we can apply the RelNN for a number of layers that is a function of n , where n is the the number of nodes in the input graph . Thus , we will use the term layers and iterations interchangeably . Handling high-arity features and using deeper models usually increase the computational cost . In appendix B.6 , we show that the time and space complexity of RelNN [ D , B ] is O ( DnB ) . Exanple ( GNN ) . The graph neural network ( GNN ) of Scarselli et al . ( 2008 ) can be described as a specific type of RelNN . First , GNNs work on graphs with maximum arity 2 . In a single iteration in a GNN , we use two message-passing subroutines : ( edge update ) the representation associated with each edge is updated by representations of its both ends , and ( node update ) : the representation of each node is updated by all edge presentations connected to it . This message passing scheme can easily fit in the RelNN : edge update can be realized by the expand and permute operations , while node update can be realized by the reduce operation . In general , hypergraph GNNs ( Morris et al. , 2019 ) are also equivalent to RelNNs . Specifically , a GNN applied to B-ary hyperedges is equivalent to an NLM applied to B + 1-ary hyperedges . We provide proof details in Appendix B.2 . There are other approaches of realizing GNN on hypergraphs such as hypergraph convolution ( Feng et al. , 2019 ; Yadati et al. , 2019 ; Bai et al. , 2021 ) , attention ( Ding et al. , 2020 ) and message passing ( Huang & Yang , 2021 ) . These approaches can be viewed as instances of RelNNs , but are less expressive than RelNNs with equal max arity . We provide proof details in Appendix B.3 . Example ( Transformer ) . The key difference between Transformers ( Vaswani et al. , 2017 ) and GNNs is the “ self-attention ” opeartion , which can be viewed as a special aggregation function . In self-attention , each “ sender ” computes the key and value , and each “ receiver ” computes the query . Xu et al . ( 2019 ) proved such aggregation can be simulated with a sum aggregation function . We will use calligraphic lettersM1 , M2 , · · · to name RelNNs . Because even when hyperparameters such as the maximum arity and the number of iterations are fixed , a RelNN is still a model familyM : the weights for MLPs will be trained on some data . Furthermore , each model M ∈M is a RelNN with a specific set of MLP weights .

Summary: This paper explores the expressiveness of neural network-based relational learning models, such as graph neural networks and Transformer. To this end, a rational neural network is built upon the hypergraph structures to unify different models. Synthetic datasets are used for performance evaluation.