On the Expressiveness and Learning of Relational Neural Networks on Hypergraphs

1 INTRODUCTION . Reasoning over graph-structured data is an important task in various applications , including molecule analysis , social network modeling , and knowledge graph reasoning ( Gilmer et al. , 2017 ; Schlichtkrull et al. , 2018 ; Liu et al. , 2017 ) . While we have seen great success of various relational neural networks , such as Graph Neural Networks ( GNNs ; Scarselli et al. , 2008 ) , Neural Logical Machines ( NLM ; Dong et al. , 2019 ) , and Transformers ( Vaswani et al. , 2017 ) in a variety of applications ( Battaglia et al. , 2018 ; Merkwirth & Lengauer , 2005 ; Veličković et al. , 2020 ) , we do not yet have a full understanding of how different design parameters , such as the depth of the neural network , affects the expressiveness of these models , or how effectively these models generalize from data . In this paper , we develop a general framework , generalized relational neural networks ( RelNNs ) , that unifies a large class of relational neural networks including GNNs , NLMs , and Transformers . Unlike previous work that analyzes the expressiveness of graph neural networks ( Xu et al. , 2019 ; 2020 ; Barceló et al. , 2020 ) , our framework applies to hypergraphs , i.e. , graphs that have representations that directly relate groups of more than two nodes . Next , we quantify the expressiveness of generalized relational neural networks in terms of their structural parameters ; in particular , the maximum arity of hyperedges and the depth of the network . We formally proved the “ if and only if ” conditions for the expressive power w.r.t . the arity . That is , k-ary hyper-graph relational neural networks are sufficient and necessary for realizing FOCk , a fragment of first-order logic which involves at most k variables . This is a helpful result because now we can determine whether a RelNN can solve a problem by thinking whether a logic formula can represent the solution to this problem . Next , we formally described the relationship between expressiveness and non-constant-depth RelNNs . We proposed a conjecture about the ” depth hierarchy , ” and we connect the potential proof of this conjecture to distributed computing literature . We also develope an upper bound for RelNNs . This is a very interesting result : applying a k-ary hyper-graph RelNN for more than O ( nk−1 ) iterations has no effect on the expressiveness . Finally , we study the learning capabilities of RelNNs . We prove , under certain realistic assumptions , it is possible to train a RelNN on a finite set of graphs , and it will generalize to arbitrarily large graphs . This is an outcome due to the weight-sharing nature of RelNNs . Next , we proved a finer-grained sample-complexity bound for RelNNs . This result theoretically suggests that using max aggregation function is provably better than sum aggregation function when realizing a first-order logic formula . We hope our work can serve as a foundation for designing relational neural networks : to solve a specific problem , what arity do you need ? What depth do you need ? What aggregation function should you use ? Will my model have structural generalization ( i.e. , to larger graphs ) ? Our theoretical results on learning are further supported by experiments , for empirical demonstration of the theorems and for exploring the tightness of the sample-complexity bounds and effectiveness of different aggregation functions . Specifically , we show that RelNNs with sufficient expressiveness power can be trained using gradient descent to realize the desired function . Meanwhile , when the training data distribution is uniformly distributed , RelNNs do demonstrate strong generalization to larger graphs than those seen during training . Finally , we also empirically show the generalization of RelNNs with different aggregation functions . 2 GRAPH REASONING AND RELATIONAL NEURAL NETWORKS . In this section , we define a very general class of relational networks and show that it can represent several important classes of relational neural networks , including graph neural networks , neural logic machines , and transformers . 2.1 GRAPH REASONING PROBLEM . A graph reasoning problem is a prediction task on a hypergraph G. It can predict a discrete or real-valued property of the whole graph , of individual nodes , or of tuples of nodes . Much of the work on graph-structured neural-network models has focused on cases with only binary edges , but many important problems are best modeled using hypergraphs , in which a hyperedge can connect multiple nodes . A further generalization is to labeled hypergraphs , in which each hyperedge can have one of a discrete set of labels . We will make use of a still more general structure , in which hyperedges can have real and/or vector-valued labels . A hypergraph representation G is a tuple ( V , X ) , where V is a set of entities ( nodes ) , and X is a set of hypergraph representation functions . Specifically , X = { X0 , X1 , X2 , · · · , Xk } , where Xj : ( v1 , v2 , · · · , vj ) → S is a function mapping every tuple of j nodes to a value . We call j the arity of the hyperedge and k is the max arity of input hyperedges . The range S can be any set of discrete labels that describes relation type , or a scalar number ( e.g. , the length of an edge ) , or a vector . In general , we will use the arity 0 representation function X0 ( ∅ ) → S to represent any global properties of the graph as a whole . A graph reasoning function f is a mapping from a hypergraph representation G = ( V , X ) to another hyperedge representation function Y on V . There are several important types : • graph classification function : computes a global label , Y = { Y0 } ; Y0 ( ∅ ) → S ′ = { 0 , 1 } for the whole graph . • graph regression function : computes a global real output , Y = { Y0 } , where Y0 ( ∅ ) → R. • node classification/regression function : computes Y = { Y1 } , where Y1 ( v0 ) → { 0 , 1 } is a binary label for each individual node . • edge classification function : extends previous prediction notions to higher-arity hyperedges . For example , asking whether a graph is fully connected is a graph classification problem ; computing its diameter is a graph regression problem ; finding the set of disconnected subgraphs of size k is a k-ary hyperedge classification problem . Example ( undirected graph connectivity ) . The input is G = ( V , X ) , where X = { X0 , X1 , X2 } , in which X2 is mapping from each pair of nodes v1 , v2 ∈ V to { 0 , 1 } , indicating whether ( v1 , v2 ) are connected . The input is an undirected graph ; thus , X2 ( v1 , v2 ) = X2 ( v2 , v1 ) for every v1 , v2 . Note that X0 and X1 are constant functions and not used for describing the undirected graph . The output is Y = { Y0 } , where Y0 ( ∅ ) → { 0 , 1 } is a binary classification label . The desired label is 1 when the graph is connected , and 0 otherwise . 2.2 RELATIONAL NEURAL NETWORKS . We now introduce introduce generalized relational neural networks ( RelNNs ) , a general framework that unifies a set of important relational neural network models that can be trained to solve graph reasoning problems . In particular , we will describe relational neural networks following the notation and computation of neural logic machines ( NLMs ; Dong et al. , 2019 ) . We choose NLM as a base model because it naturally generalizes to hyperedges with arities greater than 2 . As we will see later , other relational neural networks such as graph neural networks ( GNNs ; Scarselli et al. , 2008 ) and transformers ( Vaswani et al. , 2017 ) are equivalent to NLMs and each other in terms of expressiveness . Showing this equivalence allows us to focus the rest of the paper on analyzing a single model type , with the understanding that the conclusions generalize to a broader class of relational neural networks . A RelNN is a multi-layer neural network that operates on hypergraph representations , in which the hypergraph representation functions are represented as tensors . The input is a hypergraph representation ( V , X ) . There are then several computational layers , each of which produces a hypergraph representation with nodes V and a new set of representation functions . Specifically , a B-ary RelNN produces hypergraph representation functions with arities from 0 up to a maximum hyperedge arity of B . We let Ti , j denote the tensor representation for the output at layer i and arity j . Each entry in the tensor is a mapping from a set of node indices ( v1 , v2 , · · · , vj ) to a vector in a latent space RW . Thus , Ti , j is a tensor of j+1 dimensions , with the first j dimensions corresponding to j-tuple of nodes , and the last feature dimension . For convenience , we write h0 , · for the input hypergraph representation and hD , · for the output of the RelNN . Fig . 1a shows the overall architecture of RelNNs . It has D×B computation blocks , namely relational reasoning layers ( RRLs ) . Each block RRLi , j , illustrated in Fig . 1b , takes the output from neighboring arities in the previous layer , Ti−1 , j−1 , Ti−1 , j and Ti−1 , j+1 , and produces Ti , j . Below we show the computation of each primitive operation in an RRL . The expand operation takes tensor Ti−1 , j−1 ( arity j − 1 ) and produces a new tensor TEi−1 , j−1 of arity j . The reduce operation takes tensor Ti−1 , j+1 ( arity j + 1 ) and produces a new tensor TRi−1 , j+1 of arity j + 1 . Mathematically , TEi−1 , j−1 [ v1 , v2 , · · · , vj ] = Ti−1 , j−1 [ v1 , v2 , · · · , vj−1 ] ; TRi−1 , j+1 [ v1 , v2 , · · · , vj ] = Aggvj+1 { Ti−1 , j+1 [ v1 , v2 , · · · , vj , vj+1 ] } . Here , Agg is called the aggregation function of a RelNN . For example , a sum aggregation function takes the summation along the dimension j + 1 of the tensor , and a max aggregation function takes the max along that dimension . The concat ( concatenate ) operation ⊕ is applied at the “ vector representation ” dimension . The permute operation generates a new tensor of the same arity , but it fuses the representations of hyperedges that share the same set of entities but in different order , such as ( v1 , v2 ) and ( v2 , v1 ) . Mathematically , for tensor X of arity j , if Y = permute ( X ) then Y [ v1 , v2 , · · · , vj ] = Concat σ∈Sj { X [ vσ1 , vσ2 , · · · , vσj ] } , where σ ∈ Sj iterates over all permuations of { 1 , 2 , · · · j } . NNj is a multi-layer perceptron ( MLP ) applied to each entry in the tensor produced after permutation , with nonlinearity σ ( e.g. , ReLU ) . It is important to note that we intentionally name the MLPs NNj instead of NNi , j . In generalized relational neural networks , for a given arity j , all MLPs across all layers i are shared . It is straightforward to see that this “ weight-shared ” model can realize a “ non-weight-shared ” RelNN that uses different weights for MLPs at different layers when the number of layers is a constant . With a sufficiently large length of the representation vector , we can simulate the computation of applying different transformations by constructing block matrix weights . ( A more formal proof is in Appendix B.1 ) The advantage of this weight sharing is that the network can be easily extended to a “ recurrent ” model . For example , we can apply the RelNN for a number of layers that is a function of n , where n is the the number of nodes in the input graph . Thus , we will use the term layers and iterations interchangeably . Handling high-arity features and using deeper models usually increase the computational cost . In appendix B.6 , we show that the time and space complexity of RelNN [ D , B ] is O ( DnB ) . Exanple ( GNN ) . The graph neural network ( GNN ) of Scarselli et al . ( 2008 ) can be described as a specific type of RelNN . First , GNNs work on graphs with maximum arity 2 . In a single iteration in a GNN , we use two message-passing subroutines : ( edge update ) the representation associated with each edge is updated by representations of its both ends , and ( node update ) : the representation of each node is updated by all edge presentations connected to it . This message passing scheme can easily fit in the RelNN : edge update can be realized by the expand and permute operations , while node update can be realized by the reduce operation . In general , hypergraph GNNs ( Morris et al. , 2019 ) are also equivalent to RelNNs . Specifically , a GNN applied to B-ary hyperedges is equivalent to an NLM applied to B + 1-ary hyperedges . We provide proof details in Appendix B.2 . There are other approaches of realizing GNN on hypergraphs such as hypergraph convolution ( Feng et al. , 2019 ; Yadati et al. , 2019 ; Bai et al. , 2021 ) , attention ( Ding et al. , 2020 ) and message passing ( Huang & Yang , 2021 ) . These approaches can be viewed as instances of RelNNs , but are less expressive than RelNNs with equal max arity . We provide proof details in Appendix B.3 . Example ( Transformer ) . The key difference between Transformers ( Vaswani et al. , 2017 ) and GNNs is the “ self-attention ” opeartion , which can be viewed as a special aggregation function . In self-attention , each “ sender ” computes the key and value , and each “ receiver ” computes the query . Xu et al . ( 2019 ) proved such aggregation can be simulated with a sum aggregation function . We will use calligraphic lettersM1 , M2 , · · · to name RelNNs . Because even when hyperparameters such as the maximum arity and the number of iterations are fixed , a RelNN is still a model familyM : the weights for MLPs will be trained on some data . Furthermore , each model M ∈M is a RelNN with a specific set of MLP weights .

Summary: The paper proposes a unifying study of models for relational learning, namely Transformers, Graph Neural Networks (GNNs), and Neural Logic Machines (NLM), under the RelNN framework, and provides a series of results pertaining to their expressiveness relative to the arity of relations they support, as well as the depth, i.e., the number of iterations, they use during computation. In particular, arity and depth hierarchies are studied, and a set of arity and depth requirements are stated for a set of computational problems. Furthermore, the generalization ability of RelNN models is studied in the fixed-precision and infinite-precision settings. In the former setting, it is shown that there exists a finite size N for input graphs at which perfectly fitting all possible input graphs yields perfect generalization onto graphs of size greater than N. In the latter case, the sample complexity required to achieve certain error bounds is studied, and it is shown that more range-restricted aggregation function such as max and mean enable improved sample complexity. Finally, the paper conducts an experimental analysis on several synthetic problems, and achieves results validating the earlier provided arity and depth results for RelNN.