You are AllSet: A Multiset Function Framework for Hypergraph Neural Networks

Hypergraphs are used to model higher-order interactions amongst agents and there exist many practically relevant instances of hypergraph datasets . To enable the efficient processing of hypergraph data , several hypergraph neural network platforms have been proposed for learning hypergraph properties and structure , with a special focus on node classification tasks . However , almost all existing methods use heuristic propagation rules and offer suboptimal performance on benchmarking datasets . We propose AllSet , a new hypergraph neural network paradigm that represents a highly general framework for ( hyper ) graph neural networks and for the first time implements hypergraph neural network layers as compositions of two multiset functions that can be efficiently learned for each task and each dataset . The proposed AllSet framework also for the first time integrates Deep Sets and Set Transformers with hypergraph neural networks for the purpose of learning multiset functions and therefore allows for significant modeling flexibility and high expressive power . To evaluate the performance of AllSet , we conduct the most extensive experiments to date involving ten known benchmarking datasets and three newly curated datasets that represent significant challenges for hypergraph node classification . The results demonstrate that our method has the unique ability to either match or outperform all other hypergraph neural networks across the tested datasets : As an example , the performance improvements over existing methods and a new method based on heterogeneous graph neural networks are close to 4 % on the Yelp and Zoo datasets , and 3 % on the Walmart dataset . 1 INTRODUCTION . Graph-centered machine learning , and especially graph neural networks ( GNNs ) , have attracted great interest in the machine learning community due to the ubiquity of graph-structured data and the importance of solving numerous real-world problems such as semi-supervised node classification and graph classification ( Zhu , 2005 ; Shervashidze et al. , 2011 ; Lü & Zhou , 2011 ) . Graphs model pairwise interactions between entities , but fail to capture more complex relationships . Hypergraphs , on the other hand , involve hyperedges that can connect more than two nodes , and are therefore capable of representing higher-order structures in datasets . There exist many machine learning and data mining applications for which modeling high-order relations via hypergraphs leads to better learning performance when compared to graph-based models ( Benson et al. , 2016 ) . For example , in subspace clustering , in order to fit a d-dimensional subspace , we need at least d+1 data points ( Agarwal et al. , 2005 ) ; in hierarchical species classification of a FoodWeb , a carbon-flow unit based on four species is significantly more predictive than that involving two or three entities ( Li & Milenkovic , 2017 ) . Hence , it is desirable to generalize GNN concepts to hypergraphs . One straightforward way to generalize graph algorithms for hypergraphs is to convert hypergraphs to graphs via clique-expansion ( CE ) ( Agarwal et al. , 2005 ; Zhou et al. , 2006 ) . CE replaces hyperedges by ( possibly weighted ) cliques . Many recent attempts to generalize GNNs to hypergraphs can be viewed as redefining hypergraph propagation schemes based on CE or its variants ( Yadati et al. , 2019 ; Feng et al. , 2019 ; Bai et al. , 2021 ) , which was also originally pointed out in ( Dong et al. , 2020 ) . Despite the simplicity of CE , it is well-known that CE causes distortion and leads to undesired losses in learning performance ( Hein et al. , 2013 ; Li & Milenkovic , 2018 ; Chien et al. , 2019b ) . In parallel , more sophisticated propagation rules directly applicable on hypergraphs , and related to tensor eigenproblems , have been studied as well . One such example , termed Multilinear PageRank ( Gleich et al. , 2015 ) , generalizes PageRank techniques ( Page et al. , 1999 ; Jeh & Widom , 2003 ) directly to hypergraphs without resorting to the use of CE . Its propagation scheme is closely related to the Z eigenproblem which has been extensively investigated in tensor analysis and spectral hypergraph theory ( Li et al. , 2013 ; He & Huang , 2014 ; Qi & Luo , 2017 ; Pearson & Zhang , 2014 ; Gautier et al. , 2019 ) . An important result of Benson et al . ( 2017 ) shows that tensor-based propagation outperforms a CE-based scheme on several tasks . The pros and cons of these two types of propagation rule in statistical learning frameworks were examined in Chien et al . ( 2021a ) . More recently , it was shown in Tudisco et al . ( 2020 ) that label propagation based on CE of hypergraphs does not always lead to acceptable performance . Similarly to Chien et al . ( 2021a ) , Benson ( 2019 ) identified positive traits of CE eigenvectors but argued in favor of using Z eigenvectors due to their more versatile nonlinear formulation compared to that of the eigenvectors of CE graphs . We address two natural questions pertaining to learning on hypergraphs : “ Is there a general framework that includes CE-based , Z-based and other propagations on hypergraphs ? ” and , “ Can we learn propagation schemes for hypergraph neural networks suitable for different datasets and different learning tasks ? ” We give affirmative answers to both questions . We propose a general framework , AllSet , which includes both CE-based and tensor-based propagation rules as special cases . We also propose two powerful hypergraph neural network layer architectures that can learn adequate propagation rules for hypergraphs using multiset functions . Our specific contributions are as follows . 1 . We show that using AllSet , one can not only model CE-based and tensor-based propagation rules , but also cover propagation methods of most existing hypergraph neural networks , including HyperGCN ( Yadati et al. , 2019 ) , HGNN ( Feng et al. , 2019 ) , HCHA ( Bai et al. , 2021 ) , HNHN ( Dong et al. , 2020 ) and HyperSAGE ( Arya et al. , 2020 ) . Most importantly , we show that all these propagation rules can be described as a composition of two multiset functions ( leading to the proposed method name AllSet ) . Furthermore , we also show that AllSet is a hypergraph generalization of Message Passing Neural Networks ( MPNN ) ( Gilmer et al. , 2017 ) , a powerful graph learning framework encompassing many GNNs such as GCN ( Kipf & Welling , 2017 ) and GAT ( Veličković et al. , 2018 ) . 2 . Inspired by Deep Sets ( Zaheer et al. , 2017 ) and Set Transformer ( Lee et al. , 2019 ) , we propose AllDeepSets and AllSetTransformer layers which are end-to-end trainable . They can be plugged into most types of graph neural networks to enable effortless generalizations to hypergraphs . Notably , our work represents the first attempt to connect the problem of learning multiset function with hypergraph neural networks , and to leverage the powerful Set Transformer model in the design of these specialized networks . 3 . We report , to the best of our knowledge , the most extensive experiments in the hypergraph neural networks literature pertaining to semi-supervised node classification . Experimental results against ten baseline methods on ten benchmark datasets and three newly curated and challenging datasets demonstrate the superiority and consistency of our AllSet approach . As an example , AllSetTransformer outperforms the best baseline method by close to 4 % in accuracy on Yelp and Zoo datasets and 3 % on the Walmart dataset ; furthermore , AllSetTransformer matches or outperforms the best baseline models on nine out of ten datasets . Such improvements are not possible with modifications of HAN ( Wang et al. , 2019b ) , a heterogeneous GNN , adapted to hypergraphs or other specialized approaches that do not use Set Transformers . 4 . As another practical contribution , we also provide a succinct pipeline for standardization of the hypergraph neural networks evaluation process based on Pytorch Geometric ( Fey & Lenssen , 2019 ) . The pipeline is built in a fashion similar to that proposed in recent benchmarking GNNs papers ( Hu et al. , 2020a ; Lim et al. , 2021 ) . The newly introduced datasets , along with our reported testbed , may be viewed as an initial step toward benchmarking hypergraph neural networks . All proofs and concluding remarks are relegated to the Appendix . 2 BACKGROUND . Notation . A hypergraph is an ordered pair of sets G ( V , E ) , where V = { 1 , 2 , . . . , n } is the set of nodes while E is the set of hyperedges . Each hyperedge e ∈ E is a subset of V , i.e. , e ⊆ V . Unlike a graph edge , a hyperedge e may contain more than two nodes . If ∀ e ∈ E one has |e| = d ∈ N , the hypergraph G is termed d-uniform . A d-uniform hypergraph can be represented by a d-dimensional supersymmetric tensor such that for all distinct collections i1 , . . . , id ∈ V , Ai1 , ... , id = 1 ( d−2 ) ! if e = { i1 , ... , id } ∈ E , and Ai1 , ... , id = 0 otherwise . Henceforth , A : , ... , id is used to denote the slice of A along the first coordinate . A hypergraph can alternatively be represented by its incidence matrix H , where Hve = 1 if v ∈ e and Hve = 0 otherwise . We use the superscript ( t ) to represent the functions or variables at the t-th step of propagation and ‖ to denote concatenation . Furthermore , Θ and b are reserved for a learnable weight matrix and bias of a neural network , respectively . Finally , we use σ ( · ) to denote a nonlinear activation function ( such as ReLU , eLU or LeakyReLU ) , which depends on the model used . CE-based propagation on hypergraphs . The CE of a hypergraph G ( V , E ) is a weighted graph with the same set of nodes V . It can be described in terms of the associated adjacency or incidence matrices which we write with a slight abuse of notation as A ( CE ) ij = ∑ i3 , ... , id∈V Ai , j , i3 , ... , id and H ( CE ) = HHT , respectively . It is obvious that these two matrices only differ in their diagonal entries ( 0s versus node degrees , respectively ) . One step of propagation of a F -dimensional node feature matrix X ∈ Rn×F is captured by A ( CE ) X or H ( CE ) X ; alternatively , in terms of node feature updates , we have CEpropA : X ( t+1 ) v , : = ∑ e : v∈e ∑ u : u∈e\v X ( t ) u , : ; CEpropH : X ( t+1 ) v , : = ∑ e : v∈e ∑ u : u∈e X ( t ) u , : , ( 1 ) Many existing hypergraph convolutional layers actually perform CE-based propagation , potentially with further degree normalization and nonlinear hyperedge weights . For example , the propagation rule of HGNN ( Feng et al. , 2019 ) takes the following node-wise form : X ( t+1 ) v , : = σ ( [ 1√ dv ∑ e : v∈e we |e| ∑ u : u∈e X ( t ) u , :√ du ] Θ ( t ) + b ( t ) ) , ( 2 ) where dv denotes the degree of node v , we is a predefined weight of hyperedge e and σ is the ReLU activation function . The hypergraph convolution in HCHA ( Bai et al. , 2021 ) uses different degree normalizations and attention weights , with the attention weights depending on node features and the hyperedge features . If datasets do not contain hyperedge feature information or if the features come from a different domain compared to the node features , one can not use their attention module ( Bai et al. , 2021 ) . HyperGCN replaces each hyperedge by an incomplete clique via so-called mediators ( Yadati et al. , 2019 ) . When the hypergraph is 3-uniform , the aforementioned approach becomes a standard weighted CE . Hence , all the described hypergraph neural networks adapt propagation rules based on CE or its variants , potentially with the addition of nonlinear hyperedge weights . The described methods achieve reasonable good performance on standard cocitation and coauthor benchmarking datasets . Tensor-based propagations . As mentioned in the introduction , there exist more elaborate tensorbased propagation schemes which in some cases outperform CE-based methods . The propagation rules related to Z eigenproblems such as multilinear PageRank ( Gleich et al. , 2015 ) and spacey random walks ( Benson et al. , 2017 ) are two such examples . The Z eigenproblem for an adjacency tensor A of a d-uniform hypergraph is defined as : λx = Axd−1 , ∑ i2 , ... , id A : ,i2 , ... , idxi2 . . . xid , x ∈ Rn and ‖x‖2 = 1 . ( 3 ) Here , Axd−1 equals ∑ i2 , ... , id A : ,i2 , ... , idxi2 . . . xid , an entity frequently used in the tensor analysis literature . The Z eigenproblem has been extensively studied both in the context of tensor analysis and network sciences ; the problem is also known as the l2 eigenproblem ( Lim , 2005 ; Gautier et al. , 2019 ) . We refer the interested readers to Qi & Luo ( 2017 ) for a more detailed theoretical analysis of the Z eigenproblems and Benson ( 2019 ) for its application in the study of hypergraph centralities . By ignoring the norm constraint , one can define the following tensor-based propagation rule based on ( 3 ) according to : Zprop : X ( t+1 ) v , : = ∑ e : v∈e ( d− 1 ) ∏ u : u∈e\v X ( t ) u , : . ( 4 ) Despite its interesting theoretical properties , Zprop is known to have what is termed the “ unit problem ” ( Benson , 2019 ) . In practice , the product can cause numerical instabilities for large hyperedges . Furthermore , Zprop has only been studied for the case of d-uniform hypergraphs , which makes it less relevant for general hypergraph learning tasks . Clearly , CEprop and Zprop have different advantages and disadvantages for different dataset structures . This motivates finding a general framework that encompasses these two and other propagation rules . In this case , we aim to learn the suitable propagation scheme under such a framework for hypergraph neural networks .

Summary: Transferring standard graph operators to the hypergraph setting is non-trivial and several message passing operators have been considered in the setting of hypergraphs, including clique-expansion and tensor based. In this paper the authors propose a general framework where learnable multiset functions are used in order to learn the hypergraph propagation map from the data. In particular, this framework contains  many previously considered propagation maps as particular cases