You are AllSet: A Multiset Function Framework for Hypergraph Neural Networks
Authors: Eli Chien
Chao Pan
Jianhao Peng
Olgica Milenkovic
Venue: ICLR 2022 Poster (2022)
Keywords: Hypergraph neural networks
multiset functions
deep sets
set transformer

Abstract: Hypergraphs are used to model higher-order interactions amongst agents and there exist many practically relevant instances of hypergraph datasets. To enable the efficient processing of hypergraph data, several hypergraph neural network platforms have been proposed for learning hypergraph properties and structure, with a special focus on node classification tasks. However, almost all existing methods use heuristic propagation rules and offer suboptimal performance on benchmarking datasets. We propose AllSet, a new hypergraph neural network paradigm that represents a highly general framework for (hyper)graph neural networks and for the first time implements hypergraph neural network layers as compositions of two multiset functions that can be efficiently learned for each task and each dataset. The proposed AllSet framework also for the first time integrates Deep Sets and Set Transformers with hypergraph neural networks for the purpose of learning multiset functions and therefore allows for significant modeling flexibility and high expressive power. To evaluate the performance of AllSet, we conduct the most extensive experiments to date involving ten known benchmarking datasets and three newly curated datasets that represent significant challenges for hypergraph node classification. The results demonstrate that our method has the unique ability to either match or outperform all other hypergraph neural networks across the tested datasets: As an example, the performance improvements over existing methods and a new method based on heterogeneous graph neural networks are close to $4\%$ on the Yelp and Zoo datasets, and $3\%$ on the Walmart dataset.

Published as a conference paper at ICLR 2022
YOU ARE ALLSET: A MULTISET LEARNING FRAME- WORK FOR HYPERGRAPH NEURAL NETWORKS
Eli Chien"
Chao Pan" Jianhao Peng" Olgica Milenkovic
Department of Electrical and Computer Engineering
University of Illinois, Urbana- Chhana-Champaign (ichien3, chaopan2, j anhao2, mi Lenkolnkkv .ddu
ABSTRACT
Hypergraphs are used to model higher-order interactions amongst agents and there exist many practically relevant instances of hypergraph datasets. To enable the efficient processing of hypergraph data, several hypergraph neural network plat- forms have been proposed for learning hypergraph properties and structure, with a special focus on node classification tasks. However, almost all existing methods use heuristic propagation rules and offer suboptimal performance on benchmark- ing datasets. We propose AllSet, new hypergraph neural network paradigm that represents a highly general framework for (hyper)graph neural networks and for the first time implements hypergraph neural network layers as compositions of two multiset functions that can be efficiently learned for each task and each dataset. The proposed AllSet framework also for the first time integrates Deep Sets and Set Transformers with hypergraph neural networks for the purpose of learning mul- tiset functions and therefore allows for significant modeling flexibility and high expressive power. To evaluate the performance of AllSet, we conduct the most ex- tensive experiments date involving known benchmarking datasets and three newly curated datasets that represent significant challenges for hypergraph node classification. The results demonstrate that our method has the unique ability either match or outperform all other hypergraph neural networks across the tested datasets: an example, the performance improvements over existing methods and new method based on heterogencous graph neural networks are close to 4% on the Yelp and Zoo datasets, and 3% on the Walmart dataset. Our AllSet network implementation is available onling
INTRODUCTION
Graph-centered machine learning, and especially graph neural networks (GNNs), have attracted great interest in the machine learning community due to the ubiquity of graph-structured data and the importance of solving numerous real-world problems such as semi-supervised node classifica- tion and graph classification (Zhu 2005| Shervashidze et al.) 2011 Lü & Zhou 201T). Graphs model pairwise interactions between entities, but fail to capture more complex relationships. Hypergraphs, on the other hand, involve hyperedges that connect
than two nodes, and therefore capa- ble of representing higher-order structures in datasets. There exist many machine learning and data mining applications for which modeling high-order relations via hypergraphs leads to better learning performance when compared to graph-based models (Benson et al. 2016). For example, in subspace clustering, in order fit d-dimensional subspace, we need least d+1 data points (Agarwal al. 2005); in hierarchical species classification of a FoodWeb, a carbon-flow unit based on four species is significantly more predictive than that involving two or three entities (Li & Milenkovic 2017). Hence, it is desirable generalize GNN concepts to hypergraphs.
One straightforward way to generalize graph algorithms for hypergraphs to convert hypergraphs to graphs via tlique-expansien (CE) (Agarwal et al. 2005) Zhou et al. (006).. CE replaces hyperedges by (possibly weighted) cliques. Many recent attempts to generalize GNNs to hypergraphs can be viewed as redefining hypergraph propagation schemes based on CE its variants (adatii et al.
"Equal contribution.
htpps//gtthuh.comrjjjaaa2a/2///

---

Published as a conference paper at ICLR 2022
2019 Feng et al. 2019 Bai et al. 2021 which was also originally pointed out in (Dong et al. 2020
Despite the simplicity of CE, it is known that CE causes distortion and leads to undesired losses
in learning performance (Hein et al. 2013 Li & Milenkovic 2018 Chien al. 2019b).
In parallel, more sophisticated propagation rules directly applicable on hypergraphs, and related to tensor eigenproblems, have been studied well. One such example, termed Multilinear PageR-
ank (Gleich et al. 2013), generalizes PageRank techniques (Page et al.) 1999; Deh & Widom] (2003)
directly hypergraphs without resorting to the use of CE. Its propagation scheme is closely related
to the Z eigenproblem which has been extensively investigated in tensor analysis and spectral hyper-
graph theory (Li al., [2013] He & Huang 2014 Qi & Luo 2017] Pearson & Zhang 2014 Gautier et al., 2019). An important result of Benson et al. (2017) shows that tensor-
tensor-based based propagation out-
performs CF-based scheme on several tasks. The pros and cons of these two types of propagation rule in statistical learning frameworks were examined in Chien et al. (2021a). More recently, it was shown in [uuiicoo et al. (2020) that label propagation based on CE of hypergraphs does not always lead to acceptable performance. Similarly to Chien et al. (202la), Benson (2019) identified positive traits of CE eigenvectors but argued in favor of using Z eigenvectors due to their more versatile nonlinear formulation compared to that of the eigenvectors of CE graphs.
We address two natural questions pertaining to learning on hypergraphs: "Is there a general frame- work that includes CE-based, Z-based and other propagations on hypergraphs?" and, "Can we learn propagation schemes for hypergraph neural networks suitable for different datasets and different learning tasks?" We give affirmative answers both questions. We propose general framework, AllSet, which includes both CE-based and tensor-based propagation rules as special cases. We also propose two powerful hypergraph neural network layer architectures that can learn adequate propa- gation rules for hypergraphs using multiset functions. Our specific contributions are as follows.
1. We show that using AllSet, one can not only model CE-based and lensor-based propagation rules, but also cover propagation methods of most existing hypergraph neural networks, including Hy- perGCN (Yadati et al. 2019), HGNN (Feng et al.] 2019). HCHA (Bai et al. 202T). HNHN (Dong et al. 2020) and HyperSAGE (Arya et al. 2020). Most importantly, we show that all these prop- agation rules can be described as a composition of two multiset functions (leading to the proposed method name AllSet), Furthermore, we also show that AllSet is hypergraph generalization of Mes- sage Passing Neural Networks (MPNN) (Gilmer et al. 2017)). a powerful graph learning framework encompassing many GNNs such as GCN (Kipf & Welling[2011 (017) and GAT (Veličković al. 2018)
2. Inspired by Deep Sets (Zaheer et al., 2017) and Set Transformer [ee et al. 2019), we propose AllDeepSets and AllSerTransformer layers which are end-to-end trainable. They can be plugged into most types of graph neural networks to enable effortless generalizations to hypergraphs. No- tably, our work represents the first attempt connect the problem of learning multiset function with hypergraph neural networks, and to leverage the powerful Set Transformer model in the design of these specialized networks.
3. We report, to the best of our knowledge, the most extensive experiments in the hypergraph neural networks literature pertaining to semi-supervised node classification. Experimental results against ten baseline methods on ten benchmark datasets and three newly curated and challenging datasets demonstrate the superiority and consistency of our AllSet approach. As an example, AllSe(Trans- former outperforms the best baseline method by close to 4% in accuracy on Yelp and Zoo datasets and 3% on the Walmart dataset; furthermore, AllSerTransformer matches or outperforms the best baseline models nine out of ten datasets. Such improvements are not possible with modifications of HAN (Wang et al. 2019b), a heterogeneous GNN, adapted to hypergraphs or other specialized approaches that do not use Set Transformers.
As another practical contribution, we also provide a succinet pipeline for standardization of the hypergraph neural networks evaluation process based on Pytorch Geometric (Fey & Lenssen 2019). The pipeline is built in a fashion similar to that proposed in recent benchmarking GNNs papers (Hu et al. 2020aa Lim et al.) 2021). ). The newly introduced datasets, along with our reported testbed, may be viewed as an initial step toward benchmarking hypergraph neural networks.
All proofs and concluding remarks are relegated to the Appendix.
2

---

Published as a conference paper at ICLR 2022
f
Multiset functions
S >
b
Z (++1)
Z (++1)
CE
((+1)
Saw (12
Figure 1: Left: The difference between a hypergraph and clique-expanded graph. Right: Illustration of our AllSet framework for the hypergraph depicted on the left. Included is an example on the aggregation rule for hyperedge b and node 4. The key idea is that fv and fE-v two multiset functions, which by definition are permutation invariant with respect to their input multisets.
BACKGROUND
Notation. A hypergraph an ordered pair of sets G(V,)), where V
is the set of nodes while & is the set of hyperedges. Each hyperedge E & is subset of V, i.e., S V. Unlike a graph edge, a hyperedge e may contain more than two nodes. If one has |e| = d N, the hypergraph G is termed d-uniform. A d-uniform hypergraph can be represented by d-dimensional supersymmetric tensor such that for all distinct collections i,, da E V, Ain.....s = (a-2)
otherwise. Henceforth, A. .....k is used to denote the slice of A along the first coordinate. hypergraph can alternatively be represented by its incidence matrix and H 0 otherwise. We the superscript (t) represent the
(i.....; id} E E, and Ai.....
H. where H. functions or variables at the t-th step propagation and || to denote concatenation. Furthermore, O and b are reserved for learnable weight matrix and bias of neural network, respectively. Finally, we use (-) to denote nonlinear activation function (such as ReLU, eLU or LeakyReLUU, which depends the model used.
if
CE-based propagation on hypergraphs. The CE of hypergraph G(V,E) is a weighted graph with the same set of nodes V. It can be described in terms of the associated adjacency or incidence (CE)
matrices which we write with a slight abuse notation as A f) ((CE) HH'", respectively. entries (Os versus node degrees, respectively). One step of propagation of a F-dimensional node feature matrix E R is captured by A (CE)X or H(CE)X; alternatively, terms of node feature updates, have
"'..... Aij.is...... and is obvious that these two matrices only differ in their diagonal
CEpropA: (t+1) = Σ Σ
(4) 0,2
CEpropH: ((++))
Σ
Σ
(1)
CXPEC W.aEe
CHEE UNEE
Many existing hypergraph convolutional layers actually perform CE-based propagation, potentially with further degree normalization and nonlinear hyperedge weights. For example, the propagation rule of HGNN (Feng al.) (019) takes the following node-wise form:
(1 eVEE
X ((+1)
(0) (b()
where d, denotes the degree of node We is predefined weight of hyperedge and a is the ReLU activation function. The hypergraph convolution in HCHA (Bai et al. 2021) uses different degree normalizations and attention weights, with the attention weights depending node features and the hyperedge features. If datasets do not contain hyperedge feature information or if the features come from different domain compared to the node features, cannot their attention module (Bai
3

---

Published as a conference paper at ICLR 2022
et al. 2021
clique via so-called media- tors Yadati et al. 2019p. When the hypergraph is 3-uniform, the aforementioned approach becomes standard weighted CE. Hence, all the described hypergraph neural networks adapt propagation rules based on CE or its variants, potentially with the addition of nonlinear hyperedge weights. The described methods achieve reasonable good performance standard cocitation and coauthor benchmarking datasets.
HyperGCN replaces each hyperedge by incomplete anteeeeennntt
Tensor-based propagations. As mentioned in the introduction, there exist more elaborate tensor- based propagation schemes which in some cases outperform CE-based methods. The propagation rules related to eigenproblems such as multilinear PageRank (Gleich al., 2015) and spacey random walks (Benson et al. (017) are two such examples. The Z eigenproblem for an adjacency tensor of a d-uniform hypergraph is defined as:
A Ax d-1
Σ f2....id
Fggs x E R" and ||=||²
(3)
Here, Ard-1 equals Lig...ta Ai ... Fi. an entity frequently used in the tensor analysis literature. The Z eigenproblem has been extensively studied both in the context of tensor analysis and network sciences; the problem is also known as the 1 eigenproblem (Lim) 2005 Gautier et al. 2019). We refer the interested readers to Qi & Luo (217) for more detailed theoretical analysis of the Z cigenproblems and Benson (2019) for its application in the study of hypergraph centralities.
By ignoring the norm constraint, one can define the following tensor-based propagation rule based on according to:
Zprop: x(++1)
Ed- II X
e:NEc
Despite interesting theoretical properties, Zprop is known to have what is termed the "unit prob- lem" (Benson, 2019). In practice, the product can cause numerical instabilities for large hyperedges. Furthermore, Zprop has only been studied for the case d-uniform hypergraphs, which makes less relevant for general hypergraph learning tasks. Clearly, CEprop and Zprop have different advan- tages and disadvantages for different dataset structures. This motivates finding a general framework that encompasses these two and other propagation rules. In this case, we aim to learn the suitable propagation scheme under such a framework for hypergraph neural networks.
ALLSET: ONE METHOD TO BIND THEM ALL
We show that all the above described propagation methods can be unified within one setting, termed AllSet. The key observation is that all propagation rules equal composition of two multiset func- tions, defined below.
Definition 3.1. A function is permulation invariant and only if'v Sa where Sn denotes the symmetric group of order nl, (xx<())...,xaiia = f(xi...., Xn))
Definition 3.2. We say that a function a multiset function if is permutation invariant.
Next, let Ve,x (Xa,: the hyperedge e. Also, let Z E [REEX=F' denote the hidden hyperedge representations. Similarly, let E = (Ze,: a E <) be the multiset of hidden representations of hyperedges that contain the node v. The AllSet framework uses the update rules
E e} denote the multiset of hidden node representations contained in
((++) -V-= Z!D), ((++1) = SE-vv(F. X(!), e.:
(5)
where fV4g and fE-v are two multiset functions with respect to their first input. For the initial condition, we choose _ and x!!!: to be hyperedge features and node features, respectively (if available). If these are not available, we set both entities to be all-zero matrices. Note that we make the tacit assumption that both functions fv-e and fg-v also include the hypergraph & as an input. This allows degree normalization to be part of our framework. As one can also distinguish the aggregating node v from the multiset V x we also have the following AllSet variant:
Z ((+1),² fV-E((e (Ve)e, x
X ((+1) = (F...iii x()).
(6)
4

---

Published as a conference paper at ICLR 2022
For simplicity, we omit the last input argument (1) for fy in (6). unless explicitly needed (as in the proof pertaining to HyperGCN). The formulation (5) lends itself to a significantly more computationally- and memory-efficient pipeline. This clearly the case since for each hyperedge the expression in (5) only uses one hyperedge hidden representation while the expression (6) uses |el distinct hidden representations. This difference can be substantial when the hyperedge sizes are large. Hence, we only use (6) for theoretical analysis purposes and (5) for all experimental verifications. The next theorems establish the universality of the AllSet framework.
Theorem 3.3. CEpropil of ① is a special case of AllSet (5). Furthermore, CEpropA of and Zprop of are also special cases of AllSet (6).
Sketch of proef. If we ignore the second input and choose both fy-e and JE-V to be sums over their input multisets, we recover the CE-based propagation rule CEpropH of (1 via (5). For CEpropA, the same observation is true with respect to (6). If one instead chooses fv-E to be the product over its input multiset, multiplied by the scalar V() 1, one recovers the tensor-based propagation Zprop of 44 via (6).
Next, we show that many state-of-thh-aa hypergraph neural network layers also represent special instances of AllSet and are strictly less expressive than AllSet.
Theorem 3.4. The hypergraph neural network layers of HGNN (2). ICHAA (Bai et al. 2027)
HNHN (Dong el al. 2020) and HyperSAGE (Arya el al. 2020) are all special cases of AllSet (5) The HyperGCN layer (Yadati et al. 2019 a special instance of AllSet (6). Furthermore, all the above methods are strictly less expressive than AllSet (5) and (6). More precisely, there exists
combination of multiset functions fv-E and fe-v for AllSet and (6) that cannot be modeled by any of the aforementioned hypergraph neural network layers.
The first half of the proof is by direct construction. The second half of the proof consists of counter- examples. Intuitively, none of the listed hypergraph neural network layers can model Zprop (3) while we established in the previous result that Zprop is a special case of AllSet.
The last result shows that the MPNN (Gilmer et al. 2017 framework is also special instance of
our AllSet for graphs, which are (clearly) special cases of hypergraphs. Hence, AllSet may also be viewed as a hypergraph generalization of MPNN. Note that MPNN itself generalizes many well- known GNNs, such GCN (Kipf & Welling 2017), Gated Graph Neural Networks (Li et al. 2015)
and GAT (Veličković et al. 2018).
Theorem 3.5. MPNN is a special case of AllSet 66 when applied to graphs.
4 HOW TO LEARN ALLSET LAYERS
The key idea behind AllSet to learn the multiset functions fV-LE and fE-v on the fly for each dataset and task. To facilitate this learning process, we first have to properly parametrize the multiset functions. Ideally, the parametrization should represent a universal approximator for a multiset function that allows one to retain the higher expressive power of our architectures when compared to that of the hypergraph neural networks described Theorem 3.4 For simplicity, we focus on the multiset inputs of fv-+ and fe and postpone the discussion pertaining to the second arguments of the functions to the end of this section.
Under the assumption that the multiset size is finite, the authors of Zaheer al. (0117 and Wagstaff et al. (2019) proved that any multiset functions f can be parametrized as S(S where p and Φ are some bijective mappings (Theorem 4.4 in Wagstaff et al. 2019 In practice these mappings can be replaced by any universal approximator such as a multilayer perceptron (MLP) (Zaheer et al. 2017). This leads the purely MLP AllSet layer for hypergraph neural networks, termed AllDeepSets.
Σ
SES ((x))
AllDeepSets: ( MLP Σ MLP(s)
The authors of Lee et al. 2019) argued that the unweighted sum in Deep Set makes it hard to learn the importance of each individual contributing term. Thus, they proposed the Set Transformer
5

---

Published as a conference paper at ICLR 2022
paradigm which was shown to offer better performance than Deep Sets learnable multiset func- tion architecture. Based on this result, we also propose an attention- AllSet layer for hyper- graph neural networks, termed AllSerTransformer Given the matrix S E RISIX which represents the multiset S of F-ddimensional real vectors, the definition of AllSeTTransformer
AllSerTransformer: e = fe-vv()) = LN + (MMM)Y++)M)))
where = LN (0 MH S,S) MH, 1|__0 O
0 w (000) (K()))) VO, A a ,,,,000, K MLP (S), v (S) (8)
Here, LN represents the layer normalization (Ba et al. 2016), || denotes concatenation and 0 E R1xhPh is learnable weight: in addition. MH multihead attention mechanism with h heads and activation function w (Vaswani et al. 2017). y. Note that the dimension of the output of MH x hFn where Fú is the hidden dimension of V(i) E Rl In our experimental setting, we choose w be the softmax function which is robust in practice. Our formulation of projections in , is slight more general than standard linear projections. If one restricts MLPK,' and MLPV. to one-layer perceptrons without a bias term (i.e., including only a learnable weight matrix), we obtain standard linear projections in the multihead attention mechanism. The output dimensions of MLPK and MLP are both equal R Fl erate row-wise, which means they are applied to each multiset element (a real vector) independently and in an identical fashion. This directly implies that MLP modules are permutation equivariant.
It is worth pointing out that all the MLP modules op-
Based on the above discussion and Proposition 2 Lef et al. (019 we have the following result. Proposition 4.1. The functions fV-E and fe-v defined in AlSSetlyansformr
are permutation invariant. Furthermore, the functions JV- and Je-V defined in (8) are universal approximators
of multiset functions when the size of the inpul multisel is finile.
Note that in practice, both the size of hyperedges and the node degrees are finite. Thus, the finite size multiset assumption is satisfied. Therefore, the above results directly imply that the AllDeepSets (7) and Tranforrreeerrrmr (8) layers have the same expressive power as the general AllSet framework. Together with the result of Theorem 3.4 arrive at the conclusion that AllDeepSets and AllSet- Transformer are both more expressive than any other aforementioned hypergraph neural network.
Conceptually, one can also incorporate the Set Attention Block and the two steps pooling strategy Lee et al. (2019) into AllSet. However, it appears hard to make such an implementation efficient and we hence leave this investigation as future work. In practice, we find that even our simplified design ( already outperforms all baseline hypergraph neural networks, as demonstrated by our extensive experiments. Also, possible to utilize the information of the second argument of fv-ve and fe-v via concatenation, another topic relegated to future work.
As concluding remark, we observe that the multiset functions both AllDeepSets 7 and AllSet- Transformer 8 are universal approximators for general multiset functions. In contrast, the other described hypergraph neural network layers fail to retain the universal approximation property for general multiset functions, as shown in Theorem 3x4 This implies that all these hypergraph neural network layers have strictly weaker expressive power compared to AlleeepSess (7) and AllSetTrans- former 8 We also note that any other universal approximators for multiset functions
easily be combined with our AllSet framework (as we already demonstrated by our Deep Sets and Set Transformer examples). One possible candidate is Janossy pooling (Murphy al.) 2019) which will be examined as part of our future work. Note that more expressive models do not necessarily have better performance than less expressive models, as many other factors influence system per- formance (as an example, AllDeepSet performs worse than AllSerTransformer albeit both have the same theoretical expressive power; this is in agreement with the observation from Lee et al. (2019) that Set Transformer can leam multiset functions better than Deep Set.). Nevertheless, we demon- strate in the experimental verification section that our AllSseTransformrr indeed has consistently better performance compared to other baseline methods.
5 RELATED WORKS
Due to space limitations, a more comprehensive discussion of related work is relegated to the Ap- pendix A

---

Published as a conference paper at ICLR 2022
Hypergraph learning. Learning on hypergraphs has attracted significant attention due to its abil- ity to capture higher-order structural information (Li et al. 2017: 2019). In the statistical learning and information theory literature. researchers have studied community detection problem for hyper- graph stochastic block models (Ghoshdastidar & Dukkipati 2015 Chien et al. 2018 2019a). In the machine learning community, methods that mitigate drawbacks of CE have been reported in (Li & Milenkovic 2017 Chien et al.) 2019b) [eii et al. 2013] Chan et al., 2018). Despite these advances, hypergraph neural network developments are still limited beyond the works covered by the AllSet paradigm. Exceptions include Hyper- SAGNN (Zhang et al. 2019). LEGCN (Yang et al. 2020) and MPNN-R (Yadati) 2020), the only three hypergraph neural networks that do not represent obvious special instances of AllSet. But, Hyper-SAGNN focuses on hyperedge prediction while we focus on superviee node classification. LEGCN first transforms hypergraphs into graphs via line expansion and then applies a standard GCN. Although the transformation used in LEGCN does not cause a large distortion as CE, it significantly increases the number of nodes and edges. Hence, is not memory and computationally efficient (more details about this approach can be found in the Appendix G MPNN-R focuses on recursive hypergraphs (Joslyn & Nowak 2017). which is not the topic of this work. It is an interesting future work to investigate the relationship between AllSet and Hyper-SAGNN or MPNN-R when extending AllSet to hyperedge prediction tasks or recursive hypergraph implementations.
Furthermore, Tudisco& & Higham 2021) : Tudisco et al. (2021) also proposed to develop and analyze hypergraph propagations that operate "between" CE-prop and Z-prop rules. These works focus on the theoretical analysis of special types of propagation rules, which can also be modeled by our AllSet frameworks. Importantly, our AllSet layers do not fix the rules but instead learn them in an adaptive manner. Also, one can view Tuuiisco & Higham (2021): Tudisco et al. (2021) as the hypergraph version of SGC, which can help with interpreting the functionalities of AllSet layers.
Star expansion, UNIGNN and Heterogeneous GNNs. A very recent work concurrent to ours, UniGNN (Huang & Yang. 202T), also proposes to unify hypergraph and GNN models. Both Huang & Yang (2021) and our work can be related to hypergraph star expansion (Agarwal et al. 2006 Yang et al. 2020), which results in a bipartite graph (see Figure D One particular variant, Uni- GIN, is related to our AllDeepSets model, as both represent a hypergraph generalization of GIN (Xu
jtt al. 2019). However, UniGNN does not make use of deep learning methods for learning mul- tiset functions, which is crucial for identifying the most appropriate propagation and aggregation rules for individual datasets and learning tasks as done by AllSerTransfommer (see Section 6 for supporting information). Also, the most advanced variant of UniGNN, UniGCNII, is compara- ble to ArnSerrransformrrs. Furthermore, one could also naively try to apply heterogeneous GNNs, such as HAN (Wang et al. 20196).. to hypergraph datasets converted into bipartite graphs. This approach was not previously examined in the literature, but we experimentally tested it neverthe- less to show that the performance improvements in this case are significantly smaller than ours and that the method does not scale well even for moderately sized hypergraphs. Another very recent work (Xue et al. 202T) appears at first glance similar to ours, but solves a very different problem by transforming a heterogeneous bipartite graph into a hypergraph and then apply standard GCN layers to implement /'-g and fe-v, which is more related to HNHN (Dong et al. 2020) and UniGCN (Huang & Yang 2021 and very different from TrnseeTransfommrr.
Learning (multi)set functions. (Multi)set functions, which are also known as pooling architectures for (multi)sets, have been used in numerous problems including causality discovery (Lopez-Paz) et al. (017), few-shot image classification (Snell et al. 2017) and conditional regression and clas- sification (Garnelo et al.] 2018). The authors of Zaheer et al. (2117) provide universal way to parameterize the (multi)set functions under some mild assumptions. Similar results have been pro- posed independently by the authors of of et al. (017)) for computer vision applications. The authors of Lee et al. (2019) propose to learn multiset functions via attention mechanisms. This further in- spired the work in Back et al. (2021), which adopted the idea for graph representation learning. The authors of Wagstalf et al. (019) discuss the limitation of Zaheer et al. (2017), and specifically focus on continuous functions. A more general treatment of how to capture complex dependencies among multiset elements is available in Murphy et al. (2019). To the best of our knowledge, this work is the first to build the connection between learning multiset functions with propagations on hypergraph.
7

---

Published as a conference paper at ICLR 2022
Table 1: Dataset statistics: |e| refers to the size of the hyperedges while d, refers to the node degree.
Cora
Citesoer
Pubmed
Cora-CA
DBLP-CA
Zoo
20News
Mushroom NTU2012
ModelNer40
Yelp
House
Walmart
[>]
2708
3312
19717
2708
41302
101
16242
8124
2012
12311
50758
1290
88860
[8]
1579
1079
7963
1072
22363
43
100
298
2012
12311
679302
341
69906
fenture
1433
3703
500
1433
1425
16
100
22
100
100
1862
100
100
# class
7
6
3
7
6
7
4
2
67
40
9
2
11
max
[e|
26
171
43
202
93
2241
1808
2838
81
25
6 EXPERIMENTS
We focus on semi-supervised node classification in the transductive setting. We randomly split the data into training/vllddationness samples using (50%/25% 2 splitting percentages. We aggre- gate the results of 20 experiments using multiple random splits and initializations.
Methods used for comparative studies. We compare our rrasserran and AllDeepSets with ten baseline models, MLP,
CE++GAI HGNN (Feng et al.) (019). HCHA (Bai et al.
2021), HyperGCN (Yadati et al. 2019). HNHN (Dong et al.) 2020), UnIGCNII (Huang & Yang 2021 and HAN (Wang al., 2019b), with both full batch and mini-batch settings. All architec-
tures are implemented using the Pytorch Geometric library (PyG) (Fey & Lenssen 2019) except
for HAN, in which case the implementation was retrieved from Deep Graph Library (DGL) (Wang
ct al. 2019a). The implementation of HyperGCN is used in the same form as reported in the official repository. We adapted the implementation of HCHA from PyG. We also reimplemented HNHN and HGNN using the PyG framework. Note that in the original implementation of H propaga- tion performed via matrix multiplication which is far less memory and computationally efficient when compared to our implementation. MLP, GCN and GAT are executed directly from PyG. The UniGCNII code is taken from the official site For HAN, we tested both the full batch training setting and mini-batch setting provided by DGL. We treated the vertices (V) and hyperedges (E) as two heterogeneous types of nodes, and used two metapaths: (V + 3 + V.E + V + E} in its semantic attention. In the mini-bate setting, we used the DGL's default neighborhood sampler with a number of neighbors set to 32 during training and 64 for validation and testing purposes. In both settings, due to the specific conversion of hyperedges into nodes, our evaluations only involve the original labeled vertices in the hypergraph and ignore the hyperedge node-proxies More details regarding the experimental settings and the choices of hyperparameters
given in the Appendix !
Benchmarking and new datasets. We use ten available datasets from the existing hypergraph neu- ral networks literature and three newly curated datasets from different application domains. The benchmark datasets include cocitation networks Cora, Citeseer and Pubmed, downloaded from Ya- dati al. (2019). The coauthorship networks Cora-CA and DBLP-CA were adapted from Yadati et al. (2019). We also tested datasets from the UCI Categorical Machine Learning Repository (Dua & Graff] 2017)) including 20Newsgroups. Mushroom and Zoo. Datasets from the area of computer vision and computer graphics include ModelNet40 (Wu et al. 2015) and NTU2012 2 (Chen et al.) 2003]. The hypergraph construction follows the recommendations from Feng et al. [2019); Yang ct al. (2020). The three newly introduced datasets are adaptations of Yelp (Yelp). House (Chodrow ctt al. 2021) and Walmart (Amburg et al. 2020). Since there are no node features in the original datasets of House and Walmart,
Gaussian random vectors instead, in fashion similar to what was proposed for Contextual stochastic block models (Deshpande et al.] (2118). We use post- fix (xr) to indicate the standard deviation of the added Gaussian features. The partial statistics of all datasets are provided in Table L and more details are available in the Appendix [
Results. We use accuracy (micro-F1 score) as the evaluation metric, along with its standard devia- tion. The relevant findings are summarized in Table [ The results show that Tranforrrn is the most robust hypergraph neural network model and that has the best overall performance when compared to state-of-the art and new models such a hypergraph HAN. In contrast, all baseline meth- ods perform poorly on at least two datasets. For example, UniGCNI and HAN are two of the best performing baseline models according to our experiments. However, UniGCNII performs poorly on Zoo, Yelp and Walmart when compared to our AllSetTransformer. More precisely, AllSerTrans- former outperforms UniGCNII by 11.01% in accuracy on the Walmart(1) dataset. HAN has poor performance when compared AlISeTTransformer on Mushroom, NTU2012 and ModeINet40. Fur-
htpp://giiiiikcooommmallalllltcp http://gittahccom//oom.. aWHGNN
8

---

Published as a conference paper at ICLR 2022
Table 2: Results for the tested datasets: Mean accuracy (%) + standard deviation. Boldfaced letters
shaded grey are used to indicate the best result, while blue shaded boxes indicate results within one standard deviation of the best result. NA indicates that the method has numerical precision issue. For HAN", additional preprocessing of each dataset is required (see the Section 6 for more details).
Cora
Cieseer
Pubied
Cora-CA
DBLP-CA
Zoo
20Newsgroups
mushroom
AllSeTTransfomer
78.59 = 1.47
73.08 1.20
88.72 0.37
83.63
1.47
9
97,50
3.59
81.38 0.58
100.00 0.00
AllDeepSets
76.88 =: 1.80
70.83 + 1.63
88.75 a 0.33
81.7
a 1.50
91.27 0.27
95.39
4.77
81.06 a 0.54
99.99 1 0.02
MLP
75.17
72.67 = 70.16 1.31
87,47 =
74.31
84.83
4.44
81,42
100.00
CECGN
76.17 1.39
86.45
77.05
+ 1.26
88.00
51.54
11.19
0OM
95.27 + 0.47
CEGAT
76.41 + 1.53
70.63 = 1.30
86.81
76.16
1.19
47.88
14.03
00M
96.60 + 1.67
HNHN
76.36 1.92
72.64 + 1:57
86.90 0.30
77.19
1.49
93.59 + 5.88
81.35 0.61
100.00 0.01
HGNN
79.39 1.36
72.45 = 1.16
86.44
0.44
82.64
+ 1.65
91.03
0.20
92.50 + 4.58
80.33 # 0.42
98.73 # 0.32
HCHA
79.14
72.42 =:
86.41
90.92 0.97
93.65 6.15
98.70 0,39
HyperGCCN
78.45 + 1.26
71:28 +: 0.82
82.84 8.67
79.48
a 2.08
89.38
0.25
N/A
81.05 a: 0.59
47,90 + 1.04
UnIGCNII
78.81
773.05 +
88.25
83.60
91.69
95 4.17
81.12
99.96 0.05
HAN (full batch)"
80.18 1.15
74.05 + 1.43
86.21 0..8
84.04 + 1.02
90.89 # 0.23
85, 19 + 8.18
0OM
90,86 + 2.40
HAN (mini batch)
79.70 1.77
74.12 1.52
85.32 = 2.25
81.71 + 1.73
90.17 + 0.65
75.77 + 7.10
79.72 + 0.62
93.45 + 1.31
NTU2012
ModelNes4U
Yelp
()use(
(almart(1)
Houset0..6 6)
Wallman(0..)
avg. ranking (1)
AllSeTTransfrmer
88.69 1.24
98.20 = 0.20
36,89
0.51
69.33
65,46 = 0.25
83,14 +: 1.92
78.46
0.40
2,00
AllDeepSets
88.09 4: 1.52
96.98 =: 0.26
30.36
1.57
67.82 a 2.40
64.55 a 0.33
80.70 1.59
78.46 0.26
4.47
MLP
85.52 1.49
96.
0.36
31,96 = 0.44
67.93
2.33
45.51
0.24
81.53 + 2.26
63.28
0.37
6.27
CECGN
81.52 + 1.43
0OM
62.80 + 2.61
64.36
2.41
59.78
0.32
9.66
CEGAT
82.21 1.23
92.5220.33
0OM
3.00
0.56
77.25 2.53
59.47 1.05
8.80
HNHN
89.11 1.44
0.25
31.65 = 0.44
67.80
78.78 1.88
65.80
0.39
5.87
87.72 1.35
95.44
33.04
61.39
62.00
66.16 1.80
77.72
HCHA
87.48 =: 1.87
94.48
1: 0.28
30.99
= 0.72
61.36
62.45 1 0.26
67.91 1: 2.26
77.12 0.26
6.40
HyperGGN
56.36 +: 4.86
75.89
5.26
29.42 1.54
48.31 + 2.93
44.74 2.81
78.22 +: 2.46
55,31 0.30
9.87
UniGCNII
89.30 1.33
0.23 98.07
31.70 0.52
67.25
+ 2.57
54,45
80.65 1.96
72.08 0,28
3.87
HAN (full batch)"
83.58 + 1646
04.04:00:4
0OM
71.05
a: 2.26
0OM
83.27 # 1.62
00M
6.73
HAN (mini batch)"
80.77 2.36
91.52
0.96
26.05
62.00
48.57
82.04 68
63.10
7.60
thermore, it experiences memory overload issues on Yelp and Walmart with a full batch setting and performs poorly with mini-batch setting on the same datasets. Numerically, Tanseerrr outperforms HAN with full batch setting by 9.14% on the Mushroom and HAN with mini-batch setting and by 16.89% on Walmart(1). These findings emphasize the importance of including di- verse datasets from different application domains when trying to test hypergraph neural networks fairly. Testing standard benchmark datasets such as Cora, Citeseer and Pubmed alone is insufficient and can lead to biased conclusions. Indeed, our experiments show that all hypergraph neural net- works work reasonably well on these three datasets. Our results also demonstrate the power of Set Transformer when applied to hypergraph learning via the AllSet framework.
The execution times of all methods are available the Appendix J They show that the complexity of AllSet is comparable to that of the baseline hypergraph neural networks. This further strengthens the case for AllSet, as its performance gains do not come at the cost of high computational complexity. Note that in HAN's mini-batch setting, the neighborhood sampler is performed on CPU instead of GPU, hence its training time is significantly higher than that needed by other methods, due to the frequent 1/O operation between CPU and GPU. On some larger datasets such as Yelp and Walmart, 20 runs take more than 24 hours so we only recorded the results for first 10 runs.
The performance of AllDeepSets is not on par with that of AlISeeTransformer, despite the fact that both represent universal approximators of the general AllSet formalism. This result confirms the assessment Lee et al. 2019) that attention mechanisms are crucial for learning multiset functions in practice. Furthermore, we note that combining CE with existing GNNs such as GCN and GAT leads to suboptimal performance. CE-based approaches are also problematic in terms of memory efficiency when large hyperedges are present. This is due to the fact the CE of hyperedge e leads to @(le()) edges in the resulting graph. Note that as indicated in Table 2 CE-based methods went out-of-memory (OOM) for 20Newsgroups and Yelp. These two datasets have the largest maximum hyperedge size, as can be see from Table [ HAN encounters the OOM issue on even more datasets when used in the full batch setting, and its mini-batch setting mode perform poorly on Yelp and Walmart. This shows that a naive application of standard heterogeneous GNNs on large hypergraphs often fails and is thus not as robust our AllSeTTransfomer. The mediator HyperGCN approach exhibits numerical instabilities on some datasets (i.e., Zoo and House) and may hence not be suitable for learning on general hypergraph datasets.
9

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
The authors would like thank Prof. Pan Li Purdue University for helpful discussions. The au- thors would also like to thank Chaoqi Yang for answering questions regarding the LEGCN method. This work was funded by the NSF grant 1956384.
ETHICS STATEMENT
We do not aware of any potential ethical issues regarding work. For the newly curated three datasets, there is no private personally identifiable information. Note that although nodes in House datasets represent congresspersons and hyperedges represent members serving on the commit- tee, this information is public and not subject to privacy constraints. For the Walmart dataset, the nodes are products and hyperedges are formed by sets of products purchased together. We do not include any personal information regarding the buyers (customers) in the text. In the Yelp dataset the nodes represent restaurants and hyperedges are collections of restaurants visited by the same user. Similar to the Walmart case, there is no user information included in the dataset. More details of how the features in each datasets are generated can be found Appendix
REPRODUCIBILITY STATEMENT
We have tried our best to ensure reproducibility of our results. We built succinet pipeline for stan- dardization of the hypergraph neural networks evaluation process based on Pytorch Geometrie. This implementation can be checked by referring to our supplementary material. We include all dataset in our supplementary material and integrate all tested methods in our code. Hence, one can simply reproduce all experiment results effortlessly (by just running a one-line command). All details re- garding how the datasets were prepared stated in Appendix[] [ All choices of hyperparameters and the process of selecting them are available in Appendix J Our experimental settings including the specifications of machine and environment, the training/validatiionnnes split and the evaluation metric. Once again, refer ro the Appendix[Jame and Section (6 dependencies used in our code in the Supplementary material.
We also clearly specify all the package
REFERENCES
Sameer Agarwal, Jongwoo Lim, Lihi Zelnik-Mandn Pietro Perona, David Kriegman, and Serge Belongie. Beyond pairwise clustering. In 2005 IEEE Computer Society Conference on Computer Vision and Paltern Recognition (CVPR (05), volume 2, pp. 838-845. IEEE, 2005.
Sameer Agarwal, Kristin Branson, and Serge Belongie. Higher order learning with graphs. In Proceedings of the 23rd international conference on Machine learning, pp. 17-24, 2006.
Ilya Amburg, Nate Veldt, and Austin Benson. Clustering in graphs and hypergraphs with categorical edge labels. In Proceedings of The Web Conference 2020, WWW '20, pp. 706-717, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450370233. doi: 10.1145/
3366423.3380133 URL
3380152
Devanshu Arya, Deepak K Gupta, Stevan Rudinac, and Marcel Worring. Hypersage: Generalizing inductive representation learning on hypergraphs. arXiv preprint arXiv:2010.04538. 2020.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 1oXiv:1607.0644 2016.
Jinheon Baek, Minki Kang, and Sung Ju Hwang. Accurate learning of graph representations with
graph multiset pooling. In International Conference on Learning Representations, 2021. URL
net twtpttertttaatteee HirrmmmidddHHHHGGaiii
Song Bai, Feihu Zhang, and Philip HS Torr. Hypergraph convolution and hypergraph attention. Pattern Recognition, 10: 107637, 2021.
Austin R Benson. Three hypergraph eigenvector centralities. SIAM Journal on Mathematics of Data Science, 1(2):293-312, 2019.
10

---

Published as a conference paper at ICLR 2022
Austin R Benson, David F Gleich, and Jure Leskovec. Higher-order organization of complex net-
works. Science, 353(6295):163-166 2016.
Austin R Benson, David F Gleich, and Lek-Heng Lim. The spacey random walk: stochastic process for higher-order data. SIAM Review, 59(2):321-455 2017.
Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geomet- ric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine, 34(4):18 42, 2017.
T-H Hubert Chan, Anand Louis, Zhihao Gavin Tang, and Chenzi Zhang. Spectral properties of hypergraph laplacian and approximation algorithms. Journal of the ACM (JACM), 65(3):1-48, 2018.
Ding- Yun Chen, Xiao-Pei Tian, Yu-Te Shen, and Ming Ouhyoung. On visual similarity based 3d model retrieval. Computer graphics forum, volume 22, pp. 223-232. Wiley Online Library, 2003.
Wei-Lin Chiang, Xuanqing Liu, Si Si, Yang Li, Samy Bengio, and Cho-Jui Hsich. Cluster-gen: An efficient algorithm for training deep and large graph convolutional networks. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp- 257-266, 2019.
Eli Chien, Pan Li, and Olgica Milenkovic. Landing probabilities of random walks for seed-set expansion in hypergraphs. IEEE Information Theory Workshop (ITW), 2021a.
Eli Chien, Jianhao Peng, Pan Li, and Olgica Milenkovic. Adaptive universal generalized pagerank graph neural network. In International Conference on Learning Representations, 2021b. URL
htttps://openreviee eww. net / forum?id=n6]1
I Chien, Chung-Yr Yi Lin, and I-Hsiang Wang. Community detection in hypergraphs: Optimal sta- tistical limit and efficient algorithms. In International Conference Artificial Intelligence and Statistics, pp. 871-879. PMLR, 2018.
I Eli Chien, Chung-Y Yi Lin, and I-Hsiang Wang. On the minimax misclassification ratio of hy- pergraph community detection. IEEE Transactions Information Theory, 65(11))8095-811 81888, 2019a.
I Eli Chien, Huozhi Zhou, and Pan Li. Hs2: Active learning over hypergraphs with pointwise and pairwise queries. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 2466-2475. PMLR, 2019b.
Philip S Chodrow, Nate Veldt, and Austin R Benson. Hypergraph clustering: from blockmodels to modularity. arXiv preprint arXiv:2101.09611 2021.
Yash Deshpande, Subhabrata Sen, Andrea Montanari, and Elchanan Mossel. Contextual stochastic block models. Advances in Neural Information Processing Systems 31 (NIPS 2018), 2018.
Kaize Ding, Jianling Wang. Jundong Li, Dingcheng Li, and Huan Liu. Be more with less: Hy- pergraph attention networks for inductive text classification. arXiv preprint 1.X0822011.00387, 2020.
Yihe Dong, Will Sawin, and Yoshua Bengio. Hnhn: Hypergraph networks with hyperedge neurons. arXiv preprint arXiv:2006. 12278, 2020.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL URLpt:p://archii
ics. uci edu/ml
Yifan Feng, Zizhao Zhang, Xibin Zhao, Rongrong Ji, and Yue Gao. Gvenn: Group-view convo- lutional neural networks for 3d shape recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 272, 2018.
Yifan Feng, Haoxuan You, Zizhao Zhang, Rongrong Ji, and Yue Gao. Hypergraph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3568, 2019.
11

---

Published as a conference paper at ICLR 2022
Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometrie. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Tch, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In International Conference Machine Learning, pp. 1704-177 1713. PMLR, 2018.
Antoine Gautier, Francesco Tudisco, and Matthias Hein. A unifying perron for nonnegative tensors via multihomogeneous maps. SIAM Journal Matrix Analysis and Applications, 40(3): 2019.
perron-frobente frobenius theorem
Debarghya Ghoshdastidar and Ambedkar Dukkipati. A provable generalized tensor spectral method for uniform hypergraph partitioning. In International Conference on Machine Learning, pp. 400- 409. PMLR, 2015.
Justin Gilmer, Samuel Schoenholz, Patrick Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In International Conference Machine Learning, pp- 1263-1272. PMLR, 2017.
David F Gleich, Lek-Heng Lim, and Yongyang Yu. Multilinear pagerank. SIAM Journal on Matrix Analysis and Applications, 36(4):1507-1511, 2015.
William L Hamilton, Rex Ying. and Jure Leskovec. Inductive representation learning on large graphs. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 1025-1035, 2017.
Jun He and Ting-Zhu Huang. Upper bound for the largest z-eigenvalue of positive tensors. Applied Mathematics Letters, 38:110-114, 2014.
Matthias Hein, Simon Setzer, Leonardo Jost, and Syama Sundar Rangapuram. The total variation on hypergraphs-learning on hypergraphs revisited. In Proceedings of the 26th International Con- ference on Neural Information Processing Systems- Volume 2, pp. 2427 2435, 2013.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 22118-22133 Curran Asso- ciates, Inc., 2020a. URL Eb60d41la5c55 wwwpee27222cccce2e22222222222
neurips www.wnw.w..pppppeeeeeeee wctppaper/222//11eeee... pdf
Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. Heterogeneous graph transformer. In Proceedings of The Web Conference 2020, pp. 2704-2710. 2020b.
Jing Huang and Jie Yang. Unignn: a unified framework for graph and hypergraph neural networks. arXiv preprint arXhv:2105.00956, 2021.
Glen Jeh and Jennifer Widom. Scaling personalized web search. In Proceedings of the 12th inter- national conference World Wide Web, pp. 271-279, 2003.
Junteng Jia and Austion Benson. Residual correlation in graph neural network regression. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 388-558, 2020.
Jachyeong Jo, Jinheon Back, Seul Lee, Dongki Kim, Minki Kang, and Sung Ju Hwang. Edge representation learning with hypergraphs. Advances in Neural Information Processing Systems, 34, 2021.
Cliff Joslyn and Kathleen Nowak. Ubergraphs: A definition of a recursive hypergraph structure. arXiv preprint arXiv: 1704.05547, 2017.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net-
works. In International Conference Learning Representations (ICLR), 2017.
12

---

Published as a conference paper at ICLR 2022
Johannes Klicpera, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. In International Conference on Learning Representations, 2019.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Tch. Set transformer: A framework for attention-based permutation-invariiat neural networks. In Interna- fional Conference on Machine Learning, pp. 3744-3753. PMLR, 2019.
Guoyin Li, Liqun Qi, and Gaohang Yu. The z-eigenvalues of symmetric tensor and its application to spectral hypergraph theory. Numerical Linear Algebra with Applications, 2016):0011-1029, 2013.
Pan and Olgica Milenkovic. Inhomogeneous hypergraph clustering with applications. Advances in Neural Information Processing Systems, 2017:2309-2311 2017.
Pan Li and Olgica Milenkovic. Submodular hypergraphs: p-laplacians, cheeger inequalities and spectral clustering. In International Conference on Machine Learning, pp. 3014-3023. PMLR, 2018.
Pan Li, Hoang Dau, Gregory Pulco, and Olgica Milenkovic. Motif clustering and overlapping clustering for social network analysis. In IEEE INFOCOM 2017-IEEE Conference on Computer Communications, pp. 1-9. IEEE, 2017.
Pan Li, Gregory Pulco, and Olgica Milenkovic. Motif and hypergraph correlation clustering. IEEE Transactions Information Theory, 6665):3065-3078, 2019.
Pan Li, Yanbang Wang. Hongwei Wang. and Jure Leskovee. Distance encoding: De- sign provably more powerful neural networks for graph representation learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 4465 4478. Curran Asso- ciates, Inc., 2020. URL /tttpps:/pprcccednggs hewpoeeeeddnngs..eeii.......p httppsscc/ppppr/////// pc/pppper/222///1ee//
2f731
pdf
Yujia Li, Daniel Tarlow, Mare Brockschmidt, and Richard Zemel. Gated graph sequence neural networks. arXiv preprint arXiv:157 2015.
Derek Lim, Xiuyu Li, Felix Hohne, and Ser-Nam Lim. New benchmarks for learning on non- homophilous graphs. arXiv preprint arXiv:2104 0.40440..44.....00110 2021.
Lek-Heng Lim. Singular values and eigenvalues of tensors: a variational approach. In 1st IEEE International Workshop Computational Advances in Multi-Sensor Adaptive Processing, 2005., pp. 129-132. 132. IEEE, 2005.
David Lopez-Paz, Robert Nishihara, Soumith Chintala, Bernhard Scholkopf, and Léon Bottou. Dis- covering causal signals in images. In Proceedings of the IEEE Conference on Computer Vision and Pattem Recognition, pp. 1979-69877, 2017.
Linyuan Lü and Tao Zhou. Link prediction in complex networks: A survey. Physica A: statistical mechanics and its applications, 19006))1500-11700 2011.
Jiaqi Ma. Bo Chang. Xuefei Zhang, and Qiaozhu Mei. CopulaGNN: Towards integrating repre- sentational and correlational roles of graphs in graph neural networks. In International Confer- ence on Learning Representations, 2021. URL
net forett/forrmmsa
XI XI-OJ5yyse
Ryan L. Murphy, Balasubramaniam Srinivasan, Vinayak Rao, and Bruno Ribeiro. Janossy pooling: Learning deep permutation invaariantnnnnii functions for variable-size inputs. In International Confer-
ence on Learning Representations, 2019. URL pettps://openrette wew.rreviwww... net. www.:nnttfoommmmmm
BJ BJluy2RcFm
Lawrence Page. Sergey Brin, Rajeev Motwani, and Terry Winograd. The pagerank citation ranking:
Bringing order to the web. Technical report, Stanford InfoLab, 1999.
13

---

Published as a conference paper at ICLR 2022
Kelly J Pearson and Tan Zhang. On spectral hypergraph theory of the adjacency tensor. Graphs and Combinatorics, 30(5))1233-111 1248, 2014.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 652-660, 2017.
Liqun Qi and Ziyan Luo. Tensor analysis: spectral theory and special tensors. SIAM, 2017.
Yu Rong. Wenbing Huang, Tingyang Xu, and Junzhou Huang. Dropedge: Towards deep graph convolutional networks node classification. In International Conference Learning Repre- sentations, 2019.
Nino Shervashidze, Pascal Schweitzer, Erik Jan Van Leeuwen, Kurt Mehlhorn, and Karsten M Borg- wardt. Weisfeiler-Iemman graph kernels. Journal of Machine Learning Research, 12(9), 2011.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp.
4080-4090. 2017.
Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Lealned-Mller. Multi-view convo- lutional neural networks for 3d shape recognition. In Proceedings of the IEEE international conference on computer vision, pp. 945-953, 2015.
Francesco Tudisco and Desmond Higham. Node and edge nonlinear cigenvector centrality for hypergraphs. Communications Physics, 4(1):1-10, 2021.
Francesco Tudisco, Austin R Benson, and Konstantin Prokopchik. Nonlinear higher-order label spreading. arXiv preprint MwXiv:2006.04762 2020.
Francesco Tudisco, Konstantin Prokopchik, and Austin R Benson. A nonlinear diffusion method for semi-supervised learning on hypergraphs. arXiv preprint arXiv:2103. 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances Neural Informa- tion Processing Systems, 30:5998-6008 2017.
Petar Velickovic, William Fedus, William Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. In ICLR (Poster), 2019.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations, 2018. URLAtttss:/ wtt.::/eooom...dn.ttaaaaaalaaal rXXMpik22
Edward Wagstaff, Fabian Fuchs, Martin Engelcke, Ingmar Posner, and Michael A Osborne. On the limitations of representing functions on sets. In International Conference Machine Learning, pp. 6487-6494. PMLR, 2019.
Jianling Wang, Kaize Ding, Ziwei Zhu, and James Caverlee. Session-based recommendation with hypergraph attention networks. Proceedings of the 2021 SIAM International Conference on Data Mining (SDM), pp. 82-90. SIAM, 2021.
Minjie Wang, Da Zheng, Zihao Ye, Quan Gan, Mufei Li, Xiang Song. Jinjing Zhou, Chao Ma, Lingfan Yu, Yu Gai, Tianjun Xiao, Tong He, George Karypis. Jinyang Li. and Zheng Zhang. Deep graph library: graph-centrie, highly-performant package for graph neural networks. arXiv preprint arXiv: 1909.0/319, 2019a
Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S Yu. Heterogeneous graph attention network. In The World Wide Web Conference, pp. 2022-2032, 2019b.
Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Sim- plifying graph convolutional networks. In International conference on machine learning. pp- 6861-6871. PMLR, 2019.
14

---

Published as a conference paper at ICLR 2022
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1912-1920, 2015.
Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In International Conference on Machine Learning, pp. -462-5462. PMLR, 2018.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference Learning Representations, 2019. URL https:
//openreview.
Am
Hansheng Xue, Luwei Yang, Vaibhav Rajan, Wen Jiang, Yi Wei, and Yu Lin. Multiplex bipartite network embedding using dual hypergraph convolutional networks. In Proceedings of the Web Conference 2021, pp. 1649-1660, 2021.
Naganand Yadati. Neural message passing for multi-relational ordered and recursive hypergraphs. Advances in Neural Information Processing Systems, 33, 2020.
Naganand Yadati, Madhav Nimishakavi, Prateek Yadav, Vikram Nitin, Anand Louis, and Partha Talukdar. Hypergen: A new method for training graph convolutional networks on hyper- graphs. In H. Wallach, H. Larochelle, A. Beygelzimer, F. Alché-Buc, E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL http:: /tttpp://ppooceeddgggs...u neurips........pp..p..........s. http://wwwsss.cc/ppp//////////1
(efa39Dcaecf30000111166623333333 Paper. pdf
Chaoqi Yang, Ruijic Wang, Shuochao Yao, and Tarek Abdelzaher. Hypergraph learning with line expansion. arXiv preprint arXi::2005.04843 2020.
Yelp. Yelp business dataset. URL httpt::////// /ww::://wwwww.. yelp www.datase...........
Seongjun Yun, Minbyul Jeong, Rachyun Kim, Jaewoo Kang, and Hyunwoo J Kim. Graph trans- former networks. H. Wallach, H. Larochelle, A. Beygelzimer, F. d Alché-Band Buc, E. Fox, and
R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019. URL 9d63484abb4 770003/7777777700000000099933333 Paper pdf
Manzil Zaheer, Satwik Kottur, Siamak Ravanbhakhsh, Barnabás Póczos, Ruslan Salakhutdinov, and Alexander J Smola. Deep sets. In Proceedings of the 31 International Conference on Neural Information Processing Systems, pp. 3394-3404, 2017.
Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor Prasanna. Graph- saint: Graph sampling based inductive learning method. In International Conference Learning Representations, 2019.
Ruochi Zhang, Yuesong Zou, and Jian Ma. Hyper-sagnn: self-attention based graph neural network for hypergraphs. arXiv preprint arXiv:1911.02613. 2019.
Lingxiao Zhao and Leman Akoglu. Pairnorm: Tackling oversmoothing in gnns. In International Conference on Learning Representations, 2019.
Dengyong Zhou, Jiayuan Huang, and Bernhard Schölkopf. Learning with hypergraphs: Clustering, classification, and embedding. Advances in neural information processing systems, 19:1601- 1608, 2006.
Xiaojin Jerry Zhu. Semi-supervised learning literature survey. Technical Report, 2005.
15

---

Published as a conference paper at ICLR 2022
APPENDIX
A ADDITIONAL RELATED WORKS
We further discuss related works for completeness.
Graph neural networks on graphs. GNNs have received significant attention in the past, often under the umbrella of Geometric Deep Learning (Bronstein et al.) 2017). Many successful archi- tectures have been proposed for various tasks graphs, including GCN (Kipf & Welling 2017),
GAT (Vaswani et al., 2017). GraphSAGE (Hamilton et al. 2017). GIN (Xu et 2019)) and many
others (Klicpera et al. 2019 Wu et al.) 2019 Velickovic al. 2019 Li al. 2020 Xu et al. 2018
Chien et al.
2021b). There have also been some recent attempts to use Transformers in GNNs (Back)
al... 2021 and not to hypergraphs. Although CE can be used in all these case, this is clearly not an optimal strategy due to the previously mentioned distortion issues. There are also many techniques that can improve GNNs in various directions. For example, PairNorm (Zhao & Akoglu (2019) and DropE- dge (Rong al., 2019) allow one build deeper GNNs. ClusterGCN (Chiang et al. 2019) and GraphSAINT (Zen et al. 2019), on the other hand, may be used to significantly scale up GNNs. Residual correlation (Jia Benson (020) and CopulaGNN (Ma et al.) (021) have been shown to improve GNNs on graph regression problems. These works highlight new directions and limitations of not only our method, but also hypergraph neural networks in general. Nevertheless, we believe that AllSet layers can be adapted to resolve these issues using similar ideas those proposed for graph layers.
Yun al., 2019 Hu et al. 2020b). Nevertheless, all these methods only apply to graphs
Additional contributions to learning hypergraphs. After the submission of our manuscript, we were made aware of some loosely related lines of works. Ding et al. (2020) and Wang et al.) V formulation, spe- cialized for certain downstream tasks. However, these approaches fall under our AllSet framework and have smaller expressiveness compared AllSet. The work Jo et al. (202T) proposes learn edge representations in a graph via message passing on its dual hypergraph. Although the authors used GMT (Back et al., (022)) as their node pooling module, they do not explore the idea of treating fv-s and fg-v as multiset functions for hypergraph neural networks as AllSet does. It is also possible to define propagation schemes based on other tensor eigenproblems, such as eigenprob- lems (Qi & |uo] 2017). Defining propagation rules based on H eigenproblems can lead to imaginary features which is problematic: A more detailed discussion regarding this direction is available the
(2021) proposed hypergraph attention networks based the V - & and
Appendix H
B CONCLUSIONS
We proposed AllSet, a novel hypergraph neural network paradigm that represents a highly general framework for hypergraph neural networks. We implemented hypergraph neural network layers as compositions of two multiset functions that can be efficiently learned for each task and each dataset. Furthermore, for the first time, we integrated Deep Set and Set Transformer methods within hyper- graph neural networks for the purpose of learning the described multiset functions. Our theoretical analysis demonstrated that most of the previous hypergraph neural networks are strietly less expres- sive then our proposed AllSet framework. We conducted the most extensive experiments to date involving ten known benchmarking datasets and three newly curated datasets that represent signif- icant challenges for hypergraph node classification. The results showed that our method has the unique ability to either match or outperform all other baselines, including the state-of-the-an hyper- graph neural networks UniGNN (Huang & Yang 2021). Remarkably, our baseline also included HAN (Wang et al., 2019b). the heterogeneous hypergraph neural networks, via star expansion as the new comparative method in the hypergraph neural network literature for the first time. Our results show that leveraging powerful multiset function learners in our AllSet framework is indeed helpful for hypergraph learning. As a promising future direction, more advanced multiset function learners such as Janossy pooling (Murphy et al. 2019) can be used in our AllSet framework and potentially further improve the performance. We left this topic in future works.
16

---

Published as a conference paper at ICLR 2022
C PROOF OF THEOREM 3.3
First, we show that one can recover CEpropH I from AllSet ( By choosing fa+g and fE-vV to be the sums over appropriate input multisets and by ignoring the second input, we obtain
z(+1) = fv- (V. (())
(0)
(9)
BEV ei
uEe
x(:1) fe-v (E, 2
E
I=
Σ
Z
(10)
BEE 2,2(11)
COEER
Combining these two equations gives
((+1)
Σ
EX
X
(11)
aEe.
which matches the update equations of CEpropH
Next, we prove that one can recover CEpropA ! from AllSet (6). We once again choose fy-e and
fe-v to be the sums over appropriate input multisets and ignore the second input term. This results in the following update equations:
(1+1)), = =Iv-re(V (Ver, 2000)
T
(12)
zEV el,,(())
REelo
((+1) Je-vv( 20,
Σ
(1
(13)
BEE v,2 1)e
Combining the two equations we obtain
x(++1)
(1+1)a
Σ
Σ uEel#
X(9
(14)
which represents the update rule of CEpropA from
In the last step, we prove that one can recover Zprop (4) from AllSet ( By choosing fy-e and
fe-v to be the product and sum over appropriate input multisets, respectively, and by ignore the second input, we have:
((+1),, = -V-se(Ve (Vere.xis; Z()0)
II
x=
II X
(15)
zEV e(e,x(t)
(++1) by
(E-
Σ
x
(1+1),
(16)
#EE, 1. (12((-1)
EIVEE
Combining these two equations arrive at
II X
(17)
e:VEé WEele
which matches Zprop from This completes the proof.
D PROOF OF THEOREM 3.4 3.4
First prove that the hypergraph neural network layer of HGNN AllSet (5). By choosing fV-E and fs to be the weighted sum over appropriate input multi- sets, where the weights are chosen according to the node degree, edge degree, and edge weights, and by ignore the second input, we have:
2
special instance of
( = (V (.2)))
(18)
Vila
wse
x ((+1) Se-v
x
(1))
Ze.
(+)
(t)
(4)
(19)
Vil
ee
e:vee
17

---

Published as a conference paper at ICLR 2022
Since fv-Eg and fe-v both use the hypergraph 9 as their input, the functions the node de- grees, edge degrees and edge weights as their arguments. Also, is not hard to see that our choices for f 18 and fe-v (19) permutation invariant in V. x and E, z(+11 respectively. Plug-
ging 18 into 19
leads to:
(++1)
(0) + b
(20)
Vd
de
eleEe
UEe
which represents the HGNN layer (2).
For the HCHA layer, we have node-wise formulation that reads as follows:
( dp
X x =
We
Σ uuuE
a (!)
O + (())
(21)
BIVEE
where the attention weight Oue depends on the node features ( and the feature of hyperedge e. Here, a(-) is a nonlinear activation function such as LeakyReLU and eLU. An analysis similar to that described for HGNN can be used in this case well. The only difference is in the choice of the attention weights Oue and One. The definition attention weights for HCHA is presented below.
exp(or(a X
Oue Eeuse exp(o((a X
(22)
Clearly, the attention function in has V e() and Zo as its arguments. Furthermore, we can choose fV-E and f&- as:
z(41) fv-ee (Vex0
( Σ al
x()( U1:!!
(23)
woave Ze (t) (++1)) (++1))
(t)
((++1) x(++1)== = Se-v (Fg, x a
+ b()
(24)
0,
le]
CRIE
Note that both 23 and (24) are permutation invariant, which means that they are valid choices for fv-se and Se-vy. Combining these two equations reproduces the HCHA layer of (2T)
For the HyperGCN layer, ignoring the degree normalization, one can use the following formulation:
((
[
((+1)) 0.
(( LE: au.e
Σ
(t) X
(+) + +b)))
(25)
Here, the weight (1) depends on all node features within the hyperedge (x(()) E c}, and ((.) is nonlinear activation function (for example, ReLU). The same analysis presented for the
previous cases may used this case as well: The only difference lies the choice of the weights
wae.e. According to the original HyperGCN paper (Yadati et al. 2019). these weights are defindd as
if (ierje) or (ierje), otherwise.
w u0.e
2
(26)
where (Je, = argmax, o.e
(+))
(27)
Again, it is straightforward to that with e is function of V. Ve,r, X and x Also, is permuta- tion invariant with respect to V. Velu.x Hence, we can choose fv4e and fE
7 (++)), fv Velpxx00 700 a.
Σ
(a) (+)
(28)
w
(1+1) Se-v(En x X
= Σ
(1++)).0 +b(0)
(29)
e:vEc
18

---

Published as a conference paper at ICLR 2022
Combining the two equations leads to the HyperGCN layer from 25
Next, we show that HNHN layer introduced in Dong et al. 2020 is a special case of AllSet (5). The definition of HNHN layer is as follows
Z ((+1)
Σ
X
O + bbg)
(30)
de,i,3
NEE
x(++1) =o
denazte
(1+1) b.:
6 + bb)
(31)
duddic eioEe
where 1
Σ REe
|dul% dur, = dii dod,8 =
Σ eivE
|d duraa dz.
(32)
Note that and / are two hyperparameters that can be tuned in HNHN. As before, o-)) is non- linear activation function (c.g., ReLU). It is obvious that we can choose fv-s and fE-v according to (30) and (31), respectively. This is due the fact that these expressions only involves degree normalizations and linear transformations.
Finally, we show that the HyperSAGE layer from Arya et al. (2020) is special case of AllSet (5)
The HyperSAGE layer update rules are as follows:
1/8
(((+))
Σ
(33)
uEe
1/p
X ((+1)++
[e:vec] EIDEE
Σ (Z!!)
+
(() 0,2t
(34)
(1+1),+
X (t+1)
(((+1))
(35)
where (-) is a nonlinear activation function. The update (33) can recovered by simply choosing fV-LE to be the lp (power) mean. For the fe-y component, of another two multiset functions. The first is the lp (power) mean with respect to its first input, while the second is addition with respect to the second input. This recovers (34). The second of the two defining functions can be chosen as (35). which is also a multiset function. This procedure leads fE-v. This completes the proof of the first claim pertaining to the universality of AllSet.
first model f&-V as composition
To address the claim that the above described hypergrpah neural network layers have strictly smaller expressive power then AllSet, one only has observe that these hypergraph neural network layers cannot approximate arbitrary multiset functions in either the fv-t or fE-v component. We discuss both these functions separately.
With regards to the HGNN layer, it is clear that two linear transformations cannot model arbitrary multiset functions such as the product in Zprop (4). For the HCHA layer, we that if all node features are identical and the hyperedge features are all-zero vectors, all the attention weights Que are the same. This setting similar to that described for the HGNN layer and thus this rule cannot model arbitrary multiset functions. For the HyperGCN layer (25). consider 3-uniform hypergraph as its input. In this case, by definition, the weights Wwe.e are all equal. Again, using only linear transformations one cannot model the product operation in Zprop (4); the same argument holds even when degree normalizations are included. For the HNHN layer, note that the E + V component of (31) is just one MLP layer that follows the Σ Eencer which cannot model arbitrary multiset functions based on the results of Deep Sets (Zaheer et al. 2017). As a final note, we point out that the HyperSAGE layer involves only one learnable matrix e and hence also linear transformation, which cannot model arbitrary multiset functions.
This completes the proof.
19

---

Published as a conference paper at ICLR 2022
E
PROOF OF THEOREM 3.5
The propagation rule of MPNN reads as follows:
(++1)
Σ
M.(X X " X (X m D)
(36)
(EN(w)
Here, m denotes the message obtained by aggregating messages from the neighborhood of the node while Me and U, certain functions selected the I-th step of propagation. To recover MPNN from AllSet (6). we simply choose fV-5 and fE
((+1), SV- X ( -v- (1+1),1
z
sLuE
((+1) A (E, (()) = fe-v (1 (Z
Jewwee
=
M
().
x!! = U,
X
Σ
M((X() X( 6.11
(37)
caccc
coCc
Hence, MPNN a special case of AllSet for graph inputs.
F PROOF OF PROPOSITION 4.
The proof largely based on the proof Proposition the Set Trasnformer paper (Lee et al. 2019) but is nevertheless included for completeness. We ignore all layer normalizations in the proof (Lee al. 2019) and start with the following theorem.
Theorem F.1 (Theorem in Lee al. (2019)). Functions of the form MLP Σ (MLP) are universal approximators in the space of permutation invariant functions.
This result is based on Zaheer et al. 2017). According to Wagstaff et al. 2019), we have the constraint that the input multiset has to be finite.
Next, we show that the simple mean operator a special case of a multihead attention module. For simplicity, consider the case of single head, which corresponds to h 1, as can always set the other heads to zero by choosing MLPV. (Ppp)(()) = 0 for all 2 2. Following the proof of Lemma in Lee et al. (2019). we set the learnable weight 0 to be the all-zero vector and use w(-) = + g(-) (element- wise) activation function such that (0) = O. The MH1 module in 8 then becomes
MH (0,S,)) (1) = (0(MLPK.)S))) MLPKI (S)
Σ MLP (S).
(38)
The AllSerTranformer layer takes the form
[S]
[v-c(S) Y MLP(Y), where Y Σ MLP (S). (S).
(39)
1=1
Note that Y is vector. Clearly, we can choose MLP such that MLP(Y) equals to another MLP(Y) that results in subtracting Y. Thus, we have:
S
(S) MLP(Y) MLP
Σ MLPKIN (S)
(40)
By Theorem F.I it is clear that (S) is universal approximator for permutation invariant functions. The same analysis applies to fE (S-
G A DISCUSSION OF LEGCN
Line expansion (LE) is a procedure that transform a hypergraph or a graph into a homogeneous graph; node in the LE graph represents pair of node-hyperedge from the original hypergraph.
20

---

Published as a conference paper at ICLR 2022
Nodes in the LE graph are linked if and only if there is nonempty intersection of their associ- ated node-hyperedge pairs. As stated by the authors of LEGCN (Yang et al. 2020). for a fully connected d-uniform hypergraph their resulting LE graph has @ E) nodes and (id This expansion hence leads to very large graphs which require large memory units and have high computational complexity when coupled with GNNs such as GCN. To alleviate this drawback, the authors use random sampling techniques which unfortunately lead to an undesired information loss. In contrast, we define our AllSet layers directly on hypergraph which leads significantly more efficient approach.
edges.
H HYPERGRAPH PROPAGATION RULES BASED ON THE H EIGENPROBLEM
As outlined in the main text, in this case we associate a d-uniform hypergraph & with an adjacency tensor A. The H eigenproblem for A states that
Ax Ax(d-11,
where x! [d-1) (x.)d-1.
(41)
Similar to Zprop, can define Hprop (41 in node-wise fashion
Hprop: ((++1)
Σ
(d-
II
( U.
(42)
e:eCe
Although taking the 1/(d 1) root resolves the unit issue that exists in Zprop (4). it may lead to imaginary features during updates. It remains an open question to formulate Hprop in a manner that ensures that features remain real-valued during propagation.
I ADDITIONAL DETAILS PERTAINING TO THE TESTED DATASETS
Table 3: Full dataset statistics:
|e| refers to the size of the hyperedges while d, refers to the node
degree.
Cora
Citeseer
Pubmed
Com-CA
DBLP-CA
Zoo
20News
Mushroom
NTU2012
ModelNet40
Yelp
House
Walmart
|>]
2708
3312
19717
2708
41302
101
16242
8124
2012
12311
50758
1290
88860
[8]
1579
1079
7963
1072
22363
100
298
2012
12311
679302
341
69906
Deature
1433
3703
500
1433
1425
16
100
22
100
100
1862
100
100
# class
7
6
3
7
6
7
2
67
40
2
11
max le]
min [e]
2
2
2
2
29
2
avg
3.03
3.2
4.35
4,28
39.93
136.31
34.72
med le]
3
2
3
3
3
40
537
72
40
mux d.
145
99
23
19
30
7855
44
5733
min d,
17
avg d.
1.77
1.04
1.76
1.69
2.41
17
4.03
5
5
89.
12
9.18
5.18
d,
We use 10 available benchmark datasets from the existing hypergraph neural networks literature and introduce three newly created datasets as described in the exposition to follow. The benchmark
datasets include cocitation networks Cora, Citeseer and Pubmed obtained from Yadati et al. 2019).
The coauthorship networks Cora-CA and DBLP-CA are also adapted from Yadati et al. 2019
In the cocitation and coauthorship networks datasets, the node features are the bag-of wooddds repre- sentations of the corresponding documents. Datasets from the UCI Categorical Machine Learning Repository (Dua & Graff 2017) include 20Newsgroups, Mushroom and Zoo. In 20Newsgroups, the node features are the TF- IDF representations news messages. In the Mushroom dataset, the node features represent categorical descriptions of 23 species of mushrooms. In Zoo, the node features a mix of categorical and numerical measurements describing different animals. The computer vision and graphics datasets include the Princeton CAD ModelNet40 (Wu al.) 2015) and the NTU2012 3D dataset (Chen et al.) 2003). The visual objects contain features extracted using Group- View Convolutional Neural Network(GVCCN)) (Feng et 2018 and Multi- View Convolutional Neural
https://lings.ooee www.:uueess..ee.....ee http://ammierrrogyllllllssseeeet http:////aaatttooooaaoeaaaaaaaaa
http://ppoppe.coo http://ppppll.......... wwtpppll.....uaaa.... umass.edu/
mccallum/data.html
21

---

Published as a conference paper at ICLR 2022
Network(MVCNN (uu et al. 2013). The hypergraph construction follows the setting described in Feng et al. 2019): Yang et al. (2020).
The three newly introduce datasets are adapted from Yelp Yelp)), House (Chodrow et al., 202 and Walmart (Amburg et al., 2020). For Yelp. we selected all businesses in the "restaurant" catalog as nodes, and formed hyperedges by selecting restaurants visited by the same user. We use the number stars the average review of a restaurant as the corresponding node label, starting from and going up 5 stars, with an interval of 0.5 stars. We then form the node features from the latitude, longitude, one-hot encoding of city and state, and bag-of-word encoding of the top-1000 words in the name of the corresponding restaurants. In the House dataset, each node is a member of the US House of Representatives and hyperedges are formed by grouping together members of the same committee. Node labels indicate the political party of the representatives. In Walmart, nodes represent products being purchased at Walmart, and hyperedges equal sets of products purchased together; the node labels are the product categories. Since there are no node features the original House and Walmart dataset, ilar to what was done for the contextual stochastic block model (Deshpande et al., 2018). In both datasets, fix the feature vector dimension to 100 and use one-hot encodings of the labels with added Gaussian noise ((0, I2 as the actual features. The noise standard deviation o is chosen to be 1 and 0.6.
impute the same using Gaussian random vectors, in manner sim-
J COMPUTATIONAL EFFICIENCY AND EXPERIMENTAL SETTINGS
All our test executed on Linux machine with 48 cores, 376GB of system memory, and two NVIDIA Tesla P100 GPUs with 12GB of GPU memory each. In AllSerTransformer, we also used a single layer MLP at the end for node classification. The average training times with their corresponding standard deviations in second per run, for all baseline methods at all tested datasets, are reported Table 4 Note that the reported times do not include any preprocessing times for the hypergraph datasets as these are only performed once before training. Table 4 lists the average training time per run with the optimal set of hyperparameters obtained after tunning; the average training times per run for different sets of hyperparameters used in the tuning process are reported Table 5
Choices of hyperparameters. We tune the hidden dimension of all hypergraph neural networks over (64, 128, 256, 512], except for the case of HyperGCN and HAN, where the original imple- mentations do not allow for changing the hidden dimension. For HNHN, AllSerTransformer and AltDeepSets we use one layer (a full V methods
> V propagation rule layer), while for all the other two layers recommended in the previous literature. We tune the learning rate over f0.1, 0.01, 0.001}, ), and the weight decays over {0, 0.00001), for all models under considera- tion. For models with multihead attentions, we also tune the number of heads over the set (1, 8)- The best hyperparameters for each model and dataset are listed in Table 6 Note that we only use default setting for HAN due to facts that its DGL implemennation has all parameters as constants and its much higher time complexity per run.
22

---

Published as a conference paper at ICLR 2022
Table 4: Running times for the best hyperparameter choice: Mean (sec or hour) + standard devia-
tion.
Cora
Citeseer
Pubied
Cara CA
DBLP-CA
ZoI
AllSetTransommer
+ 0.17s
26.82s
6.37s + 0.13s
134,96 + 0.37s
5.36s
AllDeepSets
11.70: 1. 0.07s
14,56 1. 0.06s
58.26s 1 0.23s
11.23s 0.06s
145,60s 1: 11.00s
9,57s 0.75s
MLP
0.06s
0.05s
+: 0.06s
3.86x +: 0.02s
CEOGN
2.3% + 0.09s
2.06s 0.07s
3.42: = 0.30s
1.68s + 0.05s
9.19s 0.04s
1.69s + 0.10s
CEGAT
10.84s + 0.21s
5,34s 0.12s
23,20s 0.15s
4,86s + 0.13s
69,50s 0.38s
3,92s 0.13s
HNHN
2.71s +: 0.04s
2.84s 0.03s
8.92s 0.03s
2.67% 1: 0.05s
26.57s :: 0.07s
0.19 :: 0.02s
HGNN
4.16
5.03s = 0.16s
14.75s + 0.08s
3,32s + 0.15s
21,65s + 0.09s
3.18s 0.04s
HCHA
HlypeGGCN
2.91s 1: 0.02s
3.37% 0.02s
4.33 1 0.27s
2.97% 1. 0.03s
8.07s 1. 0.21s
NIA
UniGCNII
+ 0.064
0.83s
16.55x
223.574 + 0.03s
2.35x
HAN (full batch)
3.476 = 0.52s
2.58s 0.48s
7.17% 0.11s
3.09 + 0.07s
12.39s + 0,22s
2,78s 0.09s
HAN (mini batch)
99.878 17.05s
69.06s
11.69s
502.44s
161-40s
143.80s 15.48s
0.45h + 0.12h
102.98s 2.3%
20Newsgroups
Mushroom
NTU2012
ModelNer40
Yelp
Houso(1)
Walmant(1)
AlsSeTransfomer
25.47s + 0.50s
10.12s + 0.14s
7.40s = 0.28s
47.55s = 0.25%
284. +: 0.23s
8.69s + 0.32s
154.19 0.11s
AllDeepSets
46.61s = 0. 14s
14.07s 0.03s
10.25s 0.07s
49.276 0.13s
474.34s + 0.07s
4.07s 0.24s
341.18s 0.62s
P
1.68s 0.02s
1.4ss 0.06s
1.2% 0.01s
1.60s 1: 0.13s
5,70s + 0.03x
1,51s 0.05s
7,83s 0.02s
CBOGN
0OM
69.67s 1.70s
2.00s 0.02s
2.54s + 0.28s
OOM
10.40s + 0.08s
173.75s = 0.07s
CEGAT
132.53s 27.88
0OM
HNHN
9.04s +: 0.09%
2.34s :: 0.03s
2.77 = 0.03s
8.54s 1: 0.08s
111.00s 1: 0.0Ks
1,76s 1: 0.01s
$4.85s 1 0.05s
HGNN
0.03%
10.57&
0.09%
15.55s
0.07s
HCHA
0.536 + 0.03s
10.54s 0.133s
3.76=0011
15.95s 0.81s
267.836 + 0.03s
3.05s + 0.08s
104.274 0.14
HyperGCN
5.968 0.03s
5.01s 0.03s
3.0% 0.02s
4,77 + 0.02s
183.76 + 0.82s
2.98s 0.04s
16.03s + 0.34s
UniGCNII
34.58s + 0.03s
10,00s 0.03s
12.95s 0.13s
4,40s + 06
197.07s + 0.04s
10. 0.06s
104.23< 10s
HAN (full batch)
0OM
30.28s + 3.29s
3.10s = 0.11s
4,56 + 0.218
OOM
3,32s + 0.12s
OOM
HAN (mini batch)
111.53s
87.25s
5.50h
5.59s
Table 5: Average running times over all different choices of hyperparameters: Mean + standard deviation.
Cora
Citeseer
Pubmed
Cora-CA
DBLP-CA
Zoo
AllSeTTransfomer
8.18 3,59s
8.41s 4.39s
22.99
16.54s
7.21s : 2.94s
53.14s 43.49s
5.97 2.23s
AllDeepSets
5.61s + 4.47s
6.83s + 5.63s
27.05s = 23.94s
5:62s + 4.38s
57.26s 1: 53.65s
4.14s :: 3.18s
MLP
1:
1.25%
CRGCN
1.636 + 0.76s
2.24s # 1.38s
5.93 5.01s
1.79% 4: 0.99%
18.91s + 17.08s
1.41s +: 0.62s
CEGAT
6.476 3.99s
9.25s 7.60s
34.52s
35.16s
7.93s + 5.94s
53.84s :: 37.83s
5.28s 5.92s
HNHN
1.65% 1.01s
2,3xs 1: 1.62s
5.03
3.14s
1.89 1: 0.80s
14,80 1: 9.55%
1.59 0.63s
HGNN
3.426 - 1.49s
4.44s 2.03s
8.44s = 5.08s
3.376 + 1.476
21.02s 13.61s
2.75s + 1.06s
HCHA
3.32a
4.97s
3.24s
20.668 13.37s
HyperGiCN
2.136 1: 0.75s
2 1 0.76s
3.23+ 1.00s
2.276 1: 0.67s
6.13s 1.93s
1.91s 1: 0.58s
UniGCNII
13.138
13.64s
17.63s
75.63%
94.97s 81.68s
HAN (full batch)
3.476 + 0.52s
2.58s 0.48s
7.17
1.
3.09
4 0.07s
12.39 4: 0.22s
2.78s 0.09s
HAN (mini batch)
99.876
17.05s
69.06s
11.69s
502.44s
161.40s
143.80s
15.48s
0.45b + 0.12h
102.98s + 2.39s
20Newsgroups
Mushroom
NTU2012
ModelNer40
Yelp
House(1)
Walmart(1)
AllSeTTransfomme
21.97% 1. 17.96s
14.35s
11.10s
6.96s 1: 2.65s
19.9% 1 14.46s
244.61s J. 97 97.44s
6.87s 1: 3.11s
123.51s 1. 102.91s
AllDeepSets
20.88s
21.73s
12.76s 13.8%s
5.15% 4.03s
19.90s + 19.15%
196.53s
254.834
4.52s
3.01s
107.54s + 119.22
MLP
1.22s
0.54x
1.11s 0.53s
0.97 0.43s
1.14s 1 0.50s
1.93s 3.88s
0.98s
0.47%
4.44s
1: 2.72x
CRGCN
0OM
67.66s
47.17%
1.51s 1 0.69s
3.68s + 2.72s
00M
4.16s :: 3.65%
63.42s
1: 30s
CEGAT
00M
121.3% 7.43s
5.45 2.97s
19.54s + 19.68s
00M
21.94s
23.44s
85.82s
HNHN
3.218 3.18s
3.28s = 1.80s
1.68 = 0.688
4.92s + 2.99s
56.15s 77.57s
1.70s 4: 0.53s
23.62s + 18.746
HGNN
5,93%
1: 5.36%
HCHA
9.61s 4: 5.98%
6.48s 1: 3,574
1.23 - 1.37%
8.82s 4: 5.47%
135.47s 1: 187.18s
1.20 +: 1.18s
43.62s 1: 36.378
HyperGCN
4.34s 1: 1.54s
3.56s 1.25%
2.2% 0.72s
3.41s 1: 1.17%
125.67s
1 56.93,
2.39 1: 0.76x
20.11s 1: 7,88s
UniGCNII
58.01s
66.97%
35.47s
44,72s
9.22s
9.68%
46.30s 1. 54.77k
309.35s
129.68s
8.83
7.97%
119.66%
73.89s
HAN (full batch)
0OM
30.28s 4: 4.23 3.29s
3.10s 1 0.11s
4.566 4: 0.21s
00M
3.32s 4: 0.12s
O0M
HAN (mini batch)
313.28s
111.53s 30.05s
165.35s
414.50s
87.25s
5.50b + 1.48h
72.79s
5.59s
1.96h 0.44h
23

---

Published as a conference paper at ICLR 2022
Table 6: Choice of hyperparameters for each method: Ir refers to the learning rate, wd refers to the weight decaying factor, h1 refers to the dimension of MLP hidden layer and heads refers the number of attention heads. Remarkably, not all hyperparameters are used for some models. For example, MIP, CEGCN, HNHN, HGNN, HCHA and HyperGCN does not take heads as input.
AllSerTranstonne
AllDcepSets
MLP
CEGCN
beads
heads
beads
Cora
0.001
0
256
0.001
0
512
0.01
0
64
0.001
0
512
Pubmed
0.001
0
256
8
0.001
0
512
0.01
1.00E-05
64
0,01
1.00E-05
Cora-CA
0.001
128
0,001
0.01
1.00E-05
0,001
DBLP-CA
0.001
512
0.001
0
512
0.01
0
0.01
1000
Zoo
0.01
1.00E-055
0,001
O
512
0.1
0
0,001
512
20News
100IE:05
OOM
Mushroom
0.001
128
0.001
0
256
0.01
0
64
0,01
1.00
64
NTU2012
1.00
0.001
ModeINer40
0.001
512
8
0.001
0
512
0.01
1.00E-05
64
0,01
1.00E-05
64
Yelp
0.001
0,001
0.01
1.00E-055
OOM
COM
OOM
OOM
House(])
0.001
0.01
1.00E 05
0.01
0.001
512
Walmant()
0.001
256
0.001
0
512
0.001
0
0.001
512
House(0.6)
100IE:
Walmant(.66
0.001
256
8
0,001
0
512
0.01
0
64
0.001
512
CEGAT
HNNHN
HGNN
HCHA
heads
heads
beads
Ir
heads
Cora
0.001
0
256
8
0,001
0
512
0.001
0
512
0,001
256
Pubmed
0.01
1.00
0.001
512
0.001
512
0.001
512
Cora
0.01
0
0,001
512
0.001
128
0.001
128
DBI P.CA
0.01
1.000-05
0.001
512
0.001
0
256
0.001
512
Zoo
0.001
1.00E-05
64
8
0.1
64
0.001
0
512
0.001
512
20News
0OM
Mushroom
0.01
1.00E-05
64
0,001
0
128
0.001
0
512
D,001
512
NTU2012
ModelNet40
0.01
1.00E-05
0.001
0.001
0
512
0.001
512
Yelp
0OM
0OM
0OM
OOM
0.001
0
128
0.001
256
0.001
Howse(()
0.001
128
0.01
1,005-05
0.01
0.01
1.000-05
Walmant(1)
0.001
0
256
0.001
0
512
0.001
0
512
0.001
0
512
House(0.6)
10WE-00
Walmant(.66)
0.001
0
256
0.001
0
512
0.001
0
512
0.001
256
HyperGCNN
URIGCNIT
Cora
0.001
0
64
0.001
0
512
Citeseer
0.01
1.00E-05
0.001
Pubmed
0.01
1.000-05
0.001
128
Cora-CA
0.01
1.00E-05
0,001
512
DBLP CA
0.01
1.000-05
0.001
256
Zoo
0.001
0
128
0.001
O
128
100E:05
Mushroom
0.001
0
64
0.001
0
128
NTU2012
0.01
1.00E-05
0.001
256
ModelNer40
0.01
1.00E-055
0.001
128
Yelp
0.01
1.00E-05
0.001
128
(ouse(()
0.01
1.00E-05
0.001
256
Walmart(1)
0.001
()
128
0.001
0
512
Houve(0.6)
Walmant0.6)
0.01
1.00E-05
64
0.001
0
256
4
24


Reviews:
[{"review_id": "pWLo9Ys6Man", "reviewer": "Reviewer_unoj", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- Overall, I think this is a well-organized paper and I enjoyed reading it. The motivations for the model design are well discussed as is the related work. \n- The proposed idea is simple and neat. Many hypergraph Laplacian-like mappings have been considered and this gives a possible data-centric answer to the question \"which one should I use?\"\n- The experiments consider a wide range of hypergraph datasets, which is somewhat rare in hypergraph classification literature. Moreover, the authors have collected several new hypergraph datasets, which I believe will be very useful for the community\n\nWeaknesses:\n- The experiments are not completely clear to me. In particular: (1) I think you should specify more clearly what is the depth of MLP and Set Transformer layers in AllDeepSets and AllSetTransformer and (2) you should more clearly specify what is the overall amount of known labels you used in the experiments and what is the amount used for training the AllSets architectures\n- The notation in equation (8) should be better specified (e.g. what is LN or || )\n- Other propagations which are \"in-between\" CE and tensors have been considered recently, e.g. [1-3]. Would these be particular cases of your proposed framework too?    \n[1] Tudisco et al - Node and edge nonlinear eigenvector centrality for hypergraphs  \n[2] Arya et al - Adaptive Neural Message Passing for Inductive Learning on Hypergraphs  \n[3] Prokopchik - A nonlinear diffusion method for semi-supervised learning on hypergraphs  \n", "summary_of_the_paper": "Transferring standard graph operators to the hypergraph setting is non-trivial and several message passing operators have been considered in the setting of hypergraphs, including clique-expansion and tensor based. In this paper the authors propose a general framework where learnable multiset functions are used in order to learn the hypergraph propagation map from the data. In particular, this framework contains  many previously considered propagation maps as particular cases", "summary_of_the_review": "Overall, this is a solid paper which proposes one simple, yet novel and efficient idea to generalize propagation maps on hypergraphs. Solid experimental results are provided in the paper.\n\nI like this paper and I would like to see it accepted. I will raise my score if the weaknesses are properly addressed\n\n-----\n\nI am happy with the feedback provided to my concerns and those of other reviewers so I will raise my score", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "F9swjgBR_OI", "reviewer": "Reviewer_LMVx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### **Strengths**\n\n1. **Quality of Paper**\n\nThe paper is technically sound. \nThe methods used (AllSetTransformer, AllDeepSets) are appropriate for the downstream task (node classification in hypergraphs). \nIt is interesting to see that AllSet generalises several neural networks on hypergraphs and message passing neural networks on graphs. \n\n2. **Clarity of Presentation**\n\nThe paper is clearly written and well organised. \nThe appendix is comprehensive with both theoretical proofs and empirical details (e.g., dataset construction, hyperparameters, etc.). \nThe authors have also released the source code as part of the supplementary material. \n\n\n\n### **Weaknesses**\n\n1. **Originality of Contributions**\n\nThe key idea of AllSet is to use rules of the form $f_{\\mathcal{V}\\rightarrow\\mathcal{E}}$ and $f_{\\mathcal{E}\\rightarrow\\mathcal{V}}$ as given in equation 5. \nThese update rules are well-known ideas in both (i) hypergraph neural network literature [e.g., Be More with Less: Hypergraph Attention Networks for Inductive Text Classification, EMNLP'20] and (ii) bipartite graph neural network literature [e.g., Hierarchical Representation Learning for Bipartite Graphs, IJCAI'19].\nMoreover, AllSetTransformer and AllDeepSets are straightforward applications of SetTransformer and DeepSets in the AllSet framework.\n\n\n2. **Significance of Empirical Evaluation**\n\nFrom Table 2 it is clear that the proposed methods show marginal or no improvements on all but three datasets viz., zoo, a tiny dataset, and Yelp, Walmart both curated by authors.  The Yelp dataset is about review rating prediction for restaurants where all methods achieve less than 37% accuracy: it is unclear how significant hypergraph modelling and the curated node features (e.g., latitude, longitude, city, state, etc.) are for rating prediction. The Walmart (and House) datasets do not come with node features, so it would be more compelling if hypergraph-only methods are used as baselines [e.g., Re-revisiting Learning on Hypergraphs: Confidence Interval and Subgradient Method, ICML'17].\n\n\n3. **Positioning with Prior Work**\n\nExperiments are restricted to transductive node classification in hypergraphs. The idea of $f_{\\mathcal{V}\\rightarrow\\mathcal{E}}$ and $f_{\\mathcal{E}\\rightarrow\\mathcal{V}}$ update rules have been successfully used in other tasks, e.g., (i) inductive text classification  [e.g., Be More with Less: Hypergraph Attention Networks for Inductive Text Classification, EMNLP'20], (ii) hyperlink prediction [e.g., Principled Hyperedge Prediction with Structural Spectral Features and Neural Networks], and (iii) recommendation [e.g., Session-based Recommendation with Hypergraph Attention Networks].\nIt would be interesting to see how AllSetTransformer and AllDeepSets would perform in these tasks.\n\n\n### Update\nAfter reading the responses to all the reviews, I have updated my recommendation from 5 -> 6", "summary_of_the_paper": "Hypergraphs can capture group/set relationships in real-world data. \nSeveral hypergraph neural networks have been proposed in the literature to exploit both group relationships among nodes and node features for learning with hypergraph data. \nThe contributions of the paper are \n1) generalisation of most existing methods into a single framework (named AllSet), \n2) exploration of AllSet based on Deep Sets [NeurIPS'17] and Set Transformer [ICML'19], and \n3) empirical evaluation on existing benchmarks and three curated hypergraph datasets ", "summary_of_the_review": "Overall the paper is clear and of good quality in which claims are well supported by theoretical analysis. However, the novelty is incremental and experimental results are marginally significant. Positioning with missing prior work and evaluation on other hypergraph tasks would improve the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8iyk_8ei5fv", "reviewer": "Reviewer_Bdsv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths of the paper:\n\nS1. The problem is well-motivated, with significant coverage of existing work based on clique expansion and tensor-based propagation.\n\nS2. Experiments include a significant number of baselines (ten models), as well as on ten different datasets.\n\nS3. There are theoretical analyses showing how the proposed framework is a generalisation of some existing works.\n\n\nWeaknesses of the paper:\n\nW1. As the proposed framework is in some sense a generalisation or a unification of existing propagation mechanisms, such as clique expansion and tensor-based propagation, there should be an ablation analysis to understand the contribution of each component.\n\nW2. The experimental discussion are focused on just showing numerical comparisons.  A more discursive analysis that explains why the method works better than the existing methods would help to reveal where the sources of improvements are.\n\nW3. Since the core is to derive hyper graph representations, experimenting with a single task is rather limited.  It would be good to have additional evaluation tasks beyond semi-supervised classification.", "summary_of_the_paper": "The paper proposes a framework for hyper graph neural networks.  It argues that existing work propagates a hyper graph by first transforming it into a regular graph trough clique expansion, which may lose information.  Alternatively, the propagation is based on tensors.  The proposed framework seeks to combine these different ways of propagations into one unified framework.  The proposed framework is called AllSet, with two instantiations AllDeepSets and AllSetTransformer.  Experiments are conducted on several graph datasets.", "summary_of_the_review": "The paper proposes a framework for hyper graph neural networks, which is an interesting problem.  Though rigorous in its treatment and experiments, further analysis on what makes the work better then the baselines would help in better appreciating the significance of the contributions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-WLMSDaB0A", "reviewer": "Reviewer_6wpr", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes a generalization of the hypergraph neural networks using multiset functions. Instead of doing clique expansion as in many existing methods, the proposed method learns explicit hypergraph representations, and uses two multiset functions to perform message propagations, one from nodes to hyperedge and another from hyperedge to nodes. The method is shown to be a general framework that encompasses existing GNN methods. Experimental results show that the AllSetTransformer achieves competitive results or outperforms existing methods on a large set of benchmark datasets. \n\nThe proposed method unifies various methods for hypergraph neural networks, and gives a competitive formulation across a wide spectrum of problems. This is a nice contribution to the community.\n", "summary_of_the_paper": "This paper proposes a generalization of the hypergraph neural networks using multiset functions.", "summary_of_the_review": "The paper makes a nice contribution to the community.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a hypergraph representation learning based on multiset encoding, which  covers most existing propagation methods for hypergraph neural networks. The authors provide theoretical proofs that both CE-based and tensor-based propagation rules can be represented as a composition of two multiset functions, and propose two different multiset encoding functions, based on DeepSets and SetTransformer. The authors validate their method for its semi-supervised node classification performance on multiple benchmark datasets, showing that it is superior or comparable to a large number of existing works on hypergraph representation learning. 

The following is the summary of the pros and cons of the paper mentioned by the reviewers:

Pros
- The proposed framework generalizes existing message passing methods for hypergraphs, and authors provide theoretical proofs on how it can generalize to two different types of propagation rules for hypergraph representation learning. 
- The paper is well-organized and is clearly written, and the code is provided for reproduction. 
- The experiments consider a wide range of hypergraph datasets and baselines, and the proposed method either outperforms them or at least achieves comparable performance.

Cons
- It is still unclear where the benefits come from, due to lack of ablation studies and deeper analysis. 
- Experiments are only restricted to the semi-supervised node classification task. 

While the initial reviews were split due to these negative points, all reviewers unanimously recommended for acceptance after the discussion period, as they found the responses from the authors satisfactory.  

In summary, this is a well-written paper that proposes a neat framework for hypergraph representation learning that generalizes to most existing methods, backed up by compelling performance on benchmark datasets, which will make it a strong addition to ICLR. However, as mentioned by the reviewers there should be more ablation studies and in–depth analysis of what makes the proposed multiset function more effective, as this is lacking in the current version of the paper. It would be worthwhile to also validate the proposed framework on other tasks (e.g. graph classification tasks). 
 
One minor thing that I want to point out is regarding the claim that this is the first attempt to connect the problem of learning multiset functions with hypergraph neural networks.  [Jo et al. 21], which is a hypergraph-based framework for edge representation learning, utilized GMT [Baek et al. 21], which performs multiset encoding using SetTransformer for hypergraph representation learning, and thus I suggest the authors to tone down on the claim that this is the first work that connects multiset encoding with hypergraph neural networks, and properly acknowledge this. 

[Jo et al. 21] Edge Representation Learning with Hypergraphs, NeurIPS 2021
Decision: Accept (Poster)