Sheaf Hypergraph Networks
Authors: Iulia Duta
Giulia Cassarà
Fabrizio Silvestri
Pietro Lio
Venue: NeurIPS 2023 poster (2023)
Keywords: hypergraph neural networks
hypergraph
sheaf
higher-order
TL;DR: Enhancing hypergraphs with sheaf structure for improving higher-order representations

Abstract: Higher-order relations are widespread in nature, with numerous phenomena involving complex interactions that extend beyond simple pairwise connections. As a result, advancements in higher-order processing can accelerate the growth of various fields requiring structured data. Current approaches typically represent these interactions using hypergraphs.
We enhance this representation by introducing cellular sheaves for hypergraphs, a mathematical construction that adds extra structure to the conventional hypergraph while maintaining their local, higher-order connectivity. Drawing inspiration from existing Laplacians in the literature, we develop two unique formulations of sheaf hypergraph Laplacians: linear and non-linear. Our theoretical analysis demonstrates that incorporating sheaves into the hypergraph Laplacian provides a more expressive inductive bias than standard hypergraph diffusion, creating a powerful instrument for effectively modelling complex data structures.
We employ these sheaf hypergraph Laplacians to design two categories of models: Sheaf Hypergraph Neural Networks and Sheaf Hypergraph Convolutional Networks. These models generalize classical Hypergraph Networks often found in the literature. Through extensive experimentation, we show that this generalization significantly improves performance, achieving top results on multiple benchmark datasets for hypergraph node classification.

Sheaf Hypergraph Networks
Tulia Duta University of Cambridge id366@cam ac. uk
Giulia Cassarà University of Rome, La Sapienza
casaaaainaaa it
Fabrizio Silvestri
Pietro Liò University Cambridge p12190cam ac. uk
University of Rome, La Sapienza fabrizio fabrizio.silvestri it
Abstract
Higher-order relations are widespread nature, with numerous phenomena in- volving complex interactions that extend beyond simple pairwise connections. As a result, advancements higher-order processing can accelerate the growth of various fields requiring structured data. Current approaches typically represent these interactions using hypergraphs. We enhance this representation by intro- ducing cellular sheaves for hypergraphs, mathematical construction that adds extra structure to the conventional hypergraph while maintaining their local, higher- order connectivity. Drawing inspiration from existing Laplacians in the literature, we develop two unique formulations of sheaf hypergraph Laplacians: linear and non-linear. Our theoretical analysis demonstrates that incorporating sheaves into the hypergraph Laplacian provides a more expressive inductive bias than standard hypergraph diffusion, creating a powerful instrument for effectively modelling complex data structures. We employ these sheaf hypergraph Laplacians to design two categories of models: Sheaf Hypergraph Neural Networks and Sheaf Hyper- graph Convolutional Networks. These models generalize classical Hypergraph Networks often found in the literature. Through extensive experimentation, we show that this generalization significantly improves performance, achieving top results on multiple benchmark datasets for hypergraph node classification.
Introduction
The prevalence of relational data real-world scenarios has led rapid development and widespread adoption of graph- based methods in numerous domains [[А]. However, a major limitation of graphs their inability represent interactions that goes beyond pairwise relations. In contrast, real- world interactions are often complex and multifaceted. There is evidence that higher-order relations frequently occur in neuroscience d) chemistry 77. environmental science g and social networks [10]. Consequently, learning powerful and meaningful representations for hypergraphs has emerged as a promising and rapidly growing subfield of deep learning [1] However, current hypergraph-bassd models struggle to capture higher-order relan relationships effectively. As described in [17], conventional hypergraph neural networks often suffer from the problem of over-smoothing. As we propagate the information inside the hypergraph, the representations of the nodes become uniform across neighbourhoods. This effect hampers the capability of hypergraph models to capture local, higher-order nuances.
More powerful and flexible mathematical constructs required to capture real-world interactions' complexity better. Sheaves provide a suitable enhancement for graphs that allow for more diverse and expressive representations. A cellular sheaf [18] enables attach data a graph, by associating vector spaces to the nodes, together with mechanism of transferring the information along the
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

edges. This approach allows for richer data representation and enhances the ability to model complex interactions.
Motivated by the need for more expressive structures, we introduce a cellular sheaf for hypergraphs, which allows for the representation of more sophisticated dynamics while preserving the higher-order connectivity inherent to hypergraphs. We take on the non-trivial challenge to generalize the two commonly used hypergraph Laplacians [19] to incorporate the richer structure sheaves offer. The- oretically, we demonstrate that the diffusion process derived using the sheaf hypergraph Laplacians that we propose induces a more expressive inductive bias than the classical hypergraph diffusion. Leveraging this enhanced inductive bias, we construct and test two powerful neural networks capa- ble of inferring and processing hypergraph sheaf structure: the Sheaf Hypergraph Neural Network (SheafHyperGGNN) and the Sheaf Hypergraph Convolutional Network (SheafHyperGCNN)
The introduction of the cellular sheaf for hypergraphs expands the potential for representing complex interactions and provides foundation for more advanced techniques. By generalizing the hypergraph Laplacians with the sheaf structure, we can better capture the nuance and intricacy of real-world data. Furthermore, our theoretical analysis provides evidence that the sheaf hypergraph Laplacians embody a more expressive inductive bias, essential for obtaining strong representations.
Our main contributions are summarised as follow:
We introduce the cellular sheaf for hypergraphs, a mathematical construct that enhances the hypergraphs with additional structure by associating vector space with each node and hyperedge, along with linear projections that enable information transfer between them.
2.
We propose both a linear and non-linear sheaf' hypergraph Laplacian, generalizing the standard hypergraph Laplacians commonly used in the literature. We also provide theoretical characterization of the inductive biases generated by the diffusion processes of these Laplacians, showcasing the benefits of utilizing these novel tools for effectively modeling intricate phenomena.
3.
The two sheaf hypergraph Laplacians the foundation for two novel architectures tailored for hypergraph processing: Sheaf Hy Hypergraph Neural Network and Sheaf Hy Hypergraph Convolutional Network. Experimental findings demonstrate that these models achieve top results, surpassing existing methods on numerous benchmarking datasets.
2 Related work
Sheaves on Graphs. Utilizing graph structure real-world data has improved various domains like healthcare D. biochemistry
(2). social networks [20). recommendation systems 33 traffic prediction (21), with graph neural networks (GNNs) becoming the standard for graph representations. However, in heterophilic setups, when nodes with different labels are likely to be connected, directly processing the graph structure leads to weak performance. In (22), they address this by attaching additional geometric structure to the graph, in the form of cellular sheaves [8]
cellular sheaf on graphs associates a vector space with each node and each edge together with a linear projection between these spaces for each incident pair. To take into account this more complex geometric structure, SheafNN (3 generalised the classical GNNs 24 26 by replacing the graph Laplacian with sheaf Laplacian (27. diigsea-dimensional sheaf-based neural networks are explored, with sheaves either learned from the graph 22 or deterministically inferred for efficiency (28). Recent methods integrate attention mechanisms 29 or replace propagation with wave equations [30]. In recent developments, Sheaf Neural Networks have been found to significantly enhance the performance of recommendation systems, as they improve upon the limitations of graph neural networks [3I]
In the domain of heterogeneous graphs, the concept of learning unique message functions for varying edges is well-established. However, there's 's distinction how theaf-based methods approach this task compared to heterogeneous methods such as RGCN [32. Unlike the latter, which learns individual parameters for each kind of incident relationship. sheaf-based methods dynamically predict projections for each relationship, relying features associated with the node and hyperedge. As a result, the total parameters in sheaf networks do not escalate with an increase the number of hyperedges. This difference underscores a fundamental shift in paradigm between the two methods.

---

Fusu Mne
ER
IMPLICIT BIAS:
(2) [ 144 Σ Fiidl -Funii
LINEAR
(Cr)or
Σ h
NON-LINEAR LAPLACIAN
(cr)or Σ FI Fl To
tt M0g 0
U
IMPLICIT BIAS:
= 5 se0e
IFe Fn. F.aa-FFaaD 3aull
Figure 1: Visual representation of linear and non-linear sheaf hypergraph Laplacian. (Top) In the linear case, the block matrix (Cx) Jue corresponding to the pair of nodes (u, v) accumulates contributions from each hyperedge that simultaneously contains both nodes. (Bottom) In the non- linear version, for each hyperedge, we first select the two nodes that are the most dissimilar in the hyperedge stalk domain: ~e if (u, v)
F Then, the block matrix (LF)uu associated with the pair of nodes (u, only accumulates contributions from a hyperedge if u ~e v. The two operators (linear and non-linear sheaf hypergraph Laplacian) represent the building blocks for the Sheaf Hypergraph Neural Network and Sheaf Hypergraph Convolutional Network respectively and we theoretically show that they exhibit a more expressive implicit bias compared to the traditional Hypergraph Networks, leading better performance.
reel
Hypergraph Networks. Graphs, while useful, have a strong limitation: they represent only pairwise relations. Many natural phenomena involve complex, higher-order interactions B3 g requiring a more general structure like hypergraphs. Recent deep learning approaches have been developed for hypergraph structures. HyperGNN II expands the hypergraph into weighted clique and
applies message passing similar to GCNs 224] HNHN (36 improves this with non-linearities, while HyperGCN (37) connects only the most discrepant nodes using non- non-linear Laplacian. Similar to the trend in GNNs, attention models gain popularity also in the hypergraph domain. HCHA (38) uses an attention-based incidence matrix, computed based on nodes-hyperedge similarity. Similarly, HERALD 33 uses a learnable distance to infer soft incidence matrix. On the other hand, HEAT [5
creates messages by propagating information inside each hyperedge using Transformers 44)
Many hypergraph neural network (HNN) methods can viewed as two-stage frameworks: 1) sending messages from nodes to hyperedges and 2) sending messages back from hyperedges nodes. Thus,
41 proposes a general framework where the first step is the average operator, while the second stage could use any existing GNN module. Similarly, 44 uses either DeepSet functions [4] or Transformers [40] to implement the two stages, while [44] uses a GNN-like aggregator in both stages, with distinct messages for each (node, hyperedge) pair.
In contrast, we propose a novel model improve the hypergraph processing by attaching a cellular sheaf to the hypergraph structure and diffusing the information inside the model according to it. We will first introduce the cellular sheaf for hypergraph, prove some properties for the associated Laplacians, and then propose and evaluate two architectures based on the sheaf hypergraph Laplacians.
3 Hypergraph Sheaf Laplacian
An undirected hypergraph is uplle = (V, F) where V = {1,2... m. is a set of nodes (also called vertices), and E is set of hyperedges (also called edges when there is no confusion with the graph
3

---

edges), Each hyperedge is subset of the nodes set V. We denote by n |V] the number of nodes in the hypergraph H. and by m [E| the number of hyperedges. In contrast graph structures. where each edge contains exactly two nodes, in hypergraph an edge can contain any number of nodes. The number of nodes in each hyperedge ((e)) is called the degree of the hyperedge and is denoted by de. In contrast, the number of hyperedges containing each node e is called the degree of the node and is denoted by dy.
Following the same intuition from defining sheaves on graphs sheaf associated with hypergraph H.
23
22
we will introduce the cellular
Definition 1. A cellular sheaf F associated with a hypergraph H is defined as a triple ((e) Foge), where:
1. F(v) are vertex stalks: vector spaces associated with each node v;
2. F(e) are hyperedge stalks: vector spaces associated with each hyperedge e;
3. Fude F(e) + F(e) are restriction maps: linear maps between each pair 1 e, if hyperedge e contains node c.
In simpler terms, a sheaf associates space with each node and each hyperedge in a hypergraph and also provides a linear projection that enables the movement of representations between nodes and hyperedges, as long as they are adjacent. Unless otherwise specified, we assign the same d- dimensional space for all vertex stalks F(u) = I2 and all hyperedge stalks F(e) Rd. We refer to as the dimension of the sheaf.
Previous works focused on creating hypergraph representations by relying on various methods of defining Laplacian for hypergraph. In this work, we will concentrate version of the hypergraph Laplacian as used in [T], and a non-linear version of the hypergraph Laplacian in [37]. We will extend both of these definitions to incorporate the hypergraph sheaf structure, analyze the advantages that arise from this, and propose two different neural network architectures based on each one of them. For visual comparison between the two proposed sheaf hypergraph Laplacians, see Figure I
definitions: linear
3.1 Linear Sheaf Hypergraph Laplacian
Definition Following the definition of a cellular sheaf on hypergraphs, we introduce the linear sheaf hypergraph Laplacian associated with hypergraph H. (CF) vv
Σ J. F E gdxd and
ejvEe
(Cx)uu = Σ F.g.FFFe E Rdxd, where Fuse: : Rd + Rd represents the linear restriction egu
oe
maps guiding the flow of information from node e to hyperedge e.
The linear sheaf Laplacian operator for node applied on a signal x E Raxd can be rewritten as:
Cx(x), = Σ
(F FFrsoTo Fusoor)))
EiDEE
uEe
When each hyperedge contains exactly nodes (thus H is graph), the internal summation will contain single term, and we recover the sheaf Laplacian for graphs as formulated in 22
On the other hand, for the trivial sheaf, when the vertex and hyperedge stalks are both fixed to be R and the restriction map the identity Fugg = we recover the usual linear hypergraph Laplacian
I1. defined as L(x), = Emsee J. Eu - x.)). However, when we allow for nigher-dimensione stalks Rd, the restriction maps for each adjacency pair (v, e) become linear projections Fize E RRdx enabling us to model more complex propagations, customized for each incident (node, hyperedge) pairs.
In the following sections, we will demonstrate the advantages of using this sheaf hypergraph diffusion instead of the usual hypergraph diffusion.
Reducing energy via linear diffusion. Previous work [7 demonstrates that diffusion using the classical symmetric normalised version of the hypergraph Laplacian A = D where D is diagonal matrix containing the degrees of the vertices, reduces the following energy function:

---

=. Eu.eee I,., - using the linear hypergraph Laplacian leads to similar representations for neighbouring nodes. While this is desirable in some scenarios, it may cause poor performance in others, phenomenon known as over-smoothing [7]. In the following, we show that applying diffusion using the linear sheaf hypergraph Laplacian addresses these limitations by implicitly minimizing a more expressive energy function. This allows us to model phenomena that were not accessible using the usual Laplacian. Definition 3. We define sheaf Dirichlet energy of a signal I E Rnxd on a hypergraph H as:
d 2.... Intuitively, this means that applying diffusion
EL (L)
Σ 8
Σ
Fode D It
Fusle D 2 xu
where D. = Σ Ff Fude is a normalisation term equivalent to the nodes degree dy for the trivial
sheaf and D = diag( D1, Dp....D. Da) the corresponding block diagonal matrix.
This quantity measures the discrepancy between neighbouring nodes in the hyperedge stalk domain as opposed to the usual Dirichlet energy for hypergraphs that, instead, measures this distance in the node features domain. In the following, we are showing that, applying hypergraph diffusion using the linear sheaf Laplacian implicitly reduces this energy.
Proposition 1. The diffusion process using a symmetric normalised version of the linear sheaf hypergraph Laplacian minimizes the sheaf Dirichlet energy of a signal on a hypergraph H. Moreover, the energy decreases with each layer of diffusion.
Concretely, defining the diffusion process as Y = (I-AF)X where A = D E addxad represents the symmetric normalised version of the linear sheaf hypergraph Laplacian, we have that EY (Y) EY (X), with } the non-zero eigenvalues of A and X, = max, {(1 (1) 1. All the proofs are in the Supplementary Material.
This result addresses some of the limitations of standard hypergraph processing. First, while classical diffusion using hypergraph Laplacian brings closer representations of the nodes in the nodes space (In, T.), our linear sheaf hypergraph Laplacian allows us to bring closer representations of the nodes in the more complex space associated with the hyperedges (Fugette Fug. This encourages form of hyperedge agreement, while preventing the nodes become uniform. Secondly, in the hyperedge stalks, each node can have a different representation for each hyperedge it is part of, leading to more expressive processing compared to the classical methods. Moreover, in many Hypergraph Networks, the hyperedges uniformly aggregate information from all its components. Through the presence of a restriction map for each (node, hyperedge) pair, we enable the model to learn the individual contribution that each node sends to each hyperedge.
From an opinion dynamic perspective 446 when the hyperedges represent group discussions, the input space T, can be seen as the private opinion, while the hyperedge stalk F. can be seen as a public opinion (what an individual decide express in certain group e). Minimizing the Dirichlet energy creates private opinions that are in consensus inside each hyperedge, while minimizing the sheaf Dirichlet energy creates an apparent consensus, by only uniformizing the expressed opinion. Through our sheaf setup, each individual is allowed to express varying opinion in each group it is part of, potentially different than their personal belief. This introduces a realistic scenario inaccessible in the original hypergraph diffusion setup.
3.2 Non-Linear Sheaf Hypergraph Laplacian
Although the linear hypergraph Laplacian is commonly used to process hypergraphs. it falls short in fully preserving the hypergraph structure [47]. To address these shortcomings, [48] introduces the non-linear Laplacian, demonstrating that its spectral properties are more suited for higher-order processing compared to the linear Laplacian. For instance, compared to the linear version, the non-linear Laplacian leads to more balanced partition in the minimum cut problem, task known to be tightly related to the suni-supervieed node classification. Additionally, while the linear Laplacian associates clique for each hyperedge, the non-linear one offers the advantage of relying sparser connectivity. We will adopt a similar methodology to derive the non-linear version of the sheaf hypergraph Laplacian and analyze the benefits of applying diffusion using this operator. Definition We introduce the non-linear sheaf hypergraph Laplacian of a hypergraph H with respect to signal following:
much
5

---

1. For each hyperedge e, compute (Me, v2)) argmaza, Fuse - Fole the set of pairs containing the nodes with the most discrepant features in the hyperedge stalk.
2. Build an undirected graph 9и containing the same sets of nodes H. and, for each hyperedge e connects the most discrepant nodes (u, v) (from now on we will write u <e o if they are connected in the Gμ graph due the hyperedge e). multiple pairs have the same maximum discrepancy, we will randomly choose one of them.
3. Define the sheaf non-linear hypergraph Laplacian as:
[r(x)== = Σ
1 J uge 5p
(Fugoto Fusbotu).
oSe
Note that the sheaf structure impacts the non-linear diffusion in two ways: by shaping the graph structure creation (Step 1), where the two nodes with the greatest distance in the hypergraph stalk are selected rather than those in the input space; and by influencing the information propagation process (Step 3). When the sheaf restricted to the trivial case (d = and Fuge 1) this corresponds the non-linear sheaf Laplacian of a hypergraph as introduced in 48
Reducing total variation via non-linear difflusio. In the following discussion, will demon- strate how transitioning from a linear to a non-lineat sheaf hypergraph Laplacian alters the energy guiding the inductive bias. This phenomenon was previously investigated for the classical hypergraph Laplacian, with [48] revealing enhanced expressivity in the non-linear case.
Definition 5. We define the sheaf total variation of signal x E Raxd on a hypergraph H as:
DT = 1
|| Fuse D 3
xv Fuse D 3
max a
where D. Σ JI Fude is a normalisation term equivalent the node's degree in the classical setup and D diag( D1, D2 ... Da) is the corresponding block diagonal matrix.
This quantity generalised the total variance (TV) (x) = 1 max,, I|d, b -, xx,, minimized in the non- non-linear hypergraph label propagation 48 49 Unlike the TV, the sheaf total variation measures the highest discrepancy at the hyperedge level computed in the hyperedge stalk, as opposed the TV, which gauges the highest discrepancy in the feature space. We will explore the connection between the sheaf TV and our non-linear sheaf hypergraph diffusion.
The diffusion process using the symmetric normalised version of non-linear sheaf hypergraph Laplacian minimizes the sheaf total variance of a signal x on hypergraph H.
Proposition 2.
Despite the change in the potential function being minimized, the overarching objective remains akin to that the linear case: striving to achieve a coherent consensus among the representations within the hyperedge stalk space, rather than generating uniform features for each hyperedge in the input space. In contrast to the linear scenario, where a quadratic number of edges is required for each hyperedge, the non-linear sheaf hypergraph Laplacian associates single edge with each hyperedge, thereby enhancing computational efficiency.
3.3 Sheaf Hypergraph Networks
37 50 draw inspiration from a variety of hypergraph 51 giving rise to diverse message passing techniques. These techniques all involve the propagation of information from nodes to hyperedges and vice-versa. We will adopt a similar strategy and introduce the Sheaf Hypergraph Neural Network and Sheaf Hypergraph Convolutional Network, based on two pesssag-paassing schemes inspired by the sheaf diffusion mechanisms discussed this paper.
Popular hypergraph neural networks 45
diffusion operators
Given hypergraph 7 = (V, E) with nodes characterised by set of features X E Raxf we initially linearly project the input features into X E (df) and then reshape them into X E f As result, each node is represented the vertex stalk as a matrix IRdxf, where d denotes the dimension of the vertex stalk, and indicates the number of channels.

---

Table Performance on a collection of hypergraph benchmarks. Our models using sheaf hyper- graph Laplacians demonstrate a clear advantage over their counterparts using classical Laplacians (HyperGNNN and HyperGCN). Compared to other recent methods, SheafHyperGGNN and ShearHyper- GCN achieve competitive performance and attain state-of-the-art results in five of the datasets.
Name
Corn
Citeseer
Pahmed
Cara
DBLP CA
Senate
House
Congress
HCHA
79.14 +:002
72.42 +1.42
86.41
+0.36
82.55 +0.97
90.92 +0.22
48.62 +44.41
61.36 +2.53
90.43
1.20
HNHN
16.922
72.64
1.55
86.90
+0.30
77.
#1.9
86.78 =0.29
50.93
+6.33
67.8 27.59
53.35
1144
All SeeppsSSets
76.88 +1.80
70.83
+166
88.75
+0.33
81.97 +1.50
91.27
+0.27
48.17 +5.67
67.82 +2,40
91,80
+155
AllSecTransfommrs
78.58 =1.47
88.72
+0.37
83.63
91.53
51.83
69.33
92.16 11.0
UniGCNII
78.81 11.05
73.05
22221
88.25 10.33
83.60 11.14
91.69 10.19
49.30 14.25
67.25 42.57
94.81
=0.81
HyperND
79.20 +1.14
72.62
+1449
86.68
+0.43
80.62 +1.32
90.35 +-0.66
52.82 +33.0
51.70 +3.30
74.63
+3362
ED-HIN
80.31
73.70
1:338
89.03
10.53
83.97
11.55
91.90
64.79 1:514
72.45
1:2.88
1.99
HyperGCNT
78. +2.00
71.01
+:2.1
80.81
+:2,4
79.
+2.11
89.42
+0,16*
51.13++4.15
69:29 +2.05
89.67
+1.22
ShealHypaGCN
80.06
73.27
+0.50
10.71
83.26 #1.20
90.83 #0.23
66.33
72.66
#:226
90.37
=1.52
HyperGINN
19.39:11.5
72.45
1111
86.44
+0.44
82.64 1166
91.03 #:020
48.59 1:452
61.39
#22.66
91.26
1111
SheafllyperCNN
81.30 1.70
74.71
1:223
87.68
10.60
85.52 11.28
91.59 1:024
68.73 14.68
73.84 12.30
91.81
1160
A general layer of Sheaf Hypergraph Network is defined as:
Y = =oo(d - -),,, & W))
Here, Á can be either A D for the linear sheaf hypergraph Laplacian introduced in Eq. or A= = D- L D- for the non-linear sheaf hypergraph Laplacian introduced in Eq. 2 Both Wi E Raxdd and W2 E R/xf are learnable parameters, while represents ReLU non-linearity.
Sheaf Hypergraph Neural Network (SheafHyperGNN). This model utilizes the linear sheaf
hypergraph Laplacian A = A². When the sheaf is trivial (d = 1 and Fuge = 1). and W1 Ia- the SheafHyperGNN is equivalent the conventional HyperGNN architecture [TT]. However, by increasing dimension d and adopting dynamic restriction maps, our proposed SheafHyperGNN becomes more expressive. For every adjacent node-hyperedge pair (u, e). we use a matrix to discern each node's contribution instead of fixed weight that only stores the incidence relationship. The remaining operations are similar to those in HyperGNN [T]. More details on how the block matrices Fuge are learned can be found in the following subsection.
block
Sheaf Hypergraph Convolutional Network (SheafHyperGCNN) This model employs the non-linear
Laplacian A = A Analogous to the linear case, when the sheaf is trivial and Wi = Ia we obtain the classical HyperGCN architecture 37 In our experiments, we will use an approach similar to that in (37] and adjust the Laplacian to include mediators. This implies that we will not only connect the two most discrepant nodes but also create connections between each node in the hyperedge and these two most discrepant nodes, resulting in a denser associated graph. For more information on this variation, please refer to (37 or Supplementary Material.
In summary, the models introduced in this work, SheafHyperGNN and SheafHyperGCN serve as generalisations of the classical HyperGNN [II and HyperGCN (37]. These new models feature a more expressive implicit regularisation compared to their traditional counterparts.
Learnable Sheaf' Laplacian. A key advantage of Sheaf Hypergraph Networks lies in attaching and processing a more complex structure (sheaf) instead of the original standard hypergraph. Different sheaf structures can be associated with single hypergraph, and accurately modeling the most suitable structure is crucial for obtaining effective and meaningful representation. In our proposed models, we achieve this by designing learnable restriction maps. For a d-dimensional sheaf, we predict the
restriction maps for each pair of incident (vertex v, hyperedge e) Fure MIP(<>ollha [ha) E Rdd
where Tu represent node features of v. and he represents features of the hyperedge e. This vector representation is then reshaped into d x d block matrix representing the linear restriction map for the (u, e) pair. When hyperedge features he are not provided, any permutation-invarian operation can be applied to obtain hyperedge features from node-level features. We experiment with three types of d x d block matrices: diagonal, low-rank and general matrices, with the diagonal version
Results where rerun compared to 50 using the same hyperparameters, to fix an existing issue in the original
code.
7

---

Table 2: Ablation study on Restriction Maps: we explore three types of d restriction maps: diagonal, low-rank and general. Diagonal matrices consistently achieve better accuracy on most of the datasets, demonstrating a superior balance between complexity and expressivity
Name
Cora
Citeseer
Pubmed
Com CA
DBI P CA
Senate
House
Congress
Ding-hhafftyyerCCC
80:06 1.1.2
73.27 +0.50
87.00
00.71
83.26 +1
90.83 +0.23
66.33 14.58
72.66 +
90.37
11.52
JR-ShealHyperOCN
78.70 #1.14
72.14 1100
86.99 20.39
82.61 11.28
90.84 10.29
66.76-14.58
70.70
42.23
84.88 =2.31
Gen Sen-SheafFlperGCCN
79.13 10.85
72.54 #2.3
86,90
10.46
82.54
22.08
90.57 10.40
65.49 45.17
71.05
42.12
82.14 #22.1
Diag ShoufllypstGNN
74.71
11223
87.68
10.50
85.52
11228
91.59
10.24
14.66
73.62
11.29
91.81
Rt-SheafHypeeGiNN
+1.41 76.65
74.05
+1.34
87.09
+0.25
77.05
+1.00
85.13
+0.29
68.45
42.46
73.84 +2.30
74.83 +2.32
Gea-SixallyyparGNN
76.82 #1332
74.24
#1.05
87.35
=0.34
77.12 #1.14
84,99 +0.39
68.45 4:4.9
69.47
#1.97
74.52 #1.27
consistently outperforming the other two. These restriction maps are further used to define the sheaf hypergraph Laplacians (Def. or used in the final Sheaf Hypergraph Networks. Please refer to the Supplementary Material for more details on how we constrain the restriction maps.
4 Experimental Analysis
We evaluate our model on eight real-world datasets that vary in domain, scale, and heterophily level and are commonly used for benchmarking hypergraphs. These include Cora, Citeseer, Pubmed, Cora-CA, DBLP-CA [37]. House [52], Senate and Congress [53) To ensure a fair comparison with the baselines, we follow the same training procedures used in (50 by randomly splitting the data into 50% training samples, 25% validation samples and 25% test samples, and running each model 10 times with different random splits. We report average accuracy along with the standard deviation.
Additionally, we conduct experiments on a set of synthetic heterophilic datasets inspired by those introduced by 150]. Following their approach, we generate a hypergraph using the contextual hypergraph stochastic block model [54456), containing 5000 nodes: half belong to class while the other half to class 1. We then randomly sample 1000 hyperedges with cardinality 15, each containing exactly 33 nodes from class 0. The heterophily level is computed as a = min(3. 15 - B). Node fcatures are sampled from label-dependent Gaussian distribution with standard deviation of 1. As the original dataset is not publicly available, we generate our own set of datasets by varying the heterophily level a E (1...7) and rerun their experiments for fair comparison.
The experiments are executed on single NVIDIA Quadro RTX 8000 with 48GB of GPU memory. Unless otherwise specified, our results represent the best performance obtained by each architecture using soyper-paramed optimisation with random search. Details on all the model choices and hyper-parameters can be found in the Supplementary Material.
Laplacian vs Sheaf Laplacian. As demonstrated in the previous section, SheafHyperGNN and SheafHyperGCN are generalisations of the standard HyperGNN [II] and HyperGCN [37]. respec- tively. They transition from the trivial sheaf (d = 1 and Fu 1) to more complex structures (d 2 1 and Free a d x d learnable projection). The results in Table I and Table 3 show that both models significantly outperform their counterparts on all tested datasets. Among our models, the one based on linear Laplacian (SheafHyperGNN) consistently outperforms the model based on non-linear Laplacian SheaaHH)) across all datasets. This observation aligns with the performance of the models based on standard hypergraph Laplacian, where HyperGCN is outperformed by HyperGNN all but two real-world datasets, despite their theoretical advantage 48
Comparison to recent methods. We also compare to several recent models from the literature such as HCHA (38), HNHN (36), AllDeepSets (42), AlISe(Transfommer 42 UniGCNII (57). HyperND (58). and ED-HNN (50). Our models achieve competitive results on all real-world datasets, with state-of- the art performance on five of them (Table D.. These results confirm the advantages of using the sheaf Laplacians for processing hypergraphs. We also compare our models against a series baselines on the synthetic heterophilic dataset. The results are shown in Table Our best model, SheafHyperGNN, consistently outperforms the other models across all levels of heterophily. Note that, our framework enhancing classical hypergraph processing with sheaf structure not restricted to the two traditional models tested in this paper (HyperGNN and HyperGCNN. Most of the recent state-of-the-arr methods, such as ED-HNN, could be easily adapted to learn and process our novel cellular sheaf hypergraph instead of the standard hypergraph, leading to further advancement in the hypergraph field.
8

---

Figure 2: Impact of Depth and Stalk Di- mension evaluated on the heterophilic dataset
Table 3: Accuracy on Synthetic Datasets with Varying Heterophily Levels: Across all different level of heterophily (a), our theaf-based methods SheafGCN and SheafHGNN consistently outper- form their counterparts. Additionally, they achieve top results for all heterophily levels, further demon- strating their effectiveness. For each experiment, the result represents average accuracy over 10 runs.
(a 7). SheafHyperGNN's performance is unaffected by increasing depth, and high- dimensional stalks is essential for achieving top performance. The Dirichlet energy shows that, while HyperGNNN enforces the nodes be sim- ilar, our ShealHyperGNN does not suffer from this limitation, encouranging features diversity.
Influence of stalk dimension
Influence of #layers
hecerophily (a.
1000
Name
5
HyperGCCN
83.9
69.4
72.9
75.9
70.5
67.3
06.5
HypertNNN
98.4
83.7
74.5
69.3
66.9
63.8
500
98.1
74.1
71.1
ED-HNN
99.9
91.3
88.4
84.1
80.7
78.8
76.5
ShaafHOCN
100
87.1
84.8
79.2
75.1
76.6
75.5
200
SheafEIGNN
100
94.2
90.8
86.5
82.1
79.8
77.3
Dimmension stale id)
Layer
In the following sections, we conduct series of ablation studies to gain deeper understanding of our models. We will explore various types of restriction maps, analyze how performance changes when varying the network depth and study the importance of stalk dimension for final accuracy.
Investigating the Restriction Maps. Both linear and non- non-linear sheaf hypergraph Laplacians rely on attaching a sheaf structure to the hypergraph. For cellular sheaf F with vertex stalks F(v) Rd and hyperedge stalks F(e)= Rd as used in our experiments, this involves inferring the restriction maps Fore E Rdxd for each incidence pair (v, e). We implement these as a function dependent on
corresponding nodes and hyperedge features: Furste ML.(x,,l |l))) Rdd Learning these matrices can be challenging; therefore, we experimented with adding constraints to the type of matrices used as restriction maps. In Table we show the performance obtained by our models when constraining the restriction maps to be either diagonal (Diag-SheafHypeer SheafHyperNN), low-rank LLR-SheafHyperNN or general matrices Seen--SeaaHyppNNNN We observe that the sheaves equipped with diagonal restriction maps perform better than the more general variations. We believe that the advantage of the diagonal restriction maps is due to easier optimization, which overcomes the lose in expressivity. More details about predicting constrained d x d matrices can be found in the Supplementary Material.
Importance of Stalk Dimension. The standard hypergraph Laplacian corresponds to sheaf Lapla- cian with d = and Fers.e = 1. Constraining the stalk dimension to be 1, but allowing the restriction maps to be dynamically predicted, becomes similar to an attention mechanism 38). However, at- tention models are restricted to guiding information via a scalar probability, thus facing the same over-smoothing limitations as traditional HyperGNN in the heterophilic setup. Our d-dimensional restriction maps increase the model's expressivity by enabling more complex information transfer between nodes and hyperedges, tailored for each individual pair. We validate this experimentally on the synthetic heterophilic dataset, using the diagonal version of the models, which achieves the best performance in the previous ablation. In Figure 2 we demonstrate how performance significantly stalks (d > 1). These results are consistent for both
improves when allowing linear sheaf Laplacian- based models (SheafHy GNNeGGGNy)) and non-linear ones (SheafHyperGCN).
Influence of Depth. is well-known that stacking many layers in hypergraph network can lead to a decrease in model performance, especially in the heterophilic setup. This phenomenon, called over- smoothing, is well-studied in both graph (59) and hypergraph literature [17] To analyse the extent to which our model suffers from this limitation. version of the synthetic dataset (a = 7). For both ShealHyperGNN and its HyperGNN equivalent, we vary the number of layers between - 8. In Figure 2 we observe that while HyperGNN exhibits a drop in performance when going beyond 3 layers, SheafHyperGNN's performance remains mostly constant. Similar results were observed for the non-linear version when comparing SheafHyperGCN with HyperGCN (results in Supplementary Material). These results indicates potential advantages of our models in the heterophilic setup by allowing the construction of deeper architectures.
train series of models on the most heterophilic
Investigating Features Diversity. Our theoretical analysis shows that, while conventional Hyper- graph Networks tend to produce similar features for neighbouring nodes, our Sheaf Hypergraph

---

Networks reduce the distance between neighbouring nodes in the more complex hyperedge stalk space. As result, the nodes' features do not become uniform, preserving their individual identities. We empirically evaluate this, by computing the Dirichlet energy for HyperGNN and SheafHyperGGNN (shaded area in Figure 2). as a measure of similarity between neighbouring nodes. The results are aligned with the theoretical analysis: while increasing depth in HyperGNN creates uniform features, SheafHyperGNN does not suffer from this limitation, encouraging diversity between the nodes.
Conclusion
In this paper we introduce the cellular sheaf for hypergraphs, an expressive tool for modelling higher- order relations build upon the classical hypergraph structure. Furthermore, we propose two models capable of inferring and processing the sheaf hypergraph structure, based on linear and non-linear sheaf hypergraph Laplacian. respectively. We prove that the diffusion processes associated with these models induce more expressive implicit regularization, extending the energies associated with standard hypergraph diffusion. This novel architecture generalizes classical Hypergraph Networks, and we experimentally show that outperform existing methods on several datasets. Our technique of replacing the hypergraph Laplacian with sheaf hypergraph Laplacian in both HyperGNN and HyperGCN establishes a versatile framework that can be employed to "sheafify" other hypergraph architectures. We believe that sheaf hypergraphs can contribute to further advancements in the rapidly evolving hypergraph community, extending far beyond the results presented in this work.
Acknowledgment The authors would like to thank Ferene Huszár for fruitful discussions and con- structive suggestions during the development of the paper and Eirik Fladmark and Laura Brinkholm Justesen for fixing a minor issue in the original HyperGCN code, which led improved results in the baselines. Iulia Duta is a PhD student funded by a Twitter scholarship. This work was also supported by PNRR MUR projects PE0000013-FAIR SERICS (PE00000014), Sapienza Project FedSSL, and IR0000013-SoBigDataai
References
[1] Catherine Tong, Emma Rocheteau, Petar Veličković, Nicholas Lane, and Pietro Lio. Predicting Patient Outcomes with Graph Representation Learning, pages 281-293. 01 2022.
[2] Kexin Huang, Cao Xiao, Lucas M Glass, Marinka Zitnik, and Jimeng Sun. Skipgnn: predicting molecular interactions with skip-graph networks. Scientific reports, 10(1):1-16, 2020.
[3] Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L. Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 974-983, 2018
[4] Xiaolong Wang and Abhinav Gupta. Videos space-time region graphs. In Proceedings of the European Conference on Computer Vision (ECCV), pages 399-419 2018. 2018
[5] Nicolas A. Crossley, Andrea Mechelli, Jessica Scott, Francesco Carletti, Peter T. Fox, Philip K. McGuire, and Edward T. Bullmore. The hubs of the human connectome are generally implicated in the anatomy of brain disorders. Brain, 137:2382 2395, 2014.
[6] Tingting Guo, Yining Zhang. Yanfang Xue, and Lishan Qiao. Brain function network: Higher order vs. more discrimination. Frontiers in Neuroscience, 15, 08 2021.
[7] Jürgen Jost and Raffaella Mulas. Hypergraph laplace operators for chemical reaction networks, 2018.
[8] Pragya Singh and Gaurav Baruah. Higher order interactions and species coexistence. Theoretical Ecology, 14, 03 2021.
[9] Alba Cervantes-Loree Loreto, Carolyn Ayers, Emily Dobbs, Berry Brosi, and Daniel Stouffer. The context dependency of pollinator interference: How environmental conditions and co-foraging species impact floral visitation. Ecology Letters, 24, 07 2021.
10

---

[10] Unai Alvarez-Rodrigee Federico Battiston, Guilherme Ferraz de Arruda, Yamir Moreno, Matjaž Perc, and Vito Latora. Evolutionary dynamics of higher-order interactions social networks. Nature Human Behaviour, 5:1-10, 05 2021.
[11] Yifan Feng, Haoxuan You, Zizhao Zhang, Rongrong Ji, and Yue Gao. Hypergraph neural networks. Proc. Conf. AAAI Artif. Intell., 33(01):3558 335553355558888 July 2019.
[12] Ramon Viñas, Chaitanya K. Joshi, Dobrik Georgiev, Bianca Dumitrascu, Eric R. Gamazon, and Pietro Liò. Hypergraph factorisation for multi-tissue gene expression imputation. bioRxiv, 2022.
[13] Xinlei Wang, Junchang Xin, Zhongyang Wang. Chuangang Li, and Zhiqiong Wang. An evolving hypergraph convolutional network for the diagnosis of alzheimers disease. Diagnostics, 12(11), 2022.
[14) Yichao Yan, Jie Qin, Jiaxin Chen, Li Liu, Fan Zhu, Ying Tai, and Ling Shao. Learning multi- granular hypergraphs for video-based person re-identification. In 2020 IEEF/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 19, 2020, pages 2896-2905. IEEE, 2020.
[15] Dobrik Georgiev Georgiev, Mare Brockschmidr, and Miltiadis Allamanis. HEAT: Hyperedge attention networks. Transactions on Machine Learning Reseurch, 2022.
[16] Jingcheng Wang. Yong Zhang, Lixun Wang, Yongli Hu, Xinglin Piao, and Baocai Yin. Multitask hypergraph convolutional networks: A heterogeneous traffic prediction framework. IEEE Trans. Intell. Transp. Syst., 23(10):18557-1855 2022.
[17] Guanzi Chen, Jiying Zhang, Xi Xiao, and Yang Li. Preventing over-smoothing for hypergraph neural networks. 2022.
[18] Justin Curry. Sheaves, cosheaves and applications, 2014.
[19] T.-H. Hubert Chan, Anand Louis, Zhihao Gavin Tang, and Chenzi Zhang. Spectral properties of hypergraph laplacian and approximation algorithms. ACM, 65(3), mar 2018.
[20] Federico Monti, Fabrizio Frasca, Davide Eynard, Damon Mannion, and Michael M Bron- stein. Fake news detection on social media using geometric deep learning. arXiv preprint arXiv:1902.0667 2019.
[21] Bing Yu, Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks:
A deep learning framework for traffic forecasting. pages 3634-3644 International Joint Conferences on Artificial Intelligence Organization. 7 2018.
[22] Cristian Bodnar, Francesco Di Giovanni, Benjamin Paul Chamberlain, Pietro Liø, and Michael M. Bronstein. Neural sheaf diffusion: A topological perspective heterophily and oversmoothing GNNs. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[23] Jakob Hansen and Thomas Gebhart. Sheaf neural networks. In TDA & Beyond, 2020.
[24] Thomas N. Kipf and Max Welling, Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR), 2017.
[25] Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in neural information processing systems, pages 3844-3852, 2016.
[26] Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks graphs. CoRR, abs/1312.6203,, 2013.
[27] Jakob Hansen and Robert Ghrist. Toward a spectral theory of cellular sheaves. Journal of Applied and Computational Topology, 3(4):315-358, aug 2019.
[28] Federico Barbero, Cristian Bodnar, Haitz Sáez de Ocáriz Borde, Michael Bronstein, Petar Veličković, and Pictro Lio. Sheaf neural networks with connection laplacians. 06 2022.
11

---

[29] Federico Barbero, Cristian Bodnar, Haitz Sáez de Ocáriz Borde, and Pietro Lio. Sheaf attention networks. In NeurIPS 2022 Workshop on Symmetry and Geometry Neural Representations, 2022.
[30] Julian Suk, Lorenzo Giusti, Tamir Hemo, Miguel Lopez, Konstantinos Barmpas, and Cristian Bodnar. Surfing on the neural sheaf. In NeurlPS 2022 Workshop on Symmetry and Geometry in Neural Reprexentations, 2022.
[31] Antonio Purificato, Giulia Cassarà, Pietro Liò, and Fabrizio Silvestri. Sheaf neural networks for graph-based recommender systems, 2023.
[32] Michael Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks, 2017.
[33] Chen Zu, Yue Gao, Brent Munsell, Minjeong Kim, Ziwen Peng, Yingying Zhu, Wei Gao, Daoqiang Zhang. Dinggang Shen, and Guorong Wu. Identifying high order brain connectome biomarkers via learning hypergraph. Mach Learn Med Imaging, 10019:1-9 October 2016.
[34] Gregor Urban, Christophe N. Magnan, and Pierre Baldi. Sspro/acepro 6: almost perfect prediction of protein secondary structure and relative solvent accessibility using profiles, deep learning and structural similarity. Bioinform.. 38(7):2044-2065, 2022.
[35] Yoshinobu Inada and Keiji Kawachi. Order and flexibility in the motion of fish schools. Journal of theoretical biology, 214:371-87, 03 2002.
[36] Yihe Dong, Will Sawin, and Yoshua Bengio. Hnhn: Hypergraph networks with hyperedge neurons. In Graph Representation Learning and Beyond Workshop at ICML 2020, June 2020. Code available: http:s//tttt..ooom/wiieeebbttttt
[37] Naganand Yadati, Madhav Nimishakavi, Prateek Yadav, Vikram Nitin, Anand Louis, and Partha Talukdar. Hypergen: A new method for training graph convolutional networks on hypergraphs. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bus E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[38] Song Bai, Feihu Zhang, and Philip H.S. Torr. Hypergraph convolution and hypergraph attention. Pattern Recognition, 107637,, 2021.
[39] Jiying Zhang, Yuzhao Chen, Xiong Xiao, Runiu Lu, and Shutao Xia. Learnable hypergraph laplacian for hypergraph learning. In ICASSP, 2022.
[40] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez. L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, Bengio, H. Wallach, Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[41] Jing Huang and Jie Yang. Unignn: unified framework for graph and hypergraph neural net- works. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, 2021.
[42] Eli Chien, Chao Pan, Jianhao Peng, and Olgica Milenkovic. You are allset: A multiset function framework for hypergraph neural networks. In International Conference on Learning Representations, 2022.
[43] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov. and Alexander Smola. Deep sets. In Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[44] Devanshu Arya, Deepak K. Gupta, Stevan Rudinac, and Marcel Worring. Hypersage: General- izing inductive representation learning hypergraphs, 2020.
[45] Song Bai, Feihu Zhang, and Philip H. Torr. Hypergraph convolution and hypergraph attention. CoRR, abs/1901.00 000.000.1111100100 150, 2019.
12

---

[46] Jakob Hansen and Robert Ghrist. Opinion dynamics on discourse sheaves. SIAM Journal on Applied Mathematics, 81(5):0333-2660 2021.
[47] Sameer Agarwal, Kristin Branson, and Serge Belongie, Higher order learning with graphs. volume 2006, pages 17-24, 01 2006.
[48] Matthias Hein, Simon Setzer, Leonardo Jost, and Syama Sundar Rangapuram. The total variation on hypergraphs learning on hypergraphs revisited. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26, Curran Associates, Inc., 2013.
[49] Chenzi Zhang, Shuguang Hu, Zhihao Gavin Tang, and T-H. Hubert Chan. Re-revisiting learning on hypergraphs: Confidence interval and subgradient method. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 2034. PMLR, 06-11 Aug 2017.
[50] Peihao Wang, Shenghao Yang, Yunyu Liu, Zhangyang Wang, and Pan Li. Equivariant hyper- graph diffusion neural operators. arXiv preprint arXiv:2207.06680, 2022.
[51] Shota Saito, Danilo Mandic, and Hideyuki Suzuki. Hypergraph p-laplacian: A differential geometry view. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), apr 2018.
[52] Philip S. Chodrow, Nate Veldt, and Austin R. Benson. Generative hypergraph clustering: From blockmodels to modularity. Science Advances, 7(28).
[53] James H. Fowler. Legislative cosponsorship networks in the US house and senate. Social Networks, 28(4):454 oct 2006.
[54] Yash Deshpande, Subhabrata Sen, Andrea Montanari, and Elchanan Mossel. Contextual stochastic block models. S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[55] Debarghya Ghoshdastidar and Ambedkar Dukkipati. Consistency of spectral partitioning of uniform hypergraphs under planted partition model. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
[56] I Chien, Chung- Yi Lin, and I-Hsiang Wang. Community detection in hypergraphs: Optimal statistical limit and efficient algorithms. In Amos Storkey and Fernando Perez-Cruz, editors, Proceedings of the Twenty-Fiis First International Conference on Artificial Intelligence and Statistics, volume 84 of Proceedings of Machine Learning Research, pages 871 871-879. PMLR, 09-11 11 Apr 2018.
[57] Jing Huang and Jie Yang. Unignn: a unified framework for graph and hypergraph neural net- works. In Proceedings of the Thirtieth International Joint Conference Artificial Intelligence, IJCAI-21, 2021.
(58) Francesco Tudisco, Austin R. Benson, and Konstantin Prokopchik. Nonlinear higher-order label spreading. In Proceedings of the Web Conference 2021, WWW '21, page 2402-2413, New York, NY, USA, 2021. Association for Computing Machinery.
[59] Xinyi Wu, Zhengdao Chen, William Wei Wang, and Ali Jadbabaic. A non-asymptotic analysis of oversmoothing in graph neural networks. In The Eleventh International Conference on Learning Representations, 2023.
13


Reviews:
[{"review_id": "MLK7Me28km", "reviewer": "Reviewer_C7Ao", "summary": "The paper introduces the cellular sheaves to hypergraph and proposes two sheaf hypergraph laplacians. In addition, they also propose two models: sheaf HNN and Sheaf HCN. The experiments demonstrated the effectiveness of the proposed models.", "questions": "It is necessary to highlight the difficulties of using sheaf to hypergraphs and the core technological innovation of the proposed method.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written and well-organized. \n- The authors incorporate the sheaves to hypergraph laplacian, which would promote the development of hypergraph spectral theory.\n- The proposed two hypergraph neural networks provide new options for hypergraph learning.", "weaknesses": "1. The novelty of this paper is limited. It uses the techniques in graph-based methods[1] to existing hypergraph laplacian[2][3]. The Sheaf hypergraph networks also have the same form of Sheaf Convolutional Network (SCN). The Sheaf Dirichlet energy and its convergence property also can be trivially generalized from [4].\n2. The experiments don’t bring surprising results. The replacement of trivial hypergraph Laplacian with sheaf laplacian can improve the performance, which can be easily speculated from the results in GNN[1]. \n\n[1]Neural Sheaf Diffusion: A Topological Perspective on Heterophily and Oversmoothing in GNNs.\n\n[2]Hypergraph neural networks.\n\n[3]Hypergcn: A new method for training graph convolutional networks on hypergraphs.\n\n[4]Preventing over-smoothing for hypergraph neural networks\n\nMinor:\n\nSome references have incorrect author names. E.g. [43] Guanzi Chen, Jiying Zhang,et al. Preventing over-smoothing for hypergraph neural networks\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We warmly appreciate your time and effort invested in reviewing our paper. Your perceptive comments and constructive criticism are invaluable, and we are grateful. We are sorry for the incorrect reference and we will make sure it will be fixed in the final version of the paper.\n\n**Regarding the novelty**\n\nWe acknowledge your feedback on the novelty of our work. Due to their remarkable versatility and powerful representation capabilities, sheaves have become a growing field within the machine learning community. Yet, our work is pioneering in introducing the cellular sheaf *for hypergraphs*. This mathematical construct enriches hypergraphs by associating a vector space with each node and hyperedge, along with linear projections facilitating the transfer of information between them. The cellular sheaf brings a unique perspective and adds a fresh dimension to the current body of knowledge inside the hypergraph community. We emphasize the complexity of extending concepts from graphs to hypergraphs due to the latter's increased complexity. This endeavor in itself is a significant contribution to the field.\n\nMoving on to the Sheaf hypergraph networks, it's easy to spot similarities with the Sheaf Convolutional Network (SCN) in terms of form. Hypergraph represents a  generalisation of graphs, that’s why obtaining similar processing with the ones in the graph literature it’s natural when constraining the hypergraph to be a graph.  Nevertheless, the underlying mathematical structure and implementation diverge substantially. This is accounted for by the unique, higher-order nature of hypergraphs and their respective Laplacians.\n\nIn the linear case of Sheaf hypergraph Laplacian, the sheaf Laplacian is defined as $\\mathcal{L} x_v=\\sum_{e; v \\in e} F_{v  \\trianglelefteq e}^T \\sum_{u \\in e} (F_{v  \\trianglelefteq e}x_v - F_{u  \\trianglelefteq e} x_u)$. When each hyperedge contains exactly two nodes (thus, $\\mathcal{H}$ is a graph), the inner summation will have a single term, and we recover the sheaf Laplacian for graphs as formulated in [1]. However, the power of higher-order processing consists of being able to process relations beyond pairwise. We achieve this by allowing the sheaf to act on edges of arbitrary cardinality. \n\nMoreover, the sheaf Laplacian for the non-linear case differs substantially from [1]. In fact, when the sheaf is restricted to the trivial case ($d = 1$ and  $F_{v  \\trianglelefteq e}= 1$), this corresponds to the non-linear sheaf Laplacian of a hypergraph, as introduced in [2]. Our work generalises [2] by incorporating sheaf Laplacian, thereby introducing a more diverse range of inductive biases and avoiding well-known challenges in the hypergraph community such as over-smoothing.  \n\n**Similarity with hypergraph Dirichlet energy**\n\nThe main goal of our theoretical results is not just to empirically compare the SheafHNN and HNN models but to highlight the benefits SheafHNN brings on top of HNN. We do not aim to introduce a different metric to evaluate hypergraph issues as in [3]. Instead, our goal is to emphasise the differences in the minimised energy between the two methods. A comparison of the energies minimized by SheafHNN and HNN reveals the distinctive capabilities of the two models: while HNN narrows the features of the nodes, making it susceptible to over-smoothing, SheafHNN optimizes the representation of the nodes in the sheaf space, effectively circumventing over-smoothing. In Figure 2, we also provide empirical validation by plotting the Dirichlet energy, which serves as a measure of over-smoothing [3], for both models. The results demonstrate that while the Dirichlet energy of HNN decreases as the number of layers increases, the Dirichlet energy of SheafHNN remains relatively constant.\n\nThe results in [3] confine themselves to the linear case, and generalizing these results to the non-linear HGCN is a highly challenging task due to the non-differentiability of the non-linear operator. Our work, however, addresses both these aspects, providing a comprehensive and general overview and a more complete intuition over what advantages sheaf hypergraphs can bring to the hypergraph research area.\n\nConsidering these, we humbly assert that our work is indeed novel and makes a significant contribution to the discourse in this field.\n\n*[1] Cristian Bodnar, Francesco Di Giovanni, Benjamin Paul Chamberlain, Pietro Liò, and Michael M. Bronstein. Neural sheaf diffusion: A topological perspective on heterophily and oversmoothing in GNNs.*\n\n*[2] Matthias Hein, Simon Setzer, Leonardo Jost, and Syama Sundar Rangapuram. The total variation on hypergraphs - learning on hypergraphs revisited.* \n\n*[3] Guanzi Chen, Jiying Zhang, Xi Xiao, Yang Li, Preventing Over-Smoothing for Hypergraph Neural Networks.*\n"}, {"review_id": "b6GLpV8dz2", "reviewer": "Reviewer_qurz", "summary": "The paper proposes cellular sheaf to enhance the high-order relationships among hypergraphs. Specifically, it first designs two formulations, linear sheaf hypergraph Laplacian and non-linear sheaf hypergraph Laplacian to construct the hypergraph structure. Then the two sheaf hypergraph Laplacians are leveraged to design two types of models, i.e., sheaf hypergraph neural network and sheaf hypergraph convolutional networks. Detailed theoretical analysis and extensive experiments are provided to demonstrate the effectiveness of the designed model.", "questions": "1. Can you clearly explain the motivation for this work? \n2. Please discuss the existing challenges of hypergraph models in extracting the high-order relationships among hypergraphs.\n3. I would like to know the reason behind the performance gap between SheafHyperGNN and ED-HNN over Congress.\n4. Grammar mistakes and vague statements should be revised.", "limitations": "This work clearly discusses the limitation of the model design.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The topic of enhancing the hypergraph structure is impressive.\n\n2. The paper is well-organized overall and easy to follow. \n\n3. Figures are concise, enabling readers to easily catch the key points. ", "weaknesses": "1. The motivation for the sheaf hypergraph network is not clearly discussed in this work. In the section Introduction, this work does not clearly discuss the existing limitation of extracting the high-order relationships among hypergraphs. Instead, this work directly introduces sheaf for hypergraphs, which is confusing and not consistent from my point of view. \n\n\n2. The format of the equations and some notations are very confusing in this manuscript. There should have a comma or a full stop after the equations. \nFor instance, a full stop should be added at the end of equation 2. All equations in Definitions should also be numbered. \n\n3. Some statements in this manuscript are very confusing and vague. There are also a lot of typos and grammar mistakes. For instance, line 31, 'that allows for'--> 'allow for'; In the caption of Figure 1, 'Than' --> 'Then'.\n\n4. In Table 1, SheafHyperGNN gains excellent performance over most benchmark datasets, while the performance of SheafHyperGNN over Congress is not that satisfactory than ED-HNN and the performance gap (i.e., 3\\%) between these two models is large. I would like to know the reason behind that.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and constructive feedback on our paper. We appreciate the time you've taken to review our work, and we would like to address your concerns and questions.\n\n**Motivation and challenges**\n\nThe motivation for our work comes from recognizing that current hypergraph models struggle to capture higher-order relationships effectively. As described in [1], conventional hypergraph neural networks (HGNNs) often suffer from the  problem of over-smoothing [2]: as we propagate the information inside the hypergraph the representations of the nodes become overly uniform across neighbourhoods. This over-smoothing effect hampers the capability of HGNNs to capture local, higher-order nuances, especially on heterophilic node-classification tasks, when connected nodes tend to belong to different classes. \n\nOur proposed Sheaf Hypergraph Networks aim to overcome these limitations by introducing cellular sheaves into hypergraphs, providing a more expressive and nuanced representation of the underlying data structure. We achieve this by allowing each node to have a different representation for each hyperedge it is part of ($F_{v \\trianglelefteq e} x_v$). Moreover, as we shown in Proposition 1 and  2 and empirically in Figure 2, our sheaf diffusion brings closer the representation of the nodes in this hyperedge space as opposed to bringing them closer in the feature space (which is the underlying cause of over-smoothing). \n\nMoreover, conventional hypergraphs lack the additional structure to capture local, higher-order connectivity nuancedly, hence the introduction of cellular sheaves into hypergraphs which enable a more complex flow of messages compared to usual Laplacians.\n\n**Formatting and grammar**\n\nWe apologize for any confusion caused by the formatting of equations and notations. We appreciate your suggestion and will ensure all equations are appropriately numbered and punctuated in the revised manuscript.\n\nWe value your feedback on the quality of language and will undertake a thorough revision to correct any grammatical errors and confusing statements. We will particularly address the instances you've mentioned in line 31 and the caption of Figure 1.\n\n**Performance**\n\nWe believe that the performance gap between SheafHyperGNN and ED-HNN over Congress may be due to the unique structure or properties of the Congress dataset. The unique design of the Congress dataset, which contains more hyperedges and fewer nodes than other datasets, presents a specific challenge for our model. When hyperedge features are not provided, we can typically apply any permutation-invariant operation to obtain hyperedge features from node-level elements. Without a ground-truth sheaf structure associated with the data, SheafHNN heavily relies on the quality of nodes and hyperedge features to infer this structure. However, the Congress dataset does not contain either node or hyperedge attributes. We followed the method of [3] to generate node features from label-dependent Gaussian distribution, as proposed by [4] . Given that the node and hyperedge features are caused predominantly by Gaussian noise, we might end up with a noisier sheaf structure which damages the learning process. We will delve into greater detail about this in the revised manuscript. Designing better features for these highly challenging cases represent an interesting area for future study. \n\nOnce again, thank you for your valuable feedback. By addressing these points, we can significantly improve the clarity and quality of the paper.\n\n*[1] Guanzi Chen, Jiying Zhang, Xi Xiao, Yang Li. Preventing Over-Smoothing for Hypergraph Neural Networks.*\n\n*[2] Qimai Li, Zhichao Han, & Xiao-Ming Wu. Deeper Insights Into Graph Convolutional Networks for Semi-Supervised Learning.*\n\n*[3] Eli Chien, Chao Pan, Jianhao Peng, and Olgica Milenkovic. You are allset: A multiset function framework for hypergraph neural networks.* \n\n*[4] Yash Deshpande, Subhabrata Sen, Andrea Montanari, Elchanan Mossel. Contextual stochastic block models.*\n"}, {"review_id": "IPJadut0Pt", "reviewer": "Reviewer_2wJ1", "summary": "This paper generalizes hyper-graph Laplacians to hyper-graphs attached with sheaves, called Hypergraph Sheaf Laplacian, and constructued hyper-graph NNs with proposed Hypergraph Sheaf Laplacian. Experiments demonstrate the effectiveness of SheafHyperGNNs over normal HyperGNNs.", "questions": "1. I feel the proposed methods are kind of similar to heterogeneous graph NNs, for example RGCN[1]. I wonder if the authors can discuss the relationship between their proposed methods and those heterogeneous graph NNs, for example can RGCN be viewed as a special case of SheafHyperGNN?\n\n2. Given the SheafHyperGNN has learnable sheaves, will there be an overfitting risk and do the authors have any ideas on how to avoid it? For example I personally have also tried to somehow attatch parameters to each edge in message passing, and I found the GNN simply learns zero message passing weights and degenrates to an MLP and uses the MLP to overfit the training set. Theoretically, it seems the proposed SheafHyperGNN has the same risk and I wonder if the authors observed similar phenomenons in experiments? (say do you need to tune the hyper-parameters to avoid it?) If so it would be nice to discuss potential ways to address it.\n\n\n[1] Modeling Relational Data with Graph Convolutional Networks", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "I find this paper well-written and easy to follow. The paper addresses two crucial challenges in the GNN community: hyper-graph processing and heterophily graph processing. The concept of hyper-graph sheaf Laplacian looks convincing and promising. The SheafHyperGNN is derived from hyper-graph sheaf Laplacian, which provides a rigorous framework to study the properties of it (e.g. to prove it doesn't suffer from over-smoothing). I believe both the new concept of hyper-graph sheaf Laplacian and the proposed SheafHyperGNN can be benificial to the community.\n\n-----\n\nAfter reading authors' rebuttal, I decide to still keep my current rating.", "weaknesses": "See Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful that the reviewer appreciates our work and believes that the novel direction is promising and beneficial to the community. We thank the reviewer for the constructive feedback and for pointing out to us a relevant related domain. \n\nIn the following, we discuss in more detail each point raised by the reviewer. We will incorporate all the feedback and related works into the final version of the paper.\n\n**Connection with the Heterogeneous graph NN literature**\n\nBoth SheafHyperNN and heterogeneous methods for graphs such as RGCN [1] share a technical similarity: they use distinct “message functions” for each edge (each type of edge for RGCN). However, they differ in their approach. In contrast to RGCN, SheafHyperNN does not learn different parameters for each incident relationship. Instead, it dynamically predicts a projection (the reduction map) based on (node, hyperedge) features for each relationship. More specifically, the sheaf predictor takes the node and hyperedge features for each incident pair and predicts a $d \\times d$ matrix to be used as a linear projection.  The parameters that are learned in our approach are not directly the one used in projection (as in RGCN) but the parameters of the small network used to predict the projection. As a result, the number of parameters for our model does not scale with the number of hyperedges (types of edges in the RGCN scenario). Another significant distinction lies in the types of data each method addresses. SheafHNN is designed for hypergraph representation learning, whereas RGCN is applied to heterogeneous graph data.\n\nWe thank the reviewers for pointing out this line of work. We believe that, while the methods are different, there are overlapping motivations driving both directions, making SheafHNN a candidate tool for representing heterogeneous hypergraph data. Exploring this potential through experiments represents an exciting avenue for future research.\n\n**Overfitting risk due to learnable sheaves**\n\nIntroducing the learnable sheaf structure in the model leads to an increase in the number of parameters compared to the backbone.  However, we mitigate this by employing a small network to predict the projection, which is shared across all incidence relations. This approach enables us to keep the number of additional parameters relatively small, and independent of the number of relations. \n\nTo further address the risk of overfitting, we experimented with various restrictions applied to the projection matrix, such as diagonal, low-rank and orthogonal constraints. These restrictions allow us to predict a smaller number of parameters, thus requiring a smaller predictor network. In line with the reviewer's comment, the empirical advantages of using a diagonal projection over more general forms can be attributed, in part, to a reduced risk of overfitting. However, overfitting can still be a problem especially in very small datasets. We believe that exploring non-parametric ways of predicting the sheaf represents a highly interesting research direction. \n\n*[1] Michael Sejr Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, Max Welling. Modeling Relational Data with Graph Convolutional Networks.*"}, {"review_id": "5xW18UdeGC", "reviewer": "Reviewer_XTYg", "summary": "\\\nThe primary research focus of this paper is to investigate the potential of neural networks for processing hypergraph datasets.\n\nThe paper utilises the mathematical concept of sheaf, which describes how locally defined data on a space can be consistently pieced together, to enhance hypergraphs by associating vector spaces with nodes and hyperedges, facilitating the transfer of information through linear projections.\n\nThe paper lays the groundwork for \"sheaf\" hypergraph Laplacians, examining their diffusion processes to understand the inductive biases they generate, which play a pivotal role in driving the progress of hypergraph neural networks.", "questions": "* Does the paper include any experiments demonstrating that a neural network with a non-linear Sheaf Laplacian captures significantly more information than a linear operator? Please clarify if such analysis is already present, and if not, it would be valuable to include as an important contribution to the paper.\n* Can we rely solely on the increased expressiveness of sheaf Laplacians, or should we also consider the importance of good generalisability for achieving strong empirical performance on test data? \n* In terms of generalisation, while both the AllSet framework [3] and Sheaf Hypergraph Networks generalise and offer advancements over earlier models, does the Sheaf Networks framework have the capacity to encompass all instances of the AllSet framework, or are there any specific scenarios within the AllSet framework that cannot be accommodated by Sheaf Networks?\n\n3. [You are AllSet: A Multiset Function Framework for Hypergraph Neural Networks, In ICLR'22](https://openreview.net/forum?id=hpBTIv2uy_E)\n", "limitations": "Adequately addressed (e.g. see Appendix A).", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "\\\n**Clarity**\n\nThe introduction of the paper offers a concise and informative overview of the research topic, providing necessary context by proposing the generalisation of hypergraph Laplacians to advanced sheaf Laplacians for capturing intricate phenomena. \n\nIt effectively outlines the objectives of advancing neural networks on hypergraphs and sets the stage for the study and development of sheaf hypergraph neural networks.\n\nThe submitted work showcases a well-structured and organised presentation, with a logical progression of ideas, clearly defined sections and headings, and enhanced clarity through the inclusion of code and supplementary materials.\n\n\\\n**Quality**\n\nThe paper's claims are supported by sound theoretical analyses of the diffusion processes (e.g. propositions 1, 2) and empirical studies including involving different restriction maps (e.g. Table 2).\n\nBy investigating the impact of depth and heterophily levels, the paper reveals additional insights, contributing to its overall quality.\n\nIn summary, the methodology, experimental design, and theoretical framework employed to support the contributions are sound, credible, and well-founded.\n\n\\\n**Originality**\n\nBy introducing sheaf Laplacians, the paper tackles a limitation present in current hypergraph Laplacians (e.g. see lines 163-172), which have been instrumental in the development of existing hypergraph networks.\n\nBy adopting unique perspectives (e.g. opinion dynamics perspective see lines 173-180), the paper illuminates the nuanced and intricate phenomena that can be captured by the sheaf structure.\n\nOverall, the paper sheds light on a previously unexplored limitation in the existing literature, showcasing its originality.", "weaknesses": "\\\n**Significance**\n\nRecent research has explored sheaf-based approaches for graph data, resulting in the development of advanced graph neural network architectures [1, 2] that draw inspiration from sheaf Laplacians. \n\nHyperedges in hypergraph datasets can be represented as pairwise connections, treating them as graph datasets, and the linear sheaf Hypergraph Laplacian employed by SheafHyperGNN yields the best results among the models in Table 1.\n\nTo strengthen the significance and necessity of a Sheaf operator on higher-order data, it is crucial to conduct comprehensive empirical analyses, such as comparing with the best SheafGNN architecture as a baseline and exploring hypergraph datasets where the non-linear Laplacian captures substantially more information than its linear counterpart.\n\n1. [Sheaf Neural Networks, In TDA and Beyond, 2020](https://openreview.net/forum?id=GgcgIJsT8HD)\n2. [Neural Sheaf Diffusion: A Topological Perspective on Heterophily and Oversmoothing in GNNs, In NeurIPS'22](https://openreview.net/forum?id=vbPsD-BhOZ)\n\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the very detailed review and constructive feedback. We are delighted the reviewers agree that we propose a sound method, with a pivotal role for the hypergraph neural networks progress. \n\nIn the following, we discuss the main remarks raised by the reviewer, and we will incorporate all the feedback into the final version of the paper.\n\n**Expressivity vs generalisability**\n\nWe entirely agree with the reviewer’s observation that having a more expressive model does not necessarily offer better generalisability on test data and that both expressivity and generalisability are important for good performance. Our theoretical framework focuses solely on characterising the expressivity, demonstrating that SheafHNN are able to model a broader range of functions compared to classical HNN. However, that does not provide any guarantees that the optimization process will identify the optimal sheaf structure for the test data. \n\nThe empirical results consistently show an advantage of using SheafHNN compared to HNN across all tested datasets, hinting that the model is able to perform well in terms of generalisability as well. However, we do not have any theoretical analyses in this regard. We will clearly emphasise this in the limitation section. Investigating aspects like generalizability and extrapolation power of SheafHNN represents an extremely promising avenue for future research.\n\n**Importance of non-linear Sheaf Laplacian**\n\nFrom a practical standpoint, the non-linear Sheaf Laplacian offers the advantage of relying on a sparser connectivity compared to its linear counterpart. In each hypergraph, the non-linear Sheaf Laplacian propagates information along a reduced number of edges, approximately $O(\\sum_e 2|e|)$, whereas the linear Sheaf Laplacian requires a much higher number of connections, approximately  $O(\\sum_e \\frac{(|e|-1)|e|}{2})$., where $|e|$ is the cardinality of hyperedge $e$. \n\nFrom a theoretical perspective, [1] indicates that the spectral properties of the non-linear Laplacian are more suited for higher-order processing compared to the linear Laplacian. For instance, the non-linear Laplacian leads to a more balanced partition in the minimum cut problem compared to its linear version (min-cut problem is known to be tightly related to the semi-supervised node classification task).  This balanced minimum cut avoids making a cut inside a hyperedge that would result in isolating nodes.  Despite being recognized as a powerful tool for hypergraphs, we were not able to empirically observe these advantages in our experiments. We believe that this could be attributed to the scarcity of datasets specifically dedicated to higher-order analysis.\n\nIn our experiments, we observed an advantage of the non-linear SheafHyperGCN over the linear SheafHyperGNN on a subset of datasets, only in the more challenging optimization scenarios, when the predicted sheaf is either *general* or restricted to be *low-rank* (as shown in Table 2). However, we believe that this advantage does not stem from the fact that non-linear models capture significantly more information. Instead, it is more likely due to the easier optimization process facilitated by the sparser computational graph, as mentioned in the first paragraph.\n\n\n**Theoretical comparison against AllSet framework**\n\nIn our research, our primary focus was on integrating the sheaf into HGNN and HGCN architectures, enabling us to explore the inductive bias theoretically. This inductive bias is an interpretable element absent in the AllSet framework. The experiments  showing that SheafHGNN not only obtains superior performance compared to HGNN, but also surpasses the more powerful AllSet architectures (AllDeepSets and AllSetTransformers) suggests that the imposed inductive bias helps our model to more effectively learn the downstream task. \n\nThe linear SheafHNN utilises the aggregated message function in the form of $f_{e} = \\sum_{v \\in e} \\tilde{h}(x_v, z_e) \\times  x_v$, where $\\tilde{h}(x_v, z_e)$ represent the sheaf associated with the pair $(v,e)$. Comparing the capacity of SheafHNN against the AllSet framework leads to investigating the universality of this message-passing function, which is an interesting, but highly challenging task. Our theoretical framework does not provide a comparison of this sort. In the current form, there might exist functions that the general AllDeepSet architecture (with an aggregated message in the form $MLP_1(\\sum_{v \\in e} MLP_2(x_v))$) can represent, while SheafHNN does not.\n\nWhile being outside the scope of the current work,  we mention that integrating the sheaf component into AllDeepSet is an easy step forward. This integration involves replacing the propagation function $MLP_1(\\sum_{v \\in e} MLP_2(x_v))$ with $\\mathbf{MLP_1}(\\sum_{v \\in e} h(x_v, z_e) \\times \\mathbf{MLP_2}(x_v))$, where $h(x_v, z_e) \\in \\mathbb{R}^{d \\times d}$ is the learned sheaf associated with the incident pair $(v,e)$, $x_v \\in \\mathbb{R}^{d \\times f}$ and $\\mathbf{MLP_*}$ is applied row-wise. This would guarantee to preserve the theoretical universality of AllSet message passing (since the model will be equivalent to the classical AllSet for the trivial sheaf). However, it comes at the cost of losing the interpretable inductive bias that SheafHNN possesses.\n\n*[1] Hein, Matthias, Setzer, Simon, Jost, Leonardo and Rangapuram, Syama Sundar. The Total Variation on Hypergraphs - Learning on Hypergraphs Revisited.*"}]

Author rebuttal:

Meta-review: This paper introduces cellular sheaf for hypergraphs, which associates a vector space for each node and hyperedge, as well as a linear map between each incident node-hyperedge pair. The linear map is learned from the node-hyperedge features, aiming to project nodes to hyperedge-specific embedding space before performing the diffusion/message passing, thus enabling a higher expressive power compared to traditional hypergraph Laplacian that treats nodes equally within a hyperedge. The proposed methods achieve superior performance on multiple benchmark datasets. One improvement area is on the presentation: the motivation of cellular sheaf can be more clearly discussed. More explanation and intuition could be put to the introduction part, which will make the paper easier to understand for readers not familiar with cellular sheaf. Overall, an acceptance is recommended by the AC.
Decision: Accept (poster)