Title: Sheaf Hypergraph Networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: hypergraph neural networks

hypergraph

sheaf

higher-order
Abstract: Higher-order relations are widespread in nature, with numerous phenomena involving complex interactions that extend beyond simple pairwise connections. As a result, advancements in higher-order processing can accelerate the growth of various fields requiring structured data. Current approaches typically represent these interactions using hypergraphs.
We enhance this representation by introducing cellular sheaves for hypergraphs, a mathematical construction that adds extra structure to the conventional hypergraph while maintaining their local, higher-order connectivity. Drawing inspiration from existing Laplacians in the literature, we develop two unique formulations of sheaf hypergraph Laplacians: linear and non-linear. Our theoretical analysis demonstrates that incorporating sheaves into the hypergraph Laplacian provides a more expressive inductive bias than standard hypergraph diffusion, creating a powerful instrument for effectively modelling complex data structures.
We employ these sheaf hypergraph Laplacians to design two categories of models: Sheaf Hypergraph Neural Networks and Sheaf Hypergraph Convolutional Networks. These models generalize classical Hypergraph Networks often found in the literature. Through extensive experimentation, we show that this generalization significantly improves performance, achieving top results on multiple benchmark datasets for hypergraph node classification.

Sheaf Hypergraph Networks

Abstract

Higher-order relations are widespread in nature, with numerous phenomena involving complex interactions that extend beyond simple pairwise connections. As a result, advancements in higher-order processing can accelerate the growth of various fields requiring structured data. Current approaches typically represent these interactions using hypergraphs. We enhance this representation by introducing cellular sheaves for hypergraphs, a mathematical construction that adds extra structure to the conventional hypergraph while maintaining their local, higherorder connectivity. Drawing inspiration from existing Laplacians in the literature, we develop two unique formulations of sheaf hypergraph Laplacians: linear and non-linear. Our theoretical analysis demonstrates that incorporating sheaves into the hypergraph Laplacian provides a more expressive inductive bias than standard hypergraph diffusion, creating a powerful instrument for effectively modelling complex data structures. We employ these sheaf hypergraph Laplacians to design two categories of models: Sheaf Hypergraph Neural Networks and Sheaf Hypergraph Convolutional Networks. These models generalize classical Hypergraph Networks often found in the literature. Through extensive experimentation, we show that this generalization significantly improves performance, achieving top results on multiple benchmark datasets for hypergraph node classification.

1 Introduction

The prevalence of relational data in real-world scenarios has led to rapid development and widespread adoption of graph-based methods in numerous domains [1-4]. However, a major limitation of graphs is their inability to represent interactions that goes beyond pairwise relations. In contrast, realworld interactions are often complex and multifaceted. There is evidence that higher-order relations frequently occur in neuroscience [5, 6], chemistry [7], environmental science [8, 9] and social networks [10]. Consequently, learning powerful and meaningful representations for hypergraphs has emerged as a promising and rapidly growing subfield of deep learning [11-16]. However, current hypergraph-based models struggle to capture higher-order relationships effectively. As described in [17], conventional hypergraph neural networks often suffer from the problem of over-smoothing. As we propagate the information inside the hypergraph, the representations of the nodes become uniform across neighbourhoods. This effect hampers the capability of hypergraph models to capture local, higher-order nuances.

More powerful and flexible mathematical constructs are required to capture real-world interactions' complexity better. Sheaves provide a suitable enhancement for graphs that allow for more diverse and expressive representations. A cellular sheaf [18] enables to attach data to a graph, by associating vector spaces to the nodes, together with a mechanism of transferring the information along the

edges. This approach allows for richer data representation and enhances the ability to model complex interactions.

Motivated by the need for more expressive structures, we introduce a cellular sheaf for hypergraphs , which allows for the representation of more sophisticated dynamics while preserving the higher-order connectivity inherent to hypergraphs. We take on the non-trivial challenge to generalize the two commonly used hypergraph Laplacians [19, 11] to incorporate the richer structure sheaves offer. Theoretically, we demonstrate that the diffusion process derived using the sheaf hypergraph Laplacians that we propose induces a more expressive inductive bias than the classical hypergraph diffusion. Leveraging this enhanced inductive bias, we construct and test two powerful neural networks capable of inferring and processing hypergraph sheaf structure: the Sheaf Hypergraph Neural Network (SheafHyperGNN) and the Sheaf Hypergraph Convolutional Network (SheafHyperGCN).

The introduction of the cellular sheaf for hypergraphs expands the potential for representing complex interactions and provides a foundation for more advanced techniques. By generalizing the hypergraph Laplacians with the sheaf structure, we can better capture the nuance and intricacy of real-world data. Furthermore, our theoretical analysis provides evidence that the sheaf hypergraph Laplacians embody a more expressive inductive bias, essential for obtaining strong representations.

Our main contributions are summarised as follow:

We introduce the cellular sheaf for hypergraphs , a mathematical construct that enhances the hypergraphs with additional structure by associating a vector space with each node and hyperedge, along with linear projections that enable information transfer between them.

We propose both a linear and a non-linear sheaf hypergraph Laplacian , generalizing the standard hypergraph Laplacians commonly used in the literature. We also provide a theoretical characterization of the inductive biases generated by the diffusion processes of these Laplacians, showcasing the benefits of utilizing these novel tools for effectively modeling intricate phenomena.

The two sheaf hypergraph Laplacians are the foundation for two novel architectures tailored for hypergraph processing: Sheaf Hypergraph Neural Network and Sheaf Hypergraph Convolutional Network . Experimental findings demonstrate that these models achieve top results, surpassing existing methods on numerous benchmarking datasets.

2 Related work

Sheaves on Graphs. Utilizing graph structure in real-world data has improved various domains like healthcare [1], biochemistry [2], social networks [20], recommendation systems [3], traffic prediction [21], with graph neural networks (GNNs) becoming the standard for graph representations. However, in heterophilic setups, when nodes with different labels are likely to be connected, directly processing the graph structure leads to weak performance. In [22], they address this by attaching additional geometric structure to the graph, in the form of cellular sheaves [18].

A cellular sheaf on graphs associates a vector space with each node and each edge together with a linear projection between these spaces for each incident pair. To take into account this more complex geometric structure, SheafNN [23] generalised the classical GNNs [24-26] by replacing the graph Laplacian with a sheaf Laplacian [27]. Higher-dimensional sheaf-based neural networks are explored, with sheaves either learned from the graph [22] or deterministically inferred for efficiency [28]. Recent methods integrate attention mechanisms [29] or replace propagation with wave equations [30]. In recent developments, Sheaf Neural Networks have been found to significantly enhance the performance of recommendation systems, as they improve upon the limitations of graph neural networks [31].

In the domain of heterogeneous graphs, the concept of learning unique message functions for varying edges is well-established. However, there's a distinction in how sheaf-based methods approach this task compared to heterogeneous methods such as RGCN [32]. Unlike the latter, which learns individual parameters for each kind of incident relationship, sheaf-based methods dynamically predict projections for each relationship, relying on features associated with the node and hyperedge. As a result, the total parameters in sheaf networks do not escalate with an increase in the number of hyperedges. This difference underscores a fundamental shift in paradigm between the two methods.

Fuses= MLP(o ) ERdxd

IMPLICITBIAS:

(m)DFuD

LINEAR LAPLACIAN

u,vEe

u

e3

e2

(LF)uv=

Se

eEE

u,vEe

e2

ei

LAPLACIAN

(CF)uv=-

u

Se

eEE

u~eV

u ~e v if (u, v) = argmaxu,veellFueXu—Fvoex↓

NON-LINEAR

IMPLICITBIAS:

1

max/FceD-—FuseDαull12

8e

u,vEe

u~e2

Hypergraph Networks. Graphs, while useful, have a strong limitation: they represent only pairwise relations. Many natural phenomena involve complex, higher-order interactions [33-35, 9], requiring a more general structure like hypergraphs. Recent deep learning approaches have been developed for hypergraph structures. HyperGNN [11] expands the hypergraph into a weighted clique and applies message passing similar to GCNs [24]. HNHN [36] improves this with non-linearities, while HyperGCN [37] connects only the most discrepant nodes using a non-linear Laplacian. Similar to the trend in GNNs, attention models gain popularity also in the hypergraph domain. HCHA [38] uses an attention-based incidence matrix, computed based on a nodes-hyperedge similarity. Similarly, HERALD[39] uses a learnable distance to infer a soft incidence matrix. On the other hand, HEAT [15] creates messages by propagating information inside each hyperedge using Transformers [40].

Many hypergraph neural network (HNN) methods can be viewed as two-stage frameworks: 1) sending messages from nodes to hyperedges and 2) sending messages back from hyperedges to nodes. Thus, [41] proposes a general framework where the first step is the average operator, while the second stage could use any existing GNN module. Similarly, [42] uses either DeepSet functions [43] or Transformers [40] to implement the two stages, while [44] uses a GNN-like aggregator in both stages, with distinct messages for each (node, hyperedge) pair.

In contrast, we propose a novel model to improve the hypergraph processing by attaching a cellular sheaf to the hypergraph structure and diffusing the information inside the model according to it. We will first introduce the cellular sheaf for hypergraph, prove some properties for the associated Laplacians, and then propose and evaluate two architectures based on the sheaf hypergraph Laplacians.

3 Hypergraph Sheaf Laplacian

An undirected hypergraph is a tuple H = ( V, E ) where V = { 1 , 2 . . . n } is a set of nodes (also called vertices), and E is a set of hyperedges (also called edges when there is no confusion with the graph

edges). Each hyperedge e is a subset of the nodes set V . We denote by n = | V | the number of nodes in the hypergraph H and by m = | E | the number of hyperedges. In contrast to graph structures, where each edge contains exactly two nodes, in a hypergraph an edge e can contain any number of nodes. The number of nodes in each hyperedge ( | e | ) is called the degree of the hyperedge and is denoted by δ e . In contrast, the number of hyperedges containing each node v is called the degree of the node and is denoted by d v .

Following the same intuition from defining sheaves on graphs [23, 22], we will introduce the cellular sheaf associated with a hypergraph H .

Definition 1. A cellular sheaf F associated with a hypergraph H is defined as a triple ⟨F ( v ) , F ( e ) , F v ⊴ e ⟩ , where:

F ( v ) are vertex stalks: vector spaces associated with each node v ;

F ( e ) are hyperedge stalks: vector spaces associated with each hyperedge e ;

F v ⊴ e : F ( v ) → F ( e ) are restriction maps: linear maps between each pair v ⊴ e , if hyperedge e contains node v .

In simpler terms, a sheaf associates a space with each node and each hyperedge in a hypergraph and also provides a linear projection that enables the movement of representations between nodes and hyperedges, as long as they are adjacent. Unless otherwise specified, we assign the same ddimensional space for all vertex stalks F ( v ) = R d and all hyperedge stalks F ( e ) = R d . We refer to d as the dimension of the sheaf.

Previous works focused on creating hypergraph representations by relying on various methods of defining a Laplacian for a hypergraph. In this work, we will concentrate on two definitions: a linear version of the hypergraph Laplacian as used in [11], and a non-linear version of the hypergraph Laplacian as in [37]. We will extend both of these definitions to incorporate the hypergraph sheaf structure, analyze the advantages that arise from this, and propose two different neural network architectures based on each one of them. For a visual comparison between the two proposed sheaf hypergraph Laplacians, see Figure 1.

3.1 Linear Sheaf Hypergraph Laplacian

Definition 2. Following the definition of a cellular sheaf on hypergraphs, we introduce the linear sheaf hypergraph Laplacian associated with a hypergraph H as ( L F ) vv = ∑ e ; v ∈ e 1 δ e F T v ⊴ e F v ⊴ e ∈ R d × d and

( L F ) uv = -∑ e ; u,v ∈ e 1 δ e F T u ⊴ e F v ⊴ e ∈ R d × d , where F v ⊴ e : R d → R d represents the linear restriction maps guiding the flow of information from node v to hyperedge e .

The linear sheaf Laplacian operator for node v applied on a signal x ∈ R n × d can be rewritten as:

̸

When each hyperedge contains exactly two nodes (thus H is a graph), the internal summation will contain a single term, and we recover the sheaf Laplacian for graphs as formulated in [22].

On the other hand, for the trivial sheaf, when the vertex and hyperedge stalks are both fixed to be R and the restriction map is the identity F v ⊴ e = 1 we recover the usual linear hypergraph Laplacian [11, 45] defined as L ( x ) v = ∑ e ; v ∈ e 1 δ e ∑ u ∈ e ( x v -x u ) . However, when we allow for higher-dimensional stalks R d , the restriction maps for each adjacency pair ( v, e ) become linear projections F v ⊴ e ∈ R d × d , enabling us to model more complex propagations, customized for each incident (node, hyperedge) pairs.

In the following sections, we will demonstrate the advantages of using this sheaf hypergraph diffusion instead of the usual hypergraph diffusion.

Reducing energy via linear diffusion. Previous work [17] demonstrates that diffusion using the classical symmetric normalised version of the hypergraph Laplacian ∆ = D -1 2 L D -1 2 , where D is a diagonal matrix containing the degrees of the vertices, reduces the following energy function:

E L 2 ( x ) = 1 2 ∑ e 1 δ e ∑ u,v ∈ e || d -1 2 v x v -d -1 2 u x u || 2 2 . Intuitively, this means that applying diffusion using the linear hypergraph Laplacian leads to similar representations for neighbouring nodes. While this is desirable in some scenarios, it may cause poor performance in others, a phenomenon known as over-smoothing [17]. In the following, we show that applying diffusion using the linear sheaf hypergraph Laplacian addresses these limitations by implicitly minimizing a more expressive energy function. This allows us to model phenomena that were not accessible using the usual Laplacian.

Definition 3. We define sheaf Dirichlet energy of a signal x ∈ R n × d on a hypergraph H as:

where D v = ∑ e ; v ∈ e F T v ⊴ e F v ⊴ e is a normalisation term equivalent to the nodes degree d v for the trivial sheaf and D = diag ( D 1 , D 2 . . . D n ) the corresponding block diagonal matrix.

This quantity measures the discrepancy between neighbouring nodes in the hyperedge stalk domain as opposed to the usual Dirichlet energy for hypergraphs that, instead, measures this distance in the node features domain. In the following, we are showing that, applying hypergraph diffusion using the linear sheaf Laplacian implicitly reduces this energy.

Proposition 1. The diffusion process using a symmetric normalised version of the linear sheaf hypergraph Laplacian minimizes the sheaf Dirichlet energy of a signal x on a hypergraph H . Moreover, the energy decreases with each layer of diffusion.

Concretely, defining the diffusion process as Y = ( I -∆ F ) X where ∆ F = D -1 2 L F D -1 2 ∈ R nd × nd represents the symmetric normalised version of the linear sheaf hypergraph Laplacian, we have that E F L 2 ( Y ) < λ ∗ E F L 2 ( X ) , with { λ i } the non-zero eigenvalues of ∆ F and λ ∗ = max i { (1 -λ i ) 2 } < 1 . All the proofs are in the Supplementary Material.

This result addresses some of the limitations of standard hypergraph processing. First, while classical diffusion using hypergraph Laplacian brings closer representations of the nodes in the nodes space ( x v , x u ), our linear sheaf hypergraph Laplacian allows us to bring closer representations of the nodes in the more complex space associated with the hyperedges ( F v ⊴ e x v , F u ⊴ e x u ). This encourages a form of hyperedge agreement, while preventing the nodes to become uniform. Secondly, in the hyperedge stalks, each node can have a different representation for each hyperedge it is part of, leading to a more expressive processing compared to the classical methods. Moreover, in many Hypergraph Networks, the hyperedges uniformly aggregate information from all its components. Through the presence of a restriction map for each (node, hyperedge) pair, we enable the model to learn the individual contribution that each node sends to each hyperedge.

From an opinion dynamic perspective [46] when the hyperedges represent group discussions, the input space x v can be seen as the private opinion, while the hyperedge stalk F v ⊴ e x v can be seen as a public opinion (what an individual v decide to express in a certain group e ). Minimizing the Dirichlet energy creates private opinions that are in consensus inside each hyperedge, while minimizing the sheaf Dirichlet energy creates an apparent consensus, by only uniformizing the expressed opinion. Through our sheaf setup, each individual is allowed to express varying opinion in each group it is part of, potentially different than their personal belief. This introduces a realistic scenario inaccessible in the original hypergraph diffusion setup.

3.2 Non-Linear Sheaf Hypergraph Laplacian

Although the linear hypergraph Laplacian is commonly used to process hypergraphs, it falls short in fully preserving the hypergraph structure [47]. To address these shortcomings, [48] introduces the non-linear Laplacian, demonstrating that its spectral properties are more suited for higher-order processing compared to the linear Laplacian. For instance, compared to the linear version, the non-linear Laplacian leads to a more balanced partition in the minimum cut problem, a task known to be tightly related to the semi-supervised node classification. Additionally, while the linear Laplacian associates a clique for each hyperedge, the non-linear one offers the advantage of relying on a much sparser connectivity. We will adopt a similar methodology to derive the non-linear version of the sheaf hypergraph Laplacian and analyze the benefits of applying diffusion using this operator.

Definition 4. We introduce the non-linear sheaf hypergraph Laplacian of a hypergraph H with respect to a signal x as following:

For each hyperedge e , compute ( u e , v e ) = argmax u,v ∈ e ||F u◁e x u -F v◁e x v || , the set of pairs containing the nodes with the most discrepant features in the hyperedge stalk.

Build an undirected graph G H containing the same sets of nodes as H and, for each hyperedge e connects the most discrepant nodes ( u, v ) (from now on we will write u ∼ e v if they are connected in the G H graph due to the hyperedge e). If multiple pairs have the same maximum discrepancy, we will randomly choose one of them.

Define the sheaf non-linear hypergraph Laplacian as:

Note that the sheaf structure impacts the non-linear diffusion in two ways: by shaping the graph structure creation (Step 1), where the two nodes with the greatest distance in the hypergraph stalk are selected rather than those in the input space; and by influencing the information propagation process (Step 3). When the sheaf is restricted to the trivial case ( d = 1 and F v ⊴ e = 1 ) this corresponds to the non-linear sheaf Laplacian of a hypergraph as introduced in [48].

Reducing total variation via non-linear diffusion. In the following discussion, we will demonstrate how transitioning from a linear to a non-linear sheaf hypergraph Laplacian alters the energy guiding the inductive bias. This phenomenon was previously investigated for the classical hypergraph Laplacian, with [48] revealing enhanced expressivity in the non-linear case.

Definition 5. We define the sheaf total variation of a signal x ∈ R n × d on a hypergraph H as:

where D v = ∑ e ; v ∈ e F T v ⊴ e F v ⊴ e is a normalisation term equivalent to the node's degree in the classical setup and D = diag ( D 1 , D 2 . . . D n ) is the corresponding block diagonal matrix.

This quantity generalised the total variance (TV) ¯ E TV ( x ) = 1 2 ∑ e 1 δ e max u,v ∈ e || d -1 2 v x v -d -1 2 u x u || 2 2 minimized in the non-linear hypergraph label propagation [48, 49]. Unlike the TV, the sheaf total variation measures the highest discrepancy at the hyperedge level computed in the hyperedge stalk, as opposed to the TV, which gauges the highest discrepancy in the feature space. We will explore the connection between the sheaf TV and our non-linear sheaf hypergraph diffusion .

Proposition 2. The diffusion process using the symmetric normalised version of non-linear sheaf hypergraph Laplacian minimizes the sheaf total variance of a signal x on hypergraph H .

Despite the change in the potential function being minimized, the overarching objective remains akin to that of the linear case: striving to achieve a coherent consensus among the representations within the hyperedge stalk space, rather than generating uniform features for each hyperedge in the input space. In contrast to the linear scenario, where a quadratic number of edges is required for each hyperedge, the non-linear sheaf hypergraph Laplacian associates a single edge with each hyperedge, thereby enhancing computational efficiency.

3.3 Sheaf Hypergraph Networks

Popular hypergraph neural networks [45, 11, 37, 50] draw inspiration from a variety of hypergraph diffusion operators [47, 48, 51], giving rise to diverse message passing techniques. These techniques all involve the propagation of information from nodes to hyperedges and vice-versa. We will adopt a similar strategy and introduce the Sheaf Hypergraph Neural Network and Sheaf Hypergraph Convolutional Network, based on two message-passing schemes inspired by the sheaf diffusion mechanisms discussed in this paper.

Given a hypergraph H = ( V, E ) with nodes characterised by a set of features X ∈ R n × f , we initially linearly project the input features into ˜ X ∈ R n × ( df ) and then reshape them into ˜ X ∈ R nd × f . As a result, each node is represented in the vertex stalk as a matrix R d × f , where d denotes the dimension of the vertex stalk, and f indicates the number of channels.

A general layer of Sheaf Hypergraph Network is defined as:

Here, · ∆ can be either ∆ F = D -1 2 L F D -1 2 for the linear sheaf hypergraph Laplacian introduced in Eq. 1 or ¯ ∆ F = D -1 2 ¯ L F D -1 2 for the non-linear sheaf hypergraph Laplacian introduced in Eq. 2. Both W 1 ∈ R d × d and W 2 ∈ R f × f are learnable parameters, while σ represents ReLU non-linearity.

Sheaf Hypergraph Neural Network (SheafHyperGNN). This model utilizes the linear sheaf hypergraph Laplacian · ∆ = ∆ F . When the sheaf is trivial ( d = 1 and F v ⊴ e = 1 ), and W 1 = I d , the SheafHyperGNN is equivalent to the conventional HyperGNN architecture [11]. However, by increasing dimension d and adopting dynamic restriction maps, our proposed SheafHyperGNN becomes more expressive. For every adjacent node-hyperedge pair ( v, e ) , we use a d × d block matrix to discern each node's contribution instead of a fixed weight that only stores the incidence relationship. The remaining operations are similar to those in HyperGNN [11]. More details on how the block matrices F v ⊴ e are learned can be found in the following subsection.

Sheaf Hypergraph Convolutional Network (SheafHyperGCN). This model employs the non-linear

Laplacian · ∆ = ¯ ∆ F . Analogous to the linear case, when the sheaf is trivial and W 1 = I d we obtain the classical HyperGCN architecture [37]. In our experiments, we will use an approach similar to that in [37] and adjust the Laplacian to include mediators. This implies that we will not only connect the two most discrepant nodes but also create connections between each node in the hyperedge and these two most discrepant nodes, resulting in a denser associated graph. For more information on this variation, please refer to [37] or Supplementary Material.

In summary, the models introduced in this work, SheafHyperGNN and SheafHyperGCN serve as generalisations of the classical HyperGNN [11] and HyperGCN [37]. These new models feature a more expressive implicit regularisation compared to their traditional counterparts.

Learnable Sheaf Laplacian. A key advantage of Sheaf Hypergraph Networks lies in attaching and processing a more complex structure (sheaf) instead of the original standard hypergraph. Different sheaf structures can be associated with a single hypergraph, and accurately modeling the most suitable structure is crucial for obtaining effective and meaningful representation. In our proposed models, we achieve this by designing learnable restriction maps. For a d-dimensional sheaf, we predict the restriction maps for each pair of incident (vertex v, hyperedge e) as F v ⊴ e = MLP ( x v || h e ) ∈ R d 2 , where x v represent node features of v , and h e represents features of the hyperedge e . This vector representation is then reshaped into a d × d block matrix representing the linear restriction map for the ( v, e ) pair. When hyperedge features h e are not provided, any permutation-invariant operation can be applied to obtain hyperedge features from node-level features. We experiment with three types of d × d block matrices: diagonal, low-rank and general matrices, with the diagonal version

consistently outperforming the other two. These restriction maps are further used to define the sheaf hypergraph Laplacians (Def. 2, or 4) used in the final Sheaf Hypergraph Networks. Please refer to the Supplementary Material for more details on how we constrain the restriction maps.

4 Experimental Analysis

We evaluate our model on eight real-world datasets that vary in domain, scale, and heterophily level and are commonly used for benchmarking hypergraphs. These include Cora, Citeseer, Pubmed, Cora-CA, DBLP-CA [37], House [52], Senate and Congress [53]. To ensure a fair comparison with the baselines, we follow the same training procedures used in [50] by randomly splitting the data into 50% training samples, 25% validation samples and 25% test samples, and running each model 10 times with different random splits. We report average accuracy along with the standard deviation.

Additionally, we conduct experiments on a set of synthetic heterophilic datasets inspired by those introduced by [50]. Following their approach, we generate a hypergraph using the contextual hypergraph stochastic block model [54-56], containing 5000 nodes: half belong to class 0 while the other half to class 1 . We then randomly sample 1000 hyperedges with a cardinality 15 , each containing exactly β nodes from class 0 . The heterophily level is computed as α = min ( β, 15 -β ). Node features are sampled from a label-dependent Gaussian distribution with a standard deviation of 1 . As the original dataset is not publicly available, we generate our own set of datasets by varying the heterophily level α ∈ { 1 . . . 7 } and rerun their experiments for a fair comparison.

The experiments are executed on a single NVIDIA Quadro RTX 8000 with 48GB of GPU memory. Unless otherwise specified, our results represent the best performance obtained by each architecture using hyper-parameter optimisation with random search. Details on all the model choices and hyper-parameters can be found in the Supplementary Material.

Laplacian vs Sheaf Laplacian. As demonstrated in the previous section, SheafHyperGNN and SheafHyperGCN are generalisations of the standard HyperGNN [11] and HyperGCN [37], respectively. They transition from the trivial sheaf ( d = 1 and F v◁e = 1 ) to more complex structures ( d ≥ 1 and F v◁e a d × d learnable projection). The results in Table 1 and Table 3 show that both models significantly outperform their counterparts on all tested datasets. Among our models, the one based on linear Laplacian (SheafHyperGNN) consistently outperforms the model based on non-linear Laplacian (SheafHyperGCN) across all datasets. This observation aligns with the performance of the models based on standard hypergraph Laplacian, where HyperGCN is outperformed by HyperGNN in all but two real-world datasets, despite their theoretical advantage [48].

Comparison to recent methods. Wealso compare to several recent models from the literature such as HCHA[38], HNHN [36], AllDeepSets [42], AllSetTransformer [42], UniGCNII [57], HyperND [58], and ED-HNN [50]. Our models achieve competitive results on all real-world datasets, with state-ofthe art performance on five of them (Table 1). These results confirm the advantages of using the sheaf Laplacians for processing hypergraphs. We also compare our models against a series baselines on the synthetic heterophilic dataset. The results are shown in Table 3. Our best model, SheafHyperGNN, consistently outperforms the other models across all levels of heterophily. Note that, our framework enhancing classical hypergraph processing with sheaf structure is not restricted to the two traditional models tested in this paper (HyperGNN and HyperGCN). Most of the recent state-of-the-art methods, such as ED-HNN, could be easily adapted to learn and process our novel cellular sheaf hypergraph instead of the standard hypergraph, leading to further advancement in the hypergraph field.

Table 3: Accuracy on Synthetic Datasets with Varying Heterophily Levels : Across all different level of heterophily ( α ), our sheaf-based methods SheafGCN and SheafHGNN consistently outperform their counterparts. Additionally, they achieve top results for all heterophily levels, further demonstrating their effectiveness. For each experiment, the result represents average accuracy over 10 runs.

heterophily (

α

)

Name

1

2

3

4

5

6

7

HyperGCN

83.9

69.4

72.9

75.9

70.5

67.3

66.5

HyperGNN

98.4

83.7

79.4

74.5

69.5

66.9

63.8

HCHA

98.1

81.8

78.3

75.88

74.1

71.1

70.8

ED-HNN

99.9

91.3

88.4

84.1

80.7

78.8

76.5

SheafHGCN

100

87.1

84.8

79.2

78.1

76.6

75.5

SheafHGNN

100

94.2

90.8

86.5

82.1

79.8

77.3

Influenceofstalkdimension

Influenceof#layers

800

75

700

65

70

HGNN_accuracy

SheafHGNN_accuracy

SheafHGNN_energy

400

600

500

Energy

HGNN_energy

300

60

200

Diag-SheafHyperGNN

55

100

Diag-SheafHyperGCN

Dimension of stalk(d)

2

Layer idx

4

b

In the following sections, we conduct a series of ablation studies to gain a deeper understanding of our models. We will explore various types of restriction maps, analyze how performance changes when varying the network depth and study the importance of stalk dimension for final accuracy.

Investigating the Restriction Maps. Both linear and non-linear sheaf hypergraph Laplacians rely on attaching a sheaf structure to the hypergraph. For a cellular sheaf F with vertex stalks F ( v ) = R d and hyperedge stalks F ( e ) = R d as used in our experiments, this involves inferring the restriction maps F v◁e ∈ R d × d for each incidence pair ( v, e ) . We implement these as a function dependent on corresponding nodes and hyperedge features: F v ⊴ e = MLP ( x v || h e ) ∈ R d 2 . Learning these matrices can be challenging; therefore, we experimented with adding constraints to the type of matrices used as restriction maps. In Table 2 we show the performance obtained by our models when constraining the restriction maps to be either diagonal (Diag-SheafHyperNN), low-rank (LR-SheafHyperNN) or general matrices (Gen-SheafHyperNN). We observe that the sheaves equipped with diagonal restriction maps perform better than the more general variations. We believe that the advantage of the diagonal restriction maps is due to easier optimization, which overcomes the lose in expressivity. More details about predicting constrained d × d matrices can be found in the Supplementary Material.

Importance of Stalk Dimension. The standard hypergraph Laplacian corresponds to a sheaf Laplacian with d = 1 and F v ⊴ e = 1 . Constraining the stalk dimension to be 1 , but allowing the restriction maps to be dynamically predicted, becomes similar to an attention mechanism [38]. However, attention models are restricted to guiding information via a scalar probability, thus facing the same over-smoothing limitations as traditional HyperGNN in the heterophilic setup. Our d-dimensional restriction maps increase the model's expressivity by enabling more complex information transfer between nodes and hyperedges, tailored for each individual pair. We validate this experimentally on the synthetic heterophilic dataset, using the diagonal version of the models, which achieves the best performance in the previous ablation. In Figure 2, we demonstrate how performance significantly improves when allowing higher-dimensional stalks ( d > 1 ). These results are consistent for both linear sheaf Laplacian-based models (SheafHyperGNN) and non-linear ones (SheafHyperGCN).

Influence of Depth. It is well-known that stacking many layers in a hypergraph network can lead to a decrease in model performance, especially in the heterophilic setup. This phenomenon, called oversmoothing, is well-studied in both graph [59] and hypergraph literature [17]. To analyse the extent to which our model suffers from this limitation, we train a series of models on the most heterophilic version of the synthetic dataset ( α = 7 ). For both SheafHyperGNN and its HyperGNN equivalent, we vary the number of layers between 1 -8 . In Figure 2, we observe that while HyperGNN exhibits a drop in performance when going beyond 3 layers, SheafHyperGNN's performance remains mostly constant. Similar results were observed for the non-linear version when comparing SheafHyperGCN with HyperGCN (results in Supplementary Material). These results indicates potential advantages of our models in the heterophilic setup by allowing the construction of deeper architectures.

Investigating Features Diversity. Our theoretical analysis shows that, while conventional Hypergraph Networks tend to produce similar features for neighbouring nodes, our Sheaf Hypergraph

Networks reduce the distance between neighbouring nodes in the more complex hyperedge stalk space. As a result, the nodes' features do not become uniform, preserving their individual identities. We empirically evaluate this, by computing the Dirichlet energy for HyperGNN and SheafHyperGNN (shaded area in Figure 2), as a measure of similarity between neighbouring nodes. The results are aligned with the theoretical analysis: while increasing depth in HyperGNN creates uniform features, SheafHyperGNN does not suffer from this limitation, encouraging diversity between the nodes.

5 Conclusion

In this paper we introduce the cellular sheaf for hypergraphs, an expressive tool for modelling higherorder relations build upon the classical hypergraph structure. Furthermore, we propose two models capable of inferring and processing the sheaf hypergraph structure, based on linear and non-linear sheaf hypergraph Laplacian, respectively. We prove that the diffusion processes associated with these models induce a more expressive implicit regularization, extending the energies associated with standard hypergraph diffusion. This novel architecture generalizes classical Hypergraph Networks, and we experimentally show that it outperform existing methods on several datasets. Our technique of replacing the hypergraph Laplacian with a sheaf hypergraph Laplacian in both HyperGNN and HyperGCN establishes a versatile framework that can be employed to "sheafify" other hypergraph architectures. We believe that sheaf hypergraphs can contribute to further advancements in the rapidly evolving hypergraph community, extending far beyond the results presented in this work.

Acknowledgment The authors would like to thank Ferenc Huszár for fruitful discussions and constructive suggestions during the development of the paper and Eirik Fladmark and Laura Brinkholm Justesen for fixing a minor issue in the original HyperGCN code, which led to improved results in the baselines. Iulia Duta is a PhD student funded by a Twitter scholarship. This work was also supported by PNRR MUR projects PE0000013-FAIR, SERICS (PE00000014), Sapienza Project FedSSL, and IR0000013-SoBigData.it.

References

Catherine Tong, Emma Rocheteau, Petar Veliˇ ckovi´ c, Nicholas Lane, and Pietro Lio. Predicting Patient Outcomes with Graph Representation Learning , pages 281-293. 01 2022.

Kexin Huang, Cao Xiao, Lucas M Glass, Marinka Zitnik, and Jimeng Sun. Skipgnn: predicting molecular interactions with skip-graph networks. Scientific reports , 10(1):1-16, 2020.

Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 974-983, 2018.

Xiaolong Wang and Abhinav Gupta. Videos as space-time region graphs. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 399-417, 2018.

Nicolas A. Crossley, Andrea Mechelli, Jessica Scott, Francesco Carletti, Peter T. Fox, Philip K. McGuire, and Edward T. Bullmore. The hubs of the human connectome are generally implicated in the anatomy of brain disorders. Brain , 137:2382 - 2395, 2014.

Tingting Guo, Yining Zhang, Yanfang Xue, and Lishan Qiao. Brain function network: Higher order vs. more discrimination. Frontiers in Neuroscience , 15, 08 2021.

Jürgen Jost and Raffaella Mulas. Hypergraph laplace operators for chemical reaction networks, 2018.

Pragya Singh and Gaurav Baruah. Higher order interactions and species coexistence. Theoretical Ecology , 14, 03 2021.

Alba Cervantes-Loreto, Carolyn Ayers, Emily Dobbs, Berry Brosi, and Daniel Stouffer. The context dependency of pollinator interference: How environmental conditions and co-foraging species impact floral visitation. Ecology Letters , 24, 07 2021.

Unai Alvarez-Rodriguez, Federico Battiston, Guilherme Ferraz de Arruda, Yamir Moreno, Matjaž Perc, and Vito Latora. Evolutionary dynamics of higher-order interactions in social networks. Nature Human Behaviour , 5:1-10, 05 2021.

Yifan Feng, Haoxuan You, Zizhao Zhang, Rongrong Ji, and Yue Gao. Hypergraph neural networks. Proc. Conf. AAAI Artif. Intell. , 33(01):3558-3565, July 2019.

Ramon Viñas, Chaitanya K. Joshi, Dobrik Georgiev, Bianca Dumitrascu, Eric R. Gamazon, and Pietro Liò. Hypergraph factorisation for multi-tissue gene expression imputation. bioRxiv , 2022.

Xinlei Wang, Junchang Xin, Zhongyang Wang, Chuangang Li, and Zhiqiong Wang. An evolving hypergraph convolutional network for the diagnosis of alzheimers disease. Diagnostics , 12(11), 2022.

Yichao Yan, Jie Qin, Jiaxin Chen, Li Liu, Fan Zhu, Ying Tai, and Ling Shao. Learning multigranular hypergraphs for video-based person re-identification. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020 , pages 2896-2905. IEEE, 2020.

Dobrik Georgiev Georgiev, Marc Brockschmidt, and Miltiadis Allamanis. HEAT: Hyperedge attention networks. Transactions on Machine Learning Research , 2022.

Jingcheng Wang, Yong Zhang, Lixun Wang, Yongli Hu, Xinglin Piao, and Baocai Yin. Multitask hypergraph convolutional networks: A heterogeneous traffic prediction framework. IEEE Trans. Intell. Transp. Syst. , 23(10):18557-18567, 2022.

Guanzi Chen, Jiying Zhang, Xi Xiao, and Yang Li. Preventing over-smoothing for hypergraph neural networks. 2022.

Justin Curry. Sheaves, cosheaves and applications, 2014.

T.-H. Hubert Chan, Anand Louis, Zhihao Gavin Tang, and Chenzi Zhang. Spectral properties of hypergraph laplacian and approximation algorithms. J. ACM , 65(3), mar 2018.

Federico Monti, Fabrizio Frasca, Davide Eynard, Damon Mannion, and Michael M Bronstein. Fake news detection on social media using geometric deep learning. arXiv preprint arXiv:1902.06673 , 2019.

Bing Yu, Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting. pages 3634-3640. International Joint Conferences on Artificial Intelligence Organization, 7 2018.

Cristian Bodnar, Francesco Di Giovanni, Benjamin Paul Chamberlain, Pietro Liò, and Michael M. Bronstein. Neural sheaf diffusion: A topological perspective on heterophily and oversmoothing in GNNs. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Jakob Hansen and Thomas Gebhart. Sheaf neural networks. In TDA & Beyond , 2020.

Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR) , 2017.

Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in neural information processing systems , pages 3844-3852, 2016.

Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks on graphs. CoRR , abs/1312.6203, 2013.

Jakob Hansen and Robert Ghrist. Toward a spectral theory of cellular sheaves. Journal of Applied and Computational Topology , 3(4):315-358, aug 2019.

Federico Barbero, Cristian Bodnar, Haitz Sáez de Ocáriz Borde, Michael Bronstein, Petar Veliˇ ckovi´ c, and Pietro Lio. Sheaf neural networks with connection laplacians. 06 2022.

Federico Barbero, Cristian Bodnar, Haitz Sáez de Ocáriz Borde, and Pietro Lio. Sheaf attention networks. In NeurIPS 2022 Workshop on Symmetry and Geometry in Neural Representations , 2022.

Julian Suk, Lorenzo Giusti, Tamir Hemo, Miguel Lopez, Konstantinos Barmpas, and Cristian Bodnar. Surfing on the neural sheaf. In NeurIPS 2022 Workshop on Symmetry and Geometry in Neural Representations , 2022.

Antonio Purificato, Giulia Cassarà, Pietro Liò, and Fabrizio Silvestri. Sheaf neural networks for graph-based recommender systems, 2023.

Michael Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks, 2017.

Chen Zu, Yue Gao, Brent Munsell, Minjeong Kim, Ziwen Peng, Yingying Zhu, Wei Gao, Daoqiang Zhang, Dinggang Shen, and Guorong Wu. Identifying high order brain connectome biomarkers via learning on hypergraph. Mach Learn Med Imaging , 10019:1-9, October 2016.

Gregor Urban, Christophe N. Magnan, and Pierre Baldi. Sspro/accpro 6: almost perfect prediction of protein secondary structure and relative solvent accessibility using profiles, deep learning and structural similarity. Bioinform. , 38(7):2064-2065, 2022.

Yoshinobu Inada and Keiji Kawachi. Order and flexibility in the motion of fish schools. Journal of theoretical biology , 214:371-87, 03 2002.

Yihe Dong, Will Sawin, and Yoshua Bengio. Hnhn: Hypergraph networks with hyperedge neurons. In Graph Representation Learning and Beyond Workshop at ICML 2020 , June 2020. Code available: https://github.com/twistedcubic/HNHN.

Naganand Yadati, Madhav Nimishakavi, Prateek Yadav, Vikram Nitin, Anand Louis, and Partha Talukdar. Hypergcn: A new method for training graph convolutional networks on hypergraphs. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Song Bai, Feihu Zhang, and Philip H.S. Torr. Hypergraph convolution and hypergraph attention. Pattern Recognition , 110:107637, 2021.

Jiying Zhang, Yuzhao Chen, Xiong Xiao, Runiu Lu, and Shutao Xia. Learnable hypergraph laplacian for hypergraph learning. In ICASSP , 2022.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

Jing Huang and Jie Yang. Unignn: a unified framework for graph and hypergraph neural networks. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21 , 2021.

Eli Chien, Chao Pan, Jianhao Peng, and Olgica Milenkovic. You are allset: A multiset function framework for hypergraph neural networks. In International Conference on Learning Representations , 2022.

Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

Devanshu Arya, Deepak K. Gupta, Stevan Rudinac, and Marcel Worring. Hypersage: Generalizing inductive representation learning on hypergraphs, 2020.

Song Bai, Feihu Zhang, and Philip H. S. Torr. Hypergraph convolution and hypergraph attention. CoRR , abs/1901.08150, 2019.

Jakob Hansen and Robert Ghrist. Opinion dynamics on discourse sheaves. SIAM Journal on Applied Mathematics , 81(5):2033-2060, 2021.

Sameer Agarwal, Kristin Branson, and Serge Belongie. Higher order learning with graphs. volume 2006, pages 17-24, 01 2006.

Matthias Hein, Simon Setzer, Leonardo Jost, and Syama Sundar Rangapuram. The total variation on hypergraphs - learning on hypergraphs revisited. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 26. Curran Associates, Inc., 2013.

Chenzi Zhang, Shuguang Hu, Zhihao Gavin Tang, and T-H. Hubert Chan. Re-revisiting learning on hypergraphs: Confidence interval and subgradient method. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 4026-4034. PMLR, 06-11 Aug 2017.

Peihao Wang, Shenghao Yang, Yunyu Liu, Zhangyang Wang, and Pan Li. Equivariant hypergraph diffusion neural operators. arXiv preprint arXiv:2207.06680 , 2022.

Shota Saito, Danilo Mandic, and Hideyuki Suzuki. Hypergraph p-laplacian: A differential geometry view. Proceedings of the AAAI Conference on Artificial Intelligence , 32(1), apr 2018.

Philip S. Chodrow, Nate Veldt, and Austin R. Benson. Generative hypergraph clustering: From blockmodels to modularity. Science Advances , 7(28).

James H. Fowler. Legislative cosponsorship networks in the US house and senate. Social Networks , 28(4):454-465, oct 2006.

Yash Deshpande, Subhabrata Sen, Andrea Montanari, and Elchanan Mossel. Contextual stochastic block models. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

Debarghya Ghoshdastidar and Ambedkar Dukkipati. Consistency of spectral partitioning of uniform hypergraphs under planted partition model. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 27. Curran Associates, Inc., 2014.

I Chien, Chung-Yi Lin, and I-Hsiang Wang. Community detection in hypergraphs: Optimal statistical limit and efficient algorithms. In Amos Storkey and Fernando Perez-Cruz, editors, Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics , volume 84 of Proceedings of Machine Learning Research , pages 871-879. PMLR, 09-11 Apr 2018.

Jing Huang and Jie Yang. Unignn: a unified framework for graph and hypergraph neural networks. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21 , 2021.

Francesco Tudisco, Austin R. Benson, and Konstantin Prokopchik. Nonlinear higher-order label spreading. In Proceedings of the Web Conference 2021 , WWW'21, page 2402-2413, New York, NY, USA, 2021. Association for Computing Machinery.

Xinyi Wu, Zhengdao Chen, William Wei Wang, and Ali Jadbabaie. A non-asymptotic analysis of oversmoothing in graph neural networks. In The Eleventh International Conference on Learning Representations , 2023.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\n\\\nThe primary research focus of this paper is to investigate the potential of neural networks for processing hypergraph datasets.\n\nThe paper utilises the mathematical concept of sheaf, which describes how locally defined data on a space can be consistently pieced together, to enhance hypergraphs by associating vector spaces with nodes and hyperedges, facilitating the transfer of information through linear projections.\n\nThe paper lays the groundwork for \"sheaf\" hypergraph Laplacians, examining their diffusion processes to understand the inductive biases they generate, which play a pivotal role in driving the progress of hypergraph neural networks.\n\nSTRENGTHS:\n\\\n**Clarity**\n\nThe introduction of the paper offers a concise and informative overview of the research topic, providing necessary context by proposing the generalisation of hypergraph Laplacians to advanced sheaf Laplacians for capturing intricate phenomena. \n\nIt effectively outlines the objectives of advancing neural networks on hypergraphs and sets the stage for the study and development of sheaf hypergraph neural networks.\n\nThe submitted work showcases a well-structured and organised presentation, with a logical progression of ideas, clearly defined sections and headings, and enhanced clarity through the inclusion of code and supplementary materials.\n\n\\\n**Quality**\n\nThe paper's claims are supported by sound theoretical analyses of the diffusion processes (e.g. propositions 1, 2) and empirical studies including involving different restriction maps (e.g. Table 2).\n\nBy investigating the impact of depth and heterophily levels, the paper reveals additional insights, contributing to its overall quality.\n\nIn summary, the methodology, experimental design, and theoretical framework employed to support the contributions are sound, credible, and well-founded.\n\n\\\n**Originality**\n\nBy introducing sheaf Laplacians, the paper tackles a limitation present in current hypergraph Laplacians (e.g. see lines 163-172), which have been instrumental in the development of existing hypergraph networks.\n\nBy adopting unique perspectives (e.g. opinion dynamics perspective see lines 173-180), the paper illuminates the nuanced and intricate phenomena that can be captured by the sheaf structure.\n\nOverall, the paper sheds light on a previously unexplored limitation in the existing literature, showcasing its originality.\n\nWEAKNESSES:\n\\\n**Significance**\n\nRecent research has explored sheaf-based approaches for graph data, resulting in the development of advanced graph neural network architectures [1, 2] that draw inspiration from sheaf Laplacians. \n\nHyperedges in hypergraph datasets can be represented as pairwise connections, treating them as graph datasets, and the linear sheaf Hypergraph Laplacian employed by SheafHyperGNN yields the best results among the models in Table 1.\n\nTo strengthen the significance and necessity of a Sheaf operator on higher-order data, it is crucial to conduct comprehensive empirical analyses, such as comparing with the best SheafGNN architecture as a baseline and exploring hypergraph datasets where the non-linear Laplacian captures substantially more information than its linear counterpart.\n\n1. [Sheaf Neural Networks, In TDA and Beyond, 2020](https://openreview.net/forum?id=GgcgIJsT8HD)\n2. [Neural Sheaf Diffusion: A Topological Perspective on Heterophily and Oversmoothing in GNNs, In NeurIPS'22](https://openreview.net/forum?id=vbPsD-BhOZ)\n\nQUESTIONS:\n* Does the paper include any experiments demonstrating that a neural network with a non-linear Sheaf Laplacian captures significantly more information than a linear operator? Please clarify if such analysis is already present, and if not, it would be valuable to include as an important contribution to the paper.\n* Can we rely solely on the increased expressiveness of sheaf Laplacians, or should we also consider the importance of good generalisability for achieving strong empirical performance on test data? \n* In terms of generalisation, while both the AllSet framework [3] and Sheaf Hypergraph Networks generalise and offer advancements over earlier models, does the Sheaf Networks framework have the capacity to encompass all instances of the AllSet framework, or are there any specific scenarios within the AllSet framework that cannot be accommodated by Sheaf Networks?\n\n3. [You are AllSet: A Multiset Function Framework for Hypergraph Neural Networks, In ICLR'22](https://openreview.net/forum?id=hpBTIv2uy_E)\n\nLIMITATIONS:\nAdequately addressed (e.g. see Appendix A).", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper generalizes hyper-graph Laplacians to hyper-graphs attached with sheaves, called Hypergraph Sheaf Laplacian, and constructued hyper-graph NNs with proposed Hypergraph Sheaf Laplacian. Experiments demonstrate the effectiveness of SheafHyperGNNs over normal HyperGNNs.\n\nSTRENGTHS:\nI find this paper well-written and easy to follow. The paper addresses two crucial challenges in the GNN community: hyper-graph processing and heterophily graph processing. The concept of hyper-graph sheaf Laplacian looks convincing and promising. The SheafHyperGNN is derived from hyper-graph sheaf Laplacian, which provides a rigorous framework to study the properties of it (e.g. to prove it doesn't suffer from over-smoothing). I believe both the new concept of hyper-graph sheaf Laplacian and the proposed SheafHyperGNN can be benificial to the community.\n\n-----\n\nAfter reading authors' rebuttal, I decide to still keep my current rating.\n\nWEAKNESSES:\nSee Questions.\n\nQUESTIONS:\n1. I feel the proposed methods are kind of similar to heterogeneous graph NNs, for example RGCN[1]. I wonder if the authors can discuss the relationship between their proposed methods and those heterogeneous graph NNs, for example can RGCN be viewed as a special case of SheafHyperGNN?\n\n2. Given the SheafHyperGNN has learnable sheaves, will there be an overfitting risk and do the authors have any ideas on how to avoid it? For example I personally have also tried to somehow attatch parameters to each edge in message passing, and I found the GNN simply learns zero message passing weights and degenrates to an MLP and uses the MLP to overfit the training set. Theoretically, it seems the proposed SheafHyperGNN has the same risk and I wonder if the authors observed similar phenomenons in experiments? (say do you need to tune the hyper-parameters to avoid it?) If so it would be nice to discuss potential ways to address it.\n\n\n[1] Modeling Relational Data with Graph Convolutional Networks\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes cellular sheaf to enhance the high-order relationships among hypergraphs. Specifically, it first designs two formulations, linear sheaf hypergraph Laplacian and non-linear sheaf hypergraph Laplacian to construct the hypergraph structure. Then the two sheaf hypergraph Laplacians are leveraged to design two types of models, i.e., sheaf hypergraph neural network and sheaf hypergraph convolutional networks. Detailed theoretical analysis and extensive experiments are provided to demonstrate the effectiveness of the designed model.\n\nSTRENGTHS:\n1. The topic of enhancing the hypergraph structure is impressive.\n\n2. The paper is well-organized overall and easy to follow. \n\n3. Figures are concise, enabling readers to easily catch the key points.\n\nWEAKNESSES:\n1. The motivation for the sheaf hypergraph network is not clearly discussed in this work. In the section Introduction, this work does not clearly discuss the existing limitation of extracting the high-order relationships among hypergraphs. Instead, this work directly introduces sheaf for hypergraphs, which is confusing and not consistent from my point of view. \n\n\n2. The format of the equations and some notations are very confusing in this manuscript. There should have a comma or a full stop after the equations. \nFor instance, a full stop should be added at the end of equation 2. All equations in Definitions should also be numbered. \n\n3. Some statements in this manuscript are very confusing and vague. There are also a lot of typos and grammar mistakes. For instance, line 31, 'that allows for'--> 'allow for'; In the caption of Figure 1, 'Than' --> 'Then'.\n\n4. In Table 1, SheafHyperGNN gains excellent performance over most benchmark datasets, while the performance of SheafHyperGNN over Congress is not that satisfactory than ED-HNN and the performance gap (i.e., 3\\%) between these two models is large. I would like to know the reason behind that.\n\nQUESTIONS:\n1. Can you clearly explain the motivation for this work? \n2. Please discuss the existing challenges of hypergraph models in extracting the high-order relationships among hypergraphs.\n3. I would like to know the reason behind the performance gap between SheafHyperGNN and ED-HNN over Congress.\n4. Grammar mistakes and vague statements should be revised.\n\nLIMITATIONS:\nThis work clearly discusses the limitation of the model design.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper introduces the cellular sheaves to hypergraph and proposes two sheaf hypergraph laplacians. In addition, they also propose two models: sheaf HNN and Sheaf HCN. The experiments demonstrated the effectiveness of the proposed models.

STRENGTHS:
- The paper is well-written and well-organized. 
- The authors incorporate the sheaves to hypergraph laplacian, which would promote the development of hypergraph spectral theory.
- The proposed two hypergraph neural networks provide new options for hypergraph learning.

WEAKNESSES:
1. The novelty of this paper is limited. It uses the techniques in graph-based methods[1] to existing hypergraph laplacian[2][3]. The Sheaf hypergraph networks also have the same form of Sheaf Convolutional Network (SCN). The Sheaf Dirichlet energy and its convergence property also can be trivially generalized from [4].
2. The experiments don’t bring surprising results. The replacement of trivial hypergraph Laplacian with sheaf laplacian can improve the performance, which can be easily speculated from the results in GNN[1]. 

[1]Neural Sheaf Diffusion: A Topological Perspective on Heterophily and Oversmoothing in GNNs.

[2]Hypergraph neural networks.

[3]Hypergcn: A new method for training graph convolutional networks on hypergraphs.

[4]Preventing over-smoothing for hypergraph neural networks

Minor:

Some references have incorrect author names. E.g. [43] Guanzi Chen, Jiying Zhang,et al. Preventing over-smoothing for hypergraph neural networks

QUESTIONS:
It is necessary to highlight the difficulties of using sheaf to hypergraphs and the core technological innovation of the proposed method.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-07