SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Iulia Duta
University of Cambridge id366@cam.ac.uk
Giulia Cassarà
University of Rome, La Sapienza giulia.cassara@uniroma1.it
Fabrizio Silvestri
University of Rome, La Sapienza fabrizio.silvestri@uniroma1.it
University of Cambridge p1219@cam.ac.uk
Higher-order relations are widespread in nature, with numerous phenomena involving complex interactions that extend beyond simple pairwise connections. As a result, advancements in higher-order processing can accelerate the growth of various fields requiring structured data. Current approaches typically represent these interactions using hypergraphs. We enhance this representation by introducing cellular sheaves for hypergraphs, a mathematical construction that adds extra structure to the conventional hypergraph while maintaining their local, higher-order connectivity. Drawing inspiration from existing Laplacians in the literature, we develop two unique formulations of sheaf hypergraph Laplacians: linear and non-linear. Our theoretical analysis demonstrates that incorporating sheaves into the hypergraph Laplacian provides a more expressive inductive bias than standard hypergraph diffusion, creating a powerful instrument for effectively modelling complex data structures. We employ these sheaf hypergraph Laplacians to design two categories of models: Sheaf Hypergraph Neural Networks and Sheaf Hypergraph Convolutional Networks. These models generalize classical Hypergraph Networks often found in the literature. Through extensive experimentation, we show that this generalization significantly improves performance, achieving top results on multiple benchmark datasets for hypergraph node classification.
The prevalence of relational data in real-world scenarios has led to rapid development and widespread adoption of graph-based methods in numerous domains [1-4]. However, a major limitation of graphs is their inability to represent interactions that goes beyond pairwise relations. In contrast, real-world interactions are often complex and multifaceted. There is evidence that higher-order relations frequently occur in neuroscience [5, 6], chemistry [7], environmental science [8, 9] and social networks [10]. Consequently, learning powerful and meaningful representations for hypergraphs has emerged as a promising and rapidly growing subfield of deep learning [11-16]. However, current hypergraph-based models struggle to capture higher-order relationships effectively. As described in [17], conventional hypergraph neural networks often suffer from the problem of over-smoothing. As we propagate the information inside the hypergraph, the representations of the nodes become uniform across neighbourhoods. This effect hampers the capability of hypergraph models to capture local, higher-order nuances.
More powerful and flexible mathematical constructs are required to capture real-world interactions' complexity better. Sheaves provide a suitable enhancement for graphs that allow for more diverse and expressive representations. A cellular sheaf [18] enables to attach data to a graph, by associating vector spaces to the nodes, together with a mechanism of transferring the information along the
edges. This approach allows for richer data representation and enhances the ability to model complex interactions.
Motivated by the need for more expressive structures, we introduce a cellular sheaf for hypergraphs, which allows for the representation of more sophisticated dynamics while preserving the higher-order connectivity inherent to hypergraphs. We take on the non-trivial challenge to generalize the two commonly used hypergraph Laplacians [19, 11] to incorporate the richer structure sheaves offer. Theoretically, we demonstrate that the diffusion process derived using the sheaf hypergraph Laplacians that we propose induces a more expressive inductive bias than the classical hypergraph diffusion. Leveraging this enhanced inductive bias, we construct and test two powerful neural networks capable of inferring and processing hypergraph sheaf structure: the Sheaf Hypergraph Neural Network (SheafHyperGNN) and the Sheaf Hypergraph Convolutional Network (SheafHyperGCN).
The introduction of the cellular sheaf for hypergraphs expands the potential for representing complex interactions and provides a foundation for more advanced techniques. By generalizing the hypergraph Laplacians with the sheaf structure, we can better capture the nuance and intricacy of real-world data. Furthermore, our theoretical analysis provides evidence that the sheaf hypergraph Laplacians embody a more expressive inductive bias, essential for obtaining strong representations.
Our main contributions are summarised as follow:
1. We introduce the cellular sheaf for hypergraphs, a mathematical construct that enhances the hypergraphs with additional structure by associating a vector space with each node and hyperedge, along with linear projections that enable information transfer between them.
2. We propose both a linear and a non-linear sheaf hypergraph Laplacian, generalizing the standard hypergraph Laplacians commonly used in the literature. We also provide a theoretical characterization of the inductive biases generated by the diffusion processes of these Laplacians, showcasing the benefits of utilizing these novel tools for effectively modeling intricate phenomena.
3. The two sheaf hypergraph Laplacians are the foundation for two novel architectures tailored for hypergraph processing: Sheaf Hypergraph Neural Network and Sheaf Hypergraph Convolutional Network. Experimental findings demonstrate that these models achieve top results, surpassing existing methods on numerous benchmarking datasets.
Sheaves on Graphs. Utilizing graph structure in real-world data has improved various domains like healthcare [1], biochemistry [2], social networks [20], recommendation systems [3], traffic prediction [21], with graph neural networks (GNNs) becoming the standard for graph representations. However, in heterophilic setups, when nodes with different labels are likely to be connected, directly processing the graph structure leads to weak performance. In [22], they address this by attaching additional geometric structure to the graph, in the form of cellular sheaves [18].
A cellular sheaf on graphs associates a vector space with each node and each edge together with a linear projection between these spaces for each incident pair. To take into account this more complex geometric structure, SheafNN [23] generalised the classical GNNs [24-26] by replacing the graph Laplacian with a sheaf Laplacian [27]. Higher-dimensional sheaf-based neural networks are explored, with sheaves either learned from the graph [22] or deterministically inferred for efficiency [28]. Recent methods integrate attention mechanisms [29] or replace propagation with wave equations [30]. In recent developments, Sheaf Neural Networks have been found to significantly enhance the performance of recommendation systems, as they improve upon the limitations of graph neural networks [31].
In the domain of heterogeneous graphs, the concept of learning unique message functions for varying edges is well-established. However, there's a distinction in how sheaf-based methods approach this task compared to heterogeneous methods such as RGCN [32]. Unlike the latter, which learns individual parameters for each kind of incident relationship, sheaf-based methods dynamically predict projections for each relationship, relying on features associated with the node and hyperedge. As a result, the total parameters in sheaf networks do not escalate with an increase in the number of hyperedges. This difference underscores a fundamental shift in paradigm between the two methods.
Figure 1: Visual representation of linear and non-linear sheaf hypergraph Laplacian. (Top) In the linear case, the block matrix  $(\mathcal{L}_{\mathcal{F}})_{uv}$  corresponding to the pair of nodes  $(u,v)$  accumulates contributions from each hyperedge that simultaneously contains both nodes. (Bottom) In the nonlinear version, for each hyperedge, we first select the two nodes that are the most dissimilar in the hyperedge stalk domain:  $u \sim_e v$  if  $(u,v) = \arg\max_{u,v \in e}||\mathcal{F}_{u \leq e}x_u - \mathcal{F}_{v \leq e}x_v||_2^2$ . Then, the block matrix  $(\bar{\mathcal{L}}_{\mathcal{F}})_{uv}$  associated with the pair of nodes  $(u,v)$  only accumulates contributions from a hyperedge  $e$  if  $u \sim_e v$ . The two operators (linear and non-linear sheaf hypergraph Laplacian) represent the building blocks for the Sheaf Hypergraph Neural Network and Sheaf Hypergraph Convolutional Network respectively and we theoretically show that they exhibit a more expressive implicit bias compared to the traditional Hypergraph Networks, leading to better performance.
Hypergraph Networks. Graphs, while useful, have a strong limitation: they represent only pairwise relations. Many natural phenomena involve complex, higher-order interactions [33-35, 9], requiring a more general structure like hypergraphs. Recent deep learning approaches have been developed for hypergraph structures. HyperGNN [11] expands the hypergraph into a weighted clique and applies message passing similar to GCNs [24]. HNHN [36] improves this with non-linearities, while HyperGCN [37] connects only the most discrepant nodes using a non-linear Laplacian. Similar to the trend in GNNs, attention models gain popularity also in the hypergraph domain. HCHA [38] uses an attention-based incidence matrix, computed based on a nodes-hyperedge similarity. Similarly, HERALD [39] uses a learnable distance to infer a soft incidence matrix. On the other hand, HEAT [15] creates messages by propagating information inside each hyperedge using Transformers [40].
Many hypergraph neural network (HNN) methods can be viewed as two-stage frameworks: 1) sending messages from nodes to hyperedges and 2) sending messages back from hyperedges to nodes. Thus, [41] proposes a general framework where the first step is the average operator, while the second stage could use any existing GNN module. Similarly, [42] uses either DeepSet functions [43] or Transformers [40] to implement the two stages, while [44] uses a GNN-like aggregator in both stages, with distinct messages for each (node, hyperedge) pair.
In contrast, we propose a novel model to improve the hypergraph processing by attaching a cellular sheaf to the hypergraph structure and diffusing the information inside the model according to it. We will first introduce the cellular sheaf for hypergraph, prove some properties for the associated Laplacians, and then propose and evaluate two architectures based on the sheaf hypergraph Laplacians.
3 Hypergraph Sheaf Laplacian
An undirected hypergraph is a tuple  $\mathcal{H} = (V, E)$  where  $V = \{1, 2, \ldots, n\}$  is a set of nodes (also called vertices), and  $E$  is a set of hyperedges (also called edges when there is no confusion with the graph
edges). Each hyperedge  $e$  is a subset of the nodes set  $V$ . We denote by  $n = |V|$  the number of nodes in the hypergraph  $\mathcal{H}$  and by  $m = |E|$  the number of hyperedges. In contrast to graph structures, where each edge contains exactly two nodes, in a hypergraph an edge  $e$  can contain any number of nodes. The number of nodes in each hyperedge  $(|e|)$  is called the degree of the hyperedge and is denoted by  $\delta_e$ . In contrast, the number of hyperedges containing each node  $v$  is called the degree of the node and is denoted by  $d_v$ .
Following the same intuition from defining sheaves on graphs [23, 22], we will introduce the cellular sheaf associated with a hypergraph  $\mathcal{H}$ .
Definition 1. A cellular sheaf  $\mathcal{F}$  associated with a hypergraph  $\mathcal{H}$  is defined as a triple  $\langle \mathcal{F}(v),\mathcal{F}(e),\mathcal{F}_{v\triangleleft e}\rangle$ , where:
1.  $\mathcal{F}(v)$  are vertex stalks: vector spaces associated with each node  $v$ ;
2.  $\mathcal{F}(e)$  are hyperedge stalks: vector spaces associated with each hyperedge  $e$ ;
3.  $\mathcal{F}_{v\leq e}:\mathcal{F}(v)\to \mathcal{F}(e)$  are restriction maps: linear maps between each pair  $v\triangleleft e$  , if hyperedge  $e$  contains node  $v$
In simpler terms, a sheaf associates a space with each node and each hyperedge in a hypergraph and also provides a linear projection that enables the movement of representations between nodes and hyperedges, as long as they are adjacent. Unless otherwise specified, we assign the same  $d$ -dimensional space for all vertex stalks  $\mathcal{F}(v) = \mathbb{R}^d$  and all hyperedge stalks  $\mathcal{F}(e) = \mathbb{R}^d$ . We refer to  $d$  as the dimension of the sheaf.
Previous works focused on creating hypergraph representations by relying on various methods of defining a Laplacian for a hypergraph. In this work, we will concentrate on two definitions: a linear version of the hypergraph Laplacian as used in [11], and a non-linear version of the hypergraph Laplacian as in [37]. We will extend both of these definitions to incorporate the hypergraph sheaf structure, analyze the advantages that arise from this, and propose two different neural network architectures based on each one of them. For a visual comparison between the two proposed sheaf hypergraph Laplacians, see Figure 1.
3.1 Linear Sheaf Hypergraph Laplacian
Definition 2. Following the definition of a cellular sheaf on hypergraphs, we introduce the linear sheaf hypergraph Laplacian associated with a hypergraph  $\mathcal{H}$  as  $(\mathcal{L}_{\mathcal{F}})_{vv} = \sum_{e;v\in e}\frac{1}{\delta_e}\mathcal{F}_{v\leq e}^T\mathcal{F}_{v\leq e}\in \mathbb{R}^{d\times d}$  and  $(\mathcal{L}_{\mathcal{F}})_{uv} = -\sum_{e;u,v\in e}\frac{1}{\delta_e}\mathcal{F}_{u\leq e}^T\mathcal{F}_{v\leq e}\in \mathbb{R}^{d\times d}$ , where  $\mathcal{F}_{v\leq e}:\mathbb{R}^d\to \mathbb{R}^d$  represents the linear restriction maps guiding the flow of information from node  $v$  to hyperedge  $e$ .
The linear sheaf Laplacian operator for node  $v$  applied on a signal  $x \in \mathbb{R}^{n \times d}$  can be rewritten as:
$$
\mathcal {L} _ {\mathcal {F}} (x) _ {v} = \sum_ {e; v \in e} \frac {1}{\delta_ {e}} \mathcal {F} _ {v \triangleleft e} ^ {T} (\sum_ {\substack {u \in e \\ u \neq v}} (\mathcal {F} _ {v \triangleleft e} x _ {v} - \mathcal {F} _ {u \triangleleft e} x _ {u})). \tag{1}
$$
When each hyperedge contains exactly two nodes (thus  $\mathcal{H}$  is a graph), the internal summation will contain a single term, and we recover the sheaf Laplacian for graphs as formulated in [22].
On the other hand, for the trivial sheaf, when the vertex and hyperedge stalks are both fixed to be  $\mathbb{R}$  and the restriction map is the identity  $\mathcal{F}_{v\triangleleft e} = 1$  we recover the usual linear hypergraph Laplacian [11, 45] defined as  $\mathcal{L}(x)_v = \sum_{e; v \in e} \frac{1}{\delta_e} \sum_{u \in e} (x_v - x_u)$ . However, when we allow for higher-dimensional stalks  $\mathbb{R}^d$ , the restriction maps for each adjacency pair  $(v, e)$  become linear projections  $\mathcal{F}_{v\triangleleft e} \in \mathbb{R}^{d \times d}$ , enabling us to model more complex propagations, customized for each incident (node, hyperedge) pairs.
In the following sections, we will demonstrate the advantages of using this sheaf hypergraph diffusion instead of the usual hypergraph diffusion.
Reducing energy via linear diffusion. Previous work [17] demonstrates that diffusion using the classical symmetric normalised version of the hypergraph Laplacian  $\Delta = D^{-\frac{1}{2}}\mathcal{L}D^{-\frac{1}{2}}$ , where  $D$  is a diagonal matrix containing the degrees of the vertices, reduces the following energy function:
$E_{L_2}(x) = \frac{1}{2}\sum_e\frac{1}{\delta_e}\sum_{u,v\in e}||d_v^{-\frac{1}{2}}x_v - d_u^{-\frac{1}{2}}x_u||_2^2.$  Intuitively, this means that applying diffusion using the linear hypergraph Laplacian leads to similar representations for neighbouring nodes. While this is desirable in some scenarios, it may cause poor performance in others, a phenomenon known as over-smoothing [17]. In the following, we show that applying diffusion using the linear sheaf hypergraph Laplacian addresses these limitations by implicitly minimizing a more expressive energy function. This allows us to model phenomena that were not accessible using the usual Laplacian.
Definition 3. We define sheaf Dirichlet energy of a signal  $x \in \mathbb{R}^{n \times d}$  on a hypergraph  $\mathcal{H}$  as:
$$
E _ {L _ {2}} ^ {\mathcal {F}} (x) = \frac {1}{2} \sum_ {e} \frac {1}{\delta_ {e}} \sum_ {u, v \in e} | | \mathcal {F} _ {v \triangleleft e} D _ {v} ^ {- \frac {1}{2}} x _ {v} - \mathcal {F} _ {u \triangleleft e} D _ {u} ^ {- \frac {1}{2}} x _ {u} | | _ {2} ^ {2},
$$
where  $D_v = \sum_{e; v \in e} \mathcal{F}_{v \triangleleft e}^T \mathcal{F}_{v \triangleleft e}$  is a normalisation term equivalent to the nodes degree  $d_v$  for the trivial sheaf and  $D = \text{diag}(D_1, D_2 \ldots D_n)$  the corresponding block diagonal matrix.
This quantity measures the discrepancy between neighbouring nodes in the hyperedge stalk domain as opposed to the usual Dirichlet energy for hypergraphs that, instead, measures this distance in the node features domain. In the following, we are showing that, applying hypergraph diffusion using the linear sheaf Laplacian implicitly reduces this energy.
Proposition 1. The diffusion process using a symmetric normalised version of the linear sheaf hypergraph Laplacian minimizes the sheaf Dirichlet energy of a signal  $x$  on a hypergraph  $\mathcal{H}$ . Moreover, the energy decreases with each layer of diffusion.
Concretely, defining the diffusion process as  $Y = (I - \Delta^{\mathcal{F}})X$  where  $\Delta^{\mathcal{F}} = D^{-\frac{1}{2}}\mathcal{L}^{\mathcal{F}}D^{-\frac{1}{2}} \in \mathbb{R}^{nd\times nd}$  represents the symmetric normalised version of the linear sheaf hypergraph Laplacian, we have that  $E_{L_2}^{\mathcal{F}}(Y) < \lambda_*E_{L_2}^{\mathcal{F}}(X)$ , with  $\{\lambda_i\}$  the non-zero eigenvalues of  $\Delta^{\mathcal{F}}$  and  $\lambda_{*} = \max_{i}\{(1 - \lambda_{i})^{2}\} < 1$ . All the proofs are in the Supplementary Material.
This result addresses some of the limitations of standard hypergraph processing. First, while classical diffusion using hypergraph Laplacian brings closer representations of the nodes in the nodes space  $(x_v, x_u)$ , our linear sheaf hypergraph Laplacian allows us to bring closer representations of the nodes in the more complex space associated with the hyperedges  $(\mathcal{F}_{v \triangleleft e} x_v, \mathcal{F}_{u \triangleleft e} x_u)$ . This encourages a form of hyperedge agreement, while preventing the nodes to become uniform. Secondly, in the hyperedge stalks, each node can have a different representation for each hyperedge it is part of, leading to a more expressive processing compared to the classical methods. Moreover, in many Hypergraph Networks, the hyperedges uniformly aggregate information from all its components. Through the presence of a restriction map for each (node, hyperedge) pair, we enable the model to learn the individual contribution that each node sends to each hyperedge.
From an opinion dynamic perspective [46] when the hyperedges represent group discussions, the input space  $x_{v}$  can be seen as the private opinion, while the hyperedge stalk  $\mathcal{F}_{v\triangleleft e}x_{v}$  can be seen as a public opinion (what an individual  $v$  decide to express in a certain group  $e$ ). Minimizing the Dirichlet energy creates private opinions that are in consensus inside each hyperedge, while minimizing the sheaf Dirichlet energy creates an apparent consensus, by only uniformizing the expressed opinion. Through our sheaf setup, each individual is allowed to express varying opinion in each group it is part of, potentially different than their personal belief. This introduces a realistic scenario inaccessible in the original hypergraph diffusion setup.
3.2 Non-Linear Sheaf Hypergraph Laplacian
Although the linear hypergraph Laplacian is commonly used to process hypergraphs, it falls short in fully preserving the hypergraph structure [47]. To address these shortcomings, [48] introduces the non-linear Laplacian, demonstrating that its spectral properties are more suited for higher-order processing compared to the linear Laplacian. For instance, compared to the linear version, the non-linear Laplacian leads to a more balanced partition in the minimum cut problem, a task known to be tightly related to the semi-supervised node classification. Additionally, while the linear Laplacian associates a clique for each hyperedge, the non-linear one offers the advantage of relying on a much sparser connectivity. We will adopt a similar methodology to derive the non-linear version of the sheaf hypergraph Laplacian and analyze the benefits of applying diffusion using this operator.
Definition 4. We introduce the non-linear sheaf hypergraph Laplacian of a hypergraph  $\mathcal{H}$  with respect to a signal  $x$  as following:
1. For each hyperedge  $e$ , compute  $(u_{e}, v_{e}) = \text{argmax}_{u, v \in e} ||\mathcal{F}_{u \triangleleft e} x_{u} - \mathcal{F}_{v \triangleleft e} x_{v}||$ , the set of pairs containing the nodes with the most discrepant features in the hyperedge stalk.
2. Build an undirected graph  $\mathcal{G}_H$  containing the same sets of nodes as  $\mathcal{H}$  and, for each hyperedge  $e$  connects the most discrepant nodes  $(u, v)$  (from now on we will write  $u \sim_e v$  if they are connected in the  $\mathcal{G}_H$  graph due to the hyperedge  $e$ ). If multiple pairs have the same maximum discrepancy, we will randomly choose one of them.
3. Define the sheaf non-linear hypergraph Laplacian as:
$$
\bar {\mathcal {L}} _ {\mathcal {F}} (x) _ {v} = \sum_ {e; u \sim_ {e} v} \frac {1}{\delta_ {e}} \mathcal {F} _ {v \underline {{\leq}} e} ^ {T} \left(\mathcal {F} _ {v \underline {{\leq}} e} x _ {v} - \mathcal {F} _ {u \underline {{\leq}} e} x _ {u}\right). \tag {2}
$$
Note that the sheaf structure impacts the non-linear diffusion in two ways: by shaping the graph structure creation (Step 1), where the two nodes with the greatest distance in the hypergraph stalk are selected rather than those in the input space; and by influencing the information propagation process (Step 3). When the sheaf is restricted to the trivial case ( $d = 1$  and  $\mathcal{F}_{v\triangleleft e} = 1$ ) this corresponds to the non-linear sheaf Laplacian of a hypergraph as introduced in [48].
Reducing total variation via non-linear diffusion. In the following discussion, we will demonstrate how transitioning from a linear to a non-linear sheaf hypergraph Laplacian alters the energy guiding the inductive bias. This phenomenon was previously investigated for the classical hypergraph Laplacian, with [48] revealing enhanced expressivity in the non-linear case.
Definition 5. We define the sheaf total variation of a signal  $x \in \mathbb{R}^{n \times d}$  on a hypergraph  $\mathcal{H}$  as:
$$
\bar {E} _ {T V} ^ {\mathcal {F}} (x) = \frac {1}{2} \sum_ {e} \frac {1}{\delta_ {e}} \max _ {u, v \in e} | | \mathcal {F} _ {v \underline {{\leq}} e} D _ {v} ^ {- \frac {1}{2}} x _ {v} - \mathcal {F} _ {u \underline {{\leq}} e} D _ {u} ^ {- \frac {1}{2}} x _ {u} | | _ {2} ^ {2},
$$
where  $D_v = \sum_{e; v \in e} \mathcal{F}_{v \leq e}^T \mathcal{F}_{v \leq e}$  is a normalisation term equivalent to the node's degree in the classical setup and  $D = \text{diag}(D_1, D_2 \ldots D_n)$  is the corresponding block diagonal matrix.
This quantity generalised the total variance (TV)  $\bar{E}_{TV}(x) = \frac{1}{2}\sum_{e}\frac{1}{\delta_{e}}\max_{u,v\in e}||d_{v}^{-\frac{1}{2}}x_{v} - d_{u}^{-\frac{1}{2}}x_{u}||_{2}^{2}$  minimized in the non-linear hypergraph label propagation [48, 49]. Unlike the TV, the sheaf total variation measures the highest discrepancy at the hyperedge level computed in the hyperedge stalk, as opposed to the TV, which gauges the highest discrepancy in the feature space. We will explore the connection between the sheaf TV and our non-linear sheaf hypergraph diffusion.
Proposition 2. The diffusion process using the symmetric normalised version of non-linear sheaf hypergraph Laplacian minimizes the sheaf total variance of a signal  $x$  on hypergraph  $\mathcal{H}$ .