SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Despite the change in the potential function being minimized, the overarching objective remains akin to that of the linear case: striving to achieve a coherent consensus among the representations within the hyperedge stalk space, rather than generating uniform features for each hyperedge in the input space. In contrast to the linear scenario, where a quadratic number of edges is required for each hyperedge, the non-linear sheaf hypergraph Laplacian associates a single edge with each hyperedge, thereby enhancing computational efficiency.
3.3 Sheaf Hypergraph Networks
Popular hypergraph neural networks [45, 11, 37, 50] draw inspiration from a variety of hypergraph diffusion operators [47, 48, 51], giving rise to diverse message passing techniques. These techniques all involve the propagation of information from nodes to hyperedges and vice-versa. We will adopt a similar strategy and introduce the Sheaf Hypergraph Neural Network and Sheaf Hypergraph Convolutional Network, based on two message-passing schemes inspired by the sheaf diffusion mechanisms discussed in this paper.
Given a hypergraph  $\mathcal{H} = (V, E)$  with nodes characterised by a set of features  $X \in \mathbb{R}^{n \times f}$ , we initially linearly project the input features into  $\tilde{X} \in \mathbb{R}^{n \times (df)}$  and then reshape them into  $\tilde{X} \in \mathbb{R}^{nd \times f}$ . As a result, each node is represented in the vertex stalk as a matrix  $\mathbb{R}^{d \times f}$ , where  $d$  denotes the dimension of the vertex stalk, and  $f$  indicates the number of channels.
Table 1: Performance on a collection of hypergraph benchmarks. Our models using sheaf hypergraph Laplacians demonstrate a clear advantage over their counterparts using classical Laplacians (HyperGNN and HyperGCN). Compared to other recent methods, SheafHyperGNN and SheafHyperGCN achieve competitive performance and attain state-of-the-art results in five of the datasets.
Name	Cora	Citeseer	Pubmed	Cora_CA	DBLP_CA	Senate	House	Congress
HCHA	79.14 ± 1.02	72.42 ± 1.42	86.41 ± 0.36	82.55 ± 0.97	90.92 ± 0.22	48.62 ± 4.41	61.36 ± 2.53	90.43 ± 1.20
HNHN	76.36 ± 1.92	72.64 ± 1.57	86.90 ± 0.30	77.19 ± 1.49	86.78 ± 0.29	50.93 ± 6.33	67.8 ± 2.59	53.35 ± 1.45
AllDeepSets	76.88 ± 1.80	70.83 ± 1.63	88.75 ± 0.33	81.97 ± 1.50	91.27 ± 0.27	48.17 ± 5.67	67.82 ± 2.40	91.80 ± 1.53
AllSetTransformers	78.58 ± 1.47	73.08 ± 1.20	88.72 ± 0.37	83.63 ± 1.47	91.53 ± 0.23	51.83 ± 5.22	69.33 ± 2.20	92.16 ± 1.05
UniGCNII	78.81 ± 1.05	73.05 ± 2.21	88.25 ± 0.33	83.60 ± 1.14	91.69 ± 0.19	49.30 ± 4.25	67.25 ± 2.57	94.81 ± 0.81
HyperND	79.20 ± 1.14	72.62 ± 1.49	86.68 ± 0.43	80.62 ± 1.32	90.35 ± 0.26	52.82 ± 3.20	51.70 ± 3.37	74.63 ± 3.62
ED-HNN	80.31 ± 1.35	73.70 ± 1.38	89.03 ± 0.53	83.97 ± 1.55	91.90 ± 0.19	64.79 ± 5.14	72.45 ± 2.28	95.00 ± 0.99
HyperGCN1	78.36 ± 2.01	71.01 ± 2.21	80.81 ± 12.4	79.50 ± 2.11	89.42 ± 0.16*	51.13 ± 4.15	69.29 ± 2.05	89.67 ± 1.22
SheafHyperGCN	80.06 ± 1.12	73.27 ± 0.50	87.09 ± 0.71	83.26 ± 1.20	90.83 ± 0.23	66.33 ± 4.58	72.66 ± 2.26	90.37 ± 1.52
HyperGNN	79.39 ± 1.36	72.45 ± 1.16	86.44 ± 0.44	82.64 ± 1.65	91.03 ± 0.20	48.59 ± 4.52	61.39 ± 2.96	91.26 ± 1.15
SheafHyperGNN	81.30 ± 1.70	74.71 ± 1.23	87.68 ± 0.60	85.52 ± 1.28	91.59 ± 0.24	68.73 ± 4.68	73.84 ± 2.30	91.81 ± 1.60
A general layer of Sheaf Hypergraph Network is defined as:
$$
Y = \sigma ((I _ {n d} - \stackrel {\bullet} {\Delta}) (I _ {n} \otimes W _ {1}) \tilde {X} W _ {2}).
$$
Here,  $\dot{\Delta}$  can be either  $\Delta^{\mathcal{F}} = D^{-\frac{1}{2}}\mathcal{L}^{\mathcal{F}}D^{-\frac{1}{2}}$  for the linear sheaf hypergraph Laplacian introduced in Eq. 1 or  $\bar{\Delta}^{\mathcal{F}} = D^{-\frac{1}{2}}\bar{\mathcal{L}}^{\mathcal{F}}D^{-\frac{1}{2}}$  for the non-linear sheaf hypergraph Laplacian introduced in Eq. 2. Both  $W_{1}\in \mathbb{R}^{d\times d}$  and  $W_{2}\in \mathbb{R}^{f\times f}$  are learnable parameters, while  $\sigma$  represents ReLU non-linearity.
Sheaf Hypergraph Neural Network (SheafHyperGNN). This model utilizes the linear sheaf hypergraph Laplacian  $\Delta = \Delta^{\mathcal{F}}$ . When the sheaf is trivial ( $d = 1$  and  $\mathcal{F}_{v\triangleleft e} = 1$ ), and  $W_{1} = \mathbf{I}_{d}$ , the SheafHyperGNN is equivalent to the conventional HyperGNN architecture [11]. However, by increasing dimension  $d$  and adopting dynamic restriction maps, our proposed SheafHyperGNN becomes more expressive. For every adjacent node-hyperedge pair  $(v,e)$ , we use a  $d\times d$  block matrix to discern each node's contribution instead of a fixed weight that only stores the incidence relationship. The remaining operations are similar to those in HyperGNN [11]. More details on how the block matrices  $\mathcal{F}_{v\triangleleft e}$  are learned can be found in the following subsection.
Sheaf Hypergraph Convolutional Network (SheafHyperGCN). This model employs the non-linear Laplacian  $\dot{\Delta} = \bar{\Delta}^{\mathcal{F}}$ . Analogous to the linear case, when the sheaf is trivial and  $W_{1} = \mathbf{I}_{d}$  we obtain the classical HyperGCN architecture [37]. In our experiments, we will use an approach similar to that in [37] and adjust the Laplacian to include mediators. This implies that we will not only connect the two most discrepant nodes but also create connections between each node in the hyperedge and these two most discrepant nodes, resulting in a denser associated graph. For more information on this variation, please refer to [37] or Supplementary Material.
In summary, the models introduced in this work, SheafHyperGNN and SheafHyperGCN serve as generalisations of the classical HyperGNN [11] and HyperGCN [37]. These new models feature a more expressive implicit regularisation compared to their traditional counterparts.
Learnable Sheaf Laplacian. A key advantage of Sheaf Hypergraph Networks lies in attaching and processing a more complex structure (sheaf) instead of the original standard hypergraph. Different sheaf structures can be associated with a single hypergraph, and accurately modeling the most suitable structure is crucial for obtaining effective and meaningful representation. In our proposed models, we achieve this by designing learnable restriction maps. For a d-dimensional sheaf, we predict the restriction maps for each pair of incident (vertex v, hyperedge e) as  $\mathcal{F}_{v\leq e} = \mathrm{MLP}(x_v||h_e)\in \mathbb{R}^{d^2}$ , where  $x_{v}$  represent node features of  $v$ , and  $h_e$  represents features of the hyperedge  $e$ . This vector representation is then reshaped into a  $d\times d$  block matrix representing the linear restriction map for the  $(v,e)$  pair. When hyperedge features  $h_e$  are not provided, any permutation-invariant operation can be applied to obtain hyperedge features from node-level features. We experiment with three types of  $d\times d$  block matrices: diagonal, low-rank and general matrices, with the diagonal version
Results where rerun compared to [50] using the same hyperparameters, to fix an existing issue in the original code.
Table 2: Ablation study on Restriction Maps: we explore three types of  $d \times d$  restriction maps: diagonal, low-rank and general. Diagonal matrices consistently achieve better accuracy on most of the datasets, demonstrating a superior balance between complexity and expressivity
Name	Cora	Citeseer	Pubmed	Cora_CA	DBLP_CA	Senate	House	Congress
Diag-SheaFHyperGCN	80.06 ±1.12	73.27 ±0.50	87.09 ±0.71	83.26 ±1.20	90.83 ±0.23	66.33 ±4.58	72.66 ±2.26	90.37 ±1.52
LR-SheaFHyperGCN	78.70 ±1.14	72.14 ±1.09	86.99 ±0.39	82.61 ±1.28	90.84 ±0.29	66.76 ±4.58	70.70 ±2.23	84.88 ±2.31
Gen-SheaFHyperGCN	79.13 ±0.85	72.54 ±2.3	86.90 ±0.46	82.54 ±2.08	90.57 ±0.40	65.49 ±5.17	71.05 ±2.12	82.14 ±2.81
Diag-SheaFHyperGNN	81.30 ±1.70	74.71 ±1.23	87.68 ±0.60	85.52 ±1.28	91.59 ±0.24	68.73 ±4.68	73.62 ±2.29	91.81 ±1.60
LR-SheaFHyperGNN	76.65 ±1.41	74.05 ±1.34	87.09 ±0.25	77.05 ±1.00	85.13 ±0.29	68.45 ±2.46	73.84 ±2.30	74.83 ±2.32
Gen-SheaFHyperGNN	76.82 ±1.32	74.24 ±1.05	87.35 ±0.34	77.12 ±1.14	84.99 ±0.39	68.45 ±4.98	69.47 ±1.97	74.52 ±1.27
consistently outperforming the other two. These restriction maps are further used to define the sheaf hypergraph Laplacians (Def. 2, or 4) used in the final Sheaf Hypergraph Networks. Please refer to the Supplementary Material for more details on how we constrain the restriction maps.
We evaluate our model on eight real-world datasets that vary in domain, scale, and heterophily level and are commonly used for benchmarking hypergraphs. These include Cora, Citeseer, Pubmed, Cora-CA, DBLP-CA [37], House [52], Senate and Congress [53]. To ensure a fair comparison with the baselines, we follow the same training procedures used in [50] by randomly splitting the data into  $50\%$  training samples,  $25\%$  validation samples and  $25\%$  test samples, and running each model 10 times with different random splits. We report average accuracy along with the standard deviation.
Additionally, we conduct experiments on a set of synthetic heterophilic datasets inspired by those introduced by [50]. Following their approach, we generate a hypergraph using the contextual hypergraph stochastic block model [54-56], containing 5000 nodes: half belong to class 0 while the other half to class 1. We then randomly sample 1000 hyperedges with a cardinality 15, each containing exactly  $\beta$  nodes from class 0. The heterophily level is computed as  $\alpha = \min (\beta ,15 - \beta)$ . Node features are sampled from a label-dependent Gaussian distribution with a standard deviation of 1. As the original dataset is not publicly available, we generate our own set of datasets by varying the heterophily level  $\alpha \in \{1\ldots 7\}$  and rerun their experiments for a fair comparison.
The experiments are executed on a single NVIDIA Quadro RTX 8000 with 48GB of GPU memory. Unless otherwise specified, our results represent the best performance obtained by each architecture using hyper-parameter optimisation with random search. Details on all the model choices and hyper-parameters can be found in the Supplementary Material.
Laplacian vs Sheaf Laplacian. As demonstrated in the previous section, SheafHyperGNN and SheafHyperGCN are generalisations of the standard HyperGNN [11] and HyperGCN [37], respectively. They transition from the trivial sheaf  $(d = 1$  and  $\mathcal{F}_{v\triangleleft e} = 1)$  to more complex structures  $(d\geq 1$  and  $\mathcal{F}_{v\triangleleft e}$  a  $d\times d$  learnable projection). The results in Table 1 and Table 3 show that both models significantly outperform their counterparts on all tested datasets. Among our models, the one based on linear Laplacian (SheafHyperGNN) consistently outperforms the model based on non-linear Laplacian (SheafHyperGCN) across all datasets. This observation aligns with the performance of the models based on standard hypergraph Laplacian, where HyperGCN is outperformed by HyperGNN in all but two real-world datasets, despite their theoretical advantage [48].
Comparison to recent methods. We also compare to several recent models from the literature such as HCHA [38], HNHN [36], AllDeepSets [42], AllSetTransformer [42], UniGCNII [57], HyperND [58], and ED-HNN [50]. Our models achieve competitive results on all real-world datasets, with state-of-the-art performance on five of them (Table 1). These results confirm the advantages of using the sheaf Laplacians for processing hypergraphs. We also compare our models against a series baselines on the synthetic heterophilic dataset. The results are shown in Table 3. Our best model, SheafHyperGNN, consistently outperforms the other models across all levels of heterophily. Note that, our framework enhancing classical hypergraph processing with sheaf structure is not restricted to the two traditional models tested in this paper (HyperGNN and HyperGCN). Most of the recent state-of-the-art methods, such as ED-HNN, could be easily adapted to learn and process our novel cellular sheaf hypergraph instead of the standard hypergraph, leading to further advancement in the hypergraph field.
Figure 2: Impact of Depth and Stalk Dimension evaluated on the heterophilic dataset  $(\alpha = 7)$ . SheafHyperGNN's performance is unaffected by increasing depth, and high-dimensional stalks is essential for achieving top performance. The Dirichlet energy shows that, while HyperGNN enforces the nodes to be similar, our SheafHyperGNN does not suffer from this limitation, encouraging features diversity.
Table 3: Accuracy on Synthetic Datasets with Varying Heterophily Levels: Across all different level of heterophily  $(\alpha)$ , our sheaf-based methods SheafGCN and SheafHGNN consistently outperform their counterparts. Additionally, they achieve top results for all heterophily levels, further demonstrating their effectiveness. For each experiment, the result represents average accuracy over 10 runs.
Name	heterophily (α)
1	2	3	4	5	6	7
HyperGCN	83.9	69.4	72.9	75.9	70.5	67.3	66.5
HyperGNN	98.4	83.7	79.4	74.5	69.5	66.9	63.8
HCHA	98.1	81.8	78.3	75.88	74.1	71.1	70.8
ED-HNN	99.9	91.3	88.4	84.1	80.7	78.8	76.5
SheafHGCN	100	87.1	84.8	79.2	78.1	76.6	75.5
SheafHGNN	100	94.2	90.8	86.5	82.1	79.8	77.3
In the following sections, we conduct a series of ablation studies to gain a deeper understanding of our models. We will explore various types of restriction maps, analyze how performance changes when varying the network depth and study the importance of stalk dimension for final accuracy.
Investigating the Restriction Maps. Both linear and non-linear sheaf hypergraph Laplacians rely on attaching a sheaf structure to the hypergraph. For a cellular sheaf  $\mathcal{F}$  with vertex stalks  $\mathcal{F}(v) = \mathbb{R}^d$  and hyperedge stalks  $\mathcal{F}(e) = \mathbb{R}^d$  as used in our experiments, this involves inferring the restriction maps  $\mathcal{F}_{v\triangleleft e} \in \mathbb{R}^{d\times d}$  for each incidence pair  $(v,e)$ . We implement these as a function dependent on corresponding nodes and hyperedge features:  $\mathcal{F}_{v\triangleleft e} = \mathrm{MLP}(x_v||h_e) \in \mathbb{R}^{d^2}$ . Learning these matrices can be challenging; therefore, we experimented with adding constraints to the type of matrices used as restriction maps. In Table 2 we show the performance obtained by our models when constraining the restriction maps to be either diagonal (Diag-SheafHyperNN), low-rank (LR-SheafHyperNN) or general matrices (Gen-SheafHyperNN). We observe that the sheaves equipped with diagonal restriction maps perform better than the more general variations. We believe that the advantage of the diagonal restriction maps is due to easier optimization, which overcomes the lose in expressivity. More details about predicting constrained  $d \times d$  matrices can be found in the Supplementary Material.
Importance of Stalk Dimension. The standard hypergraph Laplacian corresponds to a sheaf Laplacian with  $d = 1$  and  $\mathcal{F}_{v\leq e} = 1$ . Constraining the stalk dimension to be 1, but allowing the restriction maps to be dynamically predicted, becomes similar to an attention mechanism [38]. However, attention models are restricted to guiding information via a scalar probability, thus facing the same over-smoothing limitations as traditional HyperGNN in the heterophilic setup. Our d-dimensional restriction maps increase the model's expressivity by enabling more complex information transfer between nodes and hyperedges, tailored for each individual pair. We validate this experimentally on the synthetic heterophilic dataset, using the diagonal version of the models, which achieves the best performance in the previous ablation. In Figure 2, we demonstrate how performance significantly improves when allowing higher-dimensional stalks ( $d > 1$ ). These results are consistent for both linear sheaf Laplacian-based models (SheafHyperGNN) and non-linear ones (SheafHyperGCN).
Influence of Depth. It is well-known that stacking many layers in a hypergraph network can lead to a decrease in model performance, especially in the heterophilic setup. This phenomenon, called oversmoothing, is well-studied in both graph [59] and hypergraph literature [17]. To analyse the extent to which our model suffers from this limitation, we train a series of models on the most heterophilic version of the synthetic dataset  $(\alpha = 7)$ . For both SheafHyperGNN and its HyperGNN equivalent, we vary the number of layers between  $1 - 8$ . In Figure 2, we observe that while HyperGNN exhibits a drop in performance when going beyond 3 layers, SheafHyperGNN's performance remains mostly constant. Similar results were observed for the non-linear version when comparing SheafHyperGCN with HyperGCN (results in Supplementary Material). These results indicate potential advantages of our models in the heterophilic setup by allowing the construction of deeper architectures.
Investigating Features Diversity. Our theoretical analysis shows that, while conventional Hypergraph Networks tend to produce similar features for neighbouring nodes, our Sheaf Hypergraph
Networks reduce the distance between neighbouring nodes in the more complex hyperedge stalk space. As a result, the nodes' features do not become uniform, preserving their individual identities. We empirically evaluate this, by computing the Dirichlet energy for HyperGNN and SheafHyperGNN (shaded area in Figure 2), as a measure of similarity between neighbouring nodes. The results are aligned with the theoretical analysis: while increasing depth in HyperGNN creates uniform features, SheafHyperGNN does not suffer from this limitation, encouraging diversity between the nodes.
In this paper we introduce the cellular sheaf for hypergraphs, an expressive tool for modelling higher-order relations build upon the classical hypergraph structure. Furthermore, we propose two models capable of inferring and processing the sheaf hypergraph structure, based on linear and non-linear sheaf hypergraph Laplacian, respectively. We prove that the diffusion processes associated with these models induce a more expressive implicit regularization, extending the energies associated with standard hypergraph diffusion. This novel architecture generalizes classical Hypergraph Networks, and we experimentally show that it outperforms existing methods on several datasets. Our technique of replacing the hypergraph Laplacian with a sheaf hypergraph Laplacian in both HyperGNN and HyperGCN establishes a versatile framework that can be employed to "sheafify" other hypergraph architectures. We believe that sheaf hypergraphs can contribute to further advancements in the rapidly evolving hypergraph community, extending far beyond the results presented in this work.
Acknowledgment The authors would like to thank Ferenc Huszár for fruitful discussions and constructive suggestions during the development of the paper and Eirik Fladmark and Laura Brinkholm Justesen for fixing a minor issue in the original HyperGCN code, which led to improved results in the baselines. Iulia Duta is a PhD student funded by a Twitter scholarship. This work was also supported by PNRR MUR projects PE0000013-FAIR, SERICS (PE00000014), Sapienza Project FedSSL, and IR0000013-SoBigData.it.
[1] Catherine Tong, Emma Rocheteau, Petar Velicković, Nicholas Lane, and Pietro Lio. Predicting Patient Outcomes with Graph Representation Learning, pages 281-293. 01 2022.
[2] Kexin Huang, Cao Xiao, Lucas M Glass, Marinka Zitnik, and Jimeng Sun. Skipgnn: predicting molecular interactions with skip-graph networks. Scientific reports, 10(1):1-16, 2020.
[3] Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 974-983, 2018.
[4] Xiaolong Wang and Abhinav Gupta. Videos as space-time region graphs. In Proceedings of the European Conference on Computer Vision (ECCV), pages 399-417, 2018.
[5] Nicolas A. Crossley, Andrea Mechelli, Jessica Scott, Francesco Carletti, Peter T. Fox, Philip K. McGuire, and Edward T. Bullmore. The hubs of the human connectome are generally implicated in the anatomy of brain disorders. Brain, 137:2382 - 2395, 2014.
[6] Tingting Guo, Yining Zhang, Yanfang Xue, and Lishan Qiao. Brain function network: Higher order vs. more discrimination. Frontiers in Neuroscience, 15, 08 2021.
[7] Jürgen Jost and Raffaella Mulas. Hypergraph laplace operators for chemical reaction networks, 2018.
[8] Pragya Singh and Gaurav Baruah. Higher order interactions and species coexistence. Theoretical Ecology, 14, 03 2021.
[9] Alba Cervantes-Loreto, Carolyn Ayers, Emily Dobbs, Berry Brosi, and Daniel Stouffer. The context dependency of pollinator interference: How environmental conditions and co-foraging species impact floral visitation. Ecology Letters, 24, 07 2021.