LightHGNN: Distilling Hypergraph Neural Networks into MLPs for 100x Faster Inference
Authors: Yifan Feng
Yihe Luo
Shihui Ying
Yue Gao
Venue: ICLR 2024 poster (2024)
Keywords: Hypergraph
HGNN
Knowledge Distillation
MLPs
Reliale Learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Hypergraph Neural Networks (HGNNs) have recently attracted much attention and exhibited satisfactory performance due to their superiority in high-order correlation modeling. 
However, it is noticed that the high-order modeling capability of hypergraph also brings increased computation complexity, which hinders its practical industrial deployment.
In practice, we find that one key barrier to the efficient deployment of HGNNs is the high-order structural dependencies during inference.
In this paper, we propose to bridge the gap between the HGNNs and inference-efficient Multi-Layer Perceptron (MLPs) to eliminate the hypergraph dependency of HGNNs and thus reduce computational complexity as well as improve inference speed. 
Specifically, we introduce LightHGNN and LightHGNN$^+$ for fast inference with low complexity. LightHGNN directly distills the knowledge from teacher HGNNs to student MLPs via soft labels, and LightHGNN$^+$ further explicitly injects reliable high-order correlations into the student MLPs to achieve topology-aware distillation and resistance to over-smoothing.
Experiments on eight hypergraph datasets demonstrate that even without hypergraph dependency, the proposed LightHGNNs can still achieve competitive or even better performance than HGNNs and outperform vanilla MLPs by $16.3$ on average. Extensive experiments on three graph datasets further show the average best performance of our LightHGNNs compared with all other methods.
Experiments on synthetic hypergraphs with 5.5w vertices indicate LightHGNNs can run $100\times$ faster than HGNNs, showcasing their ability for latency-sensitive deployments.

Published as a conference paper at ICLR 2024
LIGHTHGNN: DISTILLING HYPERGRAPH NEURAL NETWORKS INTO MLPs FOR 100x FASTER INFERENCE
Yifan Feng', Yihe Luo', Shihui Ying", Yue Gao-
School of Software, BNRist, THUIBCS, BLBCI, Tsinghua University *Department of Mathematics, School of Science, Shanghai University
evanfeng97@gmail com, luoyh2 @mails tsinghua. edu. cn eeyying@shu..du..oo facyue@tsinghun edu. cn
ABSTRACT
Hypergraph Neural Networks (HGNNs) have recently attracted much attention and exhibited satisfactory performance due to their superiority in high-order cor- relation modeling. However, it is noticed that the high-order modeling capability hypergraph also brings increased computation complexity, which hinders its practical industrial deployment. In practice, efficient deployment of HGNNs is the high-order structural dependencies during inference. In this paper, propose to bridge the gap between the HGNNs and inference-efffcient Multi-Layer Perceptron (MLPs) to eliminate the hypergraph dependency of HGNNs and thus reduce computational complexity as well as im- prove inference speed. Specifically, we introduce LightHGNN and LightHGNN+ for fast inference with low complexity. LightHGNN directly distills the knowl- edge from teacher HGNNs to student MLPs via soft labels, and LightHGNN+ further explicitly injects reliable high-order correlations into the student MLPSs to achieve topology-awar distillation and resistance to over-smoothing. Exper- iments on eight hypergraph datasets demonstrate that even without hypergraph dependency, the proposed LightHGNNs can still achieve competitive or even bet- ter performance than HGNNs and outperform vanilla MLPs by 16.3 on average. Extensive experiments three graph datasets further show the average best per- formance of our LightHGNNs compared with all other methods. Experiments x00x
find that key barrier the
synthetic hypergraphs with 5.5w vertices indicate LightHGNNs faster than HGNNs, showcasing their ability for latency-sensitive deployments.
INTRODUCTION
Compared to the graph with pair-wise correlation, the hypergraph is composed of degree-free hyper- edges, which have an inherent superior modeling ability to represent those more complex high-order correlations. Therefore, many Hypergraph Neural Networks (HGNNs) Feng et al. 2: Gao al.
(2022); ; Dong et al. (2020) have been proposed in the vertex classification of citation networks Bai
let al. (2021) b Yadati al. (209)). the recommendation in user-item bipartite graphs Xia et al.
(022);
Jí et al. (2020). link prediction in drug-protein networks Saifuddin et al. al.] (2023); Ruan et al.
2021
and multi-tissue gene expression imputation Viñas et al. (2023); Murgas al. (2022). However, for large-scale industrial applications, especially for those big-data, small-memory, and high-speed demand environments, the Multi-Layer Perceptrons (MLPs) remain the primary workhorse. The
main reason for such an academic-indusstin gap for HGNNs is the dependence on the hypergraph structure in inference, which requires large memories in practice. Consequently, hypergraph and the number of layers of HGNNs increase, the inference time will also exponentially increase, as shown Figure I(c) limiting the potential usage of this type of method.
the scale of the
The hypergraph dependence of HGNNs is caused by the high-order neighbor fetching in message On the one hand, some GNNs-1o-MLPs methods like Graph- Less Neural Networks (GLNN) Zhang et al. (2021) and Knowledge-inspired Reliable Distillation (KRD) Wu et al. (2023) are proposed to distill the knowledge from GNNs to MLPs to eliminate the
passing of berte--hyprredge-eertee
"Corresponding author: Yue Gao

---

Published as a conference paper at ICLR 2024
*LighHHNNN
DONNAL
+LREGINNS BGNN-
Accuracy (%)
MINN-LI
HINNN-4 128
HINNN-01566
1KONNE) *I *LghitkcNNNwwst
a HUNNLL
WIGNN-W256
Inference Time (mss
120
LighHHNNN
3.
HGNN--1
HONN"
Accracy
Pow121
2556
20-
M2P
300
200
10
101
Memory (MB3)
Rumlime (ms)
#Vertices Hlypergmphh
(a) Accuracy vs. Memory.
(b) Accuracy vs, Runtime.
(c) Runtime Hypergraph Scale
Figure| I: Performance and efficiency comparison. (a) and (b) are run on the IMDB-AW hypergraph
dataset. (c) provides the inference time comparison a series of synthetic hypergraph datasets.
graph dependency. Those methods Zhang et al. (2021): >: Wu et al. (2023) either distill the knowledge by the soft label of teacher GNNs or pull the distance of reliable vertices via pair-wise edge as the extra supervision. Unlike the pair- wise edge in the graph, each hyperedge in the hypergraph can connect more than two vertices. The hypergraph neighborhood is more complex and defined in hierarchical paradigm Gao et al.] (2022). Those GNNs-to-MI.P5 methods cannot address the high-order correlation in the hypergraph. On the other hand, compared with the HGNNs, the MLPs method performs worse (about averaged 16 decline) on hypergraph datasets, as shown in Tab] I yet has no dependency on hypergraph structure among batched samples and can be deployed to process any scale of the hypergraph. Upon the above observations, we ask: Can we fix the gap between MLPs and HGNNs to achieve hypergraph dependency-free inference and topology-an aware distillation?
Present Work. In this paper, we proposed the LightHGNN and LightHGNN climinate the de- pendence on the hypergraph in the inference of HGNNs and achieve running fast and consuming low memory like ML.ss and performing as well as HGNNs as shown in Figure [Ta] and [(b) The LightHGNN directly distills the knowledge from the teacher HGNNs into the student MLPs with the classical soft label Hinton et al. (2015) of the teacher. The LightHGNN+ further develops a topology- aware knowledge distillation supervised by both the soft labels and the proposed high- order soft labels. To generate the high-order soft labels, we first quantify the reliability of hyper- edges by the resistance to noise generate a reliable hyperedge set. Then, the high-order soft labels can be generated via propagating those soft labels from vertex to hyperedge on those sampled reli- able hyperedges. The proposed high-order soft-label constraint can explicitly inject the high-order topology knowledge from those reliable hyperedges into the student MLPs. We further design the topology-aware score quantify the relevance of features and the topology. which is the main con- sequence of over-smoo Cal & Wang 2020). The results indicate the proposed topology-awan distillation can effectively resist the over-smoothing on both graphs and hyyergraphss.Experieent on 11 graph and hypergraph datasets demonstrate the effectiveness of our LightHGNNs compared to GNNs, HGNNs, and GNNs-to-MLPs, Experiments on eight hypergraph datasets indicate that the proposed LightHGNNs can significantly outperform the MLPs with averaged 16.3 improvements and only have a slight performance decline of 0.39 compared to the HGNN. Experiments on the synthetic hypergraph datasets indicate that LightHGNNs run 100x faster than the HGNN and can potentially deploy in the large-scale hypergraph dataset. Our main contributions are as follows:
We propose the LightHGNNs to bridge the gap between MLPs and HGNNs to achieve hypergraph dependency-fin free inference and topology-aware distillation.
We design the reliable hyperedge quantification and sampling strategy to achieve topology- aware distillation, which can explicitly inject reliable high-order knowledge into student MLPs and achieve an averaged 16.3 performance gain compared to the vanilla MLPs.
We develop the topology-awe aware score to quantify the smoothig phenomenon from the relevance of features and topology perspective and demonstrate the proposed LighHHGNN can effectively resist over-smoothing.
We show that the proposed LightHGNNs can run 100x faster than the HGNN, which can be easily deployed in siency-constrained applications.
2 RELATED WORK
Hypergraph Neural Networks. The early HGNNs defined upon the spectral domain like HGNN[Feng et al. (2019) and HpLapGCN Fu et al. (2019), which conduct the feature smoothing via the hypergraph Laplacian matrix. Yadati et al. (219)) propose the HyperGCN, which designs
2

---

Published as a conference paper at ICLR 2024
strategy to reduce the hypergraph to graphs and further employ the GNNs Kipf & Welling (017)) to learn the representations. Besides, a series of spatial-based hypergraph convolutions are proposed, like the vertex-hyperedge attention mechanism (Bai et al.] )021). dynamic hypergraph construction (Jiang et al. 2019). and two-stage message passing (Gao et al. 2022 Dong et al. 2020).
GNNs-to-MLPs Distillation. To enjoy the efficiency and effectiveness of MLPs and GNNs, GLNN (Zhang et al. (02T) directly utilizes the prediction distribution of teacher GNNs as the soft target to supervise the student MLPs, which ignores the topology of the original graph in distilla- tion. Yang et al. 202T) extracts the knowledge of an arbitrary learned GNN model (teacher model) and injects into a well-designed student model achieve more efficient predictions. KRD (Wul itt al. 2023) further quantifies the knowledge of each vertex and pulls the distance between it and its Seighbors.xxssting methods are constrained in the low-order graph neural networks. In contrast, this paper aims to bridge the hypergraph neural networks and MLPs and design topology-ave aware distillation, injecting reliable high-order knowledge into MLPs for faster inference than HGNNs.
PRELIMINARIES
Notions and Problem Statement. Let {V,E) be hypergraph, set V and & denote the vertex set and hyperedge set, respectively. N = |V and M = (E). Each vertex v; is associated with a feature vector 2;, and the overall vertex feature matrix is denoted by X [z] Z2, ... .NN]
R-Xxc, In practice, the hypergraph can be represented by an incidence matrix H E {0, 10,11 X where the row and column denote the vertices and hyperedges, respectively. H(v,e) the vertex v belongs to the hyperedge e. Besides, y = {ye I V E V} denotes the vertex label set. Consider a semi-supervised vertex classification task, the vertex set V is divided into two sub-sets: labeled data D- 3 ) and the unlabeled data D = (V" y²)). The task aims to learn map Φ: V/ which can be used to predict the label of those unlabeled vertices.
1 denotes
Hypergraph Neural Networks (HGNNs). HGNN is defined based on the hypergraph Laplacian matrix, and its eigenvectors are treated as the Fourier bases for a given signal æ. Furthermore, the Chebyshv polynomial Defferrard et al. (016) is adopted for the convenience of computation. Then, the HGNN can be defined as: X a (D, 1/2 HW DE H " O'O), where D. and D. are the diagonal matrix of the degree of vertices and hyperedges, respectively. W is the diagonal matrix of the weight of hyperedges, and @ the trainable parameters.
METHODOLOGY
In this section, we introduce LightHGNN and LightHGNN+, Then, we provide the complexity analysis and discuss the relationship to existing GNNs-to-MLPs methods.
4.1 LIGHTHGNN: SOFT-TARGER GUIDED HGNNS DISTILLATION
To boost the inference performance in real-world deployment, we propose the soft-target guided HGNNs distillation, named LightHGNN, which directly distills the knowledge of HGNNs into the MLPs. Motivated by the Knowledge Distillation (KD) Hinton al. (2015) and GNNs-to-MIPS methods Zhhag et al. (2021); Wu et al. (2023), we adopt the MIPs as the student network and the well-trained HGNNs as the teacher network, and distills the knowledge with the combination objective LDH of the Cross-Entropy loss Loe and the Kullback-Lebber Divergence loss as follows:
1 D
1 Σ Lee Mo) + (1 || | DEP!
Σ DKL (.) VEV
where ye is the one-hot encoded label of vertex e. Vector Df and J are the softmax normalized prediction of vertex v from the teacher HGNNs and student MLPs, respectively. Note that the first term of the LDH only computes the typical cross-entropy loss labeled set 1/4 The second term pulls the distance between the soft target of teacher HGNNs and the prediction distribution of student MLPs on the vertex set V. The prper-parameter À is adopted to balance the two terms. The model essentially is the MLPs with cross- cross-entropy and soft target supervision. Thus, LightHGNN has no dependency the hypergraph structure and, during inference, runs fast ML.Ps
3

---

Published as a conference paper at ICLR 2024
New
No Dependency On Hypergaph
Hypergmehh Comielution
Hypergraph Convolution uttion
Reatures
Teacher
Prodic:
Mecaage houiggoon on hypoggepp
Mestage
Pissing on Hepargrash
Perceptron
Hypergraph Neural Networks
Soft Targets
Supersising
Relable Hyperadge Qualiication Sampling
Student
Prudict
Predict
Fectures
MLPs
Distilled MLPS
- 69 Hyperedgos
High Order
Hypergraph
Soft Targets
Supervising
Offline Training
Online Deploy
Figure 2: The framework of the proposed Distilled Hypergraph Neural Networks (LighHGGN
4.2 LIGHTHGNN+: RELIABLE HYPEREDGE GUIDED TOPOLOGY- DISTILLATION
The LightHGNN simply injects the soft-target information into the student MLPs and ignores the high-order correlations from the original hypergraph. Therefore, we further propose the topology- aware distillation, named LightGGNN+ offline training stage, and the right part is the online deployment with the distilled MLPs. The top of the left part the teacher HGNNs, which takes the vertex features and the hypergraph structure as inputs and is supervised by the true label of these labeled vertexes. The bottom of the left part is the student MLPs, which only use the vertex features as input. It is supervised by the true label of the labeled vertices, the soft target of the output of the teacher HGNNs, and the high-order soft target of those reliable hyperedges. In the following, we will introduce how to qualify the reliability of hyperedges, the probability model of sampling reliable hyperedges, the extra high-order soft target constraint for the distillation, and the loss function of the proposed LightHGNN
illustrated in Figure 2 In the figure, the left part the
4.2.1 RELIABLE HYPEREDGE QUALIFICATION
Hyperedges, as the core of a hypergraph, can represent the high-order correlation among vertices. However, not all hyperedges can provide reliable information for the downstream task. Thus, develop an entropy-baat
reliable hyperedge qualification method to quantify the relevance of the hyperedge to the task. depicted in Figure 3 Given the well-trained teacher HGNNs so (X, H) + Y, we add noise on the input features and measure the invariance of the hyperedge entropy to determine the reliability of hyperedges, follows:
8e
E (N.(
1 Σ H(())) C
Σ H()).)) lel DEE
WEE
where Y' foex,(XX,1I H)
and
H)
where H(p) =- P: log(p;) is the information entropy. Given a hyperedge, we calculate the average entropy of its connected vertices" prediction distribution. The variance d_ of the average entropy of the hyperedge after introducing the noise e <~ NN,,, E) used to compute the hyper- edge's reliable score Pe. The larger value of 8, indicates the hyperedge is more sensitive to the noise perturbation the downstream task. Then, we normalize the ò, with the max value and compute the reliable score of hyperedge with Pe 1 &_ of hyperedge connected vertices of teacher HGNNs noise perturbation and reflects the reliabil- ity of hyperedge with respect to the downstream task. Those hyperedges with higher reliable scores containing robust knowledge should be paid more attention in the distillation process.
8 Clearly, the Pe measures the robustness
4.2.2 SAMPLING PROBABILITY MODELING AND HIGH-ORDE
TARGET CONSTRAINT
To fully use those hyperedge reliable scores, we propose sampling probability modeling for hy- peredge selection and develop high-order soft target constraint as an additional supervision for high- order topology-awareness awareness distillation shown in Figure SI Here, the Bernoulli distribution is adopted to model the hyperedge sampling as: p(s; Pe.) ~ Bernoulli(pa.) and e; E E, where s; the sampling probability of the hyperedge e, E. Given the hypergraph g {V. VV sub-set,
4

---

Published as a conference paper at ICLR 2024
HGNNs (Teacher)
Predict
X
81
P1 P2 [PM
Noise
<- (A.
Weights
HGNNs (Teacher)
X'
Message Passing via Hyperedge V=
Variance Entropy on Hyperediee
Reliable Score of Hyperedges
Vertex
Prediction Distributions
Figure 3: The illustration of reliable hyperedge qualification.
named reliable hyperedge set E", is drawn with independent Berouulii distribution by the specific parameter Pe for each hyperedge, as shown in Figure (T] (Appendix). Each hyperedge may con- tain vertices from different categories in the hypergraph and have the unique property behind the high-order correlation. Directly distilling the knowledge from the vertex's soft target may lose the erucial high-order correlation information. Thus, we propose constructing the high-order soft target via the vertex's 's soft target and those reliable hyperedges to inject the reliable high-order infor- mation into the distilled MLPs, shown in Figure SST (b) (Appendix). Given the soft target {yt E V} and reliable hyperedge set E", the high-order soft target can be computed via a naive message passing from vertex hyperedge as follows:
Σ vENste)
y and y
Σ vENu(e)
y for EEE,
(3)
e
where (e) denotes of the connected vertices of the hyperedge y and z denote the high-order soft target and the predicted high-order distribution, respectively. Then, the additional high-order soft-target constraint can be achieved by the KD Divergence follows:
Lhe
Σ
(
(4)
E
BEE'
egrppes (pe,)
where ax-)) is the Softmax function for normalization, and is the distillation temperature coeffi- cient. The Che designed to pull the distance between the predicted high-order distribution of the student MLPs and the high-order soft label of the teacher HGNNs. By minimizing the Lhe. the dis- tilled MLPs can reserve more reliable high-order information to achieve better performance. Finally, the total loss function of LightHGNN+ is formulated as follows:
Lpntt =
1 DE
Σ VEVL
Cee v.) + (1 A)
Σ VEV
DKL H.d + Lrc
(5)
[VT
where À is a hyper-paramett to balance the information from the true labels (yo) and teacher HGNNs (the soft targets ø', and high-order soft targets y²). Note that the supervision informa- tion of the true labels is the labeled vertex set vL, The supervision information of soft targets and high-order soft targets are on the entire vertex set V and reliable hyperedge set ε", respectively.
4.3 ANALYSIS
Time Complexity Analysis. The pseudo-code of LightHGNN* framework is in Appendix A The training of LightGNNNs can be divided into two steps: pre-training the teacher HGNNs and distill- ing knowledge into student MLPs, which supervised by the cross-entropy loss and distillation loss, respectively. The training and inference complexity comparison is provided in Appendix B
Relation GNNs-to-MIPPs Methods. We further discuss the relationship between the proposed method and the related GNNs-to-MLPP methods Zhang et al. (202T): Wu et al. (2023). The Graph- Less Neural Networks (GLNN) Zhang et al.] (2021) proposes using the soft label the teacher GNNs to supervise the student MLPs. Compared to it, our LightHGNN is simple extension from GNNs to HGNNs. However, our LightHGNN further proposes the high-order soft label to help the student MLPs learn more high-order structure information from the teacher HGNNs. As for the www.ledeeinsppree. Reliable Distillation (KRD) Wu et al. (2023), its student MLPs can only be supervised by the soft label from those reliable vertices (knowledge point in their paper), which still
5

---

Published as a conference paper at ICLR 2024
lose the structure information and cannot be utilized in the hypergraph. However, our LightHGNN can quantify the reliability of the high-order correlations and inject the reliable high-order infor- mation into the student MLPs via explicit supervision from high-order soft labels of those reliable hyperedges. The proposed I LightHGNN can sufficiently take advantage of both the vertex features and high-order structure information, thus yielding better performance and faster inference speed.
EXPERIMENTS
Datasets, Compared Methods, and Settings. In experiments, we adopt three typical graph datasets: Cora [Sen et al. (008)) Pubmed McCallum et (2000), and Citeseer Giles et al. (998), and eight hypergraph datasets: News20 Asuncion & Newman (2007), CA-Cora, CC-Cora, CC- CiteseerYYdatti
Citeseer[Giles
Yadati et al. (2019). DBLP-Paper, DBLP-Term, DBLP-Coni Sun et al. (2011), and IMDB-
AW Fu et al. (020)_ More details of the datasets are in Appendix C We compare three types of methods: GNNs (including Kipf & Welling
(2017) and GAT Velickooi et (217)), HGNNs
(including HGNN Feng et al. 2201)) HGNN Gao et al. (2022). and HNHN Dong et al. (0200) ).
GNNs-to-MILPs (including GLNN Zhang et al. (2021). KRD Wu et al. (2023) and NOSMOG Tian
jet al. (2022). We adopt two experimental settings for sufficient comparison, including the trans- ductive and production settings. The transductive is classical setting for vertex classification on
graph/hyeergraph datasets. The production setting contains both the transductive and inductive pre- dictions, which is developed to evaluate the performance towards realistic deployment. More details about the two settings and splitting of different datasets are provided in Appendix [D] We run 5 times with different random seeds for each experiment and report the average and standard deviation. The hyper-parameter configuration of each method is provided in Appendix F
Table I: Experimental results on eight hypergraph datasets under transductive setting.
Dataset
MLP
HGNN
LightHGNN
LightHGNN
AMLP
ARGNN
News20
63.53
75.94_0.66
20.32
75.87
12.72
+1.70
CA-Cora
51.86 1.47
13.19-1.12
73.63
73.5443.80
21.76
+0.44
CC-Cora
49.87+11.0
68.21
+3.68
69.25+3.46
69.48+3..0
19.61
+1.27
CC-Citeseer
51.79
63.37
-2.63
62.97
63.0142.77
11.27
-0.30
DBLP-Paper
62.84++1.s
72.27
=0.91
+2.74
72.93
10.09
+0.66
DBLP-Temm
1.58
82.01
22.27
80.27
80.77-+0.65
17.93
-1.23
DBLP-Comf
12.84
94.07
=0.11
90.23
90.24
27.40
-3.83
IMDB-AW
40.87
50.47
50.19
50.78
9.90
+0.30
Avg. Rank/Avg.
4.0
1.7
2.4
1.8
16.33
0.29
5.1
How Do LIGHTHGNNS COMPARE TO MLPs AND HGNNs?
We start by comparing the LightHGNNs to MLPs and HGNNs on eight hypergraph datasets under standard transductive learning. As shown in Table I the performance of LightHGNNs improves over MLPs by large margins of about averaged 16.3. Compared with the HGNN, the proposed LightHGNNs exhibit slight performance degradation of about 0.29. Besides, we observe that the proposed LightHGNN+ obtains seven times the best or second-best performance and ranks very close the teacher HGNN on eight hypergraph datasets. However, the LighUHGNNs adopt the same architecture as MLPs without hypergraph dependency. The experimental results demonstrate the effectiveness of distilling the knowledge from teacher HGNNs student MLPs. We attribute the improvement of LightHGNN+ to the devised topology- distillation, which can further ex- tract those reliable hyperedges and explicitly inject the reliable topology knowledge into the student MLPs, thus yielding better performance than the LightHGNN without topology distillation.
In real-world applications, not all samples can seen in the training phase. Thus, to fully validate the performance of the proposed methods confronting realistic deployment, we adopt more general setting: the production setting, which contains both transductive and inductive prediction. More de- tails can be found in Appendix D As shown in Table we that the LightHGNNs still outperform MLPs by large margins of about 14.8. However, we also notice the distinct margin of the HGNN and LightHGNNs under the inductive setting, especially cline. This is because the dataset only contains 20 hyperedges with averaged linking 982.2 vertices, which leads to high dependency on the topology of the prediction. In the inductive setting, HGNN
the DBLP-Com dataset, about 11% de-

---

Published as a conference paper at ICLR 2024
Table 2: Experimental results on eight hypergraph datasets under production setting.
Dataset
Setting
MLPs
HGNN
LightHGNN
LightHGNN*
AMLP
A
Prod.
63.8643.01
75.18
75.81 11.28
74.56-1.84
11.95
+0.63
News20
Tran.
63.8043.03
1..4
75.81
74.58_1.24 1.24
12.01
+0.73
Ind.
64.1042.97
75.13
75.82
75.44=1.94
11.71
+0.68
Prod.
50.73:1.43
71.01
13.19
70.33 23.49
22.16
20.72
+0.45
CA-Cora
Tran. 50.75:1.64
20.8
2..
72.49 72.49_2.13
23.13
21.73
+1.68
Ind.
50.67:1.44
70.83
65.14
67.34_3.96
16.67
3.48
Prod.
50.73:1144
68.20
68.29 :4.47
18.29
67.89_a..
17.55
+0.09
CC-Cora
Tran. 50.75.1.64
68.2613392
69.00
68.70
18.24
+0.73
Ind.
50.67:11.4
66.00
65.46
64.61 =4.69
14.79
0.53
Prod.
54,41111.36
64.02
62.90
64.11 0.85
9.69
+0.09
CC-Citescer
Tran. 54.42.11.52
63.74 0.775
63.30
64.53 0.63
10.10
+0.79
Ind. 54.36 #1.14
63.51 11.34
61.31 +2.21
-1.91
7.56
11.88
Prod.
63.23:11.45
71.52 1.331
71.14 1:123
71.69 11.44
8.46
+0.16
DBLP-Paper
Tran.
62.97.+1.69
10.75
70.88 1.229
71.40
11.50
8.42
+0.65
Ind. 6..25+1.75
72.72
72.22 22.08
72.86 12.33
8.61
+0.14
Prod. 63, 63.56411.10
$1.08.22.61
78.39 4.3.2
78.32=2.70
14.83
2.69
DBLP-Term
Tran. 63.37+1.11
81.23
78.54
78.58
15.21
- 2.64
Ind. 64.30+1.50
81.56
77.79
77.28
13.48
3.77
Prod. 63.56441.15
94.15
89.48
89,50
25.94
-4.64
DBLP-Conf
Tran. 63.37:11.17
94.08=0.33
91.12:0.76
91 21,20-04 0.74
27.83
-2877
Ind.
64.30.11.50
94.21 0.252
82.93
82.68 0.57
18.62
11.27
Prod. 41.0542..4
50.29 1.668
49.10
49.12
8.05
-1.17
IMDB-AW
Tran. 41.16 +2.67
49.46.a143
49.39 49.39.11.66 :1.63
49.68 1.88
8.51
+0.21
Ind.
40.61 1:195
52.05 6.668
47.96
47.58=3.21
7.35
-4.08
Arg. Rank/Avg.
4.0
2.3
-1.45
can utilize the extra topology information of those unseen vertices (unseen topology) to support the prediction, while LightHGNN cannot. Therefore, the LightHGNN shows distinct performance decline. is essential to distill the general topology knowledge and learn the topology-aware ability towards the unseen topology under the inductive setting, which can be exploited further work.
5.2 How Do LIGHTHGNNS COMPARE TO GNNS AND GNNS-TO-MI.PS?
We further compare LightHGNNs to existing GNNs and GNNs-to-MLPs on both graph and hy- pergraph datasets. More details of the experimental setting are in Appendix FF As shown in Table 3] unlike the results on pure hypergraph datasets (Tables I and D. the LightHGNN achieves the average first place, which outperforms MLPs, GNNs, HGNNs, and GNNs-to-MLR methods. The LightHGNNs show comparable performance to HGNNs in the hypergraph datasets while showing better performance in the graph datasets. This is because the topology-awar distillation can adap- tively select the task-relevant low-order and high-order structures as extra supervision. As demon- strated in HGNN Gao et al. 2022), explicitly modeling those potential high-order structures can improve the model's s performance, especially those graph datasets lacking high-order correlations.
Table 3: Experimental results on graph and hypergraph datasets.
Groph Datseets Pahmerl
Hypergraph Dainsets D.L.-PPpw
Avp.
Type
Model
Caro
Gizsver
CA-Com
IMD8-AW
Rank
MLPs
49.6 49.64
66.05 66.5
$1.69 os
51.86 1.47
62. 24.4g1.
40.87
10.0
GNN:
GAT
28.35
19.90+
77.5441.88
3..54
60.58+1.89
69.38
72.87
70.73
72.07+1.44 4.
12.53
50.62-1.44
49.55 49.55
7.7
HGNNS
HGNN
80.04
76.91
19
13.19.1.122
72.27
50:47
5.3
LIGNN
78.75
77.5441..6
69.15 .08
72.79 28
13.05+1.60
50.67 50.67
5.2
GLNN
80.93
78 18.36441.90
69.88-:
72.19
72.50
50.48-1.64
4,3
GNNS-I--MPP
19.47.1..73
78.72-11.91
60.82 36
71.75 ca
12 166
2
80.12.4.0.91
$0.22g0.533
70.86 1M.6..9..3
96. 2
7144742.13
48 48.96
RONNS-I-MEP
LighHHGNN
80.36
79:151..57
69.17
73.63 1... 14
12.74 1. 07
50.19 156
LighRENN
80.68+11.74
70 10.1664...7
10.3441..03
73,54
72.93+0.07
50.78.:1.02
1.8

---

Published as a conference paper at ICLR 2024
5.3 How FAST ARE LIGHTHGNNS COMPARED TO MLPs AND HGNNS?
In the deployment environment, the timeliness of the model is crucial. A good model can achieve higher performance in less time. Thus, we conduct three experiments for the performance and effi- ciency comparison as shown in Figure Figures [[]]and and [[b] show the comparison of accuracy vs. memory and runtime on the IMDB-AW dataset, respectively. The suffixes "-w32" and "-1.3" repre- sent the dimension of the hidden feature and the number of layers of the model, respectively. The upper left is the ideal model with lowe memory, less time, and high accuracy. Obviously, the MLPs runs faster and cost lower memory but has lower accuracy. In contrast, the HGNN performs better but runs slower and consumes more memory. The proposed methods bridge the gap between MLPs and HGNN and have advantages in memory. runtime, and accuracy, as shown in the upper left of the two figures. Considering that the IMDB-AW only contains 4278 vertices and 5257 hyperedges, further generate series of larger hypergraphs investigate how fast the proposed LightHGNNs compared to the HGNNs, as shown in Figure [[c] The x-coordinate with the log scale represents the number of vertices in the hypergraph, and the y-coordinate indicates the inference time of dif- ferent models. Obviously, under logarithmic scaling of the hypergraph, the runtime of HGNN and HGNN increases exponentially time complexity, while the proposed LightHGNNs still run very fast x100x faster in the hypergraph with 5.5w vertices compared to the HGNN) and exhibit robust linear time complexity. More details of the three experiments can be found in Appendix
5.4 How LIGITHHNN++ BENEFITS FROM TOPOLOGY-AWARE DISTILLATION?
2 and 3 GNNs and HGNNs significantly outperform the MLPs on the vertex classification task, and LightHGNN exhibits comparable performance to them. With extra topology-awae aware distillation, LightHGNN* is often better than LightHGNN. We further investigate how LightHGNN+ benefits from topology
As shown in Tables 1
distillation. As all know, GNNs and HGNNs often suffer from over-smoothing Cai & Wang 2020); Chen et al. (2022), which means the higher relevance of the vertex features and the topology. This is because that task-irrelevat information Wu et al. (2020); Zhang et al. (2022), including connections and features, will amplify the noise on vertices as layers go deeper. In our topology-aware distillation, only few task-relevant hyper- edges are selected to inject reliable topology knowledge into the student MLPs. Here, we design a topology-awand score S to measure the relevance of features and hypergraph typology as:
1 Σ [81
Evee
||3, xe 2el12 and Ie de(e) VEE
S
Ee.
(6)
de(e)
CEE
where a, and a. the embeddings of vertex and hyperedge e, respectively. The de(e) denotes the degree of the hyperedge e. The vertex embedding is the output of the first convolution layer, and the hyperedge embedding is calculated by aggregating the embedding of its connected vertices. Then, the topology-awe aware score measures the average distance of vertex and hyperedge. The lower score indicates the vertex feature is closer in each hyperedge and more relevant the topology. The higher score indicates a lower relevance to the topology.
Table 4: The topology-on aware score comparison in graph and hypergraph datasets.
Graph Datasets Pubmed
Hypergraph Datasets DBLP-Paper IMDB-AW
Model
Avg.
Cora
Citeseer
CA-Cora
MLPs
3.78
2.06
2.64
2.76
1.02
1.18
2.24
GCN
0.25
0.32
0.09
0.08
0.12
0.08
0.15
HGNN
0.31
0.27
0.13
0.10
0.12
0.04
0.16
LightHGNN
1.20
1,50
0.33
0.94
0.69
0.71
0.89
LightHGNN
1.58
1.95
0.64
1.15
0.73
0.77
1.14
The topology-awan scores of different models graph and hypergraph datasets are shown in Table Since the vanilla MLPs do not utilize the topology structure, its topology-aware score is the upper bound of all models. In contrast, the GCN and HGNN explicitly smooth the vertex with neighbors on the graph/hypergraph, which is the lower bound all models. The proposed LightHGNNs achieve trade-off score since the HGNNs is the teacher and the MLPs is the architecture. We notice that the LightHGNN obtains a higher score than the LightHGNN, which indicates that learned vertex embeddings have lower topology relevance the hypergraph. In the topology-awar distillation,
8

---

Published as a conference paper at ICLR 2024
those reliable hyperedges selected by the high resistance to the noise, which is more relevant to the task than those hyperedges with lower reliable scores. Thus, the LightHGNN* can resist the over-smoothing of features and topology, thus yielding better performance than LightHGNN. In Appendix K, we further provide visualization to demonstrate that the LightHGNN+ will pull those vertices in hyperedges with higher reliable scores than those with lower scores.
5.5 ABLATION STUDIES
In this subsection, we conduct four ablation studies of the proposed LightHGNNs on the DBLP- Paper dataset as shown in Figure under production setting (Appendix and distillation with different teacher HGNNs (Appendix H
We also provide extra ablation studies on the inductive ratio
MLP
74.0
Accuracy (%)
HUNN
Accuracy (%)
LighHENNS
g 8 8 3.
Accuracy (%)
36
Accuracy
HGGN
MLP
LightHIGNN
124
LightHONN
LighHGNNN
LighHHGNN
120
50
100
11
Noise Ratio a (%)
TTmin Each Class
(a) On noise of features.
(b) On training split.
(c) On of Lon Cpu+
(d) On of LDH+.
Figure 4: Experimental results of ablation studies.
On noise of vertex features. We investigate the robustness by adding Gaussian noise the vertex features X: X a1-a)X ac, where (O, 0.01). The model is trained with the original feature X and evaluated with the noise feature X. As shown in Figure 4(a)) the LightHGNNs achieve the middle performance between MLPs and HGNN. When given a small noise ratio, the LighUHGNNs is still better than HGNN. As the noise ratio rises, the performance of LightHGNNs is marching to that of MLPs. This is because the HGNN has an extra hypergraph structure as input, while MLPs and LighHGGNNs rely only on the vertex features to make predictions. As the noise ratio rises, the input information will be submerged in the noise, which leads to worse performance.
On training split under the transductive setting. In Figure 4(b) we show the ablation study on the number of training samples under the transductive setting. The number of training samples for each class varies from to 500. Obviously, LightHGNNs exhibits competitive performance with the HGNN and has significant advantages over MLPs. As for 5 training samples for each class, the margin between HGNN and MLPs is about 12%. Nevertheless, for 500 training samples for each class, the margin is only 4%. It indicates that the correlation information supply the model better in the few information scenarios.
Hyperparameter sensitivity on À and T. In Figures 4(c) 4(d) we provide the hyperparameter sensitivity of À and т in the loss function of LightHGNNs. The À balances the weight of supervi- sion from the true labels and soft labels, respectively. Due to the soft labels from teacher HGNN containing more information compared with the true labels Hinton et al. (2015), the performance of LightHGNNs decreases the weight of the soft label decreases. However, the LightHGNN+ can still outperform the LightHGNN via the extra high-order soft labels supervision. As for the temper- ature т, we find that too large or too small are both detrimental the distillation. In practice, the T 0.5 often yields pretty good performance, which is set in all datasets for fair comparison.
6 CONCLUSION
In this paper, we propose LightHGNNs, including LightHGNN and LightHGNN+ to bridge the gap between MLPs and HGNNs to achieve fast inference with low complexity. We design the reliable hyperedge quantification and sampling strategy to inject those task-relevant topology knowledge into the student MLPs. Extensive experiments on 11 real-world graph and hypergraph datasets indicate our LightHGNNs can achieve competitive performance to HGNNs and GNNs. Besides, experiments series of larger-scale synthetic hypergraph datasets indicate that by eliminating hypergraph dependency, our LightHGNNs can achieve 100x faster inference, demonstrating the potential to deploy in realistic lotency-conttrained applications.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by National Natural Science Funds of China (No. 62088102 62021002),
Beijing Natural Science Foundation (No. 4222025).
REFERENCES
Arthur Asuncion and David Newman. UCI machine learning repository, 2007.
Song Bai, Feihu Zhang, and Philip HS Torr. Hypergraph convolution and hypergraph attention. Pattern Recognition, 110: 107:677 2021.
Chen Cai and Yusu Wang. A note on over-smoothing for graph neural networks. In 37th Interna- tional Conference on Machine Learning, 2020.
Guanzi Chen, Jiying Zhang, Xi Xiao, and Yang Li. Preventing over-smoothing for hypergraph neural networks. CoRR, abs/2203. 2022.
Eli Chien, Chao Pan, Jianhao Peng, and Olgica Milenkovic. You are AllSet: A multiset function framework for hypergraph neural networks. In International Conference on Learning Represen- tations, 2022.
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pp. 3844-3852, 2016.
Yihe Dong. Will Sawin, and Yoshua Bengio. HNHN: Hypergraph networks with hyperedge neurons. CoRR, abs/2006. 12278, 2020.
Yifan Feng, Haoxuan You, Zizhao Zhang, Rongrong Ji, and Yue Gao. Hypergraph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 3558-3565, 2019.
Sichao Fu, Weifeng Liu, Yicong Zhou, and Liqiang Nie. HpLapGCN: Hypergraph p-laplacian graph convolutional networks. Neurocomputing, 362:166-174, 2019.
Xinyu Fu, Jiani Zhang. Ziqiao Meng, and Irwin King. MaGNN: Metapath aggregated graph neural
network for heterogeneous graph embedding. In Proceedings of The Web Conference 2020, pp. 2331-23411 2020.
IEEE Transactions on Pattern Analysis and Machine Intelligence, 15(3):3181-319) 2022. Yue Gao, Yifan Feng, Shuyi Ji, and Rongrong Ji. HGNN+ General hypergraph neural networks.
C Lee Giles, Kurt D Bollacker, and Steve Lawrence. Citeseer: An automatic citation indexing system. In Proceedings of the third ACM conference on Digital libraries, pp. 89-98, 1998.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, stut, 1050:9, 2015.
Jing Huang and Jie Yang. UniGNN: A unified framework for graph and hypergraph neural networks. In Proceedings the Thirtieth International Joint Conference on Artificial Intelligence, pp. 2563- 2569, 2021.
Shuyi Ji, Yifan Feng, Rongrong Ji, Xibin Zhao, Wanwan Tang, and Yue Gao. Dual channel hyper- graph collaborative filtering. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 2029-2029, 2020.
Jianwen Jiang, Yuxuan Wei, Yifan Feng. Jingxuan Cao, and Yue Gao. Dynamic hypergraph neural networks. In IJCAI, pp. 2635-2641, 2019.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net-
works. In International Conference on Learning Representations, 2017.
Andrew Kachites McCallum, Kamal Nigam, Jason Rennie, and Kristie Seymore. Automating the construction of internet portals with machine learning. Information Retrieval, 3:127-163, 2000.
10

---

Published as a conference paper at ICLR 2024
Kevin A Murgas, Emil Saucan, and Romeil Sandhu. Hypergraph geometry reflects higher-order dynamics in protein interaction networks. Scientific Reports. 12(1):20879, 2022.
Ding Ruan, Shuyi Ji, Chenggang Yan, Junjie Zhu, Xibin Zhao, Yuedong Yang, Yue Gao, Changqing Zou, and Qionghai Dai. Exploring complex and heterogeneous correlations hypergraph for the prediction of drug-target interactions. Patterns, 2(12), 2021.
Khaled Mohammed Saifuddin, Briana Bumgardner, Farhan Tanvir, and Esra Akbas. HyGNN: Drug- drug interaction prediction via hypergraph neural network. In 2023 IEEE 39th International Conference Data Engineering (ICDE), pp. 1p.15503-4 1503-1516, 2023.
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. Collective classification network data. Al magazine, 29:93-92 93, 2008.
Yizhou Sun, Jiawei Han, Xifeng Yan, Philip S Yu, and Tianyi Wu. Pathsim: Meta path-based top-k similarity search in heterogeneous information networks. Proceedings of the VLDB Endowment, 4(L1):992-1003, 2011.
Yijun Tian, Chuxu Zhang, Zhichun Guo, Xiangliang Zhang, and Nitesh Chawla. Learning ML.s on graphs: unified view of effectiveness, robustness, and efficiency. The Eleventh International Conference on Learning Representations, 2022.
Petar Velickovie, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, Yoshua Ben- gio, et al. Graph attention networks. stat, 1050420;10-488500, 2017.
Ramon Viñas, Chaitanya K Joshi, Dobrik Georgiev, Phillip Lin, Bianca Dumitrascu, Eric R Gama- zon, and Pietro Liò. Hypergraph factorization for multi-tissue gene expression imputation. Nature Machine Intelligence, pp. 1-15, 2023.
Peihao Wang, Shenghao Yang, Yunyu Liu, Zhangyang Wang, and Pan Li. Equivariant hypergraph diffusion neural operators. The Eleventh International Conference on Learning Representa- tions, 2023.
Lirong Wu, Haitao Lin, Yufei Huang, and Stan Z. Li. Quantifying the knowledge in gnns for reliable distillation into MLPs. In Proceedings of the 40th International Conference on Machine Learning, 2023.
Tailin Wu, Hongyu Ren, Pan Li, and Jure Leskovec. Graph information bottleneck. Advances in Neural Information Processing Systems, 33:20437-20448, 2020.
Lianghao Xia, Chao Huang. Yong Xu, Jiashu Zhao, Dawei Yin, and Jimmy Huang. Hypergraph con- trastive collaborative filtering. Proceedings of the 45th International ACM SIGIR conference research and development in information retrieval, pp. 70-79, 2022.
Naganand Yadati, Madhav Nimishakavi, Prateek Yadav, Vikram Nitin, Anand Louis, and Partha Talukdar. HyperGCN; A new method for training graph convolutional networks on hypergraphs. Advances in neural information processing systems, 32, 2019.
Cheng Yang, Jiawei Liu, and Chuan Shi. Extract the knowledge of graph neural networks and go beyond it: An effective knowledge distillation framework. In Proceedings of the web conference 2021, pp. 1227-1237, 2021.
Shichang Zhang, Yozen Liu, Yizhou Sun, and Neil Shah. Graph-less neural networks: Teaching old MIPs new tricks via distillation. In International Conference on Learning Representations, 2021.
Zizhao Zhang, Yifan Feng, Shihui Ying, and Yue Gao. Deep hypergraph structure learning. CoRR, 12547, 2022.
11

---

Published as a conference paper at ICLR 2024
Reliable Hyperedge Qualification
P1
Soft Target
Soft Target
PM
e5
Hypergraph
Score
Sampling via Bernouli Distribution
Soft Target
B(p))
B(pz)
B(Pm)
Soft Target
Message Passing Hyperedge: V =
y
Hyperedge e
Hyperedge es
Hyperedge eg
High-Ordes Target Hyperedge es
Reliable Hyperedge Set g'
(b) The construction of high-order Soft target.
(a) Sampling Probability Modeling
Figure S1: The illustration of sampling probability modeling and high-order soft target.
A ALGORITHM
In this section, we provide the pseudo-code of the LightHGNN framework for better understand- ing. As shown in Algorithmi] to achieve HGNNs-to-Mil the teacher HGNNs via the cross-entropy loss on the labeled vertices. Then, the soft targets and the hyperedge reliable scores are extracted based on the teacher HGNNs. each epoch the training process of the student MIPs, we sample the reliable hyperedge set from the total hyperedges and compute the bigh-order soft-label loss via those reliable hyperedges and the prediction of student and teacher. The original true label and soft-targe supervision are also calculated for the final loss Lhe of LightHGNN+ After training, the MLPs distilled from the HGNNs can be deployed for the online fast-inference environment without dependency on hypergraphs.
knowledge distillation, we first pre-trann
Algorithm Algorithm LightHGNN* Framework.
Input: The hypergraph
EV, Vertex feature matrix X, Labeled dataset D (V=, J/43.
Number of epoch E.
Output: Predicted labels Do for unlabeled vertices, Parameters of student MLPs (0)
1: Pre-train the teacher HGNNs with label y
2: Initialize the parameters {O]= of student MLPs.
3: (if V) extract the soft targets via teacher HGNNs.
4: {pell |elEEE f- reliable hyperedge qualification.
5: 4+0
6: while < do
7:
+++++1
8: Predict the distribution of vertices { v E V} via student MI.Ps.
9:
Sample reliable hyperedge set E' le & p(e) Bernoulli(p.)) from the original
hyperedge set E.
Calculate the high-order soft targets { e 8/ of hyperedges in the reliable hyperedge set via vertices' soft targets.
Calculate the student MLPs' prediction { e E E'} of high-order soft targets of hyper- edges in the reliable hyperedge set.
12:
Calculate the total loss LDD+ of cross-entropy loss, soft-target loss, and high-order soft-
target loss.
13:
Update student MLPs' parameters [O)_ by back propagation. end while
14:
15: return Predicted labels Do for unlabeled vertices, Parameters of student MLPs (
12

---

Published as a conference paper at ICLR 2024
B TIME COMPLEXITY ANALYSIS
In this section, we provide the time complexity analysis during training and inference for deeper comparison, shown in Table SI In this table, the and F' denote the number of layers and the dimension of hidden layers, respectively. N and M are the number of vertices and hyperedges. C indicates the number of categories. Clearly, the MLPs take O F²) time complexity during the training and inference, In comparison, the HGNN propagates messages via the N x N hypergraph Laplacian matrix, which needs extra ((L F) time complexity, HGNN further develops a two- stage message propagation vertex-hyperedge-eeree vertex, which needs the ((LN time complexity.
Table S1: The time complexity comparison during training and inference.
ML.Ss
HGNN
HGNN
LighHGNN
LighHGNN
Training
O(LNF
OILN'F + LNF") OILN*F+ LNF"
OILNMP + LNFE) O(LNMP LNF*)
O(LNE O(LNF
ONNCC +UNMCLLFF))
Inference
O(LNF
O(LNP?)
As for the proposed LightHGNN, it takes the same time complexity O(LN2 F) the MLPs during the training and inference stage. During training, for better capture the high-order information, the LightHGNN+ evaluates the reliable score of hyperedges and generates the high-order soft label, which both need to propagate the label distribution from vertex to hyperedge (O(N Reliable hyperedge sampling only needs ((M). Therefore, the LightHGNN takes MC + +LN time complexity for training and runs as fast as MLPs in inference.
C DATASET
We adopt three common-used graph datasets: Cora Sen et al. 2008). Pubmed McCallum et al. (2000), and Citeseer Giles al. (T998), and eight hypergraph datasets: News20 Asuncion & New- man 2007). CA-Cora CC-Cora, CC-Citeseer Yadati et al. (2019), DBLP-Paper, DBLP- Term, DBLP-Conf Sun al. (20IT), and IMDB-AW Fu et al. (2020). Table S2 provides the summary of these datasets. The Cora, Pubmed, and Citeseer are the paper citation networks, where each vertex denotes a scientific publication, and the label is the paper's topic. Each paper has sparse bag-of-words feature vector and citation relationships among publications represented by corre- sponding edges, As for the hypergraph datasets, the CA-Cora, CC-Cora, and CC-Citeseer are also the publication datasets, where the vertices that are co-authored (CA) by an author co-cited (CC) by publication are connected in one hyperedge. The label of the vertex is also the topic of the publication. The authors the vertices in the three hypergraphs: DBLP-Paper DBL.-Conf. The hyperedge constructed by the cooperating a paper, using the same term, published in the same conference, respectively. The label of the vertex is the research of the author. As for the IMDB-AW dataset, the vertex is a movie, and the label is the corresponding category. The
BBrP-Term, and
Table S2: Staties of datasets. "#Edge" denotes the number of edges or hyperedges.
Datasets
Type
#Nodes
#Edge
#Features d,
de
#Classes
Cora
Graph Graph
2,708
7,440
1,433
4.8
Pubmed
19,717
54,944
500
5.5
2
3
Citeseer
Graph
3,327
6,590
3,703
3.7
6
News20
Hypergraph
16,342
100
100
4.0
327.7
4
CA-Cora
Hypergraph
2,708
970
1,433
1.7
3.6
7
CC-Cora
Hypergraph
2,708
1,483
1,433
2.1
2.1
CC-Citeseer
Hypergraph
3,312
1,004
3,703
1.5
1.8
6
DBLP-paper
Hypergraph
4,057
5,701
334
2.3
1.6
4
DBLP-term
Hypergraph
4,057
6,089
334
28.6
19.1
4
DBLP-Conf
Hypergraph
4,057
20
334
4.8
982.2
4
IMDB-AW
Hypergraph
4,278
5,257
3.066
3.5
2.9
3
13

---

Published as a conference paper at ICLR 2024
dataset contains two types of hyperedges: the co-actor and the co-writer relationships. For each actor or writer, participating in movies is connected by a hyperedge.
D TRANSDUCTIVE SETTING AND PRODUCTION SETTING
The transductive setting is widely adopted in the evaluation of the vertex classification on the graph and hypergraph datasets. In this setting, the vertex set V is divided into two sub-sets: labeled vertex set y and unlabeled vertex set D The unlabeled vertex set used for testing and final evaluation. At the same time, the labeled vertex is further divided into the training set V%r and validation set Ve the training phase, big hypergraph ς including vertices from V UV UV is constructed for message passing. However, only labels of those vertices from the training set V/2 are used to supervise the model's training, and the validation set VL. is adopted for "best model" selection. Finally, report the performance of the "selected best model" on the testing set y Note that, in this setting, those vertices in the testing set yo are still visible in the training phase. The labels of vertices from the validation set VLL and testing set V/2 are absolutely unknown in the training phase. The detailed splitting of different datasets under the transductive setting is provided in Table
Table S3: Splitting of datasets under the transductive setting.
Datasets
#Nodes
#Classes
#Training
#Validation
# #Testing
Cora
2,708
140
700
1,868
Pubmed
19,717
60
300
19,357
Citeseer
3,327
120
600
2,607
News20
16,342
4
80
400
15,862
CA-Cora
2,708
700
1,868
CC-Cora
2,708
7
140
700
1,868
CC-Citeseer
3,312
120
600
2,592
DBLP-Puper
4,057
4
80
400
3,577
DBLP-Term
4,057
80
400
3,577
DBLP-
4,057
4
80
400
3,577
IMDB-AW 4,278
60
300
3,918
However, the transductive setting is not the best way to evaluate deployed model in the real world, where the unseen vertices usually appear in the testing set. Consequently, in this paper, we utilize the realistic production setting, which contains both transductive and inductive predictions. In the production setting, the unlabeled set y is further divided into transductive testing set V and inductive testing set DIL In the training phase, vertices from Vot. V/d U VLL U U.P utilized to construct a sub hypergraph Grub for messages passing. The model training and selection are the same the transductive setting. Then, we can fetch three types of performance, including under transductive setting, inductive setting, and production setting, respectively. The performance under the transductive and inductive settings is obtained by the big hypergraph and the transductive testing set D as input and sub-hypergraph Gsub and inductive testing set D as input, respectively.
Table S4: Splitting of datasets under the production setting.
#Transductive Testing
#Inductive Testing
Datasets
#Nodes
#Classes
#Training
#Validation
News20
16,342
80
400
12,689
3,172
CA-Cora
2,708
140
700
1,494
373
CC-Cora
2,708
7
140
700
1,494
373
CC-Citeseer
3,312
120
600
2,073
518
DBLP Paper
4,057
80
2,861
715
DBLP-Tere Term
4,057
80
400
2,861
715
DBLP-Com
4,057
4
80
400
2,861
715
IMDB-AW
4,278
60
300
3,134
783
Recipe-1000
101,585
160
800
80,500
20,125
Recipe-200k
240,094
160
191,307
47,826
14

---

Published as a conference paper at ICLR 2024
The performance under the production setting is obtained by the big hypergraph G and the testing set you = V U. as input. In our experiments, 20% vertices the testing set V are adopted for the inductive testing and the rest for the transductive testing. The detailed splitting of different datasets under the production setting provided in Table S4
E TRAINING DETAILS
We run 5 times with different random seeds for each experiment and report the average and standard deviation. In each run, 20 samples from each category are selected for training, and 100 samples from each category selected for validation. The rest used for testing. The accuracy is adopted for performance comparison, and the model with the best performance in the validation for the results. Adam is adopted for optimization. KRD implemented
is applied the based on their released code The experiments of other baselines and our methods are implemented using Pytorch and DHG library As for the GNNs-to-MLA methods, the teacher the GCN, and the student is the MLPs. Besides, in the training phase, the same hyper- parameters are used for all methods achieve a fair comparison, as shown in Tables S3 and SS6 We run all experiments on machine with 40 Intel(R) Xeon(R) E5-2640 v4 @2.40GHZ CPUs, and single NVIDIA GeForce RTX 3090 GPU.
Table S5: Hyper-parameter configuration of GCNs and HGNNs.
GCN
GAT
HGNN
HGNN+
UniGNN
UniGAT
Learning Rate
0.01
0.01
0.01
0.01
0.01
0.01
Weight Decay
0.0005
0.0005
0.0005
0.0005
0.0005
0.0005
Dropout Rate
0.5
0.5
0.5
0.5
0.5
0.5
Hidden Dimension
32
32
32
32
8
#Attention Heads
4
#Layer
2
2
2
2
2
Table S6: Hyper-parameter configuration of GNNs-to-MLPPs and L LightHGNNs.
MLPs
GLNN
KRD
LightHGNN
LighHGGNN
Learning Rate
0.01
0.01
0.01
0.01
0.01
Weight Decay
0.0005
0.0005
0.0005
0.0005
0,0005
Dropout Rate
0.5
0.5
0.5
0.5
0.5
Hidden Dimension
128
128
128
128
128
A
0
0
T #Layer
0.5
0.5
0.5
2
2
2
Multi-Layer Perceptrons (MLPs). To achieve efficient inference, the naive MLPs is adopted. The 1-th layer of the MI.Ps is defined as:
-+1 Dropout (a(x'0')).
(7)
where 2 the embedding as the input I-th Layer, and the o()) non-linear activation function. {0')=, is the learnalle parameters of the MLPs. By default, the last layer removes the dropout and activation functions.
F GRAPH MODELS ON HYPERGRAPH DATASETS AND HYPERGRAPH MODELS ON GRAPH DATASETS
In this section, we introduce how to deploy the graph models on the hypergraph datasets and how to deploy the hypergraph models on the graph datasets. As for the graph datasets, if the method is
https://gituu..com/// wwt.://www..oocoooaiiaaaaaeeecea wotp.//ooooooooonooootoo https://gthhua.com////ooo//a//a http://///m...oooaaa/eeeeeeygnno
15

---

Published as a conference paper at ICLR 2024
hypergraph-basee model like HGNN and HGNN will construct a hypergraph upon the original graph via the concatenation the original pair-wise edge and the 1-hop neighborhood hyperedges as Feng al. (2019). Given the graph, the 1-hop neighborhood hyperedges {e, je is the union of the hyperedge from each vertex's s 1-hop neighborhood. For the vertex ei, its 1-hop neighborhood hypergraph can be defined as:
e, (v:
EN(s)))
where NV-) represents the neighbor set of the specified vertex. By constructing the hypergraph from the graph in this way, the HGNNs can fully utilize the high-order representation and learning capability in the simple graph. As for the hypergraph datasets, if the method is graph-based model like GCN and GAT, the clique expansion Gao et al. (2022) is utilized transfer the hypergraph to the graph structure. Specifically, given hypergraph 9 (V,E), for each hyperedge, we link every pair of vertices the hyperedge to generate edges as follows:
(i.j) - { if Ui Ee and 339
else
where E {0, 13 NxN the adjacency matrix of the generated graph from the hypergraph.
G DETAILS OF How FAST ARE THE PROPOSED LIGHTHGNNS
To exploit the inference potential larger hypergraph datasets, we manually synthesize a series of hypergraphs shown in Table S7 The 12 synthetical hypergraphs are generated by the function "dhg.random wat.ggganndo...egrpap..ccc Seypperraph_Gnmm of the DHG library. The number of vertices (N) varies from 1000
to 55000, and the number of hyperedges is fixed to N/8. The dimensions of vertex features and hidden layers are set to 16. Given the synthetic hypergraph, we report the inference time the entire hypergraph and vertex feature matrix input, like the transductive setting. Obviously, the scale of the hypergraph increases, the inference time HGNNs increases exponentially. In contrast, the LightHGNNs still exhibit stable inference speed. The advantage of LightHGNNs inference speed increases as the size of the hypergraph grows.
In Table S8 conduct experiments
larger hypergraph datasets: Recipe- 100k (10w vertices) and Recipe-200k (24w vertices). As shown in the table, our LightHGNNs can not only achieve better performance but also extremely reduce the inference time. As the scale of the dataset increases, the advantage of our LightHGNNs becomes more evident. Naive HGNN becomes slower and slower, even throwing the nut-Of-Memory (OOM) error confronting 10w+ vertices, while our LightHGNNs are still fast. Besides, our LightHGNNs exhibit better performance compared to the naive HGNN under the transductive testing set (about 1w or 2w vertices for testing), which demonstrates the effectiveness of our distillation framework. Besides, we provide the accuracy, memory, and runtime comparison of methods with different configurations on the IMDB-AW dataset in Table
Table S7: Inference time (ms) comparison 12 synthetic hypergraphs.
#Vertices of Hypergraph
HGNN
LightHGNNs HGNN+
Faster
1,000
0.47
0.98
0.39
1,2x
2,000
0.50
0.98
0.40
1.3x
4,000
1.10
1.02
0.42
2.6x
10,000
5.39
3.77
0.44
12.3x
20,000
19.00
14.79
0.56
33,9x
25,000
29.78
22.92
0.67
44.4x
30,000
42.44
29.60
0.78
54.4x
35,000
56.85
45.04
0.91
62.5x
40,000
73.72
57.18
1.03
71.6x
45,000
92.30
71.98
1.15
80.3x
50,000
113.20
88.42
1.26
89.8x
55,000
137.70
107.24
1.37
100.5x
16

---

Published as a conference paper at ICLR 2024
Table S8: Experimental results two larger-scale hypergraph datasets. "#Testing" indicates "The
number of vertices for testing". "#Infer" denotes the "Inference Time". "OOM" denotes the "Out of Memory" error.
Recipe- 100k
Recipe-200k
#Testing
Accuracy
#Infer
#Testing
Accuracy
#Infer
Trans. 10,063
41.2 41.2643.00 3.08
5.39
23,914
37.25
29.78
HGNN
Ind.
90,562
OOM
00
215,220
OOM
00
Prod.
100,625
0OM
239,134
0OM
00
Trans.
10,063
41.82:3.48
0.44
23,914
38.4845.59
0.67
LightHGNN
Ind.
90,562
43.25
2.18
215,220
37.62+5.89
5.05
Prod.
100,625
41.22.3.26
2.40
239,134
37.70+5.86
5.62
Trans.
10,063
42.50
0.44
23,914
38.7645.24
0.67
LightHGNN+
Ind. 90,562
42.28
2.18
215,220
38.26+5.67
5.05
Prod. 100,625
42.31
2.40
239,134
38.31
5.62
Table S9: The comparison of accuracy, memory, and runtime on the IMDB-AW dataset.
Methods
Runtime (ms)
Memory (MB)
Accuracy (%)
MLPs-L1
0.26
4.27
36.48
MLPs-L3
0.52
12.63
40.89
MLPs-w32
0.35
8.45
40.00
MLPs-w64
0,40
16,80
40.46
MLPs
0,40
33.51
40.77
MLPs-w256
0.53
66.94
41.6311.22
HGNN-L1
0.35
144.94
48.58 11.23
HGNN
0,53
289.79
50.78
HGNN-L3
0.65
434.64
49.01
HGNN-w64
0.57
300.24
50.44
HGNN-
0.83
321.13
49.77
HGNN -226
1.45
362.90
49.63
HGNN -L1
0.77
6.60
50.78
+GNN+
1.31
13.10
50.77
HGNN+ -L3
1,88
19.61
48.48
HGNN+
1.53
26.12
50.37 11.99
HGNN+-W128
2.12
52.14
50.40
HGNN
3.35
104.18
50.06
LightHGNNs-wW33
0.35
8.45
46.60
LightHGNNs W66
0.40
16.80
50.77
LightHGNNs
0,40
33.51
51.53
LighHGNNs-w2556
0.53
66.94
50.32+2.68
H ABLATION STUDY ON TEACHER HGNNS ARCHITECTURE
In this section, we further investigate the performance of the LightHGNNs with different teacher architectures on the DBLP-Paper dataset, as shown in Table S1O Among these methods, HGNN Feng et al. (2019) serves as our baseline approach. HGNN Gao et al. (2022) extends HGNN by introducing general framework for modeling high-order data correlations. UniGNN, UniGAT, and UniGCNII Huang & Yangl (2021) generalize pre-designed graph neural network architectures to hypergraphs. ED-HNN Wang et al. (2023) combines the star expansions of hypergraphs with standard message passing neural networks. AllSet[Chien al. (2022) implements hypergraph neural network layers as compositions of two multiset functions that can be efficiently learned for each task and each dataset. The "Original" in the table denotes the performance of the teacher architecture. We observe that the LightHGNNs show robust competitive performance compared with different teachers. Experimental results indicate that method adapt to different teachers, and the
17

---

Published as a conference paper at ICLR 2024
better the teacher network's performance, the better method's performance will be improved accordingly.
Table S10: Experimental results of LightHGNNs with different teacher Architectures.
Method
Setting
Original
LightHGNN+ LightHGNN
Prod.
71.52
71.14
71.69 11.44
HGNN
Tran.
70.75
70.88
71.40
Ind.
2..32
72.22
72.86
Prod. Tran.
12.91 1.22
72.63 0.88
72.65 10.66
HGNN+
71.8311..5
72.33
72.47 10.95
Ind.
74.43 12.39
73.84 1.69
74.15 1.66
Prod. 72.73
72.66 1.44
72.83 1.66
UniGNN
Tran.
71.88 1:.51
72.38 1.55
72.40 1166
Ind. 73.79
73.79
74.54 22.5
Prod. 73.00
11.40
72.74 1.20
72.59 1.32
UniGAT
Tran. Ind.
71.99
11.49
72.46 11.36
72.32
74.29
12.53
73.90 1.69
73.65 22.64
Prod. 72.01 11.06
73,45
11.48
73.53+1.61
ED-HNN
Tran. 70.64
72. 12.11:14 1.66
73.34 11.78
Ind.
73.79 11.72
74.79 11.61
74.32 1.86
Prod. 72.94
73,29
73.58
UniGCNII
Tran.
72.04 1..844.....4
1.112 72.87
73.05 11.60
Ind. 73.81 1:2.48
74.95 2.33
75.71 12.54
Prod. 71.37
73,20
73.53 1:448
AllSet
Tran. 71.15+1.93 Ind. 72.97
73.00
73.14 1.32
75.36 2200
75.10
Avg. Rank
2.43
2.1
1.43
I ABLATION STUDY ON INDUCTIVE RATIO UNDER PRODUCTION SETTING
In this section, adjust the ratio of the inductive testing from 10% 90% to investigate the in- fluence of the unseen vertices ratio on the performance in the DBLP-Paper dataset. Experimental results are shown Table [STT] Based on the results, we have three observations. Firstly, as the ratio of inductive testing increases, the performance of the three types of settings all show slight decline. This is because a task under the inductive setting inherently harder than that under the transductive setting due to the seen topology being less. Secondly, observe that the proposed LighHGNNNs often show competitive performance to the HGNNs and exhibit significant improve- ment the MLPs, demonstrating the proposed methods' effectiveness. Thirdly, LighHGNN+ shows better performance when the inductive testing ratio is small. This is because as the inductive testing ratio increases, the unseen hyperedges and vertex features also increase. This information, especially the unseen topology, will be seen in the inductive testing of the HGNNs, while still is unseen in the testing of the LightHGNNs. As the unseen information increases, the performance of our LightHGNN will decrease accordingly.
notice that
J MORE EVALUATION ON THE NUMBER OF LAYERS AND THE TOPOLOGY-AWARE SCORE.
attempt to define specific metric to quantify the over-smoothing phenomenon and design a topology-awee aware score S to measure the relevance features and hypergraph typology. Specifically, we first calculate the cosine similarity between the vertex and hyperedge features and then calculate the average cosine similarity of all vertices in the hypergraph. The topology- aware score S is defined as the average cosine similarity of all hyperedges in the hypergraph. The higher
In Section 5.4
18

---

Published as a conference paper at ICLR 2024
Table S11: Experimental results of ablation study inductive ratio under production setting.
Ind//Trans. Ratio
Setting
MLP
HGNN
LightHGNN
LightHGNN+
Prod.
63.56,1.22
71.30 11.82
70.91 11.35
71.0642.08
10%/90%
Tran.
63,4941.10
70.95+2.14
70.76+1.38
70.98
Ind.
64.20+1.92
72.04
72.26 65
71.82
Prod.
63.5641.15
71.52+1..3
71.1441.22 1.23
71.69 1.44
20%/80%
Tran. 63.37
70.75
70.88 1.29
71.40
Ind. 64.3041.50
72.72
72.22
72.86
Prod. 63.364
71.58400.7
71.69
72.08 20.70
30%/70%
Tran.
63.1041.300
70.31 70.31 1.33
71.21+1.30
71.64
Ind. 63.97
72.61
72.80+0.80
73.10 0..73
Prod. 63.29 1.22
70.44.11.61
10.73
70.93 11:38
40%/60%
Tran.
62.87+1.23
68.92
70.41 70.41
70.52
Ind. 63.93
71.03
71.20+1.30
71.55
Prod. 63.17
70.84
70.57
70.31 1:0.60
50%/50%
Tran. 63.23
69.21
70.82 1.48
70.48
Ind. 63.11
70.7640.71
70.33
70.14 70.1441.10
Prod.
63.32
70.24.0.77
69,27 20.60
70.15
60%/40%
Tran.
63.15
68.8840.00
69.53
70.30+1.30
Ind.
63.43
70.11 10.08
69.09
70.06
Prod. 63.28
70.50
1..40
69.93 1.61
69.98 L:: 1.4.22
70%/30%
Tran.
63.25
68.79 11.68
69.79 12.10
69.98 12111
Ind. 63.30+1.53
70.41 +1.2
69.98
69.99 69.98
Prod. 63.24
70. .:1.45
68.98 1.228
68.84 18.8441..11
80%
Tran.
62.79:12.20
67.51 1:1.84
68.65 12111
68.18 68.18422.99
Ind.
63.36
70.14
+ 1.47
69.06 1.37
69.01 1.24
Prod. 63.38
69.81 +:128
69.
+0.99
68.3441.47
90%/10%
Tran. 63.9642.76
68.88 2.2.28
69.94 13.12
68.43 +3.20
Ind.
63.3241.53
69.66.11.20
69.02 1.02
68.33
19

---

Published as a conference paper at ICLR 2024
the score, the more relevant the features and hypergraph typology. We first provide the accuracy of MLP, HGNN, and LightHGNNs with respect to different numbers of layers, as shown in Ta- ble ST2 In the table, the accuracy of MLP, HGNN, and LightHGNNs all significantly decrease as the number layers increases, especially for the MLP and HGNN. However, the accuracy of LightHGNN+ is more stable than that of MLP, HGNN, and LightHGNNs. This is because the proposed LightHGNN can alleviate the over-smoothing phenomenon via the hyperedge reliability sampling, thus yielding robust performance that combats the increase in the number of layers.
Table $12: Experimental results of different number of layers IMDB-AW dataset.
#layer
MLP
HGNN
LightHGNN
LightHGNN+
2
40.87 11.43
50.78
50.19 1.56
50.47 11.92
3
41,25 11.03
50.32
49.52
50.38 40.38 22.23
4
41.44
44.97
47.58
49.75
5
41.52
42.5942.45
48.55
46.97
41.18 20.93
39.7213.76
47.61 55.5
47, 47.66
7
40.23+1.48
31.47+5.61
46.334s.06
45.97
11.95
32.89+4.11
46.64
47, 47.16
9
36.81 12.22
32.14
46.24 15.50
47.69 16.03
36.75
12.35
32.91 10.89
42.64 16.11
14..8815.90
We further provide the topology-aware score S of MLP, HGNN, and LightHGNNs with respect to different numbers of layers, as shown Table S13 Based on the results the table, we have the following three observations. First, despite the increase in the number of layers, the topology-aves score of MLP is always higher than that HGNN and LightHGNNs. This is because the MLP only relies the vertex features and ignores the hypergraph typology. Second, the topology score of LightHGNN' is more stable than MLP, HGNN, and LightHGNNs. This is because the proposed LightHGNN+ sampling, thus yielding robust performance that combats the increase the number of layers. Third, find interesting phenomenon that the topology-aware score significantly decreases in #layer 6 + 7. The results explain when the accuracy of HGNN and LightHGNN significantly decreases #layer + 7, shown in the above table. This is because the phenomenon of over-smoothing has reached a critical point with the increase in the number of layers, impacting the representations for hypergraphs. As result, the performance has sharply declined. However, why the critical point is reached at this specific layer needs further investigation in future work. We believe this is an intriguing experimental result that will impact the research community's study over-smoothing.
alleviate the over-smoothing phenomenon via the hyperedge reliability
Table S13: Results of topology-awe aware scores on the IMDB-AW dataset.
#Layer
MLP
HGNN
LightHGNN
LightHGNN*
2 1.180
0.040
0.713
0.770
3 2.498
0.049
0.876
0,954
4
2.660
0.017
0.782
0.821
5
2.778
0.023
0.533
0.648
6
2.887
0.021
0.454
0.570
7
1.922
0.008
0.076
0.376
8
2.842
0.004
0.082
0.226
9
2,480
0.007
0.058
0.178
10
2.096
0.006
0.014
0.120
K VISUALIZATION
In this section, we provide the visualization of the hyperedge reliable score on different datasets as To investigate whether the topology-aware distillation adaptively inject the high-order information into the student, we calculate the distance (KL Divergence) of the cor- responding hyperedge between the student LightHGNN+ and the teacher HGNN. The x-coordinate
shown in Figure S2
20

---

Published as a conference paper at ICLR 2024
Datance lypprciio betveon LightlONN
Daasse of beowees HCNN Lighttin
D byperiign horvem HGGNS and Ligattions
2.20
19
pof Hyperaioo (a) Cora.
of Hyporeaaee (b) Pubmed.
(c) Citeseer
Diurce of lyperiigo behoom LaghilONS
Diancee Plyeredoo betwonns BOSs and LigHtONN
Dicaace of Hygurege betaonn
DCN togtightt
Hypereggee (d) CA-Cora.
per Hsperedges (c) CC-Cora.
pllpologee CC-Citeseer
Dannce lyppeiige hannen HONS Boo0--rlll LighTINN
Droioo Oypne eaggrs betwown
Dreuzee of Hyperddge hymveess LigHTINNN
HUNN and Light SONNN
2012
21
Hypeslges DBLP-Paper.
Hypeeegg
pofllypeedge IMDB-AW
(g)
(h) DBLP-Term.
Figure S2: Visualization on the hyperedge reliable score.
denotes the hyperedge reliable score p of each hyperedge, and the y-coordinate is the distance of the high-order soft targets of the teacher and student calculated by equation 3 Obviously, those hyperedges with higher reliable scores will be closer the teacher, which demonstrates that topology-awan distillation can adaptively inject reliable high-order information into the student. The teacher HGNNs blindly smooth vertex features via all hyperedges, while the student only fo- cuses on a few reliable hyperedges and uses them to guide the message prorogation. This is also the main reason why our LightHGNN+ can effectively resist over- smoothing, as stated Section 5.4
21


Reviews:
[{"review_id": "IbV1GrLmAY", "reviewer": "Reviewer_tcvF", "summary": "The authors demonstrate that a Hypergraph Neural Network (HGNN), which is specifically designed for hypergraph-structured data, can be effectively distilled into a Multi-Layer Perceptron (MLP). To this end, the authors extend a Graph-Neural-Network (GNN) distillation technique (Wu et al. 2023) to accommodate hypergraphs.", "questions": "Q1. Please address W1.\n\nQ2. Pease address W2.\n\nQ3. Please address W3.\n\nQ4. How did you apply the GNN-distillation methods to hypergraph-structured datasets in the experiments? Please provide details.\n\nQ5. Please elaborate on the distinctive challenges in distilling hypergraph neural networks compared to distilling graph neural networks.\n\nQ6. Please elaborate on the novelty of your approach and its significance when compared to (Wu et al. 2023).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "S1. The paper is well-structured and easy to follow.\n\nS2. It seems that the authors are the first to distill hypergraph neural networks into MLPs, resulting in a significant improvement in speed with only little sacrifice in accuracy.\n\nS3. The proposed method is a logical extension of (Wu et al. 2023)", "weaknesses": "W1. First and most importantly, (a) the importance of reliability-based sampling and (b) the effectiveness of the proposed methodology of measuring the reliability need to be demonstrated empirically and/or theoretically. To achieve this, a comparison should be made between the proposed method and alternative approaches, including (a) utilizing all hyperedges without reliability-based sampling and (b) relying on node reliability (Wu et al. 2023).\n\nW2. The empirical results are limited to HGNN, which is one of the most basic hypergraph neural networks. The authors need to investigate the effectiveness of the proposed distillation method across a broader range of hypergraph neural networks , including more advanced ones (e.g., UNIGCN2 and AllSet).\n\nW3. Furthermore, there is scope to explore the generalizability of the proposed method across a wider range of scenarios by incorporating additional downstream tasks (e.g., hyperedge prediction) and diverse datasets. Currently, the most hypergraphs are obtained from bibliographic data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cu4KQF0h1z", "reviewer": "Reviewer_uHuc", "summary": "The paper proposes a method called LightHGNN to enhance the efficiency of Hypergraph Neural Networks (HGNNs). The proposed LightHGNN bridges the gap between HGNNs and Multi-Layer Perceptrons (MLPs) to eliminate the dependency on hypergraph structure during inference, reducing computational complexity and improving inference speed. LightHGNN distills knowledge from teacher HGNNs to student MLPs using soft labels. Additionally, LightHGNN+ injects reliable high-order correlations into the student MLPs to achieve topology-aware distillation and resistance to over-smoothing. Experimental results show that LightHGNNs achieve competitive or better performance than HGNNs, even without hypergraph dependency.", "questions": "1. Though the  LightHGNN and  LightHGNN+ models are distilled from the HGNN model, their performance is sometimes even better than the original teacher HGNN. May the author explain why this happens?\n\n2. The author claims that LightHGNN+ is able to capture the topology information, it is expected that  LightHGNN+ should thus perform better than  LightHGNN. However, there is not general superiority of  LightHGNN+ over  LightHGNN. May the author explain why?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea of using hyperedge quantification and sampling is interesting.\n\nThe paper is clear and easy to follow.", "weaknesses": "The idea of using distillation to improve the efficiency of neural networks is not new, even in the field of Graph Neural Networks. This work is an implementation of this idea to Hypergraph Neural Networks. Though the author claims that some special designs should be considered as the technical contributions of this paper, the general novelty of this paper is borderline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QwzoeO77PT", "reviewer": "Reviewer_PX2s", "summary": "In this study, the authors introduce LightHGNN and LightHGNN+, two models aimed at enhancing the efficiency of Hypergraph Neural Networks. \n\nThese models bridge the gap with Multi-Layer Perceptrons (MLPs), eliminating hypergraph dependencies and significantly improving computational efficiency. \n\nExperimental results demonstrate that LightHGNNs achieve competitive or superior performance, showcasing their effectiveness and speed in hypergraph-based inference.", "questions": "1. What criteria were considered while selecting the baseline Hypergraph Neural Network? \n2. How does this restricted choice impact the overall diversity and representation of neural models applied to hypergraphs?\n3. Were there any specific metrics or criteria that would be considered as indicators of effective combating of over-smoothing in the context of deep LightHGNN$^{+}$ models with varying hidden layers, e.g., 2, 3, ..., 10?\n4. In the event that the experiments confirm the effectiveness of deep LightHGNN$^{+}$ models in combating over-smoothing, what implications might this have for the broader research community?\n5. How might findings on over-smoothing resistance inform future research directions or practical applications in the context of knowledge distillation from hypergraph neural networks?\n6. Given the small size of the datasets, what steps were taken to ensure that the findings and conclusions drawn from these datasets can be generalised to larger, real-world scenarios, e.g., (hyper)graphs that cannot fit into GPU memory?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-structured and the proposed methods including reliable hyperedge quantification and sampling strategy are clearly explained, with the appendix giving additional relevant details.\n2. Clearly labelled figures and visualisations, such as Figures 2, 3, S1, S2, enhance the comprehension of the presented concepts.\n3. The efficacy of LightHGNN in knowledge distillation is demonstrated across three settings: transductive, inductive, and production.\n4. The proposed methods showcase inventive problem-solving abilities by filling gaps in existing approaches, particularly in the domain of knowledge distillation for hypergraphs.", "weaknesses": "1. The paper only evaluates a single baseline model for hypergraphs, Hypergraph Neural Network [Feng et al., 2019], without exploring knowledge distillation in more recent advanced methods that compute hyperedge embeddings [e.g., Wang et al. (2023), Chien et al. (2022)].\n2. Additional experiments, particularly investigating deep LightHGNN$^{+}$ models with varying hidden layers (e.g., 2, 3, ..., 10), are necessary to substantiate the assertion that LightHGNN$^{+}$ effectively combats over-smoothing.\n3. The datasets utilised in this research, as outlined in Table S2, are relatively small in size, which diminishes the persuasiveness of knowledge distillation due to the absence of large-scale data.\n\n\nReferences\n* [Feng et al., 2019] Hypergraph Neural Networks, AAAI'19\n* [Wang et al., 2023]: Equivariant Hypergraph Diffusion Neural Operators, ICLR'23\n* [Chien et al., 2022]: You are AllSet: A Multiset Function Framework for Hypergraph Neural Networks, ICLR'22", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Sl65WBxPC2", "reviewer": "Reviewer_U1AB", "summary": "This paper extends graph knowledge distillation to hypergraphs and achieves significant improvements in both efficiency and effectiveness. In particular, experiments on synthetic hypergraphs indicate LightHGNNs can run 100× faster than HGNNs, showing their ability for latency-sensitive deployments.", "questions": "Can the author answer the question I posed in the weakness part?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- This is the first work to explore the knowledge distillation on hypergraphs.\n- The experiments in the paper are adequate, especially the results on the inference efficiency.\n- The paper is well-written and presented.", "weaknesses": "- Lack of comparison with SOTA GNN-to-MLP KD methods such as NOSMOG [1].\n- From the results in Tables 1 and 2, it seems that LightHGNN only achieves comparable rather than (significantly) better results than HGNN. Can the authors explain more about this?\n- I read subsections 4.2.1 and 4.2.2 carefully and realized that the reliability quantification and sampling (which I understand to be the core design of this paper) seem to be very similar to RKD. In my opinion, it's okay to extend designs of previous work to hypergraphs, but it's better to add proper citations and clarify which parts are similar and which parts are the main contributions of this paper.\n- Most of the datasets used in this paper are small, and the authors are encouraged to demonstrate effectiveness on more large-scale datasets.\n- Can the authors explain more about how the proposed method captures high-order correlations? I think more discussions on the differences between hypergraph KD and general graph KD can greatly enhance the readability and contribution of the paper.\n\n[1] Tian Y, Zhang C, Guo Z, et al. Learning mlps on graphs: A unified view of effectiveness, robustness, and efficiency[C]//The Eleventh International Conference on Learning Representations. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)