DistillHGNN: A Knowledge Distillation Approach for High-Speed Hypergraph Neural Networks
Authors: Saman Forouzandeh
Parham Moradi DW
Mahdi Jalili
Venue: ICLR 2025 Poster (2025)
Keywords: Knowledge Distillation
Hypergraph Neural Networks
Contrastive Learning.
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: Hypergraph Neural Networks (HGNNs) using knowledge distillation to transfer high-order information to a lightweight GCN, achieving faster inference with minimal resource use while maintaining high accuracy.

Abstract: In this paper, we propose a novel framework to significantly enhance the inference speed and memory efficiency of Hypergraph Neural Networks (HGNNs) while preserving their high accuracy. Our approach utilizes an advanced teacher-student knowledge distillation strategy. The teacher model, consisting of an HGNN and a Multi-Layer Perceptron (MLP), not only produces soft labels but also transfers structural and high-order information to a lightweight Graph Convolutional Network (GCN) known as TinyGCN. This dual transfer mechanism enables the student model to effectively capture complex dependencies while benefiting from the faster inference and lower computational cost of the lightweight GCN. The student model is trained using both labeled data and soft labels provided by the teacher, with contrastive learning further ensuring that the student retains high-order relationships. This makes the proposed method efficient and suitable for real-time applications, achieving performance comparable to traditional HGNNs but with significantly reduced resource requirements.

Published as a conference paper at ICLR 2025
DISTILLHGNN: A KNOWLEDGE DISTILLATION AP- PROACH FOR HIGH-SPEED HYPERGRAPH NEURAL NETWORKS
Saman Forouzandch, Parham Moradi & Mahdi Jalili
School of Engineering
RMIT University Melbourne, Australia (saman. forouzande,,, parpaannddhhhhrhrhaaaaae parham. parham.moradi matraddiimahi..... (ereerrittte.e .du..
ABSTRACT
This paper introduces a novel framework designed to significantly enhance the in- ference speed and memory efficiency of Hypergraph Neural Networks (HGNNs) while maintaining their high accuracy. Our approach, named DistilHHGNN, employs advanced teacher-student knowledge distillation strategy, where the teacher model comprises an HGNN and Multi-Layer Perceptron (MLP). In this setup, the HGNN generates embeddings, which the MLP subsequently processes to predict soft labels. The student model consists of a lightweight Graph Convolu- tional Network (GCN), TinyGCN, paired with an MLP and optimised for online prediction. We leverage contrastive learning to train both TinyGCN and HGNN simultaneously, facilitating the transfer of high-order and structural knowledge from the HGNN to the TinyGCN. Additionally, the teacher employs mechanism to transfer knowledge to the student model through soft labels. This dual trans- fer mechanism enables the student to effectively capture complex dependencies while benefiting from a lightweight GCN's faster inference and lower computa- tional cost. The student is trained using both labelled data and soft labels pro- vided by the teacher, with contrastive learning further ensuring that the student retains high-order relationships. This makes the proposed method efficient and suitable for real-time applications, achieving performance comparable to tradi- tional HGNNs but with significantly reduced resource requirements. Experimen- tal results on several real-works datasets demonstrate that our method significantly reduces inference time while maintaining accuracy comparable to HGNN, and it achieves higher accuracy than state-of-the-art techniques, like LightHGNN, with a similar inference time.
INTRODUCTION
Hypergraphs, with their ability to capture multi-node relationships through degree-free hyperedges, offer significant advantage over traditional graphs in modeling complex high-order interactions (Fan et al. 2021].. As result, several Hypergraph Neural Networks (HGNNs) have been developed tackle various tasks, such node classification in citation networks, recommendation in bipartite graphs, and link prediction in biological networks (Zeng et al. 2024). Despite these advances, the widespread adoption of HGNNs large-scale industrial applications still needs be improved. This is primarily due to the heavy reliance on hypergraph structures during inference, which requires substantial memory and computational resources (Yu al. 2024). As the hypergraph size and the HGNNs' depth increase, the inference time and memory requirements grow exponentially, posing significant challenges for their deployment real-world scenarios where speed and efficiency are crucial (Feng et al. 2024).. This disparity highlights the need for more lightweight and scalable solutions that can harness the representational power HGNNs while being suitable for high-speed, ceouurc-constrriied environments.
The dependency of HGNNs on the hypergraph structure arises from their message-passingg mecha- nism, which involves complex high-order interactions between vertices and hyperedges. To address

---

Published as a conference paper at ICLR 2025
this challenge, recent methods like Graph-Less Neural Networks (GLNN) Zhang et al. 2022), ) and Noise-robust Structure- aware MLPs On Graphs (NOSMOG) TTan et al. 2022 and Knowledge- inspired Reliable Distillation IKRD)(Wu et al., 2023) have aimed to eliminte graph dependencies by distilling knowledge from GNNs to MLPs. However, these methods focus on simple graph structures, using soft labels pairwise edge information
supervision, and are insufficient for hypergraphs, where hyperedges connect multiple vertices, leading to more intricate neighborhoods. Consequently, MLPs, despite their scalability and graph independence, often underperform on hy- pergraph data, showing an average decline in accuracy compared to HGNNs. Recently, LightHGNN has extended the GLNN framework to hypergraphs by distilling knowledge from HGNNs into MLPs using soft labels. While this method effectively transfers class information, it cannot convey the complex high-order relationships and structural knowledge inherent in hypergraphs. Soft labels alone are insufficient to capture these intricate dependencies. This limitation raises an important question: Can we develop a strategy that distills soft labels and transfer the hypergraph 's struc- tural and high-order knowledge the student model, ensuring a more comprehensive knowledge transfer?
Present work. In this paper, present DistilIHGNN, novel framework that significantly im- proves the inference speed and memory efficiency of HGNNs while maintaining high accuracy. Dis- tillHGNN leverages comprehensive teacher-student knowledge distillation approach. The teacher model includes an HGNN and an MLP, where the HGNN captures complex relationships within the hypergraph and generates node embeddings. These embeddings are then passed to the MLP, which produces soft labels. Together, these embeddings and soft labels form the knowledge to be trans- ferred to the student model. The student model includes lightweight GCN called TinyGCN and an MLP. TinyGCN, designed for efficient learning, consists of a single-layer GCN without non-linear activation functions to reduce computational complexity. contrastive learning strategy maximises the similarity between the embeddings generated by the HGNN and TinyGCN, effectively trans- ferring high-order structural knowledge to the student model. This strategy enables TinyGCN to replicate the behaviour of the HGNN while operating at a significantly lower computational cost. Moreover, soft labels are transferred as supplementary labelled data, further using the training pro- cess of the student model. This dual transfer mechanism allows the TinyGCN and MLP in the student model to inherit both high-order dependencies and structural knowledge from the HGNN, resulting in faster inference and improved performance. Novelties of the Proposed Method are:
1. DistilIHGNN transfers soft labels and structural knowledge from HGNN to TinyGCN, sulting in richer and more effective knowledge distillation than methods like LightHGNN.
2. DistilHHGNN utilises a contrastive learning strategy to maximise the similarity between embeddings generated by the HGNN and TinyGCN, effectively transferring high-order structural knowledge to the student model.
3. TinyGCN is streamlined to single layer without activation functions, reducing compu- tational complexity while effectively capturing the high-order relationships of the teacher HGNN.
4. The proposed method achieves inference speeds comparable to LightHGNN Feng al., 2024) but with higher accuracy, making it highly suitable for real-time and large-scale applications.
5. By integrating the representational power of HGNNs with the lightweight nature of TinyGCN, DistilIHGNN delivers fast and accurate predictions while maintaining low mem- ory and computational requirements.
2 PRELIMINARIES
Graph and Hypergraph: A graph G = (V, F) consists of set of nodes V and edges E,, where each edge Eij = (vi, v2) connects two nodes, e; and Uj. The structure of a graph can be represented by its adjacency matrix A E Ra*x,, where AÚ I if there is an edge between nodes and UU and 0 otherwise. This representation is limited to pairwise relationships between nodes. A hypergraph G = EV,E), on the other hand, generalises the concept of graph by allowing hyperedges e, E E to connect any subset of nodes, enabling the representation of complex, multi-node relationships. A hypergraph is represented by its incidence matrix H E Raxm where n is the number of nodes,
2

---

Published as a conference paper at ICLR 2025
m is the number of hyperedges, and Hij 1 if node v is connected to hyperedge ej, and 0 otherwise. This matrix effectively captures high-order interactions among nodes, making it suitable for modeling heterogeneous graphs. To convert hypergraph into a homogeneous graph, we can compute a projected adjacency matrix A E R=** as = HWH - De, where W E Rm is the hyperedge weight matrix, and De is the diagonal node degree matrix, with (t, Dド=
j=1 Haj- This transformation reduces the hypergraph to traditional graph where two nodes are connected if they share common hyperedges.
Graph Convolutional Networks (GCNs) (Kipf & Welling 2016) are designed to operate directly on graph-structured data, allowing for effective representation learning from the connections be- tween nodes. In the context of a graph, the input consists of an adjacency matrix A that encodes the relationships between nodes and a feature matrix X that contains the features of each node. The fundamental operation in a GCN is performed through a series of graph convolutional layers. The update rule for node embeddings at layer I is defined as H a (AH W where H rep- resents the node embeddings layer / (for layer O, (0) is the input feature matrix), W is the trainable weight matrix at layer I, is non-linear activation function (e.g., ReLU), and Á is the normalised adjacency matrix defined A = trix with self-loops (adding the identity matrix I to include each node's ' feature in the aggregation), and Đ is the degree matrix of Á, where D,; = Σ; A.j. The term Đ- 1/2 represents the normalised degree matrix (the square root inverse of the degree matrix). This normalisation ensures the node features are scaled properly, preventing exploding or vanishing gradients during training.
1/2 - 11/2 Here, À = A+ I the adjacency ma-
Hypergraph Neural Network (HGNN) is powerful extension of the GCN designed to capture high-order relationships among nodes by leveraging the structure of hypergraphs. In a hypergraph G, each hyperedge ej can connect multiple nodes from the set V, allowing for more complex in- teractions than simple pairwise connections. HGNNs use the hypergraph Laplacian, derived from the node degree matrix D. and the hyperedge degree matrix De. to propagate information across the hypergraph. This is achieved through a basssage-passing mechanism that updates node features using the formula:
=(+ (D:
where ( represents the learnable weights and a(-) is a non-linear activation function. This formulation allows HGNNs to effectively aggregate and propagate features, accounting for the structure of the hypergraph, thus enabling the modeling of complex dependencies between nodes. HGNNs have demonstrated their versatility and effectiveness in various applications, including recommendation systems, social networks, and biological data analysis (Feng et al. 2019)
METHODOLOGY
The overall architecture of the proposed DistillHGNN framework is shown in Figure I The pro- posed method aims enhance the inference speed and memory efficiency of HGNNs while main- taining performance comparable to HGNNs. The method is structured around a teacher-student knowledge distillation Figure T The teacher model is formed by an HGNN and a MIP. The HGNN generates node embeddings 2td E Raad by processing the node feature matrix X E g=xf and the hypergraph incidence matrix H E IR=xm, where n is the number of nodes and m is the number of hyperedges. Given the node feature matrix X, the HGNN propagates these features through the hypergraph structure using the hypergraph Laplacian:
L D. D-1/2HW >=' ",""""-11"" D/-1/2
(2)
Here, D, Raxn is the diagonal node degree matrix, D. mm is the diagonal hyperedge degree matrix, and W E Rm is the hyperedge weight matrix. The HGNN generates node em- beddings Zt E Raxd by propagating the input features X through the Laplacian as Z = H(L) where L is the number of layers in the HGNN, the update rule is defined as:
H (++ = @ (CH))(()
(3)
The embedding z' is then input into the teacher MLP to predict soft labels Yt. Let C = (c), 02,-.., [k] be set of node classes. A soft label is vector with length equal the number
3

---

Published as a conference paper at ICLR 2025
Teacher
Self supervised training
Soft Labels
CATT
MLP
Loger
(V-5) 3
Corvil 103
Carn2 W00
Cort Ay
L
Lupe + Micon
Hypergraph
Incident Matrix
Controstiv
Lean
Training Distil tnowledge
loss
Student
Soft Labelz: Disill Knowledge
Homagyeous graph
Convert
TinyGCN
MLP
9 Londent
WTX
Supervised training
Figure 1: The proposed model consists of a teacher section (HGNN) and a student section (TmnGGCN). The HGNN represents hypergraphs and generates soft labels using high-order rela- tionships, while the TinyGCN captures direct neighborhoods relations in graphs. The soft labels guide the student model via a KI. divergence term and are also used to generate labels for datasets with insufficient ground truth annotations. Both models are trained using a combination of super- vised loss and contrastive loss to align their embeddings, enabling efficient knowledge distillation from teacher to student.
of classes (i.e., (CI), where each value in the i-th cell represents the probability of the node belong- ing to the class C₁ E C. The supervised loss, called Bayesian Personalised Ranking (BPR) loss, is defined
Laps
Σ (Y, (Y)) BEVL
|V
where Y, is the true label for labelled nodes VL, and Y is the soft label predicted for the node To transfer knowledge from the teacher the student model, we propose a lightweight GCN called TinyGCN, designed to mimic the embedding generation capabilities of HGNN, TinyGCN simplifies the standard GCN architecture by removing non-linear activation functions and using only single layer. This minimalistic design allows TinyGCN to effectively capture the high-order relationships learned by the teacher model while significantly reducing inference time and computa- tional complexity. In parallel, the graph is also passed through the TinyGCN to generate embeddings 7 Raad using the node feature matrix and adjacency matrix A Raxx obtained from incidence matrix H. The node embeddings are updated using only the linear aggregation of neigh- bouring nodes' information via the adjacency matrix, represented as:
Z = A"
(5)
where Á = A + and W is the trainable weight matrix. To ensure that the TinyGCN captures similar high-order relationships the HGNN, contrastive learning approach is applied using the InfoNCE loss:
1
exp (Z /2/T)
Lewn
log VEV
(6)
Evev exp (Z 1//TT)
where is the temperature parameter that scales the similarity scores between embeddings. The total loss for the teacher model combines the supervised and contrastive losses as:
Licacher
Lbps + L
with as a hyperparameter balancing the contributions of both losses. This ensures seamless transfer of the topological structure and high-order information from the HGNN to the TinyGCN. Additionally, the teacher model generates soft labels for all nodes in the graph, representing the prob- ability distribution over classes for each node. Given the limited availability of labels, these soft labels serve valuable form of knowledge, transferred to the student model to provide richer supervisory signals. This approach helps the student model effectively learn the underlying data distribution, enabling it to perform well even with sparse set of ground truth labels.
4

---

Published as a conference paper at ICLR 2025
The student model includes the TinyGCN and separate MLP. TinyGCN produces embeddings Z for each node, which is then fed into the student MLP to predict the target labels. We use both labelled data and soft labels provided by the teacher model to train the student model. For given node v, the predicted label is:
y = MLP" MLP)
where Z is obtained by applying Tiny using Equation model is:
The loss function for the student
I Lstuddet
Σ 0 Cr-YY WEVL
+
kL((V.
V4
DEV
where Y, represents the soft label obtained from the teacher model, and KL denotes the Kullback- Leibler divergence, which measures the difference between the predicted probability distribution and the target distribution provided by the teacher. The parameter controls the influence of the distillation loss in the overall training objective, balancing the impact of learning from the teacher's knowledge against other learning signals. This approach ensures that the student model effectively assimilates the nuanced class probabilities inferred by the teacher. The algorithm of the proposed method is provided in Algorithm in the Appendix A
4 EXPERIMENTS
4.1 DATASETS
In experiments, we utilise eight well-known graph datasets. The Cora dataset, introduced by Sen et al. (Sen et al. 2008), and the Citeseer dataset, developed by Giles et al. (Giles et al. 1998), have been transformed into hypergraph datasets, namely CC-Cora and CC-Citeseer, by Yadati al. Yadati et al. 2019). In these two datasets, vertices represent academic papers, and hyperedges con- nect co-cited papers (CC). Each vertex is labelled according to the topic of its corresponding paper. We include the complete IMDB dataset and its subset IMDBAW from Fu et al. (Fu et al. (219)) which features multiple types of hyperedges: user-movie interactions, actor-movie collaborations, and director-movie relationships. While IMDBAW focuses on co-actor and co-director relation- ships, the complete IMDB encompasses a broader network of 142, 142,129 nodes across movies, users, directors, and actors. Additionally, we incorporate the complete DBLP dataset and its three subsets: DBLP-Paper, DBLP-Term, and DBLP-Conf, as introduced by Sun et al. (Sun et al. 2011). The complete DBLP dataset contains 66,543 nodes spanning papers, authors, venues, and terms, with hyperedges formed through various academic relationships. In its subsets, hyperedges are formed based collaborations (DBLP-Paper), the use of the same term (DBLP-Term), and papers pub- lished at the same conference (DBLP-Conf). The vertex labels correspond to the authors' research areas. A summary of the dataset statistics provided in Table 3 in the Appendix B
4.2 EVALUATION METHODS
We conducted each experiment five times using different random seeds, reporting both the average performance and standard deviation. Our evaluation is based on three training methods: transduc- tive, inductive, and production (Tian et al.) 2022 Feng et al. 2024). In the transductive method, the training and test data are used during evaluation, allowing the model to learn from the entire dataset. In contrast, the inductive method trains the model on one dataset and tests it a com- pletely different, unseen dataset. The production method combines both transductive and inductive predictions to simulate more realistic deployment scenario. For each dataset, 20% of the data was used for validation, and 10% was used for testing. Accuracy was the primary metric for performance comparison, and the model with the best validation performance was applied the test set for final evaluation.
4.3 BASELINE METHODS
We evaluate DistillHGNN against two different types of approaches. The first category includes graph-based methods, comprising traditional GNN models (Scarselli et al., 2008). GCN by Kipf and Welling (Kipf & Welling 2016). MLP by Taud et al. (Taud & 2as12018), and knowledge distillation methods. These include GNN-to-MI.P models like GLNN by Zhang al. (Zhang al. 2022)
5

---

Published as a conference paper at ICLR 2025
and KRD by Wu et al. (Wu et al., 2023), which operate on graph structures. The second category comprises hypergraph- based methods. We utilise the Hypergraph Neural Network (HGNN) by Feng et al. (Feng et al. 2019) and the knowledge distillation method HGNN-to-MLE (LightHGNN) by Feng et al. (Feng et al. 2024). Accordingly, we generate two versions of the data: for the first type of methods, we use the graph structure, which includes GNN, MLP, and GLNN, while for the second type, we utilise the hypergraph structure for HGNN and LighHHGNN. The baseline experiments and our methods are implemented using PyTorch and the Deep Hypergraph library | proposed method for classification based on two key aspects: accuracy and inference speed. Table [ Prresents the results based on accuracy across the three training methods.
We evaluate the
Table 1: Experimental results on eight hypergraph datasets under production setting using transduc- tive (Tran.), inductive (Ind.), and productive (Prod.) training methods.
Dataset
Setting
GNN
GCN
MLP
HGNN
GLNN
KRD
LightHGNN
DistiHHNNN
46.45.22.5
46.92,1160
43.22,1..0 95
51.45,1.77
45.88-2.33
47.33=1158
50.22,1.05
51.88+1.66 1.88+1.000
IMDB
Ind.
47. 47.33,2244
47.6741.72
44.154212
52.28
46.85 16.85-2.5
48.22.2.0
51.1842.12
52.33
Prod.
16.8% 16.88,223
17.15 17.1541..0
13.77 13.77420s
51 41.22.
46.12-.2.44
47 47.88
50. 45.45.2.0
51.92+1..0
Tran.
16.2641.66
46.7T42.12
12.70041.77
53.36_2.25
45 26
47.15=1.5x
51.1140.77
52.34,11.0
IMDB-AW
Ind.
18 18
28.6
03.35443.3
54.1242.74
46.50
49. 29.12
52.2771111
55.27.1..6
4..3
47.88, 1.90
43.15, 1,92
53.33 ja.01
45.16 9,96
48.22, 2.15
(1.84
22.14
Tran.
53. 03.0644.z
63.88,1.oz
16. 764460s
62.26....u
62.57-1..7
5-1. 11.1221..0
59 59.6542.12
61.02.1..z
CC Citeseer
Ind.
55. 55,004,3.u
65.224
19. 20
62.38
51. 61.45-...m
55.67
61.3641.27
62,18.2..
Prod.
63.60,0..4
54:05 22.1
47.20
61.39
52.08 2..5
54.33 1.00
60.1
61.88 4
Tran.
52. 62
45. 45.1642.51 52,88811555
65.17-1.0x
51.22-11.0
53, 43.45
63.6344012
65.012.1.00
CC-Cora
Ind.
24.3
24.67,
2210
66.88 2.11
53.50 1.81
35.12:1..5 1.88
65.76,1227
06.78.
Prod.
54. 15,3.4
64.55,1.22
28.
65.52,2211
53.19 2.78
54.88
64.1
65.6842.14
Dataset
Setting
GNN
GCN
MLP
HGNN
GLNN
KRD
LightHGNN
DistilHHGNN
Tran.
75.33
75.92.1
06.88+277
83.26.1.55
72.45+3.5
76.45+1.85
81.45+285
82.88-1.06
DBLP
Ind.
76.15.9.44
76.67
67.2 15
84.12.1.77
73.1242.50
77.22,2212
$2.334255
14.45
Prod.
75. 42.88
76. 16.
06.46,295
83.55..1.8
72.88,2.00
76.88.22.5
81.88,2.44
83.77.1.75
Prod.
65.1542.2
65.88
71.80.,..0 60.4742.4
62,422.4 oz
66.3142215
69. 69.124.u2
71.22+2250
DRI.P-Pape
Tran.
66. 41.23
66. 06.7772.32
62 2.
72.52.0.ax
63.8743.76
67. 22
71.22+205
72.02+1.76
Ind.
65.57 1.90
66.22
61.50 3.07
72.08 L.64
63. $3222
06.88 1.02
70.69 2.17
71.16:;1.64
Prod.
66.33.22.5
66.92....M
61.200,5.s
72.10..0.90
64.15,224
67.45.,2.15
70.164329
71.67
DeLP-Temm
Tran.
67.52249.29 22
2..8
62. 124...0
73.46.,1.8
65.2442.72
08. 08.33,,2.44
71.6442.s
73.7941.52
Ind.
66.13
68.77 oG
61.68
+..1
64.87
67.22+2.a5
71.51+211
72.45..177
Prod.
72.30 12.05
72.88
63.53 2:77
90.03
69.18
73.45 (2.15
88.10
89.27 2.17
DBLP-Coaf
Tran.
74.18 12.84
74.67 2.12
65.12; 3.44
92.22 2.78
75.22
(2.35
90.74,
92.11
74.26.4..0
74.88_2.0
64.22,4.07
91.00,,..4
71.02,2.0
75.33 .9
90.05.41.04
91.384a.25
The DistilIHGNN model demonstrates substantial advantages in classification tasks by utilising knowledge distillation to enhance its performance over traditional graph-based and hypergraph- based methods. As shown in Table [ DistilIHGNN consistently achieves either the highest or sccond-highest accuracy across various datasets and settings, including training (Tran), inductive (Ind), and production (Prod) scenarios. Although DistilIHGNN's accuracy slightly lower than HGNN's top performance, it consistently outperforms LightHGNN and shows a small but notice- able improvement in accuracy. Additionally, DistillHGNN significantly surpasses traditional models such as GNN, GCN, MLP, GLNN and KRD.
4.4 BALANCING ACCURACY AND INFERENCE TIME
In the experiments, we evaluate DistilIHGNN against other models to analyse the trade-off between predictive performance and computational efficiency. For all subsequent experiments, we employ the production method for training. Additionally, we use three versions of the DBLP dataset (based on Papers, Terms, and Conferences), running the model on all three variants and taking the aver- age to ensure consistency. This averaged result is referred to as the DBLP dataset in our analysis.

---

Published as a conference paper at ICLR 2025
In this evaluation phase, we focus two datasets IMDB-AW and DBLP and compare Distill- HGNN with other models by examining the trade-off between model accuracy and inference time. This approach allows to evaluate both the effectiveness and computational efficiency of each model, offering a comprehensive comparison across these dimensions. In real-world applications, an ideal model should maintain high accuracy while minimising inference time, particularly when computational resources and processing speed are critical. Models with high accuracy but slow inference times, such as GNNs or HGNNs, may not be practical for time-sensitive applications, while faster models like MLPs, which sacrifice accuracy, may fail to meet performance expecta- tions. The strength of knowledge distillation models lies achieving balanced trade-off, offering both competitive accuracy and reasonable computational efficiency. Higher accuracy often comes at the expense of longer inference time, but DistillHGNN is designed to balance these factors. The results, illustrated in Figure 2 demonstrate how each model's 's accuracy compares with its inference time across the datasets.
LightIGNN
Accurecy (N)
Acuundy da
ONN
GLKN
GLMN
Interence Time (ms)
Inference Time (mc]
(a) IMDB-AW
(b) DBLP
Figure 2: Comparison of model accuracy and inference time across different methods on the IMDB- AW and DBLP datasets. The figure illustrates the trade-off between predictive performance and computational efficiency, highlighting the balance achieved by DistilIHGNN.
Figure 2 demonstrates that DistilIHGNN significantly reduces inference time compared to HGNN, achieving a reduction of approximately 69%. This improvement underscores DistilIHGNN's s ef- ficiency, as it maintains high accuracy while providing much faster inference. Although MLP offers the quickest inference time at just 4 ms, its accuracy of 0.42 makes it less suitable for high-perfornance applications. DistilHHGNN's inference time is not only significantly faster than HGNN's but also comparable to LightHGNN's, s, while delivering superior accuracy. This indicates that the knowledge distillation process enhances overall performance without adding substantial computational overhead. Additionally, DistillHGNN outperforms GNN and GLNNN in accuracy, making it an effective model for large-scale tasks that require balance of performance and speed. Overall, DistillHGNN effectively balances high accuracy with computational efficiency, making it an ideal choice for scenarios where both model performance and rapid execution are crucial, such as in real-time systems or large-scale applications.
4.5 ACCURACY OF THE METHODS
A key challenge in knowledge distillation is to ensure that the student captures not only direct re- lationships between nodes but also crucial high-order relations. In this process, focus on two relationship types within the student model: direct (neighbouring) relations and high-order relations. The teacher model, leveraging hypergraph structure, excels at capturing high-order relationships but often suffers from high inference times due to the complexity of hyperedges. Previous research Tian et al. 2022 Wu al. 2023 Feng et al. 2024), has utilised soft labels to facilitate knowledge transfer. However, while soft labels approximate the teacher's ' behaviour, they alone cannot fully convey the high-order structural knowledge captured by the hypergraph. To overcome this limita- tion, we propose a combined approach that utilises both soft labels and structural knowledge from HGNN as the teacher model. To implement this combined distillation, we employ contrastive learn- ing within the TinyGCN framework. The student model receives two types of inputs: one based on the simpler TinyGCN architecture and the other derived from high-order embeddings generated by the HGNN teacher model. To evaluate the effectiveness of DistillHGNN, Order Preservation Score (Hnres). which quantifies how well the student model retains the complex, high-order relationships transferred from the teacher model. DistillHGNN addresses this challenge
propose the High-
7

---

Published as a conference paper at ICLR 2025
by incorporating both soft labels and high-order relation embeddings through contrastive learning.
H Hoss is computed as follows:
Hpres
M
E BEV
NN(V
Σ MEN(1)
Sim X
leacher Muudent
X4
(10)
Here, V the total number of nodes, N(v) is the set of k-nearest neighbours of node v, and X teacher and xaudent are the embeddings generated by the teacher and student models, respectively. The similarity measure Sim can be based on cosine similarity or Euclidean distance, depending on the experimental setup. This score evaluates the extent which the student model replicates the high-order structural information encoded in the teacher model, focusing on the similarity between the embeddings of neighbouring nodes in both models. The results indicate in Table 2 as follows: The results demonstrate a clear progression in the High-Order Relation Preservation Score (Hess)
Table 2: Comparison of knowledge distillation methods based on the preservation score.
Method
CC-Cora CC-Citeseer
IMDB-AW
DBLP
Mean
GLNN (Tian et al. 2022)
0.54
0.58
0.51
0.67
0.575
KRD (Wu et al. 2023)
0.56
0.62
0.54
0.71
0.6075
LightHGNN (Feg et al. 2024)
0.72
0.78
0.74
0.83
0.7675
DistilIHGNN
0.78
0.84
0.81
0.88
0.8275
across the evaluated methods, with DistillHGNN achieving the highest mean score of 0.8275. This indicates that methods integrating greater structural knowledge are significantly more effective at preserving high-order relationships.
4.6 INFERENCE TIME AND MEMORY EFFICIENCY ANALYSIS
Distilling knowledge to MIPs offers faster execution times and lower memory usage but typically result in reduced accuracy. In contrast, HGNNs deliver superior performance, though at the cost of slower inference times and higher memory consumption. DistilIHGNN successfully bridges this gap by achieving a balanced trade-off between memory usage, storage, and accuracy. As illustrated in Figure 2 DistilIHGNN maintains competitive inference times and reduces memory consumption through its lightweight student model design without sacrificing accuracy. The model also optimises memory storage, ensuring that it utilises less disk space compared to traditional HGNNs by pruning less critical connections while retaining important structural information. By utilising heteroge- neous graphs, such those derived from the IMDB and DBLP datasets (Fan et al., (021), the model benefits from the rich, multi-relational structures present in the data. These datasets include complex relationships where nodes-such as movies or academic papers- interact with various entities (c.g., users, directors, authors), forming hyperedges that capture higher-order connections. Detailed statis- ties of the datasets and the results from this evaluation are presented in Tables 4 and respectively, in Appendix C
4.7 ABLATION STUDY
In this section, we present findings that evaluate the effectiveness of different knowledge transfer methods employed in the DistillHGNN framework. The proposed model assessed using three distinct approaches: (1) soft labels alone, (2) structural knowledge alone, and (3) combination of both (DistillHGNN). The results are illustrated in Figure 3 which includes calculations of both accuracy and inference time for each method. The accuracy scores reveal significant improvement when using the combined approach of DistilIHGNN, which leverages both soft labels and structural knowledge, compared to using either method individually. Across all datasets, the combined method consistently outperforms the individual approaches, with accuracy improvements ranging from 3% to 10%. These findings underscore the advantages of utilising both soft labels and structural knowl- edge within the DistillHGNN framework, demonstrating a favourable trade-off between speed and accuracy.
In DistillHGNN, knowledge transfer occurs through both structural knowledge and soft labels from the teacher model to the student model. When CL is incorporated, the model achieves refined
8

---

Published as a conference paper at ICLR 2025
Accuracy Comparison
Inference Time Comparison (ms)
Structrrll Knowledge
Structural Khowledge
DistilHHGN
DistilHSNN
Inference Time (ms)
Accuracy (%)
CC Cora CC Citesoor IMDE AW
DBLP
Cora CC-Cheseer IMD8 AW
DBLP
Figure 3: The proposed model is evaluated using three different knowledge transfer methods: (1) soft labels alone, (2) structural knowledge alone, and (3) a combination of both (DistilHHGNN), The evaluation includes calculating both accuracy and inference time for each method.
alignment of node embeddings, enabling the student to effectively capture intricate high-order rela- tionships. The results comparing models with and without CL. are presented Figure as follows: The accuracy comparison between the models with and without contrastive learning across datasets
Accuracy Comparison: Without CL vs. With CL
100
Without
With CL
80
Accuracy (%)
70
11
60
50-
30
OC-Cora
CC-Citseeer
IMD8-AW
DBLP-Paper
DBLP-Term
DBLP-Con
Figure 4: The proposed model is evaluated based on the absence of contrastive learning (lack of CL) and with contrastive learning (DistilIHGNN), focusing on accuracy across four datasets.
clearly shows performance improvements when using CL. These results indicate that incorporating both structural knowledge and contrastive learning in DistillHGNN enhances generalisation and per- formance, particularly in more complex datasets like DBLP. Despite slightly slower inference times, the accuracy gains justify the use of DistillHGNN for achieving more accurate predictions.
examine the impact of increasing the num- To further evaluate the performance of DistillHGNN, ber of layers in the proposed model on both accuracy and inference time. This analysis helps de- termine the influence of model depth the quality of predictions and computational efficiency components of the proposed model. The results of this evaluation are presented in Appendix D The proposed method includes several hyperparameters that play key roles in controlling trade-offs between different components, the learning process, and overall performance. Table 6 shows the results of the evaluation of the DistillHGNN framework based on different configurations of hyper- parameters and their corresponding accuracy metrics infAppeniix El In this experiment, we evaluate the performance of DistillHGNN across six hypergraph datasets under different training sizes. The goal is to observe how varying the proportion of training data affects the model's accuracy. We com- pare the DistilIHGNN model with several baselines: GNNs, MLP, GLNN, HGNN, and LightHGNN. The results are presented in Table 7 in Appendix F To thoroughly assess the performance of Dis- tillHGNN, we conducted an ablation study focusing on the impact of different training durations, specifically varying the number of training epochs. This evaluation aims to determine how the duration training influences the model's accuracy and generalisation capability across different datasets. Each model was trained using the same dataset and hyperparameter settings ensure fair

---

Published as a conference paper at ICLR 2025
comparison. The results of this study are shown in Figure 5 which presents the accuracy achieved by DistilIHGNN for each training epoch setting.
Accuracy Epochs
Acourasy Epochs for DBLP Datasers
Accunity
DSLF
DSLP Carf
Epochs
Epocris
(a) CC-Citescer, CC-Cora, and IMDB-AW Datasets
(b) DBLP Datasets
Figure 5: Evaluation of the proposed method's performance across different numbers of epochs, ranging from 100 to 1000. This analysis examines the impact of varying training epochs on model performance, offering insights into the optimal training duration for achieving the best accuracy.
The experimental results indicate that as the number of epochs increases, model accuracy generally improves, though at varying rates depending on the dataset. This suggests that the datasets contain rich features that the model can effectively leverage, benefiting significantly from additional training epochs. Next, we conducted a comprehensive visual analysis of DistillHGNN's performance using multiple visualisation techniques to evaluate the effectiveness of knowledge transfer between the teacher and student models. The results of this evaluation are presented in Appendix G
5 CONCLUSION
In this paper, we address two key challenges in knowledge distillation for graph-based models: (1) effectively transferring high-order relationships between nodes, and (2) overcoming the limitations of using soft labels as the sole medium for transferring knowledge from the teacher model to the student model. To address the first challenge, we leveraged a hypergraph structure within the teacher model, allowing for the capture and transfer of high-order relationships that go beyond direct node connections. For the second challenge, we employed the contrastive learning (CL) framework in combination with soft labels to enhance the knowledge transfer process. The contrastive learning mechanism ensures that the embeddings generated by the student model align with those of the teacher model, thereby preserving the high-order relational knowledge encoded in the HGNN. The experimental results demonstrate that the proposed approach achieves superior accuracy compared to traditional knowledge distillation methods. Specifically, the use of hlyergraph--aasee embeddings and the integration of soft label distillation with structural knowledge led to better performance in terms of both classification accuracy and the preservation of high-order node relations. Moreover, our method strikes a desirable balance between inference speed and memory efficiency. TinyGCN significantly reduces computational complexity, its performance remains comparable to that of the more complex HGNN model. Overall, this approach opens up new possibilities for lightweight models that retain the expressiveness of more complex networks while ensuring faster inference, making it suitable for real-world applications where computational resources are limited.
ACKNOWLEDGEMENTS
This research is partially supported by Australian Research Council through projects DP240100963, LP230100439, and IM240100042.
REFERENCES
Alessia Antelmi, Gennaro Cordasco, Mirko Polato, Vittorio Scarano, Carmine Spagnuolo, and Dingqi Yang. A survey on hypergraph representation learning. ACM Computing Surveys, 56 (1):1-38, 2023.
Song Bai, Feihu Zhang, and Philip HS Torr. Hypergraph convolution and hypergraph attention. Pattern Recognition, 110: 107637, 2021.
10

---

Published as a conference paper at ICLR 2025
Yihe Dong, Will Sawin, and Yoshua Bengio. Hnhn: Hypergraph networks with hyperedge neurons. arXiv preprint arXiv:2006. 12278, 2020.
Haoyi Fan, Fengbin Zhang, Yuxuan Wei, Zuoyong Li, Changqing Zou, Yue Gao, and Qionghai
Dai. Heterogeneous hypergraph variational autoencoder for link prediction. IEEE transactions puttern analysis and machine intelligence, 44:4125-44 2021.
Yifan Feng, Haoxuan You, Zizhao Zhang, Rongrong Ji, and Yue Gao. Hypergraph neural networks. In Proceedings the AAAI conference on artificial intelligence, volume 33, pp. 3558-3565, 2019.
Yifan Feng, Yihe Luo, Shihui Ying, and Yue Gao. Lighthgnn: Distilling hypergraph neural net- works into mips for 100x faster inference. In The Twelfth International Conference on Learning Representations, 2024.
Sichao Fu, Weifeng Liu, Yicong Zhou, and Liqiang Nie. Hplapgen: Hypergraph p-laplacian graph convolutional networks. Newrocomputing, 362: 162-14,,,, 2019.
Yue Gao, Yifan Feng, Shuyi Ji, and Rongrong Ji. Hgnn+: General hypergraph neural networks
IEEE Transactions Pattern Analysis and Machine Intelligence, 45(3):3181-3199 2022.
Lee Giles, Kurt D Bollacker, and Steve Lawrence. Citeseer: An automatic citation indexing system. In Proceedings of the third ACM conference on Digital libraries, pp. 89-98, 1998.
Malik Khizar Hayat, Shan Xue, Jia Wu, and Jian Yang. Heterogeneous hypergraph embedding for node classification in dynamic networks. IEEE Transactions on Artificial Intelligence, 2024.
Jianwen Jiang, Yuxuan Wei, Yifan Feng, Jingxuan Cao, and Yue Gao. Dynamic hypergraph neural networks. In IJCAI, pp. 2635-2641, 2 2019.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. arXiv preprint arXiv: 11X9:1609000207, 2016.
Jing Liu, Tongya Zheng, and Qinfen Hao. Hire: Distilling high-order relational knowledge from
heterogeneous graph neural networks. Neurocomputing, 507:67-83, 2022.
Nicolò Ruggeri, Alessandro Lonardi, and Caterina De Bacco. Message-passing on hypergraphs: detectability, phase transitions and higher-order information. Journal of Statistical Mechanics: Theory and Experiment, 2024(4).043403, 2024.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini.
The graph neural network model. IEEE transactions on neural networks, 20(1):61-80, 2008.
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad Collective classification in network data. Al magazine, 29(3):93-93, 2008
Yizhou Sun, Jiawei Han, Xifeng Yan, Philip S Yu, and Tianyi Wu. Pathsim: Meta path-based top-k similarity search in heterogeneous information networks. Proceedings of the VLDB Endowment, 1(11)9992-1003, 2011.
Hind Taud and Jean- -ranncraccoos Mas. Multilayer perceptron (mlp). Geomatic approaches for model- ing land change scenarios, pp. 451-455, 2018.
Yijun Tian, Chuxu Zhang, Zhichun Guo, Xiangliang Zhang, and Nitesh Chawla. Learning mips on graphs: A unified view of effectiveness, robustness, and efficiency. In The Eleventh International Conference on Learning Representations, 2022.
Hanrui Wu, Yuguang Yan, and Michael Kwok-Po Ng. Hypergraph collaborative network on vertices and hyperedges. IEEE Transactions on Pallern Analysis and Machine Intelligence, 45(3):3245- 3258, 2022.
Lirong Wu, Haitao Lin, Yufei Huang, and Stan Li. Quantifying the knowledge in gnns for reli- able distillation into mlps. In International Conference on Machine Learning, pp. 37571-37581 PMLR, 2023.
11

---

Published as a conference paper at ICLR 2025
Naganand Yadati, Madhav Nimishakavi, Prateek Yadav, Vikram Nitin, Anand Louis, and Partha Talukdar. Hypergen: A new method for training graph convolutional networks on hypergraphs. Advances in neural information processing systems, 32, 2019.
Jielong Yan, Yifan Feng, Shihui Ying, and Yue Gao. Hypergraph dynamic system. In The Twelfth International Conference on Learning Representations, 2024.
Cheng Yang, Jiawei Liu, and Chuan Shi. Extract the knowledge of graph neural networks and go beyond it: An effective knowledge distillation framework. In Proceedings of the web conference 2021, pp. 1227-1237, 2021.
Nan Yin, Fuli Feng, Zhigang Luo, Xiang Zhang, Wenjie Wang, Xiao Luo, Chong Chen, and Xian- Sheng Hua. Dynamic hypergraph convolutional network. In 2022 IEEE 38th International Con- ference on Data Engineering (ICDE), pp. 1621-1634. IEEE, 2022.
Beibei Yu, Cheng Xie, Hongming Cai, Haoran Duan, and Peng Tang. Meta-path and hypergraph fused distillation framework for heterogeneous information networks embedding. Information Sciences, 667:120453, 2024.
Li Zeng, Jing-Ru Yang, Gang Huang, Xiang Ying, and Tiao-Ran LUO. Survey on hypergraph appli- cation methods: Research problem, progress, and challenges. Journal of Computer Applications, pp. 0, 2024.
Shichang Zhang, Yozen Liu, Yizhou Sun, and Neil Shah. Graph-less neural networks: Teaching old mlps new tricks via distillation. 2022.
A ALGORITHM
The proposed DistillHGNN framework is a knowledge distillation approach that leverages the power of HGNN the teacher model and lightweight TinyGCN the student model. The goal of this framework is to transfer knowledge from the HGNN, which captures high-order relations among nodes through hyperedges, to the TinyGCN, which is designed for direct and low-complexity node aggregation. By distilling the rich information from the teacher model into the simpler student model, the system ensures efficient learning while maintaining competitive performance such as node classification. The distillation process includes the generation of soft labels from the teacher model, which are used as targets for the student model in conjunction with the true labels. Additionally, a contrastive loss function ensures that the embeddings produced by the student model align with the high-order relational embeddings learned by the teacher, reinforcing the transfer of meaningful information. The combination of BPR loss, contrastive loss, and Kullback-Leibler (KL) divergence enables a robust learning process for the student model. The overall procedure of the proposed DistilIHGNN method is outlined in Algorithm 1
tasks
B DATASETS
In this work, we evaluate the performance of the proposed DistillHGNN framework on several widely used benchmark datasets. These datasets come from various domains, including citation networks, movie databases, and bibliographic datasets, providing diverse range of graph structures and node features. The IMDB dataset represents a comprehensive heterogeneous network from the Internet Movie Database, containing 142, 149 nodes across four different types: movies (40.635 nodes), users (2,113 nodes), directors (4.060 nodes), and actors (95,321 nodes). The heterogeneous nature is reflected in its three types of relationships: user-movie interactions (1,216,358 edges), firector-movie connections (15,732 edges), and actor-movic collaborations (364,058 edges). With a high average degree of 22.46, this dataset exhibits very dense connectivity patterns, making particularly challenging for graph learning tasks.
The DBLP dataset is a heterogeneous bibliographic network comprising 66,543 nodes of four types: papers (43,128 nodes), authors (14,475 nodes), venues (20 nodes), and terms (8,920 nodes). The heterogeneity is manifested in its diverse edge types: author-paper collaborations (58,592 edges), venue-paper publications (20,770 edges), and term-paper associations (195,462 edges). With an
12

---

Published as a conference paper at ICLR 2025
Algorithm DistillHGNN: HGNN with Knowledge Distillation
Input:
Hypergraph G = {V, 8) features X, incidence matrix H. labelled data Dr = (Vi, Yı}, number of epochs epocks, parameters 7;A
Output: Student model parameters
1: Initialise: Model parameters for HGNN (O!), MLP (0"), TinyGCN (W*), MLP (0"), and H!)
Step 1: Compute Laplacian and Adjusted Adjacency
2: + Dr DT SeEWD="'p 'H D 1/2 1/1
3: Á * +
Step 2: Pre-train the Teacher Model
4: for epoch to epochs do
5:
Compute HGNN embeddings: Z' t- -H(,,,, where
11+1) -
6: Generate teacher predictions: Y + (L)
7: Compute TinyGCN embeddings: Z + A'WW"
8:
Compute teacher loss:
exp( Di 72/7T E exp(Zii. (///T)
Licacher [V
E(Y, (Y,-
log
[V
VEV2
oEV
9:
Update teacher parameters O
end for
Step 3: Train the Student Model
for epoch
E do
12:
Generate soft labels (Y")) for all nodes using frosen teacher.
Compute student outputs:
1 + A'wws Y:+ (Z*)
14:
Compute student loss:
Σ (2: -Y:-Yo)' + 1 Σ KL(Ý:!1Y))
Listudent
VT
VEVi
AVV
15:
Update student parameters {W",07 0*}
16: end for
17:
Return: Student model parameters
average degree 8.26, it presents a dense, interconnected structure while maintaining clear hier- archical relationships among different node types. CC-Citeseer and CC-Cora are standard citation network datasets where nodes represent research papers, and edges represent citation links between papers. These datasets are characterised as homogeneous due their uniform node and edge types. Each paper (node) is represented by a bag-of-words feature vector, and the goal is to classify papers into different research topics. Their relatively low average degrees (3.2 and 3.8 respectively) indi- cate sparse connectivity patterns. IMDB-AW is a subset of the complete IMDB dataset that focuses on actor collaborations and relationships within the movie industry. Nodes represent actors, and edges connect actors who have appeared in the same movie. Despite being smaller than the com- plete IMDB dataset, it maintains its heterogeneous characteristies with an average degree of 8.4, indicating dense connectivity patterns.
DBLP- -apper, DBLP-term, and DBLP-Conf are subsets of the complete DBLP bibliographic dataset, each highlighting different aspects of the academic network. These subsets maintain the heteroge- neous nature of the complete dataset but with varying connectivity patterns:
DBLP exhibits moderate connectivity (degree 5.2) focusing on paper-centric rela- tionships.
13

---

Published as a conference paper at ICLR 2025
2. DBLP-term shows higher connectivity (degree 7.1) emphasising term-paper associations.
3. DBLP-Conf demonstrates sparse but hicrarchical structure (degree 284.2) concentrating on conference-paper relationships.
The summary of the dataset statistics, including the number of nodes, edges (or hyperedges), features, and classes, is provided in Table B This diverse collection of datasets, ranging from sparse bomogeneous to very dense heterogeneous networks, allows for comprehensive evaluation of hypergraph-based methods across different network structures and application domains.
Table 3: Datasets used in the experiments.
Statisties
Dataset
Characteristics
#Nodes
#Edges
#Feat
#Class
DBLP
66,543
274,824
334
Dense, heterogeneous (deg=8.26)
IMDB
142,129
148
3,066
3
Very dense, heterogeneous (deg=22.46)
CC-Citeseed
3,312
1,004
3,703
6
Sparse, homogeneous (deg=3.2)
CC-Cora
2,708
1,483
1,433
7
Mod. sparse, homogeneous (deg=3.8)
IMDB-AW
5,355
6,811
3,066
Dense, hetcrogencous (deg=8.4)
DBLP-paper
14,376
14,475
334
Moderate, heterogeneous (deg=5.2)
DBLP-term
14,376
13,789
334
4
High connect., heterogencous (dcg=7.1)
DBLP-Conf
14,376
1,612
334
4
Sparse, hierarchical (deg=284.2)
C COMPARISON OF DISTILLHHNN AND HGNN BASED ON INFERENCE TIME AND MEMORY USAGE
In this evaluation phase, assess both the inference time and memory efficiency of the Distill- HGNN model using two heterogeneous datasets: IMDB and DBLP. These datasets are particularly challenging due to their diverse node types and complex relationships, making them ideal for eval- uating both the effectiveness and computational performance of our model. The IMDB dataset comprises user, movie, director, and actor nodes, interconnected through various hyperedges, in- cluding user-movie, director-movie, and actor-movie relationships. This structure exemplifies a typ- ical heterogeneous network where multiple node types and interactions are represented. Similarly, the DBLP dataset includes paper, author, venue, and term nodes, connected through their respective relationships (author-paper paper, venue venue-paper, and term-paper). representing another common structure found in academic collaboration networks. Table summarises the key statistics of these datasets, including the number of nodes, types of nodes, and hyperedges. The complexity and scale of these datasets pose significant challenges in terms of inference time, making them ideal for evaluating DistilHGNN''s ability to balance accuracy and computational efficiency.
Table 4: Statisties of the Datasets
Dataset
Nodes
Count of Nodes
Hyperedges
Count of Edges
Movie
40.635
User
2.113
User-Movie Director-Mae Movcce
1,216.358 15,732
IMDB
Director
4,060
Actor
95,321
Actor-Movie
364,058
Total
142,129
1,596.148
Paper
43,128
Author
14,475
Author-Paper
58,592
DBLP
Venue
20
Venue-Paper
20,770
Term
8,920
Term-Paper
195,462
Total
66,543
274,824
14

---

Published as a conference paper at ICLR 2025
Given the intricacies of these datasets, is crucial to evaluate the inference time of DistillHGNN in comparison to the original HGNN model. This section aims to assess the inference performance of DistilIHGNN on both IMDB and DBLP graphs, contrasting it with the performance of the standard HGNN model. The results, which include a comparative analysis of the inference times, are sum- marised in Table 5] These findings illustrate the efficiency of DistilIHGNN in reducing inference time while maintaining competitive performance.
Table 5: Comparison of Inference Time (ms) between DistilIHGNN and HGNN for IMDB and DBLP Datasets at Different Node Levels.
Dataset
Hyperedge Count
HGNN
DistilIHGNN
Improvement (%)
10,000
8.12
0.67
12.12
20,000
37.35
1.13
33.05
IMDB
30,000
84.70
1.68
50.42
40,635
175.56
2.23
78.70
10,000
9.47
0.72
13.15
20,000
42.15
0.98
43.01
DBLP
30,000
75.33
1.51
49.88
43,128
168.84
2.06
81.97
These results highlight the scalability and computational efficccctionaleffcienncyy of DistilHHGNN, managing acaeesccal graphs. For example, with 40,635 nodes in the IMDB dataset, DistillHGNN demonstrates impressive speed improvement, being almost 79 times faster than the HGNN model. Similarly, for the DBLP dataset with 43, nodes, DistillHGNN is nearly 82 times faster. The sub- stantial speedup, particularly at larger node levels, underscores the practical advantages of Distill- HGNN in applications that require rapid inference on complex, multi-rclational data. Furthermore, the consistently higher times-faster ratios across both datasets accentuate DistilIHGNN's effective- ness in minimising computational overhead compared to the HGNN model. Overall, the analysis illustrates that DistilIHGNN not only maintains competitive performance but also significantly en- hances inference speed, making it compelling choice for real-world applications that demand ef- ficiency without sacrificing accuracy. This capability particularly valuable in scenarios where the processing of vast amounts of data is required, such recommendation systems and social network analyses.
especially when
C.I MEMORY EFFICIENCY ANALYSIS
This subsection provides a comprehensive analysis of the memory efficiency of DistilIHGNN in comparison to traditional GNNs and MI.P-based models (e.g., GINN,, KRD, and LightHGNN). While we do not claim that DistilIHGNN achieves better inference time than other distillation mod- els such as LightHGNN or GLNN, we demonstrate that it offers a favourable trade-off between memory efficiency and model performance. Here, we compare the memory required for all models during inference. GLNN, KRD, LightHGNN, and DistillHGNN are knowledge distillation methods. GLNN and KRD use GNN their teacher model and an MLP with L layers the student model. Both LightHGNN and DistillHGNN use an HGNN as their teacher. However, while LightHGNN distills knowledge into MLP-bbased student model with L layers, DistillHGNNN transfers knowl- edge to a single-layer GNN (TinyGCN) and an MI.P with L layers. Since our focus is on comparing the memory efficiency of models during inference, we do not account for the memory required for the teacher model, which is trained offline.
Let's consider uniform feature dimension d across all models and node embeddings. A typical message-passing GNN updates node representations using the equation ((++1 (AHH(WW)) where H E N is the node feature matrix at layer 1, A N is the adjacency matrix (which can be stored in a sparse format), and (()) E Rd'dd is the trainable weight matrix. The memory complexity of a GNN primarily arises from storing node embeddings, adjacency information, model parameters, and intermediate activations. The node embeddings require O memory at each layer, while the adjacency matrix, if stored in a dense format, requires O However, for large and sparse graphs, storing only the nonzero elements results in O(FE) memory usage. The weight
15

---

Published as a conference paper at ICLR 2025
matrices require O(d)) memory across L layers, which remains relatively small compared to the node and adjacency storage. Additionally, storing activations from each layer during inference adds an extra ((LNd) memory requirement. Thus, the total memory complexity of GNN with L layers is O(L(IE) + Nd + dz)).
Since all knowledge distillation models use MLPs as their student models, compute the memory required for the MLP during inference. An MLP is a fully connected neural network where each
layer performs linear transformation followed by a non-linear activation function. The forward o (HOW)() + b()), where H RNxd is the activation matrix at layer L, ((()) _ddd is the trainable weight matrix, and 6(0) E R the bias vector. The memory complexity of an MLP consists of storing model parameters (weights and biases) and activations. The weight matrices across L layers require ( memory, while storing activations during inference requires ((L)). Note that during inference, memory required for backpropagation not used. Thus, the total memory complexity of an MLP with L layers to process the embedding of node is O(Ld+ Ldd). Therefore, the memory required during inference for GLNN, KRD, and LightHGNN, as they use only an MLP in their student model, O(Ld + Ldf). Moreover, since our model incorporates single-layer GCN (TinyGCN), it requires additional memory of O| Nd + d.).
propagation of an MLP with L layers is given by H
D EVALUATION OF DISTILLHGNN BASED ON THE NUMBER OF LAYERS
To further assess the performance of DistilHHGNN, we evaluate the impact of increasing the number of layers in various sections of the proposed model. This includes the HGNN and Teacher MLP in the Teacher section, as well as TinyGCN and Distill MLP in the Student section, with a focus on accuracy and inference time. The selected layers for the proposed method include three layers for both HGNN and Teacher MLP, single layer for TinyGCN, and two layers for Distill MLP. In this section, we experiment with each model by testing different configurations with multiple layers (specifically. 2, 3, 4, and 5 layers) and comparing the results based on accuracy and inference time. Our experiments are conducted on the IMDB-AW dataset, and the results are illustrated in Figure as follows:
Thy
ThnyGCN-),
54.5
Layers
Layers
54.6
Lugerss
$4.0
Lapars
52.s
$1.5
$2.0
$3.0
52.5
Laysr
52.5
$2.0
$2.0
51.
51.5
51.0
10
20
Isfernce Time (ms)
Inferetee Time
(a) HGNN Model
(b) Teacher MLP HGNN Model
ThyyOCr HONKHO, Teacher MLP-3, Distil HUP-2
$5.0
Doesi MLP:: ThayoCh-1, 1IGAN-3,, Teacher MLP+3
@3 Layers
Layers
$4..
Layers
54.0
Lapers
$1.0
Layers
Layer
53.5
51.5
Layers
$3.0
Lapan
$3.0
22.5
52.5
38.8
32.0
51.
51.5
53.0
31.
Infarmmie Tima (mm))
Inforence Time (mm)
(c) TinyGCN Model
(d) Distill MLP Model
Figure 6: Evaluation of the Proposed Method Based on the Depth of Layers Configuration for HGNN, Teacher MLP, TinyGCN, and Distilled MLP using the IMDB-AW Dataset
In evaluating the performance of DistillHGNN, indicated in Figure of increasing the number of layers across different sections of the proposed model, including the HGNN and Teacher MI.P in the Teacher section, well as TinyGCN and Distill ML.P in the Student
analysed the impact
16

---

Published as a conference paper at ICLR 2025
section. The analysis focused on accuracy and inference time, based on the results obtained. The selected layers for the proposed method include three layers for both HGNN and Teacher MLP, a single layer for TinyGCN, and two layers for Distill MLP. This selection is driven by the need to balance model complexity with computational efficiency while ensuring optimal performance. The HGNN model demonstrates consistent upward trend in accuracy as the number of layers increases, starting from 52.37% with layer and reaching 54. 15% with 5 layers. However, the decision to limit the depth of the HGNN three layers based on observed diminishing returns in accuracy beyond this point, as indicated by the marginal improvement from 54. 12% at 4 layers to 15.15% at 5 layers. This suggests that three layers are sufficient to capture the essential relationships in the data while minimising inference time, which escalates from 4 ms to 39 ms with increased depth.
Similarly, the Teacher MI.P model shows a relatively stable accuracy range, peaking at 54. with 4 layers before slightly dropping to 54.02% at 5 layers. Selecting three layers for the Teacher MLP balances complexity and performance, as allows the model to learn effectively without succumb- ing overfitting or unnecessary computational overhead, particularly since inference time increases significantly from ms at 2 layers to 40 ms at 5 layers. For TinyGCN, which reached its highest layers, single layer was
accuracy of 54.03% at layers before slightly declining to 53.90% selected to maintain efficiency in inference time while still leveraging the strengths of graph convo- lution. The increase in accuracy from 53.91 % with single layer 54.03% 3 layers is marginal, indicating that a more straightforward architecture is sufficient to capture the essential features of the dataset without incurring excessive computational costs, as inference time ranges from 10 ms to 44 ms with increasing depth. In the case of the Distill MLP model, selecting two layers allows for enhanced performance, reaching an accuracy of 54.33% at 4 layers while still being computationally feasible. The results indicate that two layers strike an effective balance between learning capacity and inference efficiency, given that the inference time increases significantly from 10 ms 51 ms as layers are added.
In summary, the proposed model's architecture is optimised through careful selection of layer depths tailored to each component's strengths and limitations. Thrre layers for both HGNN and Teacher MLP allow for capturing complex relationships while avoiding overfitting and excessive inference time. A single layer for TinyGCN maximises efficiency, while two layers for Distill MLP ensure suf- ficient capacity for accurate predictions without significant computational overhead. This thoughtful arrangement provides a robust framework for model performance, emphasising the critical balance between accuracy and operational efficiency. These findings are visually represented in Figure 6 providing further insights into how model architecture affects performance.
E SENSITIVELY ANALYSIS OF HYPERPARAMETERS
In the context of the proposed DistilIHGNN method, several hyperparameters play key roles in con- trolling the trade-offs between different components, the learning process, and overall performance. Below is brief explanation of the most important hyperparameters:
1. Temperature for contrastive learning (r): This temperature parameter scales the similar- ity scores between student and teacher embeddings. A lower value of results in sharper, more distinct similarity scores, while a higher value leads to softer comparisons.
2. Contrastive loss weight (y): This hyperparameter balances the importance of the con- trastive loss relative to the BPR loss in the teacher model. higher value of 7 increases the influence of contrastive learning.
3. Distillation loss weight (A): This hyperparameter controls the contribution of the distilla- tion loss in the student model. A higher value emphasises learning from the teacher model's soft labels, while lower value focuses more on the ground truth labels.
4. Learning Rate f:: The rate at which the model updates its parameters during training. A lower learning rate ensures more gradual convergence but may slow down the training process.
5. Embedding Dimension: This determines the size of the embedding vectors generated by both the teacher and student models. Higher dimensions can potentially capture more information, but at the cost of increased computation.
17

---

Published as a conference paper at ICLR 2025
Table 6 shows the results of evaluating the DistillHGNN framework based different configura- tions of hyperparameters and their corresponding accuracy metrics.
Table 6: Hyperparameter Configurations for DistilIHGNN on the DBLP
Config
t
X
Embed Dim
Learning Rate (lr)
Accuracy (%)
0.1
76.20
2
0.2
76.85
3
0.3
77.79
0.4
78.05
5
0.5
78.33
6
0.6
78,02
7
0.7
77.64
Config
?
Accuracy (%
0.1
77.55
2
0.2
77.83
3
0.3
78.11
4
0.4
78.33
5
0.5
77.80
6
0.6
77.61
Config
X
Accuracy (%)
0.1
77.87
2
0.2
78.33
3
0.3
78.30
4
0.4
78.21
Config
Embed Dim
Accuracy (%)
32
74.48
2
64
77.75
3
128
78.33
4
256
77.17
512
76.59
Config
1r
Accuracy (%)
0.1
76.25
2
0.5
75.15
3
0.001
78.33
4
0.005
77.23
0.0001
77.61
Best Hyperparameters Combinations
0.5
0.4
0.2
128
0.001
78.33
The evaluation of the DistillHGNN framework, based on various hyperparameter configurations, reveals interesting patterns in model performance. The temperature for contrastive learning (7) plays a crucial role in the accuracy, with an optimal value of 0.5 yielding the highest accuracy of 78.33%. Similarly, the contrastive loss weight (r) shows an increase in performance as its value rises, peaking at 0.4. Distillation loss weight (A) has more consistent influence, with the best result observed at 0.2. When analysing embedding dimensions, 128 emerges as the most effective, balanc- ing representation richness with computational cost. Finally, the learning rate (lr) of 0.001 proves optimal, enabling the model to converge effectively. The best-performing configuration, combining these hyperparameters, demonstrates the importance of fine-tuning to achieve maximal accuracy. This suggests that the interaction between the temperature, loss weights, embedding dimension, and learning rate crucial for optimising the DistillHGNN framework.
18

---

Published as a conference paper at ICLR 2025
F TRAINING SIZE SETTINGS
In this section, we evaluate the performance of DistilIHGNN under varying training size settings across six hypergraph datasets. The primary objective is to understand how increasing the amount of training data influences the model's 's accuracy compared to several baseline models: GNNs, MLP, GLNN, HGNN, and LightHGNN. By setting the training sizes to 20%, 40%, 60%, and 80%, we aim to observe the progression of each model's performance as more data becomes available. This analysis helps us assess the scalability and effectiveness of each method as the volume of training data increases. The table presents the results of our experiments, where report the accuracy for each model across different datasets and training sizes.
Table 7: Experimental results on six hypergraph datasets under various training settings.
Dataset
Train size
GNNs
MLP
GLNN
HGNN
LightHGNN
DistilIHGNN
20%
46.40
40.29
44.27
56.42
54.76
55.18
40%
49.66
43.18
48.17
58.77
58.11
58.42
CC-Citeseer
60%
52.14
45.65
50.87
61.13
60.19
60.34
80%
55.20
48.34
53.77
62.23
62.05
62.44
20%
48.32
41.74
46.92
60.17
59.05
58.34
40%
50.61
44.22
49.19
62.54
61.58
61.56
CC-Cora
60%
52.06
46.16
51.50
64.69
64.01
64.83
80%
55.50
48.58
54.79
66.04
65.28
66.81
20%
39.27
35.76
40.77
48.50
47.69
47.16
40%
43.82
39.27
42.66
49.42
49.08
49.15
IMDB-AW
60%
46.42
41.29
44.33
52.08
51.65
52.26
80%
49.65
45.25
47.44
55.03
53.86
55.35
20%
61.53
55.80
59.47
70.41
69.02
69.44
40%
65.21
57.17
63.43
74.73
73.73
74.65
DBLP
60%
68.12
61.40
67.00
78.63
77.55
78.16
80%
70.14
64.09
69.04
80.42
79.35
80.61
The experimental results demonstrate that as the training size increases from 20% to 80%, all models generally show improved performance, with DistilIHGNN consistently outperforming the baseline methods across all datasets. Notably, the DBLP dataset, DistilIHGNN achieves the highest ac- curacy at 80% training size with a value of 80.61% %. Similarly, in CC-Citeseer and CC-Cora, the performance gap between DistillHGNN and the other models widens as the training size increases. This suggests that DistillHGNN benefits more from additional training data compared to the other models, particularly on larger datasets. While HGNN and LightHGNN also perform well, Distill- HGNN's knowledge distillation, which combines structural knowledge with soft labels based high-order relations, contributes to its superior performance, making the most effective model across various training settings.
G VISUAL ANALYSIS OF KNOWLEDGE TRANSFER
To comprehensively assess the effectiveness of knowledge transfer in DistillHGNN, we employ visual analysis key evaluation tool. visualisation offers intuitive insights into how well the student model learns from the teacher model, complementing numerical performance metrics. By examining embedding spaces, structural relationships, and feature similarities, evaluate the
19

---

Published as a conference paper at ICLR 2025
extent to which the student model captures the essential patterns and relationships embedded in the teacher's s representations.
This section provides detailed visual analysis, focusing on embedding space visualisation crit- ical dimension of knowledge transfer. These visualisations not only allow us to verify the alignment between teacher and student models but also uncover deeper nuances in their respective feature representations. Through these analyses, we aim to highlight the strengths and limitations of the distillation process, offering a comprehensive understanding of the student model's performance. We then present findings from each visualisation approach applied to two datasets, IMDB-AW and DBLP, demonstrating how DistilIHGNN achieves effective and efficient knowledge transfer.
The t-SNE visualisations illustrate the model's ability to preserve class relationships and structural information. The student model maintains clear class separations, forming well-defined clusters that closely resemble the teacher's ' representation. The consistent spatial arrangement of classes between the teacher and student embeddings across the IMDB-AW and DBLP datasets, as shown in Fig indicates effective knowledge transfer while preserving the essential topological structure of the data.
IMDB-AW Dataset Analysis The t-SNE visualisation of the IMDB-AW dataset in Fig. 7 reevals distinct embedding patterns across both models. The teacher model exhibits well-defined clustering for three classes, with Class 2 (blue) showing the most extensive distribution, indicating rich feature diversity within the class. Class 0 (green) demonstrates interesting sub-cluster formations, suggest- ing underlying structural patterns in the data, while Class I (red) maintains moderate cohesion with clear boundaries. The student model successfully preserves these fundamental class relationships while introducing its structural interpretations. Notably, it maintains class separability while showing a more continuous distribution pattern, with Class 2 adopting an elongated formation and Class O demonstrating enhanced cluster cohesion. The slight increase in inter-class mixing, particu- larly visible in Class I's boundaries, suggests a balanced trade-off between feature preservation and model simplification.
DBLP Dataset Analysis In the DBLP dataset, both models demonstrate sophisticated four-class separation patterns with distinct characteristics. The teacher model establishes clear class bound- aries with unique spatial distributions: Class 0 (green) exhibits multiple sub-clusters in the upper region, indicating complex internal structure; Class (red) forms compact, well-isolated clusters; Class 2 (blue) shows concentrated distribution on the left; and Class 3 (yellow) presents an elongated central formation. The student model transforms this representation into a more globally coherent structure, organising the classes in distinctive crescent-like pattern. This arrangement maintains clear class separation while achieving smoother transitions between clusters. The student's repre- sentation demonstrates particular effecttveness in boundary definition, with each class occupying specific region: Class 0 forms a curved structure in the upper left, Class maintains concentration in the lower right, Class 2 shows compact clustering in the upper right, and Class 3 creates a central curved formation.
Both datasets demonstrate successful knowledge transfer between teacher and student models, with the student model consistently achieving more regularised and structured representations. In the IMDB-AW case, the preservation of three-class separation with modified spatial arrangements in- dicates effective feature learning while maintaining essential data relationships. The DBLP visuali- sation further reinforces this finding across four classes. where the student model's more organised spatial arrangement suggests enhanced feature generalisation without loss of discriminative power. These results validate the effectiveness of our knowledge distillation approach, demonstrating that the student model can capture and, in some aspects, enhance the structural understanding of the data despite its simplified architecture. These visual analyses provide strong empirical evidence for the success of our knowledge distillation framework. The student model's ability to maintain clear class separation while developing more structured representations suggests effective compres- sion of the teacher's 's knowledge into a more efficient form. This is particularly noteworthy given the architectural simplification, indicating that our approach successfully preserves essential feature relationships while reducing model complexity.
20

---

Published as a conference paper at ICLR 2025
Teacher Embecidings (t-SNE)
Student Embeddings (t-SNE)
Clesess Cam
dasses
75
-0
-40
-23
(a) IMDB-AW dataset
leacher Emberdding: (t SNE)
Student Embeddi ngs (t (t-SNE)
Caec
Closs
Class
C200
closs
Cass
Class 2
20-
LONE
20
-10
-20
14M Dimemion 1
H-M Dimeendoon
(b) DBLP datasets
Figure 7: I-SNE visualisations comparing teacher and student embeddings for two datasets: (a) IMDB-AW dataset shows scattered and overlapping class distributions in the teacher embeddings, which are effectively refined into cohesive and distinct clusters in the student embeddings, indicating successful knowledge transfer. (b) DBLP dataset reveals fragmented and abnormal class clusters in the teacher embeddings, likely due to the hypergraph model's high-dimensional feature complexities and high-order relationships. well-organised and continuous class regions, demonstrating the effectiveness of knowledge distillation simplifying and structuring complex representations into interpretable embeddings.
contrast, the student embeddings display
21

---

Published as a conference paper at ICLR 2025
H RELATED WORKS
Hypergraph Neural Networks (HGNN). HGNNs extend traditional graph neural networks (GNNs) by capturing complex high-order interactions among multiple nodes through hypergraphs (Attemi et al. 2023). Unlike conventional graphs, where edges connect only two nodes, hyper- graphs allow hyperedges to connect multiple nodes, making HGNNs particularly effective in do- mains where higher-order relationships are crucial (Wu et al. 2022). Early models like HGNN (Feng et al. 2019) and HpLapGCN (Fu et al., 2019) utilised the hypergraph Laplacian matrix for efficient representation learning by smoothing node features across hyperedges. HyperGCN (Yadati al., 2019) further simplified hypergraphs into conventional graphs, applying established GNN techniques to learn node representations, thus leveraging the structural richness of hypergraphs. In addition to spectral approaches, spatial-base hypergraph convolution methods have emerged to -hyeeedge attention mechanism that enhances the focus on critical node hhpeeredgpegeee interactions. Research such as (Yan al. 2024 Jiang et al., 2019; Yin et al. 2022 Hayat et al. 2024) has explored dynamic hypergraph construction, allowing for flexible representations that adapt to specific datasets. Inno- vations like two-stage
overcome earlier limitations. For instance, Bai al. (Bai et al. (02T) introduced
pesssage-psssing strategies proposed by Gao et al. (Gao et al. 2022), Dong et al. (Dong et al. 2020), and Ruggeri et al. (Ruggeri et al. (2024) have enabled more efficient in- formation flow across nodes and hyperedges. These advancements have broadened the application of HGNNs, allowing them to effectively model intricate, higher-order relationships in diverse fields such as social networks, recommendation systems, and biological networks. By capturing complex interactions, HGNNs outperform traditional GNNs, which are limited to pairwise interactions.
Distillation from GNNs and HGNNs to MLPs. Knowledge distillation involves transferring knowledge from larger, more complex model (the teacher) smaller, more efficient model (the student). Previous distillation methods, such as GLNN (Zhang et af.] (022)) and NOSMOG (Tian) al., 2022), primarily use the prediction distribution of teacher GNNs as soft labels to guide stu- dent MLPs. However, these approaches often fail to consider the original graph's 's structure, limiting their ability capture intricate relationships within the data. For example, while Yang et al. (Yang et al. (2021) extracts knowledge from a trained GNN model and transfers it to a student model for more efficient predictions, the method still lacks full integration of structural information into the distillation process. KRD (Wu et al. 2023) improves this by quantifying each vertex's ' knowledge and considering its proximity to neighbours, but it remains limited to low-order graph structures. Additionally, Liu et al. (Liu et al., (202)) introduced the High-order RElational (HIRE) knowledge distillation framework for heterogeneous graphs, which captures both first- and second-order infor- mation using soft labels. For HGNN-to-MLP distillation, Feng et al. (Feng et al. 2024) proposed the LightHGNN model, which incorporates reliable hyperedges to support high-order relations in the distillation process. However, they still relied soft labels. Similarly, Yu et al. (Yu et al.) 2024) developed a method to distill knowledge from meta-paths into hypergraphs in heterogeneous graphs, again using soft labels for transferring knowledge from the teacher to the student. This paper aims to address the limitations of prior methods by focusing on hypergraph neural networks (HGNNs). The key challenge in earlier techniques that soft labels alone do not capture the high-order depen- dencies inherent in hypergraphs.
22


Reviews:
[{"review_id": "Y01AOc9ZLc", "reviewer": "Reviewer_6QuE", "summary": "This paper proposed DistillHGNN, a novel framework designed to improve the inference speed of Hypergraph Neural Networks (HGNNs) without sacrificing accuracy. DistillHGNN utilizes a teacher-student knowledge distillation approach, where the teacher model comprises an HGNN and a Multi-Layer Perceptron (MLP), while the student model, TinyGCN, is a lightweight Graph Convolutional Network (GCN) paired with an MLP. The framework incorporates a dual knowledge transfer mechanism, passing both soft labels and structural knowledge from the teacher to the student. Trained on labeled data and teacher-provided soft labels with contrastive learning to retain high-order relationships, DistillHGNN achieves performance comparable to traditional HGNNs with significantly lower resource requirements.", "questions": "- Figure 2 shows comparisons of models based on accuracy and inference time. Does that mean accuracy and inference time are dependent on each other? If not, these graphs create confusion. Currently, from the graphs it seems that if the inference time is longer, the accuracy could be higher, and vice versa. If this is not the case, I suggest keeping inference time separate from accuracy.\n\n- Line 669 of Algorithm 1 states, \"Update teacher model parameters via backpropagation\". Typically, knowledge distillation keeps the teacher fixed once pre-trained, so if this algorithm intends otherwise, the authors should provide a justification.\n\n- In Figure 6, can the authors keep the range of accuracy axis the same for all the graphs? It would be easier to visualize the differences. \n\n- It would be interesting to see how an adaptive knowledge distillation approach performs compared to the proposed fixed distillation approach. For instance, the model could prioritize contrastive learning for highly connected hypergraphs to capture complex relationships and rely more on soft labels for simpler ones to reduce model complexity even further.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The dual knowledge distillation approach (soft labels and high-order structural knowledge) effectively transfers complex relationships, addressing limitations found in existing methods.\n\n- Comparative accuracy with minimal inference time.\n\n- The paper is well structured and well written.\n\n- The paper includes a thorough experimental setup, covering multiple datasets and evaluation metrics.", "weaknesses": "- The authors claim the proposed approach is memory efficient everywhere in the paper. In line 416, they mention, \"As illustrated in Figure 2, DistillHGNN maintains competitive inference times and memory efficiency without sacrificing accuracy\"; however, Figure 2 shows accuracy vs. inference time. Moreover, no other evidence shows that the proposed approach is memory efficient.\n\n- Among the previous works in knowledge distillation, the authors only compared with LightHGNN. Since the authors compared with GNN and GCN, the authors are suggested to compare with GNN and GCN-based distillation approaches as well. \n\n\nMinor:\n\n- In line 152, the authors mention \"Framework 1\". Since there is no Framework 1 in the paper, I suppose it should be \"Figure 1\".\n\n- The caption for Figure 1 is too big. The authors should keep it concise and incorporate the details into the actual text of Section 3. Also, the authors didn't mention what happens with the soft labels after they are generated from the teacher model in the caption.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b0t4KorybW", "reviewer": "Reviewer_ErwD", "summary": "This paper introduces DistillHGNN, a novel knowledge distillation framework designed to enhance the inference speed and memory efficiency of Hypergraph Neural Networks (HGNNs) while maintaining high accuracy. DistillHGNN employs a teacher-student model where the teacher consists of an HGNN and a Multi-Layer Perceptron (MLP) that generates soft labels. The student model uses a lightweight Graph Convolutional Network (TinyGCN) paired with an MLP, optimized for faster online predictions. Additionally, contrastive learning is utilized to transfer high-order and structural knowledge from the HGNN to the TinyGCN. Experimental results on various real-world datasets demonstrate that DistillHGNN significantly reduces inference time and achieves accuracy comparable to or better than state-of-the-art methods like LightHGNN.", "questions": "1. The proposed method converts a hypergraph into a homogeneous graph, potentially leading to the loss of high-order structural information. How do you address this information loss, and can you incorporate mechanisms similar to LightHGNN’s high-order soft-target constraints to better preserve high-order knowledge? Additionally, could you provide visualizations to enhance the interpretability of your method?\n2. Since DistillHGNN depends on homogeneous graphs during inference, how does it handle inductive learning scenarios where new samples are added? Specifically, what strategies do you propose to minimize the need for graph reconstruction and complex preprocessing steps in such cases?\n3. The experimental evaluation primarily focuses on datasets from academic papers and movie databases. Do you have plans to evaluate DistillHGNN on larger-scale or synthetic datasets to demonstrate its scalability and effectiveness across a broader range of applications?\n4. Could you provide more technical details and visualizations regarding how contrastive learning and soft labels facilitate the transfer of high-order knowledge from the teacher model to the student model? This would help in better understanding the mechanisms behind knowledge preservation and transfer.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method combines knowledge distillation with contrastive learning in an innovative way, allowing for a more comprehensive transfer of both soft labels and high-order structural knowledge from the complex HGNN to the lightweight TinyGCN. This dual transfer mechanism enhances the student model's ability to capture intricate dependencies while maintaining computational efficiency.\n2. The experimental evaluation is thorough and spans multiple real-world datasets, including CC-Citeseer, CC-Cora, IMDB-AW, and various DBLP datasets. The results consistently demonstrate that DistillHGNN achieves a favorable balance between accuracy and inference speed, often outperforming existing methods such as LightHGNN in both metrics.\n3. The paper is well-structured and provides a clear and detailed explanation of the methodology, including the architecture of both the teacher and student models, the training process, and the loss functions used. This clarity facilitates understanding and potential replication of the proposed framework.", "weaknesses": "1. The method involves converting a hypergraph to a homogeneous graph, which can lead to the loss of high-order structural information inherent in hypergraphs. While contrastive learning helps mitigate this issue, the paper does not sufficiently address how the conversion process preserves the complex relationships that hypergraphs naturally capture.\n2. DistillHGNN relies on homogeneous graphs during the inference phase, which poses a significant limitation for inductive learning scenarios. When new samples are introduced, the need to reconstruct the graph and perform complex preprocessing steps can result in increased computational and time costs, limiting the model's applicability in dynamic or real-time environments.\n3. The evaluation is primarily conducted on datasets related to academic papers and movie databases. There is a lack of assessment on larger-scale or synthetic datasets, which raises concerns about the method’s scalability and generalizability to other domains or more complex real-world applications.\n4. The selection of baseline methods, while covering traditional GNNs and some knowledge distillation approaches, does not include more recent or advanced hypergraph neural network models. This omission may limit the understanding of DistillHGNN's relative performance against the latest state-of-the-art techniques.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2pWWSUrHLj", "reviewer": "Reviewer_49KC", "summary": "This paper proposes **DistillHGNN**, a knowledge distillation framework designed to accelerate the inference of hypergraph neural networks (HGNNs) while maintaining their high accuracy. DistillHGNN leverages a teacher-student structure where the teacher model, an HGNN, captures complex relationships within hypergraphs and generates soft labels. The student model, a simplified TinyGCN, learns from these soft labels and employs contrastive learning to capture high-order structural knowledge.", "questions": "See Weaknesses part", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The dual use of soft labels and structural knowledge distillation through contrastive learning enhances the capability of the student model, addressing limitations of previous distillation methods.\n2. DistillHGNN achieves significant reductions in inference time (up to 80%) compared to traditional HGNNs, making it suitable for real-time applications.", "weaknesses": "1. **(Minor)** The writing could be improved for clarity and simplicity. For instance, the caption in Figure 1 (lines 175–188) could be condensed to improve readability, ideally to half its current length. If extensive detail is necessary, consider moving some information to the main text.\n2. **(Minor)** The paper could use `\\citet{}` for citations when referring to the authors directly. For example, in lines 280–281, instead of \"Multilayer Perceptron (MLP) by \\cite{author2020},\" using `\\citet{}` could enhance readability.\n3. When mentioning the use of six popular datasets, it would be helpful to list the features and characteristics of each dataset to provide more context for the experiments. For instance, some datasets may have dense connections while others are sparse; some may benefit more from heterogeneous graphs compared to homogeneous ones. Without a detailed explanation of the datasets' biases or inductive properties, understanding the generalization of the proposed method becomes challenging.\n4. **Scalability**: The scalability of the proposed method is unclear. According to Table 3, all datasets used in the experiments are relatively small, with node counts between 3k and 6k, similar in scale to the Cora dataset. It would be beneficial to provide insights on how the method scales to larger datasets, such as ogbn-arxiv or ogbn-products, if they have hypergraph structures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)