Pre-trained Hypergraph Convolutional Neural Networks with Self-supervised Learning
Authors: Yihe Deng
Ruochi Zhang
Pan Xu
Jian Ma
Quanquan Gu
Venue: Accepted by TMLR ()

Abstract: Hypergraphs are powerful tools for modeling complex interactions across various domains, including biomedicine. However, learning meaningful node representations from hypergraphs remains a challenge. Existing supervised methods often lack generalizability, thereby limiting their real-world applications. We propose a new method, Pre-trained Hypergraph Convolutional Neural Networks with Self-supervised Learning (PhyGCN), which leverages hypergraph structure for self-supervision to enhance node representations. PhyGCN introduces a unique training strategy that integrates variable hyperedge sizes with self-supervised learning, enabling improved generalization to unseen data. Applications on multi-way chromatin interactions and polypharmacy side-effects demonstrate the effectiveness of PhyGCN. As a generic framework for high-order interaction datasets with abundant unlabeled data, PhyGCN holds strong potential for enhancing hypergraph node representations across various domains.

Published in Transactions on Machine Learning Research (08/2024)
Hypergraph Convolutional Neural Networks with Self-supervised Learning
Yihe Deng*
yihedeng@es.ucler ucla edu
Deportment of Computer Science University of California, Los Angeles
Ruochi Zhang"
shangro@@rruudiisttttt...rr
Eric and Wendy Schmidd Center Broad Institute of MIT and Horvard
Pan Xu
pan.zw@dukee edu
Department of Biostatistics and Bioinformatics Duke University
Jian Ma
janma@cs. cma.. edu
Roy and Stephanie Lane Computational Biology Department School of Computer Science
Carnegie Mellon University
Quanquan Gut
qgu gguBcs. Bes, www..claaaaa........c...c.oaaaee edu
Department Computer Science University of California, Los Angeles
Reviewed on OpenReview: https: //t::///ppee openreview. net/ forum? id= OWWXWPmctm
Abstract
Hypergraphs are powerful tools for modeling complex interactions across various domains, including biomedieine. However, learning meaningful node represenlations from hyper- graphs remains challenge. Existing supervised methods often lack generalizability, thereby limiting their real-world applications. We proposc new method, Pre-trained Hypergraph Convolutional Neural Networks with Self-supervised Learning (PhyGCN), which leverages hypergraph structure for self-supervision to enhance node representations. PhyGCN intro- duces a unique training strategy that integrates variable hyperedge sizes with self-supervised learning, enabling improved generalization to unseen data. Applications on multi way chro- matin interactions and polypharmacy side-effects demonstrate the effectiveness of PhyGCN. As generic framework for high-order interaction datasets with abundant unlabeled data, PCyGCN holds strong potential for enhancing hypergraph node representations across ious domains.
1 Introduction
Hypergraphs are essential data structures adept at modeling complex multi-entity relationships. Conven- tional graph neural networks (GNNs), limited to pairwise interactions, cannot effectively capture the higher-
order interactions inherent hypergraphs. This shortfall led to previous methods (Agarwal al. 2005
2006
Satchidanand al. 2015) expanding hypergraphs to graphs using clique expansion (Sun el. al. 2008)
More recent works (Yadtti et al. 2019 Dong al. 2020 Arya et al. 2020 Bai et al. 2021 Yi & Park
2020
Sun et al. 2021 have adopted GNN structure such as graph convolutional networks (GCNs) and
"Equal Contribution Co-correspooning Authors

---

Published in Transactions on Machine Learning Research (08/2024)
extended them to the hypergraph setting. HyperGCN (Yadati et al. 2019) and HNHN Dong al. 2020
in particular, achieved state-of-the-art performances on hypergraph benchmark datasets. However, these supervised methods heavily rely on labeled nodes, which are often scarce in practical scenarios. In situations with insufficient labels, these methods fail to fully utilize the rich structural information of hypergraphs, leading to less effective and poorly generalizable node representations.
Self-supervised learning (SSL), an approach that extract meaningful knowledge from abundant unlabeled data to enhance model generalization (Doersch et al. 2015 Kolesnikov et al., 2019), presents promising strategy to address these challenges. SSL creates pretext tasks from unlabeled data to predict unobserved input based on the observed part, enhancing the generalizability of models trained for computer vision (c.g..
ImageNet pre-training GGirshick et al. 2014 He et al. 2019)) and natural languages (e.g., BERT (Devlin
et al.
2019)). While SSL methods have been proposed for GNNs (Hu et al.
2020a You et al.
2020b Hu
et al.
2020b Wu et al. 2021: Jin et al.] 2022 Hwang et al. 2020 Hao et al., 2021 Sun et al. 2020). their
application to hypergraph learning underexplored. Noteworthy works (Xia et al. 2021 Yu al., 2021
Xia et. al. 2022) have targeted specific applications such as recommender systems, while others (Du et al.
2021) propose pre-training GNNs on hypergraphs, primarily focusing on hyperedge prediction. A detailed
discussion of related works can be found in Appendix A
In this work, we introduce PhyGCN to address the urgent need for a general method that can effectively leverage unlabeled data from hypergraphs to enhance node representation learning. This self-supervised method extracts knowledge from the hypergraph structure using self-supervised tasks, generating robust node representations for diverse downstream tasks. Constructing superviid tasks in hypergraphs poses unique challenge due to the variable sizes of hyperedges. We tackle this challenge by designing a self- supervised task that predicts masked hyperedges from observed ones and by incorporating an attention mechanism into our model architecture (Zhang et al.) 2020) to predict variable-sized hyperedges, which allows the size of the hyperedge to vary (e.g., the size k can arbitrarily be 2, 3, or 4). In contrast, fixed-sized hyperodge prediction involves predicting hyperodges that have a single, constant size.
Link prediction, an effective pre-training task for traditional graphs, typically involves designing a neural network with fixed-length input to predict edges. Hyperedge prediction, however, is more challenging due to variable sizes. Simple average pooling of node embeddings has been found insufficient to model a hyper-
edge (Tu et al. 2018). Although some works (Xia el. al. 2021 Yu el al. 2021 Xia el al. 2022 Du el al., 2021) have explored the potential of self-supervised learning on hypergraphs, our PhyGCN method is the first straightforward and effective approach to utilize hypergraph structure and can be generally applied to many tasks.
We demonstrate the effectiveness of PhyGCN through various evaluations across multiple tasks and datasets, showing its advantage over state-of-the--rt hypergraph learning methods. Notably, PhyGCN has been ap- plied to study multi-way chromatin interactions and polypharmacy side-effect network data, confirming its advantages in producing enhanced node embeddings and modeling higher-order interactions. Together, PhyGCN, a self-supervised method for hypergraph representation learning, can be applied broadly to a variety of problems.
1.1 Most Related Work
In this subsection, we present the most related work to ours that focus on hypergraph pretraining/sell selff supervised learning. Self-supervised learning and pre-training for hypergraphs remain largely unexplored. The majority of works (Xia et al.) [2021] Yu et al. 2021 Xia et al. 2022) focused on specific applications in recommender systems. Xia et al. (2021) introduced Dual Channel Hypergraph Convolutional Networks (DHCN), explicitly designed for bussion-based recommendation. DHCN's self-supervised learning task seeks to maximize mutual information between session representations learned by a graph convolutional network and DHCN, aiming to enhance hypergraph network learning. Yu et al. (2021) also focused on recommender systems, leveraging higher-order interactions to improve recommendaation quality. They introduced a self- supervised task to compensate for the loss incurred by their multi-channel hypergraph convolutional network, aiming to maximize mutual information concerning the representations of the user node, the neighboring sub-hypergraph of the user, and the entire hypergraph. Xia et al.] (2022) proposed to enhance recommen- dation and user representations via hypergraph transformer network that uses cross-view generative
2

---

Published in Transactions on Machine Learning Research (08/2024)
self-supervised learning for data augmentation. Lastly, Du et al. (2021) proposed pre-training with GNNs hypergraphs for hyperedge classification. This method uses GNNs with self-supervised pre-training tasks, achieving state-of-the-art performances on both inductive and transductive hyperedge classification. [Behrouz ct al. (2024) studied temporal hypergraphs and used random walk to automatically extract temporal, higher- order motils. (elyatnikov et al. (2023) studied how the concept. of homophily can be characterized in liy- pergraphs, and proposed a novel definition of homophily to effectively describe HNN model performances. Both works are related to our study on hypergraphs but orthogonal to our focus on the performance im- provement by self-supervised learning on hypergraphs. Lee & Shin 2023) studied self-supervised learning on hypergraphs and utilized a tri-directional contrastive loss function consisting of node-, hyperedge-, and membership-level contrast. Very recently, Kim et al. (2024) studied generative self-supervised learning on hypergraphs and proposed a new lask called hyperedge filling for hypergraph representation learning. Lastly, we take an additional note on Hyper-SAGNN (Zhang et al., 2020), which provides the attention mechanism used in PhyGCN. HyperSAGNN is method designed for hyperedge prediction, and is therefore only con- sidered as a baseline on hyperedge prediction tasks. For node classification tasks and other tasks we focus on, comparing with HyperSAGNN is not possible.
2 Preliminaries
2.1 Graph Neural Networks
We denote graph by G EV, where V is the vertex set with node u V and E is the edge set with pairwise edge vu E E. Graph neural networks (GNNs) mostly follow the scheme (Xu et al. 2019):
X = COMBINE") (x! (11)) AGGG (fx [x" u E N(v)))), where x!! is the embedding vector of node v at layer I of the network, x is the input data, and (w)) is the set of nodes that are adjacont to v. Here AGG(.) and COMBINE(.) are two functions usually used to aggregate the neighboring embeddings and combine the aggregated embedding with the original node embedding to output the embedding of given node. Different GNN models may have different choices of COMBINE and AGG(-) functions. As a most popular approach to aggregate and combine the node vectors, GCN (Kipf & Welling 2017) averages over all the first-order neighboring nodes including the node itself. Tet x()) represent the embedding matrix of all nodes at layer I, and the equation boils down to ( = a (x where a is some activation function, A E R XN is the normalized adjacency matrix with N denoting the number of nodes. O E R 1xh is the weight matrix at layer 1 - where ly the hidden embedding size at each layer 1. And (0) E Nxdo is the input node features of all nodes where dg is the size of the input node feature size.
2.2 Hypergraph Convolutional Networks
In an attempt to adapt GCN for tasks on hyporgraph, the problem naturally ariscs: how do we appropriately adapt the adjacency matrix for hypergraphs? Adjacency matrix is natural to construct for graphs: setting the entry Aij = 1 for each edge (v,, v,). However, hyperedge involve more than two nodes and constructing A will thus be tricky. As Feng et al. (2019) proposed, for hypergraph G (V, E), hypergraph can be reprosented by incidence matrix H E RNM with N as the number of nodes and M as the number of hyperedges. When hyperedge ej E E is incident with a vertex "; E V, H,j = 1, herwissee equals O. We denote W as the hyperedge weight matrix, and then a straightforward definition for the adjacency matrix will be A = HWH And to normalize it similarly as in GCN, we take À D. D/ 'H" D., where the diagonal matrices D, E R D, E N respectively represent the degree of hyperedges and the degree of vertices (enng et al. 2019) and A E N A layer of such hypergraph GCN model will then be similar to the regular GCN but with different adjacency matrix ( = a (AX""") @ E 1xh is the weight matrix at layer I and (() Nxdo the input node features of all nodes. We adopt such simple architectures in our work, along with several techniques in GNNs to enhance the model's learning on both self-supervised and main tasks.
Similarly,
3 Our Proposed Method
In this section, we present our framework of learning from Pre-trained Hypergraph Convolutional Neu- ral Networks with Self-supevvised Learning (PhyGCN). We start with an overview of the workflow of our
3

---

Published in Transactions on Machine Learning Research (08/2024)
framework in Section 3.1 We then present the detailed network structure design and the self-supervised
pre-training pipeline in Sections
and 3.3 respectively.
3.1 Overview of PhyGCN
a.
Node classes
Drug Side effect Drug-drug side-effect interaction
Hyperedge Prediction
Node classification
Self-supervised
Hypergraph Data
PhyGCN
Downstream
tasks
Continual Signal Prediction
b.
Hypergraph convolutional
Embeddings
Fully-connected layer
Training
network
Static embeddings
80%
Hyperedges
Dynamic embeddings
Negative samples
C(p, y)
Self-attention
Self-supervised
Validation
Output
Downstream tasks
20%
Softmax
Fully-connected layer
Figure Overview of PhyGCN. a. The general workflow of PhyGCN begins with a hypergraph and its hyperedge information. Hyperedge prediction is employed self-supervieed task for model pre-training. The pre-trained embedding model is then used for downstream tasks. b. This illustration elaborates PhyGCN's detailed workflow, which involves generating labeled data for self supervipe learning by randomly taking 80% of the hyperedges as positive samples, and creating negative samples by changing the nodes in each positive sample. This data facilitates the training of the base hypergraph convolutional network with the attention network. The pre-trained base model subsequently used for downstream tasks with a fully connected layer.
Fig. 1 provides a detailed depiction of PhyGCN's model architecture and general workflow. PhyGCN aims enhance node representation learning in hypergraphs by effectively leveraging abundant unlabeled data. To this end, we propose 1) a self-supervised lcarning strategy that pre-trains the model directly on the hypergraph structure before proceeding with any downstream tasks, and 2) corresponding model archi- tecture comprising base hypergraph convolutional network for representation learning, and attention network /Zhang et al. 2020 for executing the self-supervieed task.
As shown Fig. 14, to pre-train the model, we establish a self-supervised hyperedge prediction task using randomly masked hyperedges (20%). The neural networks are then trained to reconstruct them using only the remaining hyperedges (80)). We emphasize that, the pre-training is done on the available hyperedges with negative samplings to construct the negative examples (Section 3.3 provides the formal definition). The pre-trained neural network then be used for downstream tasks such node classification and
4

---

Published in Transactions on Machine Learning Research (08/2024)
continual signal prediction. See Section 3.3 for the details of the pro-training scheme. Fig. 1 details the architecture of the neural network model, consistent of a base hypergraph convolutional network and two end networks for different tasks. The base convolutional network computes the embedding of each node in the hypergraph, which is then directed either to the attention network for variable-sized hyperedge prediction, or to a fully connected layer for node-level downstream tasks. During the pre-training phase, the base model and the attention network are trained while the fully connected layer for downstream tasks remains randomly initialized. We select and retain the base model that exhibits the best performance on the pre-training task (hyperedge prediction) the validation set, and subsequently fine-tune the base model together with the fully connected layer for downstream tasks. During this stage, we incorporate the original hypergraph data with its complete hyperedge information, and use the downstream task labels as the supervisory signals for training.
To calculate the embedding for a target node, the hypergraph convolutional network aggregates information from neighboring nodes connected it via hyperedges, and combincs with the target node embedding to output final embedding. These computations are carried out at each layer of the nefwork. The base hypergraph convolutional network is stack of such hypergraph convolutional layers (see Section 3.2 To mitigate over-smoothing (Chen et al. 2020) and enhance the performance of the base model, we concatenate the output of each layer in the end to generate the final embeddings for the nodes. Furthermore, we adapt DropEdge (Rong et al. 2020) and introduce DropHyperedge, where we randomly mask the values of the adjacency matrix for the base hypergraph convolutional network during each training iteration to prevent overfitting and improve generalization the pre-training phase. By integrating the model architecture and pre-training scheme, PhyGCN effectively learns from hypergraph structure and applies the learned knowledge to downstream tasks.
3.2 Base Hypergraph Convolutional Network a.
DropHyperedge
Training data
b.
80%
Hyperedges
Skip/Dense Connection
Negative samples
CONCAT
C. Node representation
Fully-connected layer Static embeddings
Hyperedges
20% (masked the network)
Negative Sampling
ts
Average
Negative samples
?
Validation data
Dynamic embeddings
Self-attention
Figure 2: Illustration of the method. For the base hypergraph convolutional network, propose two adapted strategies (DropHyperodde and Skip/Dense Connection) to further improve the pre-training of PCyGCN. b. For the self-supervised pre-training task, we randomly mask 20% of the hyperedges in the hypergraph and generate negative samples with regard to each positive training example. See Section 3.3 for more details. c. The pre-training scheme requires prediction hyperedges that are arbitrarily sized. Thercfore, we adopt an attention network (Zhang et al. 2020 to fulfill the task.

---

Published in Transactions on Machine Learning Research (08/2024)
We thus propose self-supervised learning strategy, along with simple but effective hypergraph con- volutional network architecture, to better capture the structural knowledge of a hypergraph. Our base convolutional model follows from the previous setup of a hypergraph convolutional network, where x() a (Ax(t=))(( And we further introduce the following modifications to the base hypergraph convolu- tional model.
Skip/Dnnse Connection. GCNs have been known to the have vanishing gradient problem where stacking multiple convolution layers in GCN model causes the derivative of the loss function to approach zero and thus hard to train (Wu et al. 2020 To make GCNs deeper and prevent the vanishing gradient problem, Li ct al. 2019) proposes several adapted strategies based on the methods frequently used for CNNs. Similar vanishing gradient problems have also been observed in hypergraph convolutional networks (Chen & Zhang 2022). Therefore, we design a similar but more ceaputing-efficienn strategy where we concatenate the outputs of all layers of the model only after the last layer (instead of concatenating the previous outputs at each layer as in Li al.] (2019)), as illustrated in Figure 2p. That is, we keep all the previous layers unchanged and set the last layer of the convolutional model as:
(
((-))) X
(1)
And we denote x) as the j-th row of x which is the embodding at layer I for node j. Such design helps us save computation time while preventing the model's performance from dropping with more layers. The eventual base model we use is then the n-layer convolutional model with concatenation at the last layer. We either add an attention layer for the self-supervised task or an MLP layer for the main task.
DropHyperedge. Overfitting is common issue in the use of neural networks, where the network learns to fit the training data too well and fails to generalize to test data. To mitigate such an effect, dropout has been a popular technique for many fields and is also commonly used for GCNs. To further reduce overfitting and improve generalization in GCNs, Rong et al. (2020) introduced a new regularization technique designed for GCNs called DropEdge, which randomly mask the edges of the graph at each iteration. In this work, to address the over-fitting issucs with PhyGCN and encourage better generalization in the pre-training stage, we adapt the DropEdge strategy our model by randomly masking the values on the adjacency matrix A at each iteration (as illustrated in Figure Pb). Note that such strategy is only used for the self-supervised hyperedge prediction task. We use the same dropout strategy in the main task the other baselines.
3.3 Pre-training with Self-supervisony Signals
Self-supervised task. As shown in Figure P,, given a hypergraph G (V, we randomly divide the set of hyperedges E into the training set Eurain and the validation set Evalid- Moreover, we generate 5 negative samples for each hyperedge in Eusin and Evaldd, and denote the negative sample sets as E'alid and Evalia' To distinguish these generated samples, we refer to Etrain and Evalid as the positive training set and the positive validation set respectively. Combining these datasets, we obtain the final training and validation set follows.
Strain {Ei, 2:) E, E Etrain: for = 1; E Frrain for 2, = }
and
Syalid = {FE, 2;)
F2 E Fratid, for 2 = 1; Frg E Fealld: for 2; = }:
where E, (ui...., ] is a set of nodes of arbitrary size k. We then train the model by the hyperedge
prediction task: given a set of node features (xx)) (x)"))eet }yetE, with arbitrary size k and x E R&, the goal is to prodict 2, E {0, 1}: whether the nodes form a hyperedge. Therefore, given the training set Stman we aim
minimize the following cross-entropy loss with regard to each training batch S:
Spretrain = - TST Eiess],sESuann 24 log (x(1 )EEE)
(2)
Here, f(:) Rd + Rdi denotes the base hypergraph convolutton model, where dg is the input feature size and dj is the size of the output embedding x (m) of the network. Moreover, g(-) : (Rd1), E. + [0, 1] denotes the attention network, which takes a arbitrary-size set of node embeddings x and output the probability p of whether the nodes form hyperedge. The base hypergraph convolutional model f(-) and attention
6

---

Published in Transactions on Machine Learning Research (08/2024)
network 9()) are trained together, and the base model's weight parameters are saved for further fine-tuning in wwwnstream tasks.
Negative sampling. We generate negative samples similarly to Zhang et al. 2020 Tu et al. (2018) Specifically, for each hyperedge E, = {ui...., b6 E Eurain, which is referred to as the positive sample, we generate negative samples. Fach negative sample E is obtained by randomly alternating one or multiple of the nodes in Es such that E does not belong to the existing hypergedge set Evain or Evalid, as shown in Figure 2b. Moreover, vary the sampling strategy for different hypergraphs based their node types. For hypergraphs with only node type (e.g., citation networks), E & U(Errann, Evalid). For hypergraphs with multiple node types (e.g., polypharmacy side-effect data), we randomly alter a node within either node type. Take a hypergraph with two node types e and u for example. The positive hyperedge takes the form of E, = fVi...., Ukg U1 UI....,uyy u.. The negative samples are then designed as
have E (Vi,...,'',, U'....,,'k)
E = (11,-.. 2/... Uk u1........ m E & UfErrains Evalid]; E (u.,..., 2k. u1, u/ w E $ Uf (Etrains Evalid).
Finally, for hypergraphs with fixed-length hyperedges, such as the polypharmacy dataset (Zinikk et al. 2018), we propose an enhanced negative sampling strategy. Assume the fixed length is k = 3 and the positive sample is FA- = {ei, 02, va)- We generate the negative sample F {v t2. u3) by altering two nodes such that (1) E & Eraai,, Evalidt and (2) there exists positive sample Ej {or 22 ag} E Uf Evand). This ensures that the altered nodes in the negative sample interact in another positive sample.
Our enhanced negative sampling strategy is novel in self-supervised learning and provides more accurate prediction in differentiating positive and negative samples in hypergraph applications. For example, in the polypharmacy side effect dataset, where we aim to predict whether a drug combination causes a specific side effect, traditional approaches (Zitnik et al.) 2018 Nováček & Mohamed, 2020)p might generate random drug entities that do not cause any side effect. In contrast, our enhanced negative sampling strategy generates drug entities that cause other side effects, forcing the models to learn more subtle differences between the positive and negative hyperedges. See Section 4.3 and Figure 5 for more details.
Attention Network. As in Figure 2p, the attention network follows the design in Zhang et al. (2020) with weight matrices Wq E gdixds, Wx E Rd ddxid, and Wy E R xds Given the a node set E of size k and the node embeddings (x f(x) f(x)))))ye JRyEs, the normalized attention coefficient ag regarding node oe and node dj is computed as ag = exp (WTx("") (WIx(")) )/E exp (WW Jx, (n)) ((W (W(xx(())) Then, the dynamic embedding of each node is E is computed as d; = tanh (Ein oa W (xx) E Rd And with weight matrix Ws ddxx the static embedding of each node 2 E is computed as s, W E R Finally, with another layer Wo E Rds, the output of the attention network is then the probability that the node set is a hyperedge p 1/k o (W](d. -s), =:) +). where o is the sigmoid function and b R. Notation denotes element-wise power operation.
4 Experiments
Based on the pre-trained model f(-) in Section 3 that generates node embeddings with the information it leveraged from the pre-training task, we conduct various experiments on node-level downstream tasks in- cluding classification problems on citation networks, biological discoveries in multi-way chromatin interaction datascts, and predicting polypharmacy side effocts. In particular, we load the saved base model and add a fully connected layer for prediction. Then we fine-tune the model by the downstream task.
Let f(-) denote the fully connected layer that is task-specific for different data. For node classification tasks with C classes as in Section 4.1 we have h(-) : R4 + RC, Given the training set Strain with input feature x! E d for each node i the label ys E [C] representing the node class, we minimize the following cross-entropy loss with regard to each training batch S:
Lmain = Eie(s) log (exp (f( ,,) (Eg ((f(x())))))
(3)
7

---

Published in Transactions on Machine Learning Research (08/2024)
For contimual signal prediction tasks as in Section 4.2 we have h(-) :R + R. Given the training set Strain
with input feature x E dR for each node the label yi E R, we minimize the following mean squared error with regard to each training batch S:
Lmain = 1/|S| Eits) (M([(xx)))) - y:)².
(4)
In the downstream tasks, we keep fine-tuning the base convolutional network f(-) with the task-specific fully connected layer h(-).
4.1 Self-supervised learning improves performance
We start with the fundamental question: can PhyG CN more effectively ulilize the structural information in a hypergraph? To assess this, we benchmarked PhyGCN against the current sfate-of-the-ary hypergraph learning methods, including HyperGCN (Yadati et al. 2019 and HNHN (Dong et al. 2020). We also estab- lished a baseline by merely decomposing the hypergraphs into graphs and applying a standard GCN (Kipf & Welling 2017). We evaluated these methods using the benchmark node classification task on citation networks (including Citeseer, Cora, DBLP,, and PubMed) whose hyperedges denote either co-authorship co-citation relationships. We note that, citation networks are modeled as hypergraphs where hyperedges contain publications sharing the same citation (co-citation) or the same author (co-authorship). Since multi- ple publications can cite the same paper or share the same author, they form hyperedges with arbitrary sizes (i.e., variable-sized hyperedge). Given that the baselines utilize different data splits for their evaluations, we consistently evaluated our method along with all the baselines on the same random data splits across a range of training data, ratios. Detailed statistics, experiment settings, and results are provided in Appendix B.2
In order to ascertain how each hypergraph learning method performss with limited training data, we first set the training data ratio to 0.5% across the five datasets, The results are shown in Fig. 3 Across 10 random data splits, we found that both HyperGCN and HNHN fail to outperform the standard GCN model, which merely transforms the hypergraphs into conventional graphs. Conversely, PhyGCN, equipped with pre-training on the self-supervisory signals obtained directly from the hypergraph, performs significantly better when dealing with a small training data ratio. It outperforms GCN across all datasets, particularly small networks such as Citeseer and Cora. The improvement over GCN suggests that PhyGCN in more adept at learning from the higher-order hypergraph structure as opposed to merely decomposing it into binary graph. Furthermore, the pre-training scheme allows PhyGCN to extract more information for node representation learning, especially when dealing with small hypergraphs.
To evaluate PhyGCN against the baselines on a wider range of training data ratios, we generated data splits with increasing training data ratios of 1%, 2%. 4%, 8% and 16%. The results on the Cora (co-authorship) dataset are shown in Fig. 3p, while comprehensive results of all models on the five datasets are provided in Appendix B.2 Overall, PhyGCN outperforms the other methods, demonstrating more robust performance under random data splits. Across varying training data ratios, PhyGCN consistently outperforms the three basclines. At training data ratios of 2%, 4%, 8% and 16%, PCyGCN's performance is considcrably more stable than that of the baselines, as indicated by the significant decrease in the variance of PhyG accuracy across 10 random data splits.
These observations suggest that the pro-training stage enables the model to more effectively gather informa- tion from unlabeled data, allowing it to perform well even when small amount of data is available. The results underscore the potential of PhyGCN in learning with higher-order interactions without the strong requirement of ample labeled data for a specific task. By fully leveraging the hypergraph structure for self- supervised training and designing a simple but effective model architecture, PhyGCN can achieve superior and more stable performances on downstream tasks compared to existing baselines.
Datasets. The details of the datasets used for the evaluation of node classification are shown in Table I It is noleworthy that the data used by Yadati el al. (219)) contains a lot of isolated nodes, implying that a significant portion of nodes, not part of any hyperedge, are classified solely based on their feature information. This is not an ideal setting for evaluating the capacity of a graph/hyperge hypergraph neural network in capturing structural information. We opt for similar setting to Dong et al. (2020). where we remove the isolated nodes from the raw data. Moreover, since the compared works Yadati et. al. (2019) Dong et al.
8

---

Published in Transactions on Machine Learning Research (08/2024)
a
b
HNHN
HyperGON
GCN
PhyGCN
Test accuracy on Cora (Co-authorship)
80 0
80.0
70
70.0
Accuracy (%)
60.0
Accuracy (%)
60 0
50.0
50.0
40.0
PhyGCN
30.0
40
GCN
20.0
HNHN
30
HyperGCN
10.0
0.0
Citeseer
PubMed
Cora cco-auttonshiip/cco-ctttttn) Dataset
Cora
DBLP
0.5%
1%
2%
4%
8%
16%
Training data ratio
Figure 3: Evaluation of our model on node classification on citation networks with variable hyperedge sizes (Citeseer, Cora (co-authorship), Cora (co-citation), DBLP, and PubMed). The baseline methods include HNHN (Dong et al. 20200, HyperGCN (Yadati et al. 2019), and vanilla GCN (Kipf & Welling 2017). a. Comparison of PhyGCN's accuracy with the baseline methods when the training data is minimal (0.05%). b. On the Cora (co-citation) dataset, PhyGCN consistently outperforms the baselines across various training data ratios (0.05%, 1%, 2%, 4%, 8%, 16%).
2020) use different train-test ratios for each dataset, we make a uniform setting of ratios for all data and models: 0.5%, 1%, 2%, 4%, 8% and 16%. Specifically, we conduct four experiments on each dataset for each model, incrementally increasing the training data ratio from 0.5% to 16%. Training data with larger proportion includes the nodes from the training data of a smaller proportion. Lastly, perform 10 random splits for each ratio of each dalaset to evaluate the average performance alongside the standard deviation.
DBLP
Pubmed (co-citation)
Cora (co-authorship)
Cora
Citescer
(co-authorship)
(co-citation)
(co-citation)
# nodes, [V] hyperedges, [E]
41302
3840
2388
1434
1458
22363
7963
1072
1579
1079
avg hyperedge size 4.5 5.4
4.3 + 5.7
4.3 4.2
3.0 L.0
3.2 2.0
features,
1425
500
1433
1433
3703
# classes, q
3
The statistics for the citation dataset used in node classification, where each data has the hyperedge either as a co-authorship relation (papers written by the same author) or as co-citation relation (papers cited by the same paper).
Table 1:
Additional Baselines. We additionally discuss the recent methods HypeBoy (Kim et al. 2024 and TriCL (Lee & Shin 2023), which were developed coneurrently or after our work. Table 2 presents per- formance comparison of these methods. The results indicate that all three methods perform comparably, with each method excelling in different datasets. PhyGCN generally ranks as either the best second-best method across various datascts and datasct splits. Notably, HypeBoy necessitates extensive softmax com- putation for nodes against all other nodes, resulting in significant GPU memory consumption and limited scalability to large datasets. Conversely, PhyGCN's self-supervised learning framework is lightweight and scalable, requiring minimal computational resources. TriCL, on the other hand, involves several hyperpa- rameters that need tuning for each dataset. In contrast, PhyGCN only requires tuning the number of layers for different datasets.
9

---

Published in Transactions on Machine Learning Research (08/2024)
a
b
Classification on GM12878
AUROC score on different datasets
AUPR score different datasets
100
1.000
1.000
MATCHA PhyGCN
0.950
Accuracy (%)
0.900
60
AUROC
0.850
0 800
0 750
0.750
0.700
0.700
plain
pre-trained
size=2
size=3
size=4
size=2
size=3
size=4
d
Regression on GM12878
Regression on HFFc6
Regression on HFFc8
0.90
0.85
0.9
Pearson Correlation
Spearman Correlation
0.75
PhyGON
0.65
0.6
0.55
0.5
0.5
0.6
0.7
0.8
0.9
0.45
A/B Compartments Regression
PhyGCN Compartments Regression
PhyGON Compartments
Regression
Chromosome
(every 5mb)
0.04
Comoo
0.02
0.01
Grondd
PhyGCN Compartments Regression
Figure 4: Application to the SPRITE data of chromatin interactions. a. Classification accuracy for subcom- partment labels of the GM12878 cell line. AUROC and AUPR scores for predicting multi-way chromatin interactions, where we compare our method with MATCHA (Zhang Ma 2020) across hyperedges sizes 2, 3, and Regression results on the GM12878 cell line, compared against the baseline A/B Compartments. d. Regression results on the HFFc6 cell line, where we compared our transferred PhyGCN results with the bascline A/B Compartments. HFFс6). The boxplot shows the MSE distribution between the ground truth and the result from PCyGCN or the baseline. MSE is calculated within each region of Mb.
Browser shot chromosome 1 for Repli-scq prediction task (regression
10

---

Published in Transactions on Machine Learning Research (08/2024)
Training Ratio
Dataset
Model
0.3%
1%
2%
4%
8%
16%
HypeBoy
48.87 + 10.3
56.88 + 7.4
66.33 + 3.1
68.84 + 1.5
69.89 + 1.
71.75 # 1.0
Citeseer
TriCL
39.84 1 9.5
48.01 +: 7.7
54.31
1 4.2
60.25 1 2.7
64.24
1 1.1
68.91 1 1.6
PhyGCN
47.35 + 8.4
54,22 + 7.2
62.87 + 4.3
67.02 + 1.0
70.07 + 1.3
71.62 + 1.0
Cora
HypeBoy
62.60 6.2
60.26 = 7.6
68.96 + 1.3
72.29 + 1.5
73.31 1 1.3
74.00 + 1.2
(co-anthorship)
TriCL
50.94 + 6.7
63.51 +: 6.3
70.58 +: 3.6
76.71 + 1.3
78.82 +: 0.7
80.80 =: 0.8
PhyGCN
49.43 1 6.5
60.75 1: 5.8
67.21 1 3.6
71.86 1 1.9
75.93 1. 1.3
78.39 1 0.8
Cora
llypeBoy
47.17 1 3.9
56.77 4.5
62.83 1 5.1
67.80 1 3.1
69.65 1 2.2
71.59 1 2.5
(co-citation)
TriCL
49.75
7.2
56.29 = 5.9
66.27 + 4.3
73.23 + 1.7
77.72 + 1.3
80.42 + 1.3
PhyGCN
48.82
8.2
58.46 + 5.5
65.71 + 4.5
T2.35 + 3.2
T7.33 + 1.5
79.34 + 0.8
HypeBoy
85.12
0.6
85.84 :: 0.2
86.07 1: 0.1
86.16 1 0.2
86.26 1 0.3
86.27 1 0.2
DBLP
TriCL
87.99
0.7
89.06 +: 0.3
89.95 +: 0.2
90.37 1: 0.1
90.61 1: 0.1
90.70 1: 0.1
PhyGCN
84.69 = 0.6
86.12 = 0.4
86.96 + 0.2
87.80 + 0.2
88.19 + 0.2
88.23 + 0.1
HypeBoy
67.84 = 5.1
69.81 = 3.0
69.52 + 3.4
71.02 +
71.48 +
7L.45 + 0.8
PuhMed
TriCL
69.57 + 7.8
75.10 + 3.4
78.96 + 1.9
80.43 + 0.8
81.68
+ 0.5
82.04 + 0.5
PhyGCN
74.66 +: 3.9
78.00 +: 1.6
79.09 +: 1.6
79.96 1: 0.9
82.08 4: 0.6
82.84 #: 0.5
Table 2: Test accuracy for different datasets at different data split ratio (training data % = 0.5% 1% 2%. 4%, 8%, 16%). We report mean test accuracy + standard deviation among 10 random train-test split for each split ratio. We underscore the best performing method and underline the sccond-best performing method.
4.2 Exploring multi-way chromatin interaction
To further demonstrate the advantages of PhyGCN, particularly its potential for advancing discoveries in biological data, we applied it to multi-way chromatin interaction datasets. Such datasets reflect simul- tancous interactions involving multiple genomic loci within the same muclci, providing an opportunity to probe higher-order genome organization within a single nucleus. However, due to the limitation of cur- rent mapping technology and the exponentially growing combination space of genomic loci, these datasets tend to be extremely sparse and noisy. We evaluated our method SPRITE [Quinodoz et al.] 2018) data from the GM12878 cell line, where the aim is to predict the Hi-C subcompartment label for each genomic bin. The details of the experimental setup, including the tasks, baselines, and datasets, can be found in Appendix B.3
node classification problem the
Why pre-training? We first evaluated how the pre-training scheme improves the learned representations for the downstream tasks with a node classification problem on the SPRITE data. In this task, we aim to predict the Hi-C subcompartment label for each genomic bin and we split the data by chromosome indices. Fig. 1e shows the results of the node classification problem. Without pre-training, our "plain" model is directly trained on the subcompartment labels to make predictions. With self-supervised learning that extracts the hyperedge information, the pro-trained model performs much better on the cross-chromosome subcompartment label prediction task.
Why the hypergraph convolutional network? In Fig. 4p, we investigate the effectiveness of PhyGCN's hypergraph convolutional architecture in lcarning multi-way chromatin interactions of variable sizes, which is the pre-training task of the model. Specifically, we compared PhyGCN against MATCHA (Zhang & Ma 2020 which uses an autoencoder to generate node representations. As in the same setting in Zhang & Ma (2020), we consider hyperedges with the size of 2, 3, and 4 and use the AUROC (area under the receiver operating characteristic) and AUPR (area under precision-recall) scores as the evaluation metrics. We found that, with the hypergraph convolutional architecture, PhyGCN outperforms MATCHA on the three different hyperodge sizes with a small fraction of training data provided. It should be noted that the hyperedge prediction lask aims provide a pre-trained model for downstream tasks. To further evaluate the effectiveness of our base model, we performed several other hyperedge prediction tasks thorough ablation study. The results, detailed in Appendix B355 confirm that the introduced convolutional architecture enables the model to consistently and stably capture hyperedge information across different datasets, providing a better pre-training stage.
Comparison with previous methods. To investigate whether our pre-trained model captures informative patterns of multi-way chromatin interactions, we conducted a downstream regression task to predict the DNA replication timing of each genomic bin using corresponding embeddings input. We fit the signals on even
11

---

Published in Transactions on Machine Learning Research (08/2024)
odd chromosomes given the training labels on odd or even chromosomes. As baseline, we used A/B compartment scores [[ieberman-Aiden et. al. 2009). The results are shown in Fig. 4 where the z-axis represents the Pearson correlation score of A/ compartment scores and Repli-seq signals, and the y-axis denotes the Pearson correlation score of the output of PhyGCN and ground truth labels. Each data point on Fig. 4 corresponds to a chromosome. Based on the downstream lasks, we found that with pre-training on the hyperedge interactions, our model can effectively capture the underlying patterns of 3D genome organization, generating more informative embeddings that are relevant to important biological functions such DNA replication.
Transferring knowledge to a different hypergraph structure. We further explored the potential of PhyGCN on learning and adapting to new hypergraph structures. Specifically, we evaluated our method on transferring the association between multi-way chromatin interactions and DNA replication timing learned from the SPRITE data of the GM12878 cell line to that of the HFFc6 cell line. In this task, the node set (genomic loci) remain the same across two cell lines, while the observed hyperedges among those nodes (multi-way chromatin interactions) have drastically changed. However, since the label we try to predict is associated with the genome structure that can be depicted by the higher-order chromatin interactions, we reason the transfer learning across hypergraphs would be feasible. To achieve this, we first pre-trained our base model and the attention network to predict multi-way chromatin interactions of the GM12878 cell line using the self-supervised learning task. We then fixed the base model and fine-tune the fully connected layer to predict replication timing of genomic loci in the GM12878 cell line. Next, with the attention network fixed, we fine-tuned the base model with the task of predicting multi-way chromatin interactions in the HFFc6 cell line using the hypergraph constructed from the SPRITE data of HFFc6 following the same SSL setting as the datasct of the GM12878 cell line. With an additional regularizer the basc model weights, we expect our base model to learn the subtle knowledge in hypergraph structural differences and combined with the fixed fully connected layer to make accurate predictions of Repli-seq signals in HFFc6 cell line. The results are shown in Fig. 4d, where we use boxplots to show the Pearson and Spearman correlation scores on each chromosome and compare them with the A/B compartment score calculated from the contact of SPRITE data in HFFc6. In Fig. 4p, we show an example from chromosome where we compared the ground truth Repli-seq signals to the predictions from A/B compartments regression and PhyGCN with PhyGCN reaching an overall smaller MSE. Together, our results demonstrate that PhyGCN can better learn the representation based on mulli-way chromatin interactions and utilize it to make downstream inferences.
Additional note. Our problem setting is unique and different from the traditional inductive learning setting. The traditional inductive learning setting involves inference or transfer learning on a new graph with new nodes and new edges/hy hhypeeegges among them. In our chromatin interaction network, across different cell lines, the nodes remain the same (as they are the same genomic region and correspond to the same DNA sequence), but the hyperedges among them have changed in different cell lines.
4.3 Application to predicting polypharmacy side effect
We also apply PhyGCN on the polypharmacy side effect dataset and demonstrate the advantages of learning such data as hypergraph rather than graph. Typically, the association between drugs and side effects is represented as a pairwise drug-side-elfect network. However, during disease treatment, multiple drugs are often used in combination, and undesirable combinations can cause side effects not known to either individual drug in the combination (polypharmacy side-effect) (Zitnik et al. 2018). Recent works like Zitnik et al. (0088): ; Nováček & Mohamed (2020) have used GNNs to predict polypharmacy side-effects with noteworthy performance. These methods treat drugs as nodes and side effects associated with a drug pair as the attributes for the corresponding edge. However, the number of unique side effects is comparable to or even exceeds the number of unique drugs, making it inefficient to model the data as multiple graphs for each side effect. Furthermore, real-world data likely involve drug combinations with more than two drugs that cause sidc-cffects, which GNN methods may not be able to handle. In contrast, hypergraph methods like PhyGCN can easily adapt to such datasets. Since the polypharmacy side-effect data involves interactions beyond pairwise, it can be naturally learned as a hypergraph with drugs and side effects as the nodes. We investigated how PhyGCN Icarns such interactions and compared it with statc-of-the-art methods, including graph- methods such as Decagon (Zitnik et al. 2018) and ComplEX (Nováčck & Mohamed] (2020). and hypergraph-baser method Hyper-SAGNN (Zhang et al. 2020 Further details are provided in Appendix B.4
12

---

Published in Transactions on Machine Learning Research (08/2024)
In Zitnik al.) (2018) Nováčck & Mohamed (2020), only side-effects known to be associated with more than 500 drug combinations were used for test data. In addition to this original setting, we report results on ran- dom data splits where any side effect can be used for testing. We presented the results in Fig. 5a. In terins
of AUROC scores, methods based on hypergraph modeling of the data (using Hyper-SAGNN (Zhang et al., 2020) and our PhyGCN) yield better performance than those based on graph modeling (using Decagon ZZi-- nik et al., 2018) and ComplEX (Nováčck & Mohamed, 2020)) in both the original and random settings. Moreover, our algorithm PhyGCN achieves the best performance on all datasets in both settings.
We conducted additional experiments to investigate how PhyGCN benefits from learning the data hypergraph by testing it on different types of data splits. Specifically, we considered interactions that involve a known pairwise relationship as "expected" and those that do not involve any known relationship as "unexpected" in the original dataset, as some side-effects are already known to be associated with a single drug (sce Appendix B.4. We empirically studied how the baselines and our method performed when testing only on expected or unexpected data. As shown in Fig. 5p. GNN methods such as Docagon and ComplEX showed a drop in performance from "expected" to "unexpected", while PhyGCN maintained a consistent performance on both types of data. As the unexpected polypharmacy side-effect triplet does not contain any potential pairwise interactions (while "expected" ones have), the "uncxpeeted" test data serves as a useful indicator of how well a method captured such high-order interactions. The consistency of PhyGCN and Hyper-SAGNN on the different data splits further demonstrates the advantage of modeling complex interactions as a hypergraph. Additionally, our results showed that PhyGCN outperformed Hyper-SAGNN, indicating its superior capability to learn from hypergraphs.
Enhanced negative sampling strategy. To investigate the effect of negative sampling on the polyphar- macy side effect dataset, we conducted an experiment to compare the traditional random negative sampling method with our proposed negative sampling strategy. The traditional approach (Zitnik et al.) 2018 Nováček & Mobamed| 2020) generates negative samples by randomly sampling two drugs (nodes) that do not cause the specific side effect (node) in the training data. These sampled drugs mostly do not cause any side effect, simplifying the prediction so that a model which can predict if a drug pair causes ANY side effect will have satisfactory performance. Therefore, we hypothesize that this strategy may lead to a model that learns the bias of the likelihood that drug combination causes any side effect rather than the specific side effect in question. To address this, we propose a stronger criterion for negative sampling (see Section 3.3 for details), which involves sampling a drug combination that is known to cause other side effects and combining it with the given side effect as a negative sample. Our enhanced negative sampling strategy, instead of sampling random drug entitics, selects an incorrect pair of drugs known to cause other side effects and combines it with the given side effect that it does not cause, therefore increasing the prediction diffi culty. This strategy forces the model to differentiate between the positive and negative samples and learn the subtle differences between them. The results of our experiment are presented in Fig. 5p. We found that our proposed negative sampling strategy poses greater challenge for all methods to perform well, with an average test accuracy below 80%. However, we observed that PhyGCN outperforms the baselines and performs stably well across different data splits. This strongly suggests that PhyGCN is better able to grasp the complex interactions without bias, unlike the baselines. We further present detailed examples in Fig. 5e to demonstrate the expected and unexpected side effect, and our enhanced negative sampling stratogy. Together, the results of this experiment highlight the importance of carcful negative sampling in achieving good performance in learning hypergraphs.
Additional note the two negative sampling strategy. We reason our negative sampling strategy resembles a harder task as supported by lower AUROC when evaluating performance. The task for predicting drug pair side effects can actually be seen as a two-step task: 1. A binary classification task, of whether a drug pair can cause any side effect or not: 2. A multi-class classification task, of what specific side effects a drug pair can cause if it will cause side effects. The first task due to the inherent imbalance in the dataset can result in much higher AUROC scores for all methods, and the original negative sampling strategy would be biased towards that. The second task is harder and is what our enhanced negative sampling strategy is trying to highlight.
13

---

Published in Transactions on Machine Learning Research (08/2024)
(Original Negalive Sampling)
b
(Enhanced Negative Sampling)
AUROC score different datasets
AUROC score different datasets
1.0
1.0
Decagon
ComplEX
0.9
0.9
HyperSAGNN
PhyGCN
AUROG
0.8
AUROC
0.8
0.7-
0.7
0.6
0.6
0.5
0.5
Original
Random
Expected
Unexpected
Original
Random
Expected
Unexpected
C
Expected:
Unexpected:
Drug Side effect
D1
D5
D2
D2
Known
D6
Original negative sampling:
Enhanced negative sampling:
D3
E1
D5
E1
D2
D4
Figure 5: Application to the polypharmacy side effects dataset. a. We display the AUROC score for polypharmacy side effect prediction across different data splits, distinguishing between performances on "cxpected" and "uncxpected" polypharmacy side effects. b. The AUROC score for polypharmacy side effect prediction across different data splits under an enhanced negative sampling strategy. c. We concretely demonstrate our definition of expected/ unexpected relations. In the expected interactions, knowledge that one of the drugs causes the side effect, as indicated by the blue arrow. Furthermore, we illustrate the difference between the original negative sampling and the enhanced negative sampling. The latter selects drug pairs that generate other side effects to be combined with the current example. More discussions on the results in this figure are provided in Section 4.3
have prior
5 Discussion
In this paper, we introduced PhyGCN, hypergraph convolutional network model with self-supervised pre-training scheme, designed to effectively capture higher-order information of hypergraphs. We demon- strated the effectiveness of PhyGCN on node classification tasks of benchmark citation networks and con- duct comprehensive studies on a multi-way chromatin interaction dataset, highlighting the contribution of each component to capturing informative interaction patterns. We also showed the advantages of modeling polypharmacy side effects as hypergraphs and propose an improved negative sampling scheme to evaluate models with less bias. With its ability to model higher-order interactions in various graph-structured data, such social networks with group interactions, PhyGCN has great potential for wide range of applications. Many real-world graph-structured data possess higher-order interactions. For example, social networks have group interactions that are inherently hypergraph--srrue tions may lose valuable information. By modeling such graph data as hypergraph, PhyGCN can provide valuable node representations by extracting information from the structure. This offers opportunities for leveraging the method in various applications, from drug discovery to social network analysis, where complex, multi-way interactions play a critical role.
and decomposing them into pairwise interac-
14

---

Published in Transactions on Machine Learning Research (08/2024)
Acknowledgments
We would like to thank the anonymous reviewers and the editor for their helpful comments. Y. Deng and Q. Gu were supported in part by the National Science Foundation IIS-1855099 and IIS- IIS-1903202. J. Ma was supported, in part, by National Institutes of Health Common Fund 4D Nucleome Program grant UM1HG011598; National Institutes of Health Common Fund Cellular Senescence Network Program grant UH3CA268202; National Institutes of Health grants R01HG007352 and R01HG012303; a Guggenheim Fel- lowship from the John Simon Guggenheim Memorial Foundation; a Google Research Collabs Award; and Single-Cell Biology Data Insights award from the Chan Zuckerberg Initiative. P. Xu was supported in part by the National Science Foundation (DMS-23231DD and the Whitehead Scholars Program at the Duke R. Zhang was additionally supported by funding from the Eric and Wendy Schmidt Center at the Broad Institute of MIT and Harvard.
University School of Medicine.
References
Nezar Abdennur and Leonid A Mirny. Cooler: scalable storage for hi-c data and other genomically labeled arrays. Bioinformatics, 36(1):311-315 2020.
Sameer Agarwal, Jongwoo Lim, Lihi Zelnik-Manor, Pietro Perona, David Kriegman, and Serge Belongie. Beyond pairwise clustering. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) volume 2, pp. 838-845. IEEE, 2005.
Sameer Agarwal, Kristin Branson, and Serge Belongie. Higher order learning with graphs. In Proceedings of the 23rd international conference on Machine learning, pp. 17-24, 2006.
Devanshu Arya, Deepak K Gupta, Stevan Rudinac, and Marcel Worring. Hypersage: Generalizing inductive representation learning on hypergraphs. orXiv preprint vrXiv:2010.04558, 2020.
Song Bai, Feihu Zhang, and Philip HS Torr. Hypergraph convolution and hypergraph attention. Pattern Recognition, 110:107637, 2021.
Ali Behrouz, Farnoosh Hashemi, Sadaf Sadeghian, and Margo Seltzer. Cat-walk: Inductive hypergraph learning via set walks. Advances in Neurol Information Processing Systems, 36, 2024.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. Advances in newral information processing systems, 26, 2013.
Alain Bretto. Hypergraph theory. An introduction. Mathematical Engineering. Cham: Springer, 2013.
Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. Mcasuring and relieving the over-smoothing problem for graph neural networks from the topological view. Proceedings of the AAAT Conference on Artificial Intelligence, volume 34, pp. 3438 3445, 2020.
Guanzi Chen and Jiying Zhang, Preventing over-smogthing for hypergraph neural networks. arXiv preprint arXiv:2208.171 2022.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidi- rectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume (Long and Short Papers), pp. 1771-186,, Minneapolis, Minnesola, June 2019. Association for Computational Linguistics. doi: 10. 10.11653//11/119
Carl Doersch, Abhinav Gupta, and Alexei Efros. Unsupervised visual representation learning by context prodiction. In Proceedings of the IEEE international conference on computer vision, pp. 1422-1430, 2015.
Yihe Dong, Will Sawin, and Yoshua Bengio. Hnhn: Hypergraph networks with hyperedge neurons. arXiv preprint ar Xiv:20006. 12278, 2020.
15

---

Published in Transactions on Machine Learning Research (08/2024)
Boxin Du, Changhe Yuan, Robert Barton, Tal Neiman, and Hanghang Tong. Hypergraph pre-training with graph neural networks. arxiv preprint or Xrx:2:2105 00.2205.00002, 2021.
Yifan Feng, Haoxuan You, Zizhao Zhang, Rongrong Ji, and Yue Gao. Hypergraph neural networks. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 3558-3565, 2019.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the TEFE conference on computer vision and pattern recognition, pp. 580-587, 2014.
Bowen Hao, Jing Zhang, Hongzhi Yin, Cuiping Li, and Hong Chon. Pre-training graph neural networks for cold-start users and items representation. Proceedinggs of the 14th ACM International Conference on Web Search and Data Mining, pp. 265 (65-273, 2021.
F Maxwell Harper and Joseph A Konstan. The moviclens datasots: History and context. Acm transactions on interactive intelligent systems (liis) 5(4):1 19, 2015.
Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning on graphs. Tn Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 4116 4126. PMLR, 13 18 Jul 2020.
Kaiming He, Ross Girshick, and Piotr Dollar. Rethinking imagenet pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
Danfeng Hong, Lianru Gao, Jing Yao, Bing Zhang, Antonio Plaza, and Jocelyn Chanussot. Graph convo- lutional networks for hyperspectral image classification. IEEE Transactions on Geoscience and Remole Semsing, 59(6): 5978, 2020.
W HU, B Liu, J Gomes, M Zitnik, Liang, V Pande, and J Leskovec. Stratogics for pro-training graph neural networks. In International Conference on Learning Representations (ICLR), 2020a.
Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai- Wei Chang, and Yizhou Sun. Gpt-gnn: Generative pre-training of graph neural networks. In Proceedings of the 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2020b.
Dasol Hwang, Jinyoung Park, Sunyoung Kwon, KyungMin Kim, Jung- Woo Ha, and Hyunwoo Kim. Self- supervised auxiliary learning with meta-paths for heterogeneous graphs. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and Н. Lin (eds.), Advances in Newral Information Processing Systems, vol- ume 33, pp. 10294-10305. Curran Associates, Inc., 2020.
Wei Jin, Xiaorui Liu, Xiangyu Zhao, Yao Ma, Neil Shah, and Jiliang Tang. Automated self-supervisel learning for graphs. In International Conference on Learning Representations, 2022.
Sunwoo Kim, Shinhwan Kang, Fanchen Bu, Soo Yong Lee, Jacmin Yoo, and Kijung Shin. Hypeboy: Gener- alive self-supervised representation learning on hypergraphs. In The Twelfth International Conference on Learning Representations, 2024. URL http:///oopaeeeeeeeeeeeeoorrrraa net//forumidd-D2U00EEE
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXio preprint
arXiv:1412. 6980, 2014.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. International Conference on Learning Representations (ICL.R, 2017.
Alexander Kolesníkov, Xiaohua Zhai, and Lucas Beyer Revisiting self-supervised visual represcntation learning. In Proceedings of the IEEE/CVF conference on computer vision and pottern recognition, pp. 1920- 1929, 2019.
16

---

Published in Transactions on Machine Learning Research (08/2024)
Dongjin Lee and Kijung Shin. I'm me, we're us, and i'm us: Tri-directional contrastive learning hy- pergraphs. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 37, pp- 8456-8464, 2023.
Guohao Li, Matthias Müller, Ali Thabet, and Bernard Ghanem. Deepgens: Can gens go as deep as cnns?
In The IEEE International Conference on Computer Vision (ICCV), 2019.
Yujia Li, Richard Zemel, Marc Brockschmidt, and Daniel Tarlow. Gated graph sequence neural networks. In Proceedings of ICLR'16, April 2016.
Erez Lieberman- Aiden,, Nynke L Van Berkum, Louise Williams, Maxim Imakaev, Tobias Ragoczy, Agnes Telling, Tdo Amit, Bryan R Lajoie, Peter J Sabo, Michael O Dorschner, et al. Comprehensive mapping of long-range interactions reveals folding principles of the human genome. Science, 326(5950):289 293, 2009.
Claire Marchal, Takayo Sasaki, Daniel Vera, Korey Wilson, Jiao Sima, Juan Carlos Rivera-Mulia, Claudia Crevilla-Garcia, Coralin Nogues, Ebtesam Nafie, and David M Gilbert. Genome-wide analysis of replication timing by next-generation sequencing with e/l repli-seq. Nature protocols, :3(5):819 839, 2018.
Andrew Kachites McCallum, Kamal Nigam, Jason Rennie, and Kristie Seymore. Automating the construc- tion of internet portals with machine learning. Information Retrieval, 3 163, 2000.
Vit Nováček and Sameh K Mohamed. Predicting polypharmacy side-effects using knowledge graph embod- dings. AMIA Summits on Translational Science Proceedings, 2020:449, 2020.
Jiczhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. Gee: Graph contrastive coding for graph neural network pre-training. arXiv preprint ar 200:0000.99963 2020.
Sofia A Quinodoz, Noah Ollikainen, Barbara Tabak, Ali Palla, Jan Marten Schmidt, Elizabeth Detmar, Mason M Lai, Alexander A Shishkin, Prashant Bhat, Yodai Takei, et al. Higher-order inter-chromosomal hubs shape 3d genome organization in the nucleus. Cell, 174(3):744 757, 2018
Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang. Dropedge: Towards deep graph convolutional networks on node classification. In International Conference on Learning Representations, 2020.
Sai Nageswar Satchidanand, Harini Ananthapadmmanaaan, and Balaraman Ravindran. Extended discrim- inative random walk: hypergraph approach to multi-view multi-relational transductive learning. In Twenty-Fourth International Joint Conference on Artificial Intelligence, 2015.
Ke Sun, Zhouchen Lin, and Zhanxing Zhu. Multi-stage self-supervised learning for graph convolutional networks on graphs with few labeled nodes. Proceedings of the AAAI Conference on Arlificial Intelligence, 34(04):5892-5888 Apr. 2020. doi: 1.1609/aaai v.4.i.334004.604.
Liang Sun, Shuiwang Ji, and Jieping Ye. Hypergraph spectral learning for multi-label classification. In KDD, 2008.
Xiangguo Sun, Hongzhi Yin, Bo Liu, Hongxu Chen, Jiuxin Cao, Yingxia Shao, and Nguyen Quoc Viet Hung. Heterogeneous hypergraph embedding for graph classification, 2021.
Lev Telyatnikov, Maria Sofia Bucarelli, Guillermo Bernardez, Olga Zaghen, Simone Scardapanc, and Pictro Lio. Hypergraph neural networks through the lens of message passing: a common perspective to homophily and architecture design. arXiv preprint arXiv:2310.0P1 0.07684, 2023.
Ke Tu, Peng Cui, Xiao Wang, Fei Wang, and Wenwu Zhu. Structural deep embedding for hyper-networks. In AAAI, 2018.
Petar Veličkovié, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Liò, and Yoshua Bengio. Graph Attention Networks. International Conference on Learning Representations, 2018. accepted as poster.
17

---

Published in Transactions on Machine Learning Research (08/2024)
Jiancan Wu, Xiang Wang, Fuli Feng, Xiangnan He, Liang Chen, Jianxun Lian, and Xing Xie. Self-supervised graph learning for recommendation. In Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval, pp. 726 735, 2021.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. comprehensive survey on graph neural networks. IEEE transactions on newral networks and learning systems, 12(1):4-24 24, 2020.
Lianghao Xia, Chao Huang, and Chuxu Zhang. Self-supervised hypergraph transformer for recommender systems. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2100 2109, 2022.
Xin Xia, Hongzhi Yin, Junliang Yu, Qinyong Wang, Lizhen Cui, and Xiangliang Zhang. Self-supervised hypergraph convolutional networks for session-based recommendation. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 35, pp. 4503-4511. 4511, 2021.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations, 2019.
Naganand Yadati, Madhay Nimishakavi, Prateek Yadav, Vikram Nitin, Anand Louis, and Partha Talukdar. Hypergen: A new method for training graph convolutional networks on hypergraphs. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bue, E. Fox, and R. Garnett (eds.), Advances in Neural Infor- mation Processing Systems, volume 32. Curran Associates, Inc., 2019.
Liang Yao, Chengsheng Mao, and Yuan Luo. Graph convolutional networks for text classification. In Proceedings of the AAAT conference on artificial intelligence, volume 33, pp. 7370-7377, 2019.
Jachyuk Yi and Jinkyoo Park. Hypergraph convolutional recurrent neural network. Tn Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 3366-3376. 2020.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 5812-5823. Curran Associates, Inc.,
2020a.
Yuning You, Tianlong Chen, Zhangyang Wang, and Yang Shen. When does sclf-supervision help graph convolutional networks? In International Conference on Machine Learning, pp. 10871- -00800 PMLR, 2020b.
Junliang Yu, Hongzhi Yin, Jundong Li, Qinyong Wang, Nguyen Quoe Viet Hung, and Xiangliang Zhang, Self-supervised multi-channel hypergraph convolutional network for social recommendation. In Proceedings of the Web Conference 2021, pp. 413-424, 2021.
Ye Yuan and Ziv Bar-Joscph. Geng: graph convolutional networks for inferring gene interaction from spatial transeriptomics data. Genome biology, 31(1):1-16, 2020.
Muhan Zhang, Zhicheng Cui, Marion Neumann, and Yixin Chen. An end-to-end deep learning architecture for graph classification. In Proceedings of the AAAT conference on artificial intelligence, volume 32, 2018.
Ruochi Zhang and Jian Ma. Matcha: Probing multi-way chromatin interaction with hypergraph represen- tation learning. Cell Systems, 10(5):397 307, 2020.
Ruochi Zhang, Yuesong Zou, and Jian Ma. HYper-SAGNN: self-attention based graph neural network for hypergraphs. In International Conference on Learning Representations, 2020.
Vincent Zheng, Bin Cao, Yu Zheng, Xing Xie, and Qiang Yang. Collaborative filtering moets mobile recom- mendation: user-centered approach. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 24, 2010.
18

---

Published in Transactions on Machine Learning Research (08/2024)
Marinka Zitnik, Monica Agrawal, and Jure Leskovec. Modeling polypharmacy side effects with graph con- volutional networks. Bioinformalics, 34(13):i457-i46) 2018.
A Additional Related Work
Interactions naturally occur among objects in real-world datasets. The pairwise interactions are conveniently modeled as graphs, facilitating their analysis through deep learning models such as graph neural networks (GNNs) (Li et al.) 2016 Kipf & Welling 2017 Voličković et al.) 2018 Xu et al. 2019). GNNs leverage the graph structure to generale representations of unknown nodes for various downst.ream lasks such as node classification (Yao et al., 2019 Hong et al. 2020), predicting possible unknown interactions between two nodes Yuan & Bar-Joseph 2020), and classification of graphs (Zhang et al., 2018). However, real-world data often exhibit relationships that are more complex than pairwise interactions; higher-order interactions involve more than two nodes. For example, in the Cora co-citation network (McCallum et al., 2000), multiple papers are connected by citations from a common paper. Similarly, in the drug side-effect network (Zitnik) ctt al. 2018), interactions may exist among multiple drugs and side effects. Such graph structures with higher-order interactions defines as hypergraph (Bretto, 2013). extending the concept of conventional graphs with pairwise interactions. In a hypergraph, these interactions are termed as hyperodges. Being a more complex data structure, hypergraphs contain richer information comapred to graphs with only pairwise edges. However, this complexity poses challenges to classical GNN models, which requires the development of new methods to represent and learn the interactions among arbitrarily many nodes instead of just pairwise.
Reflecting on the well-explored domains GNNs, a mainstream approach for utilizing graph structures is Graph Convolutional Networks (GCNs) (Kipf & Welling, 2017). TGCNs aim to learn node representations by aggregating information from neighboring nodes and combining it with the node's own information. A detailed illustration of GCNs is Section / While GCNs have proven effective in representation learning for graphs, the difference in structures between graphs and hypergraphs poses a challenge when adapting GCNs to hypergraphs, especially in capturing the additional information brought by the hyperedges. A straightforward approach to tackle this challenge is to transform the hypergraph into conventional pairwise graph through clique expansion (Sun et al.] 2005), then applying standard GCN methods. Essentially, this entails treating each pair of nodes within a hyperedge as an edge in conventional graphs. Analogous to such expansion, simple convolutional architectures for hypergraphs have been proposed and used in several works (Feng et al.) 2019 Bai et al.] 2021). ) Feng et al.] (2019) is the first work to introduce a simple neural network structure with a convolutional layer for hyperedges.
Several other works aimed to enhance the performance of hypergraph GCNs by altering the aggregation and combination of neighboring node information. Technically, this variety stems from different designs of the hypergraph's Laplacian matrix (elaborated in Section 2 Yadati et al. (2019) also approximated hyperedges with a set of pairwise edges, proposing the HyperGCN scheme that achieves the best result and FastHyperGCN that trains faster albeit with compromised performance. Subsequently, to better capture hyperedge information, Dong et al. (2020) proposed Hypergraph Networks with Hyperedge Neurons (HNHN), hypergraph convolutional network applying nonlinear activation functions to both nodes and hyperodges. This model computes embeddings for nodes and hyperedges. However, under varied train-test data splits on multiple hypergraph data, these methods could exhibit instability and perform worse than a common GCN learning on hypergraphs expanded as regular graphs. Other attempts to adapt graph architectures to hypergraphs either achieved similar or slightly improved performance compared to HyperGCN (Arya et al., 2020 Bai et al. 2021 Yi & Park 2020) or were evaluated under different conditions such as a larger amount of training data (Sun et al.) 2021
Expansions like clique expansion assume a hyperedge's decomposability, implying that any subset of nodes in hyperedge can form another hyperedge. However, this assumption becomes unreasonable when hy- peredge contains nodes of different types. For instance, hyperedge may represent a triplet relationship between a user, drug, and reaction. Treating such a hyperedge as pairwise edges could be detrimental to downstream tasks. This issue was similarly noted in Tu et al. (2018). where the authors attempted to model the hypergraph with a simple encoder-decoder model. Yet, as evidenced in our empirical analysis for the multiway chromatin interaction data, such a model does not match the capability of a deep hypergraphgraph convolutional model.
19

---

Published in Transactions on Machine Learning Research (08/2024)
While relatively few works explore self-supervised training or pre-training hypergraphs, the field of graphs has seen many mature works. Comprehensive studies (Hu et al. 2020a You et al.
2020b Wu et. al.
2021
Jin et al.
2022) have introduced and examined multiple self-supervised tasks for graphs at both the node
and graph level. Additionally, some works (Hu et al. 2020b Hwang et al. 2020 Hao et al.) 2021 Sun
et al. 2020 were tailored for specific set tings, employing one or more similar sell sell-supervised tasks to enhance model performance and adapt to chosen scenarios. In another subfield of self-supervised training, contrastive
learning for GNNs has also been explored in recent studies (Hassani & Khasahmadi] 2020;
You al. 2020a
Qiu et al. 2020
B Experimental Details
B.1 Model Parameter Setting in Pre-training
For PhyGCN, consider three hypergraph convolutional layers, each with uniform hidden layer size of 128 and an output embodding size of 64. In the multi-hoad attention layer used for pre-training, we set the nunber of heads to 8, aligning with setting in Zhang et al. (2020). The number of training epochs is set to be 300, and the training batch the pre-training task contains 32 positive samples while we generate 5 negative samples with regard to each positive sample. We use Adam optimizer of learning rate 0.001 with weight decay 5e-4. The dropout rate is set to 0.5, and the coefficient for dropedge to 0.3: masking 30% of the values of the adjacency matrix A at each training iteration.
B.2 Supplementary Information for the Citation Networks Data
Baselines. We compare approach with two recent baselines on hypergraph neural networks alongside an additional bascline derived from modifying the graph convolutional network:
HyperGCN, proposed by Yadati et al. 2019 represents their best performing model. This method incorporates node features when calculating the Laplacian to improve performances. However, such computation significantly increases the computation cost for the method.
HNHN, introduced by Dong et al. (020); divides the aggregation process into two steps, computing embeddings for hyperedges and nodes separately.
GCN, here we expand the hypergraph into graphs with pairwise edges through clique expansion and apply a straightforward GCN model Kipf & Welling (2017) for prediction.
Training. All baselines are trained using the Adam optimizer Kingma & Ba 2014 specified in their respective papers. The selected hyperparameters for each baseline also adhere to those suggested in the respective papers. For PCyG we use SGD of learning rate 0.003 with momentum 0.9 for the node classification task.
Detailed Empirical Results Detailed results of the average test accuracy + standard deviation for all models on all datasets are shown in Table 3 We found that PhyGCN outperforms all baselines on each dataset regardless of the training data ratios. On smaller hypergraphs like Cora and Citeseer, PhyGCN is able loverage more information with self-supervised lcarning and outperforms all basclines by a large margin. Similarly, in scenarios with low training data ratio on large networks, PCyGGN maintains remarkable robustness and exemplary performance, while other hypergraph methods such as HyperGCN and HNHN falter. An ablation study of the pre-training scheme presents results of our hypergraph convolutional network both with and without pre-training, A direct comparison shows the substantial benefit of pre-training. For instance, on the Cora dataset, where the plain model underperforms when compared to the vanilla GCN, pre-training enables the hypergraph convolutional network to better capture higher-order information, and significantly outperforming GCN.
A Ablation Study on Base Network. In table g we additionally investigate the effect of changing our base hypergraph convolutional network to one that is used in HyperSAGNN (Zhang et al. 2020) (encoder- based). We note that, HyperSAGNN is a method designed for hyperedge prediction, and is therefore only considered a baseline hyperedge prediction tasks. The investigation downstream tasks such
20

---

Published in Transactions on Machine Learning Research (08/2024)
Training Ratio
Dataset
Model
0.5%
1%
2%
4%
8%
16%
HyperGCN
30.71 # 7.0
41.02 + 10.6
51.50 + 8.5
63.10 = 4.6
68.93 = 2.1
71.10 = 1.2
HNHN
26.33 + 7.5
28.61
5.9
31.65
1 6.9
15.35 + 6.5
53.62 - 5.2
58. 12
2.1
Citeseer
GCN
32.66 = 10.4
44.02
= 9.7
53.91
+ 6.4
62.54 = 3.7
69.70 =
71.47 -
PhyGCN (plain)
32.74 6.2
38.99
6.5
49.21
0.5
56.72 =
PhyGCN
47.35 8.4
54.22
+ 7.2
62.87 + 4.3
67.02 + 1.0
70.07 1.3
71.62
HyPerGCN
33.08 8.6
39.50 4.7
49.24 + 4.8
57.36 + 5.2
65.38 + 3.8
70.39
3.5
Cora
HNHN
30.48 +: 5.5
38.23 +: 10.4
53.53 + 4.0
63.51 + 3.1
70.01 + 2.2
74.51 + 1.8
(co-authorship)
GCN
51.31 = 7.4
60.35 + 5.3
69.00 = 2.2
73.48 = 1.3
76.01 + 0.8
PhyGON (plain)
38.21 + 7.6
50.21 + 5.6
59.66 + 5.7
68.53 + 2.6
71.73 + 1.4
78.99 + 1.5
PhyGON
49.43 6.5
60.75 + 5.8
67.21 + 3.6
71.86 + 1.9
75.93 + 1.3
78.39 0.8
HyperGCN
33.06 6.2
37.31 7.3
45.97
56.72 =
65.12
Cora
HNHN
26.16 4.5
28.44 4.7
38.51 + 7.5
55.60 + 3.3
64.13 + 1.8
65.29 + 1.6
(co-citation)
GCN
39.52 :: 9.7
48.37 = 6.8
58.72 + 7.0
71.15 - 3.5
74.28 + 1.2
76.83 - 1.3
PhyGCN (plana)
33.37 + 6.6
38.31 5.6
48.77 + 5.5
62.28 = 3.2
68.51 + 1.3
76.18 = 1.0
PhyGCN
48.82 8.2
58.46 + 5.5
65.71 + 4.5
72.35 + 3.2
77.33 + 1.5
79.34 0.8
HyperGCN
65.35 4.3
66.30 = 6.2
63.79 1 4.7
65.84 = 6.4
70.72 = 5.2
73.93 - 1.5
HNHN
74.41 1.9
79.86 0.9
82.19 + 0.4
83.73 + 0.2
84.66 +
85.05 + 0.2
DBLP
GCN
81.11 2.9
83.65 2.0
85.32 0.7
85.84 =
86.51 0.2
86.82 0.3
PlyGCN (plain)
81.90 = 0.6
83.71 = 0.4
85.21 + 0.2
86.52 = 0.2
86,95 = 0.3
87.43 = 0.2
PhyGCN
84.69 + 0.6
86.12 + 0.4
86.96 + 0.2
87.80 + 0.2
88.19 + 0.2
88.23 + 0.1
Hy HyPerGCN
67.84 = 6.3
72.29 = 5.4
77.10 + 2.1
79.63 = 1.0
82.05 + 1.0
83.56 + 0.6
HNHN
63.61 2.9
68.86
3.9
72.24 + 3.2
76.49 + 1.6
79.72 + 1.2
80.74 + 0.9
PubMed
GON
70.29 4.4
74.31
2.8
77.92 +
79.66 +
80.81
81.51
PhyGCN (plain)
70.96 3.7
75.07
2.2
78.01
79.84
PhyGCN
74.66 = 3.9
78.00
+ 1.6
79.09
+ 1.6
79.96
+ 0.9
82.08 + 0.6
82.84 + 0.5
Table 3: Test accuracy for different datasets at different data split ratio (training data % = 0.5% 1% 2%
4%, 8%, 16%). We report mean Lest accuracy + standard deviation among 10 random train-lest split for each split ratio.
node classification is considered as an ablations study on the benefits of different base networks for feature learning. In the later sections, we further include a comprehensive discussion on the effect of the two different base networks (encoder vs. hypergraph convolution) on hyperedge prediction (Table 10 and
Training Ratio
Base
0.5%
1%
2%
4%
8%
16%
Encoder
45.89 + 7.9
51.76 + 7.3
61.04
+ 4.1
66.85 + 1.9
69.97 + 1.3
71.55 + 0.8
Conv
47.35 + 8.4
54.22 + 7.2
62.87 + 4.3
67.02 + 1.0
70.07 + 1.3
71.62 + 1.0
Table 4: Test accuracy for Citeseer at different data split ratio (training data % = 0.5%. 1% 2% 4%, 8%, 16%). We report mean test accuracy + standard deviation among 10 random train-test split for each split ratio.
B.3
Supplementary Information for the Multiway Chromatin Interaction Data
Baselines. For the hyperedge prediction task on multi-way chromatin interaction, we evaluated against a recent work, MATCHA (Zhang & Ma 2020), which employs hypergraph representation learning techniques on a specially constructed hypergraph from multi-way chromatin interaction data, aimed denoising the data and making de-novo predictions of yet undetected multi-way chromatin interactions. For the bascline of Fig. 4c and Fig. 4d, used A/B compartment scores (Lieberman-Aiden et al.) 2009),, a 3D genome features that is known to correlate with DNA replication timing. The A compartment scores are calculated using the Cooler software [Abdennur & Mirny 2020 by taking the SPRITE contact map as input.
Dataset. To systematically evaluate if our proposed new approach could improve the analysis of multi-way chromatin interaction data, use the SPRITE data (Quinodoz et al., 2018) from the GM12878 lymphoblas- toid human cell line following the same processing procedure in MATCHA (Zhang & Ma. 2020). Specifically, genomic bins at the 1Mb resolution are considered as nodes, and interactions involving multiple loci are treated as hyperedges. In Fig. 4a, a split. by chromosome indices is performed, where nodes odd chrorossome indices constitute the training data, and those from even indices form the test data. For cross-validation a swapping of the training and test set is conducted, followed by the reporting of the average accuracy for each method. Fig. 4b displays a comparison against MATCHA (Zhang & Ma (020) under a more challenging training split, constructing the hypergraph from SPRITE data with 20% designated training set and the
21

---

Published in Transactions on Machine Learning Research (08/2024)
Accuracy
Plain Pre-trained
28.42
80.79 %
Table 5:
Classification on GM12878.
MATCHA
Our Method
Size= 2
(0.898, 0.787)
(0.934, 0.827)
Size= 3
(0.988, 0.962)
(0.996, 0.977)
Size= 4
(0.996, 0.981)
(0.998, 0.992)
Table 6: Hyperedge prediction on SPRITE data. Size denotes the length of the hyperedge. We report the (AUROC, AUPR) scores.
remaining 80% as test data. Fig. 4c and Fig. 4d, the replication timing for each genomic bin is quantified using Repli-seq [Marchal et al. 2018
Detailed Empirical Results. Table shows the detailed results for Fig. 4a. Table shows the detailed results for Fig. 4b.
B.4
Supplementary Information for Polypharmacy Side Effect Data
Baselines. We compared our approach with several methods on polypharmacy side effect prediction:
Decagon, proposed by Zitnik et al. (0088 as the initial work that studies the polypharmacy side- effect data.
ComplEX, proposed in Nováček & Mohamed (0020) that has the best reported performances.
HyperSAGNN, with the encoder-decoder model for representation learning proposed in Zhang et al. (2020).
Dataset. We use the polypharmacy side-effect dataset Zitnik al. 2018 which contains 645 unique drugs and 63,473 drug-drug interactions. Within the hypergraph framework, there are 4,651,131 hyperedges representing the drug-drug side-cffect associations, featuring 1,317 unique side effects. The evaluation on side-effect prediction was conducted across four distinct train-test split scenarios:
Original: The same as in Zitnik et al. (2018), where the test set only contains side effects associated with more than 500 drug combinations, whilst side-effects liked with less than 500 drug combinations are allocated to the training set.
Random: All side-effects are utilized for both training and testing, distributed randomly across training and test sets.
Expected: A subset of 30 side-effects, already associated with a single drug, were deemed as expected, leading to 20,787 expected drug-drug side-effect associations. Despite adhering to a random split for training and test data, the "expected" scenario only tests on the expected triplets within the test set.
Unexpected: Similarly, whilst maintaining a random split for training and test data, testing is conducted solely on the unexpected triplets in the test set.
For the side effect categorization task, we consider side effects with known categories from Zitnik et al. 2018), yielding 561 side effects with 37 categories.
Detailed Empirical Results. In Table and Table g we show detailed results of the AUROC and AUPR scores for all models on the four different evaluation sets. The AUROC scores in Table S correspond Fig. 4 and the AUROC scores Table @ableecooreee correspond Fig. Ar. The AUPR scores of Table showed similar pattern as AUROC scores, where hypergraph methods tend to maintain their performance on expected and unexpected test data while graph methods exhibit decline in performance. Similarly, for the
22

---

Published in Transactions on Machine Learning Research (08/2024)
Model and training scheme
Accuracy
Plain
21.49
With pre-training, random negative sampling
33.65
With pre-training, modified negative sampling
35.23 %
Table Classification of side-eflects.
Deragon
ComplEX
Hyper-SAGNN
Our Method (0.945, 0.931)
Original
(0.814, 0.737)
(0.929, 0.914)
(0.938, 0.921)
Random
(0.801, 0.729)
(0.920, 0.903)
(0.933, 0.912)
(0.943, 0.926)
Expected
(0.807, 0. 0.733)
(0.943, 0.931)
(0.938, 0.924)
(0.947, 0.932)
Unexpected
(0.786, 0.713)
(0.917, 0.890)
(0.935, 0.918)
(0.944, 0.922)
Table 8: Performances on different data split. Random negative sampling. We report the (AUROC, AUPR) scores for cach model on each data split.
AUPR scores of Table g the enhanced negalive sampling strategy brings more challenge to the task and forces the networks to learn the more subtle differences between positive and negative samples.
Our results demonstrated the adeptness of our model in learning higher-order interactions effectively. We also studied how learning within the hypergraph structure could improve our model's performance on lasks pertinent to the nodes (drugs, side effects). We used the side effect category information from Zitnik et al. (2018), where a proportion of the side effects within the network were classified into high-level categories. As shown in Table 7 with pre-training on the hypergraph structure, our model exhibited enhanced cate- gorization capabilities, even in the absence of additional information. Furthermore, the performance metric could be improved through the implementation of a better negative sampling strategy.
B.5 The Base Convolutional Model Learns Hyperedge Information Well
We further evaluated how well the hypergraph convolutional network in PhyGCN captures the hyperedge information. We conducted evaluation on the the four datasets used in the most current works on hyperedge prediction TTu et al. 2018 Zhang et al. 2020). These datasets, shown below, have uniform-length hyperedges and no node features or node labels.
GPS [Zhngg et al. 2010 hyperedges represent (user, location, activity) relations.
MovicLens (Harper & Konstan
2015):
hyperedges represent (user, movic, tag) relations.
drug: hyperedges represent (user, drug, reaction) relations.
wordnet (Bordes et al. 2013): hyperedges represent (head entity, relation, tail entity) relations within words.
Hyperedge prediction methods need to use structural information as the features to make predictions. More- over, these datasets are heterogencous, meaning that the node types within a hyperedge are different and therefore one could not simply expand the hyperedge into pairwise edges. Details of the experiment results
are reported in Table 10 We evaluated our base convolutional model against the encoder-decoder model
used in Tu et al. 20118 Zhang et al. (2020) for representation lcarning, using the same attention network
from Zhang et al. (2020) for combining the node embeddings and making the prediction. The performance
is evaluated by the AUROC and the AUPR score. We found that, with the introduction of convolutional
architecture, our model performs either better than or on par with the encoder-decoder model. We note that the performances for the four datasets are already very high with the encoder-decoder model, our base model still exhibit enhanced capability in capturing the hyperedge information.
ComplEX
Hyper-SAGNN
Our Method
Original
(0.724, 0.702)
(0.716, 0.702)
(0.744, 0.731)
Random
(0.723, 0.699)
(0.715, 0.703)
(0.743, 0.730)
Expected
(0.739, 0.722)
(0.719, 0.709)
(0.746, 0.732)
Unexpected
(0.703, 0.686)
(0.718, 0.703)
(0.742, 0.729)
Table 9: Performances on different data split. Modified negative sampling. We report the (AUROC, AUPR) scores for each model on each data split.
23

---

Published in Transactions on Machine Learning Research (08/2024)
GPS
MovieLens
drug
wordnet
Encoder
(0.930, 0.744)
(0.926, 0.793)
(0.961, 0.888)
(0,890, 0.694)
Conv
(0.941, 0.738)
(0.938, 0.801) (0.962, 0.890)
(0.896, 0.710)
Table 10: The experiment results for the hyperedge prediction task over the four heterogeneous hypergraph data with uniform-lenge hyperedges. We report (AUROC, AUPR) to evaluate how well our model learns the structural knowledge. Our convolutional model is compared with the encoder-decoder model used in Zhang et al. (2020), both using the atlention layer for prediction.
Citeseer
Cora (co-authorship)
Cora (co-cocitation)
DBLP
PubMed
Encoder-1
(0.804, 0.573)
(0.827, 0.626)
(0.809, 0.591)
(0.825, 0.655)
(0.906, 0.722)
Encoder-2 (0.833, 0.612)
(0.798, 0.553)
(0.849, 0.590)
(0.920, 0.727)
(0.919, 0.730)
Encoder-3 (0.803, 0.583)
(0.826, 0.660)
(0.839, 0.607)
(0.930, 0.773)
(0.920, 0.752)
Conv (0.866, 0.605) (0.861, 0.657) (0.861, 0.597)
(0.923, 0.755)
(0.926, 0.727)
Table 11: The experiment results for hyperedge prediction over the five homogeneous datasets with variable- length hyperedges. We report (AUROC, AUPR) similarly. We compare our convolutional model with the encoder-decoder model that takes in three different features. Encoder-1: original features computed as in Zhang et al. (2220). Encoder-2: node features provided by the data. Encoder-3: concalenation of the two.
In Table 11 we extended our evaluation to five datasets for the main task of node classification. These datasets are citation networks where each node represents a publication and each hyperedge represents either co-citation or co-authorship relation. These data are different from the above setting, where the hypergraph is homogeneous with variable hyperedge size. We note that, for the previous hyperedge prediction data, the methods typically features constructed from the hypergraph structure: detailed formulation of the construction of input features are presented in Sections and However, these five datasets have node features themselves, additionally add two modified baselines of the encoder-decoder model for a more comprehensive evaluation. Specifically, we compare with encoder-decoder that takes in the constructed features, the node features, and concatenation of the two. Our model soley utilizes the node feature for these five datasets. We found that our convolutional base model outperforms the three baselines on AUROC score consistently across the five datasets. Such results underscore the potential of our base model effectively and efficiently utilize both the node features and the structural information of the hypergraph. For the hyperedge prediction task only, the base convolutional model has stably better performance compared to the simple model used in previous works. Such evaluation confirms that the model learns the structural knowledge well in this pre-training task.
24


Reviews:
[{"review_id": "qD7LmMDndP", "reviewer": "Reviewer_scvy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Despite the facts that the paper is well-written, easy to follow, has discussed an important problem, and is also interesting for at least some individuals in TMLR's audience, there are some unclear points that require more discussion:\n\n1. The paper lacks clear research questions: Generally, the paper is not well-motivated and requires more discussion on the limitations of existing methods. What are the main messages of this study? Why do we need PhyGCN, and what does it have that existing methods don’t? I realized that the authors several times discussed the importance of predicting variable-sized hyperedges.  While it had been a serious challenge back in 2021-2022, several existing methods are capable of this. \n\n2. There is a lack of discussion on recent studies! This paper has been submitted in 2024, and there is no discussion on even a single paper from 2023. This lack of discussion on recent studies has caused several unclear points about the main messages of this paper. For example, several recent methods are capable of self-supervised training and also can be generalized to unseen data, see [1, 2, 3]. These studies also are capable of handling hyperedges with variable sizes. Other claims are also discussed in several studies. For example, [1] also theoretically discussed the fact that using simple averaging of nodes’ encodings is not enough for hyperedge prediction tasks. \n\n3. As discussed above, since there are no clear research questions in the paper, it is hard to evaluate if the experiments support the claims or not. \n\n\n--- \n[1] CAT-Walk: Inductive Hypergraph Learning via Set Walks. Behrouz et al. NeurIPS 2023.   \n[2]  Hypergraph neural networks through the lens of message passing: a common perspective to homophily and architecture design. Telyatnikov et al. 2023.  \n[3] I’m me, we’re us, and i’m us: Tri-directional contrastive learning on hypergraphs. Lee and Shin. AAAI 2023.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This study presents a new hypergraph convolutional neural networks as well as a pre-training and self-supervised learning framework, called PhyGCN. PhyGCN is based on a self- supervised task that predicts masked hyperedges from observed ones. To this end, it uses an attention module to predict variable-sized hyperedges.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Detailed discussion of **recent** related work.\n2. Clearly mention the main messages of this paper and discuss how this method can address existing limitations of **recent** studies. \n3. Clearly mention that which of the experiments are designed to support the above messages.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EVi3XBEFIh", "reviewer": "Reviewer_d2MZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is overall well written, except from a few points (see below)\n- Experiments are sound. (I have a few important demands, see below).\n- Methodology is serious too (e.g. using the same train/val splits to compare various methods)\n\nWeaknesses\n- The paper novelty is quite limited, but credit is given appropriately, so it seems fine (I didn't search to check if other works applied SA-GNN as a pretraining step for hypergraphs related tasks).\n- Maybe hypergraphs is a niche subject ? But it sounds general enough to be relevant to TMLR.\n- some sentences are useless and carry no information, are tiring to read. Please remove them (see below)\n\n\nComment:\n- Skip/Dense connection (to avoid vanishing gradient): What about a ResNet structure ? Did you try that ?\nYour architecture is nice in terms of less resources used, but then, you are demanding the network to produce intermediate representations that are useful for the (pretext) task, instead of allowing intermediate steps to be hard-to-decode, but maybe useful to build high-order representations.\nAnyway, that's your choice.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this work, the orignal hypergraph-able GNN called Hyper-SAGNN (Zhang 2020) and the corresponding hyperedge prediction task (including the negative sampling strategy for performing SSL) are used as pre-training for improving prediction accuracy on various downstream tasks.\nIt is shown that the pre-training improves the accuracy.\n\nFurthermore, a few points seem to be original:\n- The way to fight vanishing gradient is somehow original but has most liekly been used before elsewhere (it's not an auxiliary loss but close to it, as each layer's representation is concatenated for final decoding).\n- Also, for the pretraining, Dropout is replaced with DropHyperEdge (slightly different from the previously known DropEdge).\n- A new negative sampling strategy, specific to the hyperedge geometry, is introduced, and studied empirically.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Important points:\n- In fig 3, why is SAGNN (Zhang 2020) not presented ? It seems to me it must be, for fair comparison.\n- In the experiment section, the size of the pre-training dataset is not mentionned. For instance, when training with 0.5% of the available data, how is the pre-training done ? With the same amount of raw hypergraphs, but using numerous randomized samplings to self-supervise in many ways ? Or, was a larger corpus of hypergraphs used ?\nThis is a very important point, that must be mentionned explicitly, regardless of the choice you made.\n- Why limit to 4% of the data at maximum ?\nI guess PhyGCN becomes less good than other models when data is abundant. That's ok. I think it would be more honest, and interesting, to report also larger values of N_train.\n- The pre-training loss is not written down in the main text. It should be, with a clear role of the positive/negative samples. This is important because in a first read, it seems the only task is to recover missing Hyperedges (as a pretext task).\n- The outcome of the experiment presented in fig 5 is insufficiently commented on. As I read it, it goes like this:\n- AUROC means larger is better.\n- you perform the pretext task of hyperedge prediction using 4 different methods, for 2 kinds of negative sampling.\n- Results are WORSE using your enhanced sampling strategy.\n- conclusion: the enhanced sampling is more challenging to figure out, thus, a better way to pre-train.\n- side remark: PhyGCN is not affected by the affected/unaffected split, whereas other methods are, a bit, which means that PhyGCN is especially good when hyperedges are not simply two-node edges.\nIf I'm correct, good for me, but, it'd be better to be slightly more explicit about the reasoning. You never write explicitly what is the task performed by these nets, in ML terms. (I can't read the biology jargon). Also the conclusion you draw is not clear.\nThis is a pity because as I gather, this original negative sampling strategy is one of the genuinely new ideas presented in the paper.\n\nModerately important points:\n- First read: it's not immediately clear what you mean by variable-sized hyperedge prediction (I mean, why is it non trivial, why and how is attention used/helpful for that?)\nMaybe a small comment to outline the intuition would be helpful.\n(After looking at the equations again, and remembering that attention can be input-size-independent, it makes perfect sense - but one has to do the reasoning by themselves).\n- Negative sampling strategy:\n- I do not fully understand the last Negative sampling strategy, (text just a bit before Eq 8). It's not very important, but probably it could be explained more clearly.\n- Again, Enhanced negative sampling: figure 5:\nhonestly, this is still not very clear to me.\n- BUT, I did understand when reading \"To address this, we propose a stronger criterion for negative sampling, which involves\nsampling a drug combination that is known to cause other side effects and combining it with the given\nside effect as a negative sample.\"\n\n\nMinor points:\n\n- AUROC and AUPR are acronyms and are not defined.\nArea Under the ROC curve ?\nAUPR : ??\n\n- fine-tuning: it is sometimes slightly unclear whether fine-tuning means training the decoder (h) weights only, or also those of the base model f (and/or those of the attention network g).\nIt would be better to explicitly state \"keeping the base model fixed\" or \"further training the base model\" each time.\nIndependently from that, I was unable to understand how the results fo fig 4d, 4e were obtained, from the following sentence (which clearly aims at being explicit, but.. I failed to understand):\n\"We then fixed the base model and fine-tune the\nfully connected layer to predict replication timing of genomic loci in the GM12878 cell line. Next, with the\nattention network fixed, we fine-tuned the base model with the task of\"\n\n- The adjacency matrix $A=H.W.H^T$ corresponds to a graph where any two nodes connected by an hyperedge are now connected by an edge. Correct ?\nThis is quite a reduction of the hypergraph.\nMaybe a small comment about this would be helpful for the reader.\n\n- sec 3.1 : you could insert somewhere, at beginning of a paragraph \"Here we recall the basic steps of SSL applied to our hypergraph problem, for completeness\". I say this because there is basically no new information in that part, for a reader who knows the basic working of SSL.\nYet, I found it's well written and pedagogical: I would keep it, but put this disclaimer at the beginning, to save time for most readers.\n\n- \"By integrating the model architecture\nand pre-training scheme, PhyGCN effectively learns from a hypergraph structure and applies the learned\nknowledge to downstream tasks.\"\n-> useless, informative-less sentence. I suggest to delete.\n\n- \"Based on the pre-trained model f (·) that generates node embeddings with the information\nit leveraged from the pre-training task, we can utilize it to do node-level downstream tasks in a wide range\nof applications.\"\n-> again, useless sentence.\n\n- page 6, top, typo $E'_valid$ and $E'_valid$\n\n- I infer from the equation of page 6 (top) that z_i is the binary variable that identifies the positive and negative samples (hyperedges belonging to the positive or negative set, for a given hypergraph).\nIt would be easier to write it down.\n\n- continual or continuous ? Define continual.\n\n- by the red arrow\n->\nby the blue arrow\n\n- typo, titles of fig 5a, 5b:\nAUROC score on different datasets\n->\nAUROC score with different models", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YZ827NDo6v", "reviewer": "Reviewer_ySwi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengthes:\n\nS1. This manuscript systematically shows how to pretrain hypergraphs and how to use pretrain hypergraph embeddings to enhance the performance of downstream tasks.\n\nS2. Overall, this manuscript is well-structured and well presented, providing a detailed description of the proposed method.\n\nWeaknesses:\n\nW1. The proposed method has limited technical contributions. From a broader view of graph neural network research, the used techniques, such as the link prediction style pretraining, MLP finetuning and negative sampling strategy, come from existing experience.\n\nW2. I have several concerns about the experiments. It seems (from the related work section) that there are some hypergraph pretraining methods in the literature, such as Du et al. (2021) , but they were not selected as baselines. Besides, it is not clear to me how to extend the proposed method to unseen graphs. It seems that in the experiment \"Transferring knowledge to a different hypergraph structure\", the base model is finetuned using the target data, which is different from the conventional setting of inductive learning for graphs. More explanations or details are necessary.\n\nW3. From the related work section, it seems that there are hypergraph neural networks and hypergraph neural network pretraining methods in the literature. What is the key contribution of the proposed method compared to the existing work? It would be better to highlight the difference between the proposed method and other work.\n\nW4. I also have a concern regarding the used datasets. In my view, the used the citation data (such as DBLP) is a multi-relational graph, where each edge links to nodes and has a label denoting the relation type. Is a multi-relational graph also a hypergraph? It would be be better to add more introductions about the datasets, including what is the data structure and node feature.\n\nW5. The proposed hypergraph neural network may face scalability challenges when dealing with large hypergraphs. It represets the graph using an incidence matrix $H \\in R^{N×M}$, with N as the number of nodes and M as the number of hyperedges. In general, the number of edges is much larger than the number of nodes in a graph. The computational cost increases as the number of nodes and hyperedges grows.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This manuscript proposes a graph neural network pretraining method for hypergraphs. In a hypergraph, each hyperedge involves more than two nodes. The proposed hypergraph neural network represents a hypergraph using an incidence matrix and uses a GCN-style aggregation scheme to encode the graph. To train the hypergraph neural network in a self supervised learning manner, the manuscript proposes hyperedge prediction as the pretraining task, resulting in enhanced node representations. Given a downstream task,  such as node classification, the pretrained node representations are fed into a MLP finetuned for prediction. Experiments on the node classification task show that pretrained node representations can improve the performance of the task.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Below is my suggestions to improve the manuscript:\n\n(1) Add more GNN or pretraining baselines in the experiments to make the work more solid (if possible).\n\n(2) Add more details on the used datasets.\n\n(3) Add more explanations about the experiment \"Transferring knowledge to a different hypergraph structure\".\n\n(4) Highlight the difference between the proposed method and existing work (in terms of techniques).", "broader_impact_concerns": "NA.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers acknowledge that while this work may lack novelty, it presents a nice combination of existing ideas, offering statistical distinctions from current hypergraph pre-training efforts. The research questions tackled are significant for a variety of hypergraph-related downstream tasks. The authors have effectively demonstrated use cases and provided comprehensive dataset descriptions. All claims have been substantiated, and the distinctions between this work and existing literature are well delineated after revision. Therefore, I recommend acceptance.
Decision: Accept (poster)