SHINE: SubHypergraph Inductive Neural nEtwork
Authors: Yuan Luo
Venue: NeurIPS 2022 Accept (2022)

Abstract: Hypergraph neural networks can model multi-way connections among nodes of the graphs, which are common in real-world applications such as genetic medicine. In particular, genetic pathways or gene sets encode molecular functions driven by multiple genes, naturally represented as hyperedges. Thus, hypergraph-guided embedding can capture functional relations in learned representations. Existing hypergraph neural network models often focus on node-level or graph-level inference. There is an unmet need in learning powerful representations of subgraphs of hypergraphs in real-world applications. For example, a cancer patient can be viewed as a subgraph of genes harboring mutations in the patient, while all the genes are connected by hyperedges that correspond to pathways representing specific molecular functions. For accurate inductive subgraph prediction, we propose SubHypergraph Inductive Neural nEtwork (SHINE). SHINE uses informative genetic pathways that encode molecular functions as hyperedges to connect genes as nodes. SHINE jointly optimizes the objectives of end-to-end subgraph classification and hypergraph nodes' similarity regularization. SHINE simultaneously learns representations for both genes and pathways using strongly dual attention message passing. The learned representations are aggregated via a subgraph attention layer and used to train a multilayer perceptron for subgraph inferencing. We evaluated SHINE against a wide array of state-of-the-art (hyper)graph neural networks, XGBoost, NMF and polygenic risk score models, using large scale NGS and curated datasets. SHINE outperformed all comparison models significantly, and yielded interpretable disease models with functional insights.

SHINE: SubHypergraph Inductive Neural nEtwork
Yuan Luo Feinberg School of Medicine Northwestern University Chicago, II. 60611 yua..Ioo@horttwesteen. edu
Abstract
Hypergraph neural networks can model multi-way connections among nodes of the graphs, which are common in real-world applications such as genetic medicine. In particular, genetic pathways gene sets encode molecular functions driven by multiple genes, naturally represented as hyperedges. Thus, hypergraph-guided embedding capture functional relations in learned representations. Existing hypergraph neural network models often focus on node-level or graph-level infer- ence. There is an unmet need in learning powerful representations of subgraphs of hypergraphs in real-world applications. For example, a cancer patient can be viewed subgraph of genes harboring mutations the patient, while all the genes are connected by hyperedges that correspond to pathways representing spe- cific molecular functions. For accurate inductive subgraph prediction, we propose SubHypergraph Inductive Neural nEtwork (SHINE). SHINE uses informative ge- netic pathways that encode molecular functions as hyperedges to connect genes as nodes. SHINE jointly optimizes the objectives of end-to-end subgraph classifica- tion and hypergraph nodes' similarity regularization. SHINE simultaneously learns representations for both genes and pathways using strongly dual attention message passing. The learned representations are aggregated via subgraph attention layer and used to train a multilayer perceptron for inductive subgraph inferencing. We evaluated SHINE against a wide array state-of-the-art (hyper)graph neural net- works, XGBoost, NMF and polygenic risk score models, using large scale NGS and curated datasets. SHINE outperformed all comparison models significantly, and yielded interpretable disease models with functional insights.
1 Introduction
Hypergraph neural networks have recently emerged as a series of successful methods to model multi-way connections that are beyond pairwise associations among nodes of the graphs. Multi-way connections are common in many real-world applications and, in particular, genetic medicine. From genetic medicine's perspective, pathways broadly speaking gene sets encode the relationship among multiple genes that collectively correspond to molecular function [1] which can be used in machine learning models account for disease mechanisms more intuitively and accurately than individual genes [2). Genetic pathways or gene sets encode functional relations among multiple genes (see Appendix for detailed explanations), which can be naturally modeled as hyperedges connecting all involved nodes (c.g., genes). Thus, hypergraph-guicn embedding can capture functional relations learned representations.
Existing hypergraph neural network models often adopt the semi-supervised learning (SSL) paradigm to assign labels to initially unlabeled nodes in hypergraph 5 Other methods have focused
learning graph representations 66, [7]. Node-level and graph-level representations give either local overarching views of the graphs, i.e., at the two extremes of hypergraph topological structures.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

There is an unmet need in learning powerful representations of subgraphs in hypergraphs. Such capabilities are important in real-world applications such as genetic medicine. For example, cancer patients can be viewed as subgraphs of genes that harbor mutations, while all the genes are connected by hyperedges that correspond to pathways or gene sets representing specific molecular functions. Powerful subgraph representations will lead to the capability to more accurately account for the patient's pathophysiology For regular graphs where edges connect node pairs, several subgraph representation learning algorithms were proposed, including methods that can use the learned representations to make predictions for subgraphs with fixed sizes [8] or varying sizes [9]. There are currently few if any work inductive inference for varying-sized subhypergraphs. In this work, we propose a new framework named SHINE: SubHypergraph Inductive Neural nEtwork. We share our source code at http:///gttuubcooo/uuuuaauaala
Our contributions are as follows:
To the best of our knowledge, SHINE is the first model to effectively learn subgraph represen- tations for hypergraphs, use the learned representations (for seen suhgraphs) and inductively infer representations (for unseen subgraphs) for downstream subgraph predictions.
Novel applications in the field of genetic medicine Next Generation Sequencing (NGS) datasets across diverse diseases show significant performance improvements by SHINE over wide array of state-of- the-art baselines.
In addition learning and inductively inferring subgraph representations, SHINE simul- taneously learns the representations of nodes and hyperedges. This brings interpretation advantages, allowing assessing pathways (hyperedges) correlations and reasoning on multi- ple molecular functions mutually interacting and collectively contributing to disease onset and progression.
2 Related Work
Graph Neural Networks. Graph representation learning maps graphs or their components vector representations and has attracted growing attention over the past decade. Recently, graph neural networks (GNNs), which can learn a distributed representation for graph or node in a graph, are widely applied to a variety of areas including computer vision and image processing [IO M.]. molecular structure inference [12] [33, natural language processing (14 and healthcare [17 [8]
GNN recursively updates the representation of node in a graph by aggregating the feature vectors of then be obtained through set pooling (e.g.. (20) or graph coarsening (e.g.. (21) to aggregate the node representations in the graph.
its neighbors and itself, e.g. [19]. The graph-level representations
The reader is referred to a comprehensive book 22 the topic of graph neural networks.
7 have become a popular approach for learning on multi-way relations from data. Early work on hypergraph learning, e.g., [23. formulated hypergraph message passing using spectral theory of hypergraphs. This formulation essentially adopted clique expansion to extend graph convolutional network (GCN) for hypergraph learning. Others methods applied attention mechanism to aggregate the information across the hypergraph [6 or directly learned node representations to preserve the proximity of nodes sharing a hyperedge or having similar neighborhoods [28). In both formulations, messages were passed to the node of interest from its immediate neighbors, and added layers allow propagation of messages to a farther neighborhood. A very recent model implements hypergraph neural network layers in generalized way compositions of two multiset functions that approximated by neural networks 29
Hypergraph neural network. Hypergraph neural networks 6
and its variants 3 4 27
Subgraph representation learning and prediction. Recent studies on subgraph embeddings and prediction starts with learning representations of small subgraphs. [8] encoded small fixed-sized subgraphs for subgraph evolution prediction. SubGNN [9] learned representations for varying-sized subgraphs through neighborhood, position and structure channels using random patches distributed throughout the graph. [30 31] learned subgraph representations by pooling local structures to aid the predictions of the entire graphs. Note modeling hyperedges as another type of nodes turns hypergraphs into bipartite (and heterogeneous) graphs, making SubGNN potential baseline for subhypergraph inferencing. On the other hand, most of the existing general heterogeneous graph neural network models do not support subgraph inferencing 36)
The intersection of hypergraph neural network and subgraph representation learning currently underexplored. While the above methods focus on either hypergraph learning subgraph learning,

---

Table 1: Common notations used throughout the paper.
Symbol
Definition
Symbol
Definition
H
An undirected hypergraph N,
[
Set of hypergraph nodes and size
E. [
Set of hypergraph hyperedges and size Hyperedge incidence matrix Number subgraphs
d
Hidden layer size
H
Operation composition
n
Flement-wise multiplication
none of the methods consider subgraph prediction for hypergraphs. Technically, subgraphs can be viewed hyperedge and studies link prediction could predict the existence of hyperedge [77. However, few if any such studies addressed the problems of differentiating the classes of the subgraphs, which is especially important in genetic medicine where subgraphs and hyperedges have different real-world meanings. For example, a hyperedge corresponds to a genetic pathway from curated knowledge and subgraph corresponds a patient with mutated genes as its nodes.
To sum, there is a major unmet need regarding varying-sized subgraph inference for hypergraphs, and even more so in the inductive learning setting. Our proposed framework SHINE provides an end-to-end framework that operates on hypergraphs and performs inductive subgraph inferencing.
Methods
We first outline the workflow of SHINE, see Table for symbol definitions. We develop a strongly dual attention message passing algorithm to propagate information between nodes and hyperedges, and across layers. We develop weighted subgraph attention mechanism to learn the subgraph representation by integrating representations of hypergraph nodes. We next explain each step.
Tor
3.1 Collecting Genetic Pathways
We use the Molecular Signatures Database (MSigDB) 1 and focus on MSigDB's ' curated pathway (gene set) collection, which contains human gene sets that canonical representations of biological process compiled by domain experts. There are 21,587 genes in MSigDB Pathways. Some pathways may overlap with others and have been filtered by MSigDB to remove interset redundancy. Genes with unknown functions are not included in the pathways and not used for classification, as our focus here on interpretable modeling through inferencing with known molecular functions. Adding genes with unknown functions to study their impact will be our future work.
3.2 Hypergraph Learning
We first review the hypergraph analysis basies. Different from simple graph, a hyperedge
hypergraph connects two or more vertices. A hypergraph is defined H (N, E), which includes set of nodes N, set hyperedges E. the case of genetic medicine, we model genes as hypergraph nodes, i.e., .V (91....,IW J. and pathways as hyperedges, i.e., & = {PI. )(s)) where [M], [8] are sizes of the nodes and hyperedges respectively. The hypergraph's 's topological structure can be denoted by an |M| x |g] incidence matrix H. whose entries are defined
H {
1 if node 9s hyperedge Pj 0 if node gs hyperedge Pj
Generally speaking, each node in the hypergraph may be accompanied by d-dimensional node feature/embedding matrix N E kllixx, where each row corresponds to node's feature/embeddiing The hypergraph with its topological structure and node features can be represented succinetly as H. N).
Fig. (a) shows a schematic of the constructed genome hypergraph with nodes denoted by circles and hyperedges denoted by colored arcs. While pathway can contain multiple genes, gene can also contribute to multiple pathways. That is, we can have multiple hyperedges incident on the same node (gene), can be seen in Fig. (a) nodes 722 R5, N10-

---

Srogepp adartion
122
Bungpap
ntC
startin
Cancer
100
Geneme hyporgraph with pathways as hyperscgos
Ceno pathway represemations
(a)
(b)
Figure 1: SHINE's strongly dual attention mechanism for message passing for the genome hypergraph, and its use of subgraph attention to integrate gene nodes in the feature learning for subgraphs.
3.3 Strongly Dual Attention Message Passing
Hyperedge attention over nodes. For a hyperedge Py E & in order to update its hidden representa- tion layer k. we aggregate the information from its incident nodes using the following attention mechanism. We first calculate the hyperedge attention over nodes as in
/ (Egrens exp (c" s(ps spp).ge))
ag ag (p). 9s) = exp (c's((p).)))
(2)
where e a trainable context vector and the attention ready state s(pj: 9s) for hyperedge-node pair (py.9) 9:) is calculated from the (Is - 1)th layer in
spp; 9i) LeakyReI.U (W wh (g:) bN) (Weh bE)
where * denotes element- wise product, W N and by (WE and by) are the transformation weights and bias of the nodes (the hyperedges) for the attention ready state. This is motivated by the observa- tion that different nodes (genes) contribute differently to the hyperedges (pathways), thus need proper attentions across the nodes to up- or down-weight their contributions when aggregating their representations to compute the representation of the hyperedge. Once we have the hyperedge attention over nodes, we calculate the hyperedge's representation in layer from the nodes' representations in layer (equation where is the nonlinearity layer (ReLU in experiment).
h (p,) (Eorery ar(pj.9 95) by (
(4)
Node attention over hyperedges. For a node 9, E W, in order to update its hidden representation layer k. aggregate the information from its incident hyperedges using the following attention mechanism. We first calculate the node attention over hyperedges as
aN (v(.)))) P2) = exp (cT s(p), / (Ep/ss. exp (cTs(py
(5)
where is the same trainable context vector used in hyperedge attention calculation and the attention ready state s(p,, 9i) for hyperedge-node pair (pj, ()). is calculated as in equation This allows us to have the node's. 's attention over the hyperedges, i.e., we can weight hyperedges' contributions when aggregating their representations to compute the representation of the node. We calculate the node's representation in layer i from the hyperedges' representations layer
as
(gs (En, aN (w(gi,)) P2) h b (p,)
Note that different from HyperGAT, here the calculation of hyperedge's 's and node's attentions share the same underlying dual-attention matrix as shown in Fig. (b). which is essentially unstandardized covariance matrix. Such parameter sharing across hyperedges and nodes allows us cross-regulate

---

the learning of their mutual attentions to prevent overfitting, This difference not only allows for simplification of the model, but also is more consistent with the notion of duality. The dual H of the hypergraph H is hypergraph with H's vertices and edges interchanged, and we should have (Hr)* H. easily provable that the dual-attentions for (H-*)* are the same as those for H. Such a self-dual statement is generally not true for the attentions proposed in HyperGAT due to their unsymmetrical way of calculating the node-level and the edge-level attentions, despite that the HyperGAT attention was termed as "dual" attention. For this reason, we term our attention message passing scheme as strongly dual attention message passing.
Hypergraph regularization. One important intuition about graph and hypergraph convolutional network is that the learned representations for nodes with similar context of (hyper)edges should be similar. In the case of a simple graph G = (V, E). this is to minimize the summed distance Exu.jee || hell its weighted variants. Instead of using it an explicit regularizer, graph hypergraph convolutional networks leverage an appropriately defined graph hypergraph Laplacian. As noted in [23]. the hypergraph Laplacian is A = I- O where I is the identity matrix and O is defined as (let W be diagonal matrix with diagonal entries hyperedge weights)
θ = D D 'H"D."'
Here, different from hypergraph convolutional networks, we use explicit regularization on the similarity of representations of nodes with similar hyperedge context. Let X be the matrix of the learned nodes' representations, where row X; = h (gi) and the K'th layer is the last hypergraph message passing layer. We can define the regularizer as
Lreg- = Eig ((X.X - -X + X,,3)) * O.))
Intuitively, the more hyperedges are incident on the node pair j. the more we should penalize their representational differences. On the other hand, the regularizer down-weights the penalization hyperedge connects many nodes or if node has many incident hyperedges, indicating lack of specificity for hyperedges and nodes, respectively.
3.4 Weighted Subgraph Attention
The multiple layers of strongly dual attention message passing allow learning the nodes' and hyper- edges' representations. However, the instance for the classification algorithm is a subgraph (e.g., a patient, who has mutations in multiple genes (nodes)). From the hypergraph perspective, a patient j(1 S j S n) can be considered as subhypergraph Gj whose nodes (genes) have mutations in j and are a subset of those of H. This is shown in Fig. (c) where different node colors in the hypergraph denote different patients. In order to calculate the subgraph's 's representation from its component nodes' representations at the K layer, we use the following weighted subgraph attention (WSA) mechanism, inspired by (38). In fact, none of the previous hypergraph methods support subgraph inferencing, and we had to add our WSA module to those models for subgraph inferencing as well. We first compute the subgraph attention over nodes (e.g.. 9:'s) in
1K (g:)) / (Earre, exp (M b b (gr))
a(G .i) - exp (M₃.b h
where is a trainable context vector, M is the mutation rate feature matrix with each row corre- sponding to a patient and each column corresponding gene. Thus, equation D mutation rate weighted subgraph attention mechanism. This choice conforms to the intuition that the rate of a mutation is more informative than categorical indicator of the mutation's 's occurrence. With these subgraph level attentions, we compute the patient (subgraph) representation from the Kth layer's gene representations as in
h(G),) o (En.es, a(Gy, 9t) h (a:))
(10)
We then stacked the learned patient representations to form the new patient feature matrix, as in
S = ["( h(G))" I... h(
(11)
where each row is a patient (subgraph) embedding.
5

---

3.5 Inductive Classification on Subgraphs
Let the learned feature matrix be and feed it into a softmax classifier
Z = softmax (ReLU (S) + w
(12)
where (2) in the superscript indicates two MIP layers (FC=Fully Connected layer). The loss function is defined as the cross-entropy error over all subjects in all classes as in
P= -=-E,ev.
JENDD Ej Yis In Z,s + Lreg
(13)
where V'D is the training set of subjects that have labels and F is the dimension of the output labels, which equal to the number of classes. Y is the label indicator matrix. Note that the subgraph attention layer allows us to compute any patient's representation, which effectively eliminates the need for access to test set patient features during training, making the model inductive. Existing models such HyperGCN and HGNN are transductive, we cascade our subgraph attention layer on top of these models and make them inductive to serve as our comparison models.
4 Experiments
We conducted experiments on real-world datasets in genetic medicine. Both datasets have more than 20 different classes, indicating significant complexity of the prediction tasks. These datasets are different in nature, e.g., curated from literature vs. obtained directly from high-throughput sequencing, and multi-class vs. multi-class multi-label. Our experiments genomic data call for novel methods and present unique technical challenges, in this case inductive subgraph inferencing on hypergraph. The summary statistics for each dataset shown in Table [ and the description of each dataset is as follows. Most pathways have small to medium sizes, see Table pathway sizes IQR. In fact, even at the 95th percentile, the pathway size is just over 200. On the other hand, we observed that the larger the pathway (hyperedge), the more subgraph it is incident on, and the less attention our model will give it as a discriminative feature. The DisGeNet and the TCGA-MC3 datasets are publicly available" and this study is approved by Northwestern University Institutional Review Board.
motivated by the fact that massive
4.1 Disease Type Prediction with DisGeNet Data
In this experiment, we have used the DisGeNet dataset [3] that is collection of mutated genes involved in human diseases compiled from expert curated repositories, GWAS catalogs, animal models and the scientific literature. In the following text, we abuse terminology use "gene" to really mean "variants in the gene". We model genes as hypergraph nodes and diseases as hyperedges. Each disease is labeled with one more of 22 MeSH codes, and the task is multi-class multi-label classification problem. We used 6:2:2 vrainivalid partition, and the split distribution is shown in Appendix. The DisGeNet dataset has 6226 pathways and 9133 genes involved in 8383 diseases.
4.2 Cancer Type Prediction with NGS Somatic Mutations Data
In this experiment, we have used the consensus somatic mutations for TCGA subjects produced by the Multi-Center Mutation Calling in Multiple Cancers (MC3) project [40]. Aiming to enable robust cross-tumor-type analyses, the MC3 approach applied an ensemble of 7 mutation-calling algorithms and assigned a PASS identifier to a mutation that was called by 2 or more variant callers out of the total 7 callers [4]] The MC3 approach accounted for variance and batch effects introduced by the rapid advancement of DNA extraction, hybridization- capture and sequencing over time. Following this approach, we restricted our analysis to PASS calls in order to maintain sample sizes and uniformity in mutation calling. Each subject is labeled with one of 25 cancer types, and the task is a multi-class classification problem. We used 6:2:2 train:validation.test partition, stratified by cancer types, and the split distribution is shown in Appendix. The TCGA-MC3 dataset has 6229 pathways and 18059 genes involved in 9012 subjects in total.
DisGeNett http://www.disgeeneee TCGA-MCS http:///dcccaappeooo//oooooooout
2017

---

Table 2: Real-workd hypergraph datasets for subgraph inference used in our work. For hyperedge sizes, shown are medians and interquartile ranges. From the distribution, it is clear that the hyperedge size has a positive skewness.
Dataset
# hypermodes
# hyperedges
Hyperedge size
# classes
# subgraphs
DisGeNet
9133
6226
25 (12-57) 57)
22 25
8383
TCGA-MC3
18059
6229
33 (15-1 (15-77)
9012
4.3 Baselines
We compared SHINE with the following state-of-the-are baselines, we use validation datasets to tune parameters and hyperparameters, please see Appendix for details.
Hypergraph neural networks (HGNN) uses clique expansion to transform the hypergraph to graph and uses Chebyshev approximation to derive a simplified hypergraph convolution operation.
HyperGCN 4 represents a hyperedge by selected pairwise simple edge connecting two most unlike nodes, and adds the remaining nodes in the hyperedge as mediators.
HyperGAT 7 learns node representations by aggregating information from nodes to edges and vice versa. Different from SHINE, HyperGAT uses alternating attention instead of strictly dual attention and has no regularization on nodes with similar context of hyperedges.
AlSeeTraansfomer and AltDeepSets respectively) of set-based methods (e.g., compositions of two multiset functions that are permutation invariant their input multisets) for exploiting hyperedges in hypergraphs (2)]
two variants (attention- based and MLP-based
SubGNN g was applied to the hypergraph by viewing the nodes and hyperedges two types of vertices of a bipartite graph
The multilayer perceptron (MLP) baseline evaluates how simple ferd-forward neural network with hypergraph regularization and subgraph attention performs, by replacing dual attention with MLP.
Polygenic risk score (PRS) is currently widely used standard practice in genetic medicine and calculates disease risk from genotype profile using regression [42).
Non-negative matrix factorization (NMF) discovers now-dimensional structure from high- dimensional multi-omis data and enables inference of complex biological processes [43
XGBoost is an end-to-end tree boosting system and a state-of-ther ahet machine learning method 44
that frequently achieves the top results on many machine learning challenges.
To assess whether performance changes are due added information (e.g., pathway information) and/or better utilization the added information, we run PRS, NMF, and XGBoost in the following three settings: gene features only, pathway features only, and both gene and pathway features.
5 Results
The held-out test set micro-averaged Fl scores (micro-F1) for our proposed method SHINE and all the other comparison models are in Table 3 Comparing all the models, we can that SHINE clearly outperforms a comprehensive array of state-of-the-art baselines various configurations, with noe-overlgp standard deviation intervals. PRS is indeed competitive baseline, can be seen from its close performance compared with XGBoost that frequently topped many machine learning challenges' leaderboards. Previously state-of-the-ar hypergraph neural network models (HyperGCN, HGNN, HyperGAT) do not always outperform PRS and XGBoost (e.g., on the TCGA- MC3 dataset). On the other hand, pathway as features could improve performance if used properly. whether alone or jointly with genes. This comparison shows that genetic pathway information is useful to disease type classification, consistent to the intuition that pathways encode molecular functional mechanisms that underlie the disease etiology. However, properly utilizing such information is non-trivial, as evidenced by the difficulty to outperform PRS and XGBoost models by NMF models and hypergraph models including HyperGCN, HGNN and HyperGAT. Given that difficulty, SHINE
7

---

Table 3: Held-out test set micro-Fl on real-world datasets. Standard deviations are provided from runs with 10 random seeds. SHINE significantly outperforms all the state- of-tthe-rt comparison models. PRS: polygenic risk score. NMF: non-negative matrix factorization. Best model in bold.
Model
Feature
DisGeNet Dataset
TCGA-MC3 Dataset
Metrics
Test Micro F1
Test Micro F1
PRS
gene
0.6303
0.4981
PRS
pathway
0.6461
0,5047
PRS
gene+pathway
0.6512
0,5042
XGBoost
gene
0.6259
0.0012
0.4927 1 0,0058
XGBoost
pathway
0.6467
0.0035
0.4936 0.0092
XGBoost
gene+pathway
0.6486 = 0.0036
0.5117 + 0.0084
NMF
gene
0.6167 1 0.0040
0,4181 + 0.0125
NMF
pathway
0.5867 0.0039
0.4842 1 0.0057
NMF
gene+pathway
0.5847 0.0045
0.4839 + 0.0032
SubGNN (bipartite)
gene+pathway
0.6137 1 0.0097
0,4025 1 0.0049
HyperGCN
gene+pathway
0.6638 + 0.0028
0,4384 1 0.0095
HGNN
gene+pathway
0.6809 0.0027
0,4504 + 0.0042
HyperGAT
gene+pathway
0.6495 0.0050
0.4721 + 0.0032
MLP
gene+pathway
0.6331 = 0.0056
0.4249 + 0.0165
AllDeepSets
gene+pathway
0.6309 = 0.0147
0,4324 + 0.0220
rrasfforrrr
gene+pathway
0.6355 0.0160
0,4904 + 0.0158
SHINE
gene+pathway
0.6955 0.0034
0.5319 0.0049
still attained the best performance on each dataset. Intuitively speaking, HyperGCN and HGNN focus on similarity regularization: hypergraph nodes with similar context of hyperedges should have similar representations. HyperGAT's attention mechanism gears more towards minimizing the classification loss. SHINE, in an attempt to balance the similarity regularization with the end-to-end classification task via strongly dual attention mechanism, achieved better trade-off between the two objectives and effectively integrated the functional pathway's 's (hyperedge) information with individual gene's 's (node) information.
The importance of the strong duality follows naturally from that SHINE outperforms SubGNN (bipartite) with non-overlapping standard deviation intervals and wide separation. In addition, although MLP has been frequently used to approximate a target function, in the setting of large hypergraph (e.g., both hypergraphs have a few thousand-nodes hyperedges), it can still be quite challenging approximate an ideal target function and explicit dual attention formulation wins out. The results from both AllDeepSets and AllSerTransforme have non-overlapping standard deviation intervals, in fact wide separation, with their counterparts from SHINE. These results echo with our observations that strongly dual attention explores the hypergraph propagation from a different angle than both AllDeepSets and AllSetTransformer, and suggest that effectively combining both angles could be an interesting future direction. Also note that in general, we have some performance drop when moving from the DisGeNet dataset to the TCGA-MC3 dataset, likely due to the fact that the former uses genetic features from curated literatures and the latter is from high throughput sequencing intended for data-driven discovery. Both complex classification tasks (>20 classes) are uniquely challenging because discases may have overlapping disrupted molecular functions (genetic pathways, hyperedges), especially for the TCGA-MC3 experiment that is distinguishing subcategories of similar diseases they are loosely all cancers. In addition, both tasks exhibit class distributional shift between the train and the test datasets, as shown in Appendix Tables and 2, and have been designed subgraphs with highly variable hyperedge sizes. Strong performance
to require inductive inference of SHINE on these tasks thus suggests that our model can leverage its relational inductive biases for more robust generalization. Ablation studies further confirmed the contributions from each of the key components, including strictly dual attention massage passing, weighted subgraph attention and hypergraph regularization (see Appendix for details).
Model interpretation. SHINE simultaneously learns the representations of nodes and hyperedges, which are then used to learn and inductively infer subgraph representations. This brings some interpretation advantages as it allows assessing pathways (hyperedges) correlations and reasoning multiple molecular functions mutually interacting and collectively contributing to the disease onset and progression. In addition, SHINE has built-in measures to prevent or discourage genes belonging
8

---

Table 4: Top enriched genetic path associated with different cancer risks. The text color indicates the source database for pathways that MSigDB integrated: BioCarta, Reactome, WikiPathways, Pathway Interaction Database, KEGG.
BRCA
LUAD
LGG
HNSC
Stress pathway
PTK6 stabilizes HIFla
Citrate cycle TCA cycle
Apoptotic factor response
4-1BB pathway
ErbB3 pathway
Cytosine methylation
Programmed cell death
VIP pathway
Hypertrophic
TCA cycle and deficiency
MECP2 regulates neuronal
cardiomyopathy
of pyruvate dehydrogenase
receptors and channels
CD40 pathway
Discases of metabolism
Glutathione metabolism
FRA pathway
TOLL pathway
TFAP2 regulates growth
Digestion of dietary carbohydrate
Caspase activation via
factors transcription
extrinsic apoptotic signalling
to the same functional class (c.g., promoting immunc reactions) from having drastically different representations (c.g.. opposite directions), a phenomenon that will pose interpretation difficulty to other models that do not employ SHINE's hypergraph regularization. We identify the top pathways that enriched in different cancers using the attention weights learned for SHINE, shown in Table 4 From the table, we see that many of the listed pathways reflect innate key events in the development of individual or multiple types of cancers, consistent with genetic and medical knowledge from wet lab (e.g.. TNF/Stress Related Signaling (45). We showcase interpretations for breast cancer and lung cancer here, and refer the reader to the Appendix for full interpretation of Table 4 For breast cancer. TNFa is not only closely involved in its onset. progression and in metastasis formation. but also linked to therapy resistance (45). Regarding the 4-1BB pathway, studies have suggested HER2/4-IBB bispecific molecule as a candidate of alternative therapeutic strategy to patients in -posiiivee breast cancer [46). VIP/PACAP and their receptors have prominent roles in transactivation of the Epidermal growth factor (EGF) family and growth effects in breast cancer 47 For lung cancer, the ErbB3 receptor recycling controlled by neuroregulin receptor degradation protein- is linked to lung cancer and small inhibitory RNA (siRNA) to ErbB3 shows promise as a therapeutic approach to treatment of lung adenocarcinoma
48
Lung cancer is also
modulated by multiple miRNAs interacting with the TFAP2 family 49
6 Discussion, Limitation and Future Work
In addition to being significantly more accurate and interpretable. SHINE uses inductive subgraph inferencing that works well with minibatch, and scales well to large scale problems, as showcased by real-world experiments. It is known that GNN suffers from over-smoothing when the number of layers increases, as increasingly globally uniform representation of nodes may be developed. On the other hand, attention could limit this phenomenon by limiting to a restricted set of nodes. The effect of hypergraph regularization, while also smoothing, happens on a local scale as part of a direct optimization objective and does not accumulate with increasing number of layers. Such decoupling between attention and local smoothing allows SHINE to better explore the optimization landscape.
Our work has limitations. We assumed that the hyperedges are known in advance. However, in reality, as our domain knowledge increases and evolves, we need to account for unknown hyperedges and, better, simultaneously discover novel hyperedges from data while predicting disease classes. Such a task has important clinical utilities in genetic medicine to discover new genetic pathways that may underlie disease etiology, and will be our future work. Moreover, strongly dual attention explores the hypergraph propagation from different angle than both AllDeepSets and AllSerTransformer, and effectively combining both angles could be an interesting future direction. Another line of future work to derive a hypergraph coarsening model on top of SHINE. SHINE currently has flat hypergraph layout and does not learn hierarchical representations of hypergraphs. The emerging technique of spatial transcriptomies can enable discovery of localized and hierarchical gene expression patterns 50 51.. A flexible hypergraph coarsening model that can effectively learn hierarchical network structure out of the hypergraphs can shed light on the organizations of the hyperedges (e.g.. pathways representing synergistic molecular functions in certain tissue context).

---

From the application point of view, detecting tumor subtypes is often interesting, and we expect to extend our method to such detections using multi-modal data when large shared datasets will become available [52]. To certain extent, the TCGA labels we used reflect subtypes of organ-specific primary tumors, e.g., LUAD vs. LUSC lung cancer, KIRC vs. KIRP kidney cancer. On the other hand, identifying drivers genes and pathways for cancer types and other disease subtypes continue to be biologically important [53] and will be increasingly fruitful with simultaneously collected deep genetic and phenotypic data on the same patients [54 [55]
The field of genetic medicine encompasses areas of molecular biology and clinical phenotyping to explore new relationships between disease susceptibility and human genetics. Though appearing as a single field. revolutionizes the practice of medicine preventing, modifying and treating many diseases such as cardiovascular disease and cancer [56]. We expect SHINE to be useful tool in the quest of broadly advancing the knowledge on disease susceptibility. In these real-world applications. subject's s genetic profile may contain individual characterizing information. Thus, this work should never be used in violation of an individual's privacy, and the necessary steps of IRB review and execution of data user agreement need to be properly completed prior to the study.
Conclusions
We proposed novel framework termed SubHypergraph Inductive Neural nEtwork (SHINE) for inductive subgraph inferencing on hypergraphs, designed for jointly optimizing the objectives of end-to-end subgraph classification and similarity regularization for representations of hypergraph nodes with similar context of hyperedges. We showed that SHINE improved the performance (micro-FI) of the learned model for disease type prediction for complex (>20 classes) genetic medicine datasets of different characteristics and under different settings (e.g., multi-class and/or multi-label). Genetic pathways directly correspond to molecular mechanisms and functions, which are more informative than individual genes and are represented as hyperedges in SHINE. The novel formulation of disease classification as a subgraph inferencing problem allows a hypergraph neural network to link correlated pathways, i.e interacting molecular mechanisms, to disease etiology. This leads to better performance with added interpretability. We compared SHINE with wide array of state-of-the-arr (hyper)graph neural networks, XGBoost, NMF, and PRS models with different configurations of genes and pathways as features. SHINE consistently outperformed all state-of the art baselines significantly in each of the discase classification and cancer classification tasks. Feature analysis of the learned pathway groups that are automatically identified by SHINE in a data-driven fashion offered significant clinical insights about multiple molecular mechanisms that interact and are associated with disease types and status.
Acknowledgments and Disclosure of Funding
This work was supported in part by NIH grants R0ILM013337 and UOITR003528.
References
[1] Arthur Liberzon, Chet Birger, Helga Thorvaldsóóttr, Mahmoud Ghandi, Jill Mesirov, and Pablo Tamayo. The molecular signatures database hallmark gene set collection. Cell systems, 1 (6):417-425, 2015.
[2] Yuan Luo and Chengsheng Mao. Panther: pathway augmented nonnegative tensor factorization for higher-order feature learning. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 371-3801 2021.
[3] Yifan Feng, Haoxuan You, Zizhao Zhang, Rongrong Ji, and Yue Gao. Hypergraph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3558 3565, 2019.
[4] Naganand Yadati, Madhav Nimishakavi, Prateek Yadav, Vikram Nitin, Anand Louis, and Partha Talukdar. Hypergen: A new method of training graph convolutional networks on hypergraphs. arXiv preprint arXiv:1809.022 02589, 2018.
10

---

[5] Naganand Yadati. Neural message passing for multi multi-relational ordered and recursive hyper- graphs. Advances in Neural Information Processing Systems. 33, 2020.
[6] Ruochi Zhang, Yuesong Zou, and Jian Ma. Hyper-sagnn: a self-attention based graph neural network for hypergraphs. In International Conference on Learning Representations, 2020.
[7] Kaize Ding, Jianling Wang, Jundong Li, Dingcheng Li, and Huan Liu. Be more with less: Hy- pergraph attention networks for inductive text classification, arXiv preprint arXiv:201 1,00001,.077 2020.
[8] Changping Meng, S Chandra Mouli, Bruno Ribeiro, and Jennifer Neville. Subgraph pattern neu- ral networks for high-order graph evolution prediction. Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[9] Emily Alsentzer, Samuel G Finlayson, Michelle M Li, and Marinka Zitnik. Subgraph neural networks. arXiv preprint arXiv:2006. /0538, 2020.
[10] Xin Wei, Ruixuan Yu, and Jian Sun. View-gen: View-based graph convolutional network for 3d shape analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1850-1859, 2020.
[11] Chengsheng Mao, Liang Yao, and Yuan Luo. Imagegen: Multi-relational image graph convo- lutional networks for disease identification with chest x-rays. JEEE Transactions on Medical Imaging, 2022.
[12] David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alán Aspuru-Guzik, and Ryan P Adams. Convolutional networks on graphs for learning molecular fingerprints. In NeurlPS, pages 2224-2232, 2015.
[13] Justin Gilmer, Samuel S Schoenholz, Patrick Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In ICML, pages 1263-1272. JMLR. org, 2017.
[14] Liang Yao, Chengsheng Mao, and Yuan Luo. Graph convolutional networks for text classifica- tion. In AAAI, 2019.
[15] Hao Peng, Jianxin Li, Yu He, Yaopeng Liu, Mengjiao Bao, Lihong Wang, Yangqiu Song, and Qiang Yang. Large-scale hierarchical text classification with recursively regularized deep graph-enn. In WWW, pages 1063-1072, 2018
[16] Yifu Li, Ran Jin, and Yuan Luo. Classifying relations in clinical narratives using segment graph convolutional and recurrent neural networks (seg-gerns). Journal of the American Medical Informatics Association, 26(3):222-688, 2019.
[17] Marinka Zitnik, Monica Agrawal, and Jure Leskovee. Modeling polypharmacy side effects with graph convolutional networks. Bioinformatics, 34(13)i457-ii46, 2018.
[18) Chengsheng Mao, Liang Yao, and Yuan Luo. Medgen: Graph convolutional networks for multiple medical tasks. arXiv preprint arXiv: 2019.
[19] Thomas N Kipf and Max Welling. Semi-sppervised classification with graph convolutional networks. arXiv preprint arXiv: 1w09.66697.007 2016.
[20] Oriol Vinyals, Samy Bengio, and Manjunath Kudlur. Order matters: Sequence to sequence for sets. arXiv preprint 151 1.069), 2015.
[21] Rex Ying, Jiaxuan You, Christopher Morris, Xiang Ren, William L Hamilton, and Jure Leskovec. Hierarchical graph representation learning with differentiable pooling, arXiv preprint arXiv:1806.0888 08804, 2018.
[22] William Hamilton. Graph representation learning. Synthesis Lectures on Artifical Intelligence and Machine Learning, 14(9):1-159, 2020.
[23] Dengyong Zhou, Jiayuan Huang, and Bernhard Schölkopf. Learning with hypergraphs: Clus- tering, classification, and embedding. Advances in neural information processing systems, 19: 1601-1608, 2006.
11

---

[24] Taisong Jin, Liujuan Cao. Baochang Zhang, Xiaoshuai Sun, Cheng Deng, and Rongrong Ji. Hypergraph induced convolutional manifold networks. In IJCAI, pages 2670-2676, 2019.
[25] Jianwen Jiang, Yuxuan Wei, Yifan Feng, Jingxuan Cao, and Yue Gao. Dynamic hypergraph neural networks. In IJCAI, pages 2635-2641, 2019.
[26] Sai Nageswar Satchidanand, Harini Ananthapadmanaban, and Balaraman Ravindran. Extended discriminative random walk: A hypergraph approach to multi-view multi-relational transductive learning. In IJCAI, pages 3791-3797, 2015.
[27] Fuli Feng, Xiangnan He, Yiqun Liu, Liqiang Nie, and Tat-Seng Chua. Learning on partial-order hypergraphs. In Proceedings of the 2018 World Wide Web Conference, pages 1523-1532, 2018.
[28] Ke Tu, Peng Cui, Xiao Wang, Fei Wang, and Wenwu Zhu, Structural deep embedding for hyper-networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[29] Eli Chien, Chao Pan, Jianhao Peng, and Olgica Milenkovic. You are allset: A multiset function framework for hypergraph neural networks. arXiv preprint arXiv:2106. 132644 2021.
[30] Kexin Huang and Marinka Zitnik. Graph meta learning via local subgraphs. Advances in Neural Information Processing Systems, 33, 2020.
[31] Qingyun Sun, Jianxin Li, Hao Peng. Jia Wu, Yuanxing Ning, Phillip S Yu, and Lifang He. Sugar: Subgraph neural network with reinforcement pooling and self-supervised mutual information mechanism. arXiv preprint arXiv:2 08/770 2021.
[32] Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S Yu. Het- erogeneous graph attention network. In The world wide web conference, pages 2022-2032, 2019.
[33] Chuxu Zhang, Dongjin Song, Chao Huang, Ananthram Swami, and Nitesh V Chawla. Het- erogeneous graph neural network. Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 793-803, 2019.
[34] Yuwei Fu, Yun Xiong, S Yu Philip, Tianyi Tao, and Yangyong Zhu. Metapath enhanced graph attention encoder for hins representation learning. In 2019 IEEE International Conference on Big Data (Big Data), pages 1103-1110. IEEE, 2019.
[35] Xinyu Fu, Jiani Zhang, Ziqiao Meng, and Irwin King. Magnn: Metapath aggregated graph neural network for heterogeneous graph embedding. In Proceedings of The Web Conference 2020, pages 2331-2341, 2020.
[36] Ziniu Hu, Yuxiao Dong. Kuansan Wang, and Yizhou Sun. Heterogeneous graph transformer. In Proceedings of The Web Conference 2020, pages 2704-2710, 2020.
[37] Muhan Zhang, Zhicheng Cui, Shali Jiang, and Yixin Chen. Beyond link prediction: Predicting hyperlinks in adjacency space. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[38] Yujia Li, Daniel Tarlow, Mare Brockschmidt, and Richard Zemel. Gated graph sequence neural networks. arXiv preprint 15 2015.
[39] Janet Piñero, Alex Bravo, Núria Emilio Centeno, Javier Garcfa-Garcis García, Ferran Sanz, and Laura I. Furlong. DisGeNET: a comprehensive platform integrating information on human discase-associited genes and variants. Nucleic Acids Research, 45(1)) 10 2016. ISSN 0305-1048. doi: 10. 10.1093/nar/ gkw943. hwtp::ttttttoooooo http:////abbb.ogo......//////// 1093///////////33333 gkw943
Alba Gutiéreez-Sccristán, Jordi Deu-Pons,
[40] Kyle Ellrott, Matthew H Bailey, Gordon Saksena, Kyle R Covington, Cyriac Kandoth, Chip Stewart, Julian Hess, Singer Ma, Kami E Chiotti, Michael McLellan, et al. Scalable open science approach for mutation calling of tumor exomes using multiple genomic pipelines. Cell systems, 6(3): 2018.
12

---

[41] Xiao Wang, Deyu Bo, Chuan Shi, Shaohua Fan, Yanfang Ye, and S Yu Philip. A survey on heterogeneous graph embedding: methods, techniques, applications and sources. IEEE Transactions on Big Data, 2022.
[42] Shing Wan Choi, Timothy Shin-Heng Mak, and Paul F F'RReilll. Tutorial: guide performing polygenic risk score analyses. Nature Protocols, 15(9):275--2722 2020.
[43] Genevieve L Stein- O' Brieen, Raman Arora, Aedin C Culhane, Alexander V Favorov, Lana X Garmire, Casey S Greene, Loyal A Goff, Yifeng Li, Aloune Ngom, Michael F Ochs, et al. Enter the matrix: factorization uncovers knowledge from omics. Trends in Genetics, 34(10):70-805, 2018.
[44] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785-794, 2016.
[45] María Florencia Mercogliano, Sofía Bruni, Patricia V Elizalde, and Roxana Schillaci. Tumor necrosis factor blockade: opportunity to tackle breast cancer. Frontiers in oncology, 10, 2020.
[46] Marlon J Hinner, Rachida Siham Bel Aiba, Thomas J Jaquin, Sven Berger, Manuela Carola Dürr, Corinna Schlosser, Andrea Allersdorfer, Alexander Wiedenmann, Gabriele Matschiner, Julia Schüler, et al. Tumor-localized costimulatory t-cell engagement by the 4- bbbher2 bispecific antibody-anticall fusion prs-343. Clinical Cancer Research, 25(19):5878 -58777858899-8 2019.
[47] Terry W Moody, Bernardo Nuche-Berenguer, and Robert T Jensen. Vip/pacap, and their receptors and cancer. Current opinion in endocrinology, diabetes, and obesity. 23(1):38, 2016.
[48] Sithanandam and LM Anderson. The erbb3 receptor in cancer and cancer gene therapy. Cancer gene therapy, 15(7):413-445, 2008.
[49] Damian Kotat, Zaneta Kahuzinska, Andrzej Bednarek, and Elžbieta Plucienník. The biological characteristics of transcription factors ap-2a and ap-2 and their importance in various types of cancers. Bioscience reports, 39(3), 2019.
[50] Zexian Zeng, Yawei Li, Yiming Li, and Yuan Luo. Statistical and machine learning methods for spatially resolved transcriptomics data analysis. Genome biology, 23(1):1-23, 2022.
[51] Anjali Rao, Dalia Barkley, Gustavo S França, and Itai Yanai. Exploring tissue architecture using spatial transcriptomics. Nature, 596(7871):211 12200 2021.
[52] Adrienne Kline, Hanyin Wang, Yikuan Li, Saya Dennis, Meghan Hutch, Zhenxing Xu, Fei Wang, Feixiong Cheng, and Yuan Luo. Multimodal machine learning in precision health. arXiv preprint 2xXiv:2204.0477 2022.
[53] Matthew H Bailey, Collin Tokheim, Eduard Porta-Pardo. Sohini Sengupta, Denis Bertrand, Amila Weerasinghe, Antonio Colaprico, Michael Wendl, Jaegil Kim, Brendan Reardon, et al. Comprehensive characterization of cancer driver genes and mutations. Cell, 173(2):371-385 2018.
[54] Y Luo, Mao, Y Yang. F Wang, FS Ahmad, D Arnett, MR Irvin, and SJ Shah. Integrating hyper- tension phenotype and genotype with hybrid non-negative matrix factorization. Bioinformatics (Oxford, England), 1395
2019.
[55] Marylyn D Ritchie, Emily R Holzinger, Ruowang Li, Sarah A Pendergrass, and Dokyoon Kim. Methods of integrating data to uncover genotype penooyp--phenotppe interactions. Nature Reviews Genetics, 16(2):85-97, 2015.
[56] Eric D Green, Chris Gunter, Leslie G Biesecker, Valentina Di Francesco, Carla L Easter, Elise A Feingold, Adam L Felsenfeld, David J Kaufman, Blaine A Ostrander, William Pavan, et al. Strategic vision for improving human health at the forefront of genomics. Nature, 586(7831): 683-692, 2020.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the appendixal material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the appendixal material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] We discuss the IRB approval process Section 4. (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [Yes]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "uZUAcvyQtOf", "reviewer": "Reviewer_FtLi", "summary": "Hypergraph neural networks can exploit multi-way connections in relational datasets but they are underexplored in domains such as genetic medicine. In this paper, a hypergraph attention-based message passing neural network is proposed for sub(hyper)graph-level tasks, e.g., \n* genes: nodes, \n* pathways: hyperedges, \n* patients: subgraphs, \n* predict cancer type of patient: task. \n\nExperiments on genetic medicine datasets demonstrate the effectiveness of the proposed method SHINE: SubHypergraph Inductive Neural nEtwork. ", "questions": "1. How are the attention mechanisms in HyperGAT and SHINE different from existing attention mechanisms in heterogeneous graph neural networks with two different node types, e.g., treat genes and pathways as two different node types?\n2. How does a simple feed-forward neural network with hypergraph regularisation and subgraph attention perform? This baseline is basically an ablated baseline with strongly dual attention message passing removed from SHINE and replaced with an MLP.\n3. Is the strongly dual attention message passing scheme of SHINE an instantiation of the multiset formulation of \"You are AllSet: A Multiset Function Framework for Hypergraph Neural Networks, In ICLR'22\"?\n4. What are the training times of all the methods (SHINE and all baselines)?\n5. How sensitive are the hyperparameters (e.g., hidden dimensions) to the performance of SHINE? What is the optimal number of hidden layers in SHINE?", "limitations": "The authors have addressed the limitations and potential negative societal impacts adequately.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Originality**\n\nEven though the paper explores an underexplored research topic in an interesting domain (subgraph representation learning for hypergraphs in genetic medicine), the methods proposed are incremental extensions of existing methods and not novel combinations of existing techniques.\n\nSpecifically in section 3.3., the ideas of hyperedge attention over nodes and node attention over hyperedges with parameter sharing are incremental extensions of well-known hypergraph attention networks.\n\nBy viewing the nodes and hyperedges as two types of vertices of a (bipartite) heterogeneous graph, the ideas of strongly dual attention mechanisms would be incremental extensions of existing attention-based methods for heterogeneous graphs, e.g., see \"A Survey on Heterogeneous Graph Embedding: Methods, Techniques, Applications and Sources\"\n\n\\\n**Quality**\n\nThe authors have discussed interesting weaknesses of their work (in addition to highlighting the strengths).\n\nMoreover, baseline comparison (Table 3), interpretability analysis (Table 4), and ablation study (Table 3 in supplementary) support the claims made in the paper empirically to an extent.\n\nHowever, formalising the key differences with existing similar methods (e.g., HyperGAT in lines 159-169) and confirming the differences with convincing (synthetic/real-world) experiments, e.g., on a dataset chosen cleverly to show clear failure of HyperGAT but success of SHINE, would improve the paper's quality. \n\n\\\n**Clarity**\n\nThe paper is well organised.\n\nDetails on datasets and hyperparameter tuning could help an expert to reproduce the results of the paper and build effective models (those with the best hyperparameters) from scratch.\n\nA discussion on computational complexity and an algorithm/pseudo code would further enhance the clarity of the paper.\n\n\\\n**Significance**\n\nIt is unclear from the paper why modelling genetic medicine datasets with hypergraphs, despite being a natural choice, is the best choice compared to straightforward alternatives.\n\nMore specifically, it is unclear why a (bipartite) heterogenous graph with genes: nodes of type 1, pathways: nodes of type 2, patients: (sub) heterogeneous graph would not be a reasonable choice.\n\nThe paper can be improved by positioning and comparing with set-based methods for exploiting hyperedges in hypergraphs, e.g., You are AllSet: A Multiset Function Framework for Hypergraph Neural Networks, In ICLR'22.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "cIwGbi4eZg", "reviewer": "Reviewer_CYi2", "summary": "The paper proposes a hypergraph neural network model exploiting a double attention mechanism in the message passing scheme. The overall architecture is designed to process sub-hypergraphs once representations are computed for the nodes and edges. The learning objective includes a regularization term based on the hypergraph laplacian.\nThe proposed model is evaluated on disease classification based on gene-genetic pathways data, showing higher F1 values with respect to a set of competitors.\nFinally, due to the attention mechanism intepretions in terms of gene pathways (hyperedges) can be derived from the model outputs.", "questions": "How performances are affected by the number K of layers?", "limitations": "The authors discuss some limitations that need futher work in section of the paper both for the model architecture and the specific application considered for the evaluation.\nAs listed in the weaknesses, a more general description (not tailored for the considered task in genetic medicine) would have improved the presentation of the proposed hypergraph neural network architecture.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- the model incorporates a dual attention mechanism applied to nodes and hyperedges respectively, exploiting the same attention context vector. This desing choice is claimed to prevent overfitting reducing the number of parameters\n- the architecture includes an attention module to derive subgraph representations. This scheme allows the application of the mode in a inductive setting\n- the model compares favourably with respect to the considered competitors on two benchmarks in genetic medicine\n\nWeaknesses\n- The presentation of the model is tightly interconnected with the proposed application in genetic medicine, making it appear less general\n- Given the focus on bioinformatics, the paper is hard to follow for readers not completely familar with this topic.\n- The effect of the number K of layers is not investigated (maybe I missed it, but the number of layers used in the experiments is not reported). It is known that (Convolutional) Graph Neural Network suffer from oversmoothing when the depth of the network increases. This may hinder the results in some applications since uniform representation of nodes are developed. Attention may perhaps limit this phenomenon, but on the other side node/edge regularization may produce add a related effect. The ablation study show a positive effect of regularization, but some discussion/analysis should be provided.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5YZI5BfHtM", "reviewer": "Reviewer_noVi", "summary": "This paper suggests sub-hypergraph representation learning, a niche problem related to subgraph and hypergraph learning. To tackle this problem, the authors propose the SHINE (SubHypergraph Inductive Neural nEtwork) model, which consists of three modules: strongly dual attention message passing, hypergraph regularization, and weighted subgraph attention. Experiments on two real-world datasets demonstrate the superiority of SHINE on performance (against baselines including GNNs for hypergraphs) and interpretation (using attention).", "questions": "My questions are summarized in the weaknesses section.\n", "limitations": "The authors do not address the potential negative societal impact of their work. This paper targets a high-level machine learning problem called sub-hypergraph representation learning; however, all datasets are related to a particular area, genes, pathways, and diseases. There could be a potential societal impact that should be considered in real-world applications in this area (e.g., privacy). It would be nice if the authors addressed this point.\n", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nThe authors present a novel and niche problem of sub-hypergraph representation learning which has not been explored in the GNN community. A specific example (cancer patients as subgraphs of genes in hypergraphs) can be a practical application for this task. The performance improvement by the authors’ approach is significant.\n\n## Weaknesses\n\nHowever, I think this paper is not ready for publication for the following reasons.\n\nFirst, the technical novelty of SHINE is limited. This model consists of several parts, and each of them is a slightly modified version of existing approaches. Using the attention to both nodes and (hyper) edges is presented in HyperGAT, and the authors are aware of it. Nevertheless, the idea of (strongly) dual attention is aligned with the dual form of hypergraphs, and I can see the novelty of this paper here. However, explicit regularization by Laplacian (Hypergraph regularization) [1] and pooling by attention weights (Weighted Subgraph Attention) [2] are well-known methods in using GNNs. In this case, SHINE's novelty is limited to Strongly Dual Attention, and a more detailed analysis of this part is required.\n\nSecond, related to the first paragraph, there are no rigorous ablation studies on the architecture. As many submodules make up the model, it is necessary to study where the performance gain comes from. In the supplementary material, only the ablation study on hypergraph regularization is presented, and the study on dual attention message passing is presented by comparison with HyperGAT. However, there are other differences in attention forms between HyperGAT and SHINE, and comparing these two does not provide a fully controlled ablation study of dual attention message passing. I recommend authors retain all parts except parameter-sharing in the attention. In addition, the performance comparison between SHINE with/without WSA and other GNNs with/without WSA also should be presented.\n\nThird, it is skeptical that model interpretation by attention is an exclusive strength of SHINE. There are learned attentions in other attentional models like HyperGAT. Can these models provide interpretation at the same level as SHINE? Can you compare interpretations between models? Does SHINE give more precise explanations than other models?\n\nLastly, there are missing baselines for subgraph classification; in particular, the SubGNN can be a strong baseline. Of course, SubGNN is not designed for hypergraphs, but it is straightforward to create graphs from hypergraphs such as clique expansion. The transformation from hypergraphs to graphs is done only once before training; thus, it has a low overhead. Comparing SHINE and GNNs-for-subgraphs can justify that these specific problems in this work should be represented as a hypergraph.\n\n## References\n\n- [1] Learning with Hypergraphs: Clustering, Classification, and Embedding\n- [2] GATED GRAPH SEQUENCE NEURAL NETWORKS\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-Ms8W_5J3x", "reviewer": "Reviewer_tNZm", "summary": "The authors propose a GNN approach to learn embeddings of sub-hyper-graphs. The approach has an explicit treatment of hyper edges, e.g., it does not resort to clique expansion, and makes use of a regulariser based on the hyper graph laplacian.  The application chosen is that of disease prediction for patients (modelled as sub hyper graphs) given a pathway network (modelled as a hyper graph with genes as nodes and sets/pathways as hyper edges). ", "questions": "Additional experiments to quantify the   relative importance of the various ideas introduced would strengthen the paper.\nIn particular consider offering support to reply to the points 1, 2 and 3 raised as negative elements in the Strengths And Weaknesses section.", "limitations": "yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pos\nExplicitly treating hyper edges as first class citizens in the GNN modelling is of interest, since in this was hyper edges can be the subjects of notions of regularisation or attention.\n\nNeg\nThe relative importance of the various ideas introduced is not clear, i.e., a better experimental design with clearer baselines and an ablation study is warranted. More specifically:\n\n1. is the regularisation effective or of importance? (ablation case)\n\n2. is the proposed architecture much different from using a standard graph neural network with attention on a pre-processed hyper graph? In particular the pre-processing could consist in representing an hyper graph as a bi-partite graph where hyper edges are materialised as the nodes of one part and the genes are the nodes of the other part.  (experiment)\nNote that the whole discussion regarding strong duality would follow automatically in this case.\n\n3. is the introduction of WSA (the weighted  subgraph attention) needed? what happens if we replace the whole subgraph treatment by Mji directly? I.e., what if we consider a subgraph simply the sum of the nodes (genes) that are of interest (with mutations) for each patient (subgraph), that is, we could learn directly the embedding of the nodes/hyperedges for the classification task when they are simply summed up for each patient. (experiment/ablation)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept