HypeBoy: Generative Self-Supervised Representation Learning on Hypergraphs
Authors: Sunwoo Kim
Shinhwan Kang
Fanchen Bu
Soo Yong Lee
Jaemin Yoo
Kijung Shin
Venue: ICLR 2024 poster (2024)
Keywords: Hypergraph
Self-supervised learning
Hypergraph neural network
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose a hypergraph generative self-supervised learning strategy.

Abstract: Hypergraphs are marked by complex topology, expressing higher-order interactions among multiple nodes with hyperedges, and better capturing the topology is essential for effective representation learning. Recent advances in generative self-supervised learning (SSL) suggest that hypergraph neural networks (HNNs) learned from generative self-supervision have the potential to effectively encode the complex hypergraph topology. Designing a generative SSL strategy for hypergraphs, however, is not straightforward. Questions remain with regard to its generative SSL task, connection to downstream tasks, and empirical properties of learned representations. In light of the promises and challenges, we propose a novel generative SSL strategy for hypergraphs. We first formulate a generative SSL task on hypergraphs, hyperedge filling, and highlight its theoretical connection to node classification. Based on the generative SSL task, we propose a hypergraph SSL method, HYPEBOY. HYPEBOY learns effective general-purpose hypergraph representations, outperforming 15 baseline methods across 11 benchmark datasets. To our knowledge, this is the first study on generative SSL on hypergraphs, and we demonstrate its theoretical and empirical strengths for hypergraph representation learning.

Published as a conference paper at ICLR 2024
HYPEBOY: GENERATIVE SELF-SUPERVISED REPRE- SENTATION LEARNING ON HYPERGRAPHS
Sunwoo Kim Shinhwan Kang Fanchen But Soo Yong Leef Jaemin You Kijung Shin
i Kim Jaechul Graduate School of AI, School of Electrical Engineering
Korea Advanced Institute of Science and Technology (KAIST) (kswoo97, shinhwan.kang boqvezen97, beretolow.jeemiiii kijungs) @ kaist.ac.kr
ABSTRACT
Hypergraphs are marked by complex topology, expressing higher-order interac- tions among multiple nodes with hyperedges, and better capturing the topology is essential for effective representation learning. Recent advances in generative self- supervised learning (SSL) suggest that hypergraph neural networks learned from generative self-supervision have the potential to effectively encode the complex hypergraph topology. Designing generative SSL strategy for hypergraphs, how- ever, is not straightforward. Questions remain with regard to its generative SSL task, connection to downstream tasks, and empirical properties of learned repre- sentations. In light of the promises and challenges, we propose novel generative SSL strategy for hypergraphs. We first formulate generative SSL task on hyper- graphs, hyperedge filling, and highlight its theoretical connection to node classifi- cation. Based on the generative SSL task, we propose a hypergraph SSL method, HYPEBOY. HYPEBOY learns effective general- purepoe hypergraph representa- tions, outperforming 16 baseline methods across benchmark datasets. Code and datasets are available https: pettps:/ggithee /ttpttgtttttt///t///////ttethhtt wooo/kswwo9777 htyp:/oyyyy......o//
INTRODUCTION
Many real-world interactions occur among multiple entities, such as online group discussions on social media, academic collaboration of researchers, and joint item purchases (Benson et al., 2018; Kim et al., 2023a; Lee et al., 2024). Representing such higher-order interactions with an ordinary graph can cause topological information loss (Dong al., 2020; Yoon et al., 2020). Thus, hyper- graphs have emerged as an effective tool for representing high-order interactions in various domains, including recommender systems (Xia et al., 2022; Yu et al., 2021). financial analysis (Sawhney et al., 2021; 2020), and drug analysis (Ruan et al., 2021; Saifuddin et al., 2023).
For representation learning such complex topology, hypergraph neural networks (HNNs) have been developed. Training HNNs has primarily relied on task-related label supervision, such as exter- nal node labels. However, simply learning from external supervision may limit HNNs in capturing more complex patterns in hypergraph topology. Incorporating self-supervisio related to topology, hence, can substantially improve HNNs' representation learning.
Particularly, generative self-supervised learning (SSL) holds promise for effective hypergraph repre- sentation learning. Generative SSL has recently shown remarkable success in encoding complex pat- terns in multiple domains. GPT (OpenAI, 2023) in natural language processing and Masked Autoen- coder (He al., 2022) in computer vision are notable examples. With generative self- supervisei,, HNNs may encode complex topology more effectively, leading to improved representation learning.
However, designing a generative SSL strategy for hypergraphs is not straightforward. First, ques- tions remain unanswered for generative SSL tasks: (1.a) What should be the target generative SSL task for HNNs? (1.b) How does the generative SSL task relate to downstream tasks (e.g., node clas- sification with external labels)? Second, even after determining the task, details of the SSL method (based on the task) remain unspecified. (2.a) What empirical properties should the method aim to satisfy? (2.b) How the method achieve effective ceneral-purposs hypergraph representations? Moreover, if not carefully designed, the generative SSL strategy can suffer from severe computa- tional burden, the number of potential hyperedges grows exponentially with the number of nodes.

---

Published as a conference paper at ICLR 2024
In light of these promises and challenges, we systematically investigate and propose a hypergraph generative SSL strategy. Our contributions and the rest of the paper are organized as follows:
SSL Task: We formulate generative SSL task, hyperedge filling, for hypergraph representation learning. Notably, we establish its theoretical connections node classification (Section 3).
SSL Method: Based on the hyperedge filling task, we propose HYPEBOY, a novel hypergraph SSL method. HYPEBOY is designed to satisfy desirable properties of hypergraph SSL, mitigating (1) over-emphasized proximity, (2) dimensional collapse, and (3) non- non-uniformityy-lignnne of learned representations (Section 4).
Experiments: We demonstrate that HYPEBOY learns effective general-purpose hypergraph rep- resentations. significantly outperforms SSL-based HNNs in both node classification and hy- peredge prediction across 11 benchmark hypergraph datasets (Section 5; code and datasets are available at https: : /ightuu.bb..../.gg com/ http:///wwww///////////e/
RELATED WORK
In this section, we review the literature on hypergraph neural networks and self-supervised learning.
Hypergraph neural networks (HNNs). HNNs learn hypergraph representations. Converting hy- peredges into cliques (fully connected subgraphs) allows graph neural networks be applied to hypergraphs (Feng et al., 2019; Yadati et al., 2019). Such conversion, however, may result in topo- logical information loss, since high-order interactions (hyperedges) are reduced to pair-wise interac- tions (edges). As such, most HNNs pass messages through hyperedges to encode hypergraphs. Some notable examples include HNHN (Dong et al., 2020) with hyperedge encoders, UniGNN (Huang & Yang, 2021) with generalized message passing functions for graphs and hypergraphs, AllSet (Chien et al,, 2022) with set encoders, ED-HNN (Wang et al., 2023a) with permutation-equiaarine diffusion operators, and PhenomNN (Wang et al., 2023b) with hypergraph-regullriized energy functions.
Self-supervised learning (SSL). SSL strategies aim to learn representation from the input data itself, without relying on external labels. They can largely be categorized into contrastive or gener- ative types. Contrastive SSL aims to maximize the agreement between data obtained from diverse views (Chen et al., 2020; Grill al., 2020; You al., 2020). Generative SSL, on the other hand, predicts or reconstructs parts of the input data. The success of generative SSL demonstrates its strengths in learning complex input data, in domains including natural language processing (Devlin et al., 2019; OpenAl, 2023) and computer vision (He et al., 2022: Tong et al., 2022). Recently, gen- erative SSL for graphs has gained significant attention, with their main focuses on reconstructing edges (Tan et al., 2023; Li et al., 2023) node features (Hou et al., 2022: 2023).
Self-supervised learning on hypergraphs. The interest SSL for hypergraphs on the rise. Early hypergraph SSL strategies mainly targeted specific downstream tasks, such as group (Zhang et al., 2021) and session-based recommendation (Xia et al., 2022). Recent ones aim to obtain general- purpose representation. TriCL (Lee & Shin, 2023a) utilizes tri-directional contrastive loss, which consists of node-, hyperedge-, and membership-lewe contrast. Kim et al. (2023c) enhances the scala- bility of TriCL with partitioning technique. HyperGCL (Wei et al., 2022) utilizes neural networks to generate views for contrast and empirically demonstrates its superiority in node classification, outperforming rule-based view generation methods. HyperGRL (Du al., 2022) is generative SSL method sharing similar spirits with our approach; however, the underlying motivations and methodological designs are markedly distinct. We provide a detailed comparison Appendix C.I.
PROPOSED TASK AND THEORETICAL ANALYSIS
In this section, after providing some preliminaries, we formulate hyperedge filling, our generative SSL task on hypergraphs. Then, we establish theoretical connection between hyperedge filling and node classification, which is a consie-considered important downstream task.
Preliminaries. A hypergraph g (V,E) is defined by node set V and hyperedge set E. Each hyperedge e, E & is a non-empty set nodes (i.c., @ A ej C V, Ve, E E). Each node v; E V is equipped with a feature vector x; e Idd and X E V denotes the node feature matrix where the i-th row X; corresponds to æ;;
2

---

Published as a conference paper at ICLR 2024
9
5
5
?
6
X
Hypergraph 9
Augmented
hypergraph
Objective of filling
Node and its features
Augmentation
Encoding
mmedding
(a) Hyperedge filling task
(b) Visual illustration of HyPEBOY
Figure I: Overview of (a) the hyperedge filling task and (b) HYPEBOY, our proposed SSL method based on the task. The goal of the task is to find the missing node for given query subset (i.e., the other nodes a hyperedge). HYPEBOY trains HNNs aiming to correctly predict the missing node.
A hypergraph neural network (HNN) fe is a function that receives node feature matrix X and set of hyperedges as inputs to return node embeddings Z E V (i.e., Z = fo(X, E), where is set of learnable parameters) !
3.1 PROPOSED TASK: HYPEREDGE FILLING
We formulate hyperedge filling, generative SSL task for hypergraph representation learning. We first define the task and discuss the superiority of the hyperedge filling task over alternatives. An illustration the hyperedge filling task is provided Figure ((a).
Task definition. Given set of nodes, hyperedge filling aims to predict node that is likely to form a hyperedge together. Specifically, for each hyperedge e, E E, we divide it into a (missing) node uy E Ej and (query) subset qú E₁ {ey]. The target of the task is to correctly fill the missing node i for each given subset 9iz. This can be formalized by maximizing the probability of e; correctly aim to
completing fij. which is denoted Pix. (Ma 4:z). where @ is set of parameters design of P(X,E,0) (-) in Section 4.3.
optimize in this task. We will further elaborate
Advantage over alternatives. Potential alternatives include naive extensions of generative SSL tasks for ordinary graphs: (a) generating hyperedges from scratch and (b) classifying given sets of nodes into hyperedges and non-hyperedges. Compared to (a), by shifting the focus of prediction from the set level (hyperedge itself) to the node level, the hyperedge filling task reduces the pre- diction space from computationally prohibitive 0(2 Vi) to affordable O(V)). Compared to (b), the hyperedge filling task provides richer and more diversified generative SSL signals. Specifically, for each hyperedge ej. our task offers lcyl distinct node-subset combinations that serve SSL signals. In contrast, classifying the mere existence of ej yields a singular and, thus, limited signal.
3.2 THEORETICAL RESULTS ON HYPEREDGE FILLING
To demonstrate the effectiveness of hyperedge filling general SSL task, present its theoretical connection to node classification. In essence, we demonstrate that node representations optimized for the hyperedge filling task can improve node classification accuracy.
3.2. BASIC SETTING
First, we assume a data model a hypergraph (V,E) where (1) each node belongs to a single class, (2) the features of each node are generated from Gaussian distribution, and (3) each hyperedge is generated according to given hyperedge affinity parameter P E [0, 1].
Assumption (Node classes and features), Assume that there are 2N nodes and node classes C₁ and |C1| |Cal N. Each node feature vector æ; is
and Co such that C C, V. n independently generated from Nx;; μ1, Σ) if 0; E C1, and N (xx; μ4g Σ) if Us E Cg. For simplicity, assume 1 (0.5) Mo (-0.5) and Σ I, where I is the d-by-d identity matrix.
In this paper, assume an HNN returns only vector representations of nodes unless otherwise stated, while lcknowledge that some HNNs return embeddings of hyperedges well.
3

---

Published as a conference paper at ICLR 2024
Assumption (Hypergraph topology). Assume that the number of hyperedges and the size of each hyperedge are given, and there is no singleton hyperedge (i.e., leyl 2 2, Ve, E E). Let denote the binomial distribution. Each hyperedge ej E E has a class membership Cj E {0. 1}. where Cj ~ (1, 0.5). Given the number legl of nodes and the class oj E (0,1) of a hyperedge, the number of its members belonging to C1 (i.e., le,
B(ls,,, (1-P) 1)1-5)))
Note that the tendency of each hyperedge containing nodes of the same class symmetric about P 0.5 under the binary class setting. In addition, when P approaches 1, each hyperedge ej more likely to contain nodes of the same class (spec., node class cj); as P approaches O, each hyperedge ej is again more likely to contain nodes of the same class (spec., node class 1 - cy).
Second, we describe how node representations are updated via the hyperedge filling task. In this theoretical analysis, we define the updating process of node representations follows:
(F1) Filling probability P.x. (-) is defined on each node-subset pair as follows:
exp(x (Eor Egsy 2x))
P(x,E,0) (e: 9:)) = Emery exp(ar (Ev, mk))
(1)
(F2) Node representation 2; is obtained via gradient descent with respect E; from L, which is the negative log-likelihood of Eq. (1), (i.e., L = log P(X,E,)) (e; 4ty)). For ease of analysis, we assume 2, 3. 7 m where y I + is fixed constant.
At last, assume a Gaussian naive Bayes classifier F (Bishop, 2006), which is defined as: F(x:)= = arg max f(z;; Pk. I), where f is the probability density function of N(x;; Pa. I). (2) (0
3.2.2 HYPEREDGE FILLING HELPS NODE CLASSIFICATION
Our goal is to show that for accurate classification of v, the representation z;, which is obtained for hyperedge filling as described in (F1) and (F2), is more effective than the original feature æ;. First, assume node V; belonging the class C1 (i.c., U, C1). and later generalize the result to Co. The effectiveness of an original feature is defined as the expected accuracy of a classifier FF with x; (i.e., Ex ==a == Pa (f M1, I) > fxx; No, I))). Similarly, that with a derived representation defined as Ez [=] = Pz (f(z M1, I) > f(z po. I)). Below, we show that the effectiveness of a derived representation is greater than that of an original feature I; under certain condition (Theorem 1).
Theorem 1 (Improvement in effectiveness). Assume hyperedge ej S.I. Ej O Ci & 9 and node features X that are generated under Assumption 1. For any node vs E Eej n nCi, the following holds:
I E ug. E0:3
Ks 00 + Ez (x Ex Ex(1)(mm)=1) where denotes (1)
(3)
Proof. A full proof is provided in Appendix A.I.
Theorem states that when a certain condition (specified in the parentheses in Eq. (3)) is met, the effectiveness of z, is greater than that of x4, This result implies that node representations, when refined using the objective function associated with the hyperedge filling task, are more proficient in performing accurate node classification compared to the original node features.
While the finding in Theorem demonstrates the usefulness of the hyperedge filling task in node classification, its validity relies on the condition (in the parentheses in Eq. (3)). We further analyze the probability that the condition is met by stochastic G under Assumptions and 2 for given P.
Theorem 2 (Realization of condition). Assume node features X. and hyperedge ej s.f. (i) gener- ated under Assumption 1 and 2 respectively, and (ii) e, C1 # O. For any qı where vs e, 0 nC, the following holds:
1. Pa.c (1H (1'' I6 >0 200) 2 0.5, VP E [0,
4

---

Published as a conference paper at ICLR 2024
Unitormity
Class 0
Class
Class
Class 4
Log of singular values
-2
W/O Projection head
Random
initialization
W/
Projection head
-10
Our method
-18
60
120
Singular value rank index
(a) Representation spectrum analysis. A sudden singular- value drop implies dimensional collapse.
(b) Representations on the unit hypersphere. Uniformly distributed representations achieve uniformity, and representations of nodes of the class located close each other achieve alignment.
Analysis regarding Property 2 (prevention of dimensional collapse) and Property 3 (rep- resentation uniformity and alignment) of HYPEBOY. As shown in (a), while HYPEBOY without projection heads (red) suffers from dimensional collapse, HYPEBOY (blue) does not, demonstrating the necessity the projection head. Furthermore, as shown (b), representations from an HNN trained by HYPEBOY meet both uniformity and alignment, justifying our design choice of the loss function. Experiments are conducted on the Cora dataset.
Figure 2:
2. Pæçe (1' Enneos 2k 00>0 P is strictly decreasing function w.r.t. P [0, 0.5] and strictly increasing function w.rt. P [0.5, 1].
Proof. A full proof is provided in Appendix A.2.
Theorem 2 states that the probability of the condition being satisfied is least 0.5 for any affin- ity parameter P E [0, 1]. Moreover, the probability strictly increases w.r.t P E [0.5, 1] and that strictly decreases w.r.t. P [0, 0.5). Thus, peredge including nodes the same class increases, so does the likelihood of the condition being met. Notably, many real-world group interactions exhibit homophilic traits (Laakasuo et al., 2020; Khanam et al., 2023). Therefore, the hyperedge filling task can improve node classification in many real-world scenarios, evidenced by our theoretical findings and real-world characteristics.
be inferred that the probability of each hy-
Generalization to the class Co- The above results can be easily generalized to the class Co due to the symmetry. Specifically, for each node v, Co. the effecttivnness (spec., the expected accuracy of the classifier F) of a derived representation z; is greater than that of an original feature under a certain condition. The probability for such a condition to hold strictly increases w.r.t. [0.5, 1] and strictly decreases w.r.t. P E [0, 0.5]. We theoretically show this in Appendix A.1 and A.2.
4 PROPOSED METHOD FOR HYPEREDGE FILLING
In this section, we present HyPEBOY (Hypergraphs, build own hyperedges), a hypergraph genera- tive SSL method based the hyperedge filling task (Section 3). HYPEBOY exhibits the following desirable properties of hypergraph generative SSL, as empirically demonstrated later:
Property 1. Does over-emphasize proximity information (Veličković al., 2019).
Property 2. Avoids dimensional collapse (Jing et al., 2022) in node representations.
Property 3. Learns node representation to be aligned and uniform (Wang & Isola, 2020).
Our proposed method, HYPEBOY, illustrated in Figure 1(b). After presenting each of its steps, we discuss its role in satisfying the above properties. Lastly, introduce a two-stage training scheme for further enhancing HYPEBOY.
4.1 STEP 1: HYPERGRAPH AUGMENTATION
HYPEBOY first obtains augmented feature matrix X' and hyperedge set E' by using augmentation functions Ta and Tg, respectively. The feature augmentation function certain entries of X based on Bernoulli sampling (spec., X' = X M, where O is an element- wise product and Mjj B(1, 1 po), [[V]], [d). The topology augmentation function
(X, Po) + X' masks
5

---

Published as a conference paper at ICLR 2024
TE (E) + E' samples [] pe)l hyperedges uniformly at random from &. Note that the magnitudes feature and topology augmentations are proportional to pe. Pe E [0,1], 1], respectively.
Role. For hypergraph SSL, augmentations crucial for mitigating overly- werly-emppassied proximity information. That is, using all hyperedges and/or features for both message passing and objective- function construction may risk HNNS heavily rely on direct neighborhood information (Tan al., 2023). Many graph SSL. strategies mask certain input edges and/or node features, preventing their encoder models from overfitting to the neighbor distribution and features (Hou et al., 2022; Li al., 2023; Tan et al., 2023). Motivated by their findings, we use the augmentation step. In Appendix F.I, we demonstrate that augmentation enhances node classification performance of HYPEBOY.
4.2 STEP 2: HYPERGRAPHI ENCODING
After augmentation, HYPEBOY obtains node and query subset representations. First, HYPE- BOY obtains node embeddings Z E R/V by feeding the augmented hypergraph into en- coder HNN: = fo(X','") Then, HYPEBOY obtains projected representations of query subsets (i.e., Rej. 04 E ej, E) and nodes. To this end, we utilize a node projection head f and set projection head f. Specifically, projected embeddings of node V; and query subset Kú h, = (z,) IRt and q; = f= (",,,,, 2,) E TR. respectively. Here, the design choice of the set projection head is motivated by Deep Sets (Zaheer et al., 2017).
Role. We investigate the role of the projection heads, which are non-trivial components, in the text of the dimensional collapse embeddings. Dimensional collapse is a phenomenon in which embedding vectors occupy only the lower dimensional sub-space of their full dimension (Jing al., 2022). This is identified by observing whether or not certain singular values of the embedding co- variance matrix drop zero. To prevent dimensional collapse, we employ projection heads in HY- PEBOY, and this is in line with the prior findings of Jing et al. (2022) and Song et al. (2023). Fig- ure 2(a) illustrates that an HNN trained using HYPEBOY avoids dimensional collapse, whereas an HNN trained with a HYPEBOY variant (without projection heads) does not. Results on more datasets are in Appendix F.2. Furthermore, we provide a theoretical analysis of why HYPEBOY without pro- jection heads may suffer from dimensional collapse in Appendix B.2. Note that this distinction leads performance discrepancy node classification (Section 5.3).
4.3 STEP HYPEREDGE FILLING LOSS
The last step is to compute the SSL loss based on the hyperedge filling probability. We design P(x,E,e) (U% 9.y) to be normalized over V (i.c., Ener P(x. (vx 9.y) 1). To this end, we utilize a Softmax function to model probabilities. sum, with projected embeddings h, and zi (Section 4.2), the probability of node completing query subset d; is defined as follows:
exp(sim(h;, 4:z))
P(X,E,) (e: (g) Ener exp(sim(hu: q:))
(4)
where sim is the cosine similarity function (other similarity functions are also appli- cable). Lastly, HYPEBOY is optimized for all possible hyperedge filling cases (i.e., eel,eell,,e,,Pas P(x.E,,) (ox 4s))). To sum up, HYPBBOY minimizes the negative log-likelihood of all possible cases as follows:
h:=- Σ Σ log exp(sim(h,, q:)))
(5)
Ev.ey exp(ssiml 9:z))
Note that the set θ of all parameters consists of the parameters of the encoder HNN fe, the node projection head f. and the set projection head f (i.e., e (0, &, p)). They updated by gradient descent, aiming to minimize the loss L defined in Eq. (5).
Role, Our design choice of (.). ensures that representations learned by HYPEBOY achieve both alignment and uniformity (Wang & Isola, 2020). In our case, alignment indicates that node embeddings belonging to the same class are closely located to each other, and uniformity indicates that embeddings are uniformly distributed over the embedding space. Through the numerator of Eq. (5), HYPEBOY pulls representations a missing node and a query subset, promoting the align- ment discussed in theoretical analysis (Section 3.2). At the same time, the denominator of

---

Published as a conference paper at ICLR 2024
Eq. (5) pushes away every node representation from each query subset representation, encouraging that representations are uniformly distributed. This intuition is supported by the findings of Wang & Isola (2020): the denominators of their contrastive loss, which push away representations from each other, encourage uniformity. As shown in Figure 2(b), we verify that node representations from HYPABOY achieve both alignment and uniformity. Results more datasets in Appendix F.3.
4.4 TWO-STAGE TRAINING SCHEME FOR FURTHER ENHANCEMENT
In our preliminary study, we observed that with a randomly initialized encoder, HyPEBoYtends to rely too heavily on the projection heads rather than the encoder parameters, leading to sub-optimal training. Thus, we propose a two-stage training scheme to enhance the effectiveness HYPEBOY. In nutshell, we first train an HNN So to reconstruct masked node features (inspired by Hou et al. (2022)). Then, we use the trained fe parameters for the initialization of HYPEBOY's 's encoder to leam hyperedge filling (Sections 4.1.4.3). This feature warm-up for the HNN encoder repeats the following three steps for fixed number of epochs, which we set to 300 in all our experiments.
Step 1: Encoding. For each feature-reconsstuuction training epoch, we sample a certain number of nodes uniformly at random from V/ which we denote as V C V. Then, we mask the features of sampled nodes by using a learnable input token (1) E IRd and use the resulting masked node feature matrix X" E RIV]xd, as the input node feature matrix (i.e., X" = mm()) Ve, E VV and X = Xj Vuz E V \ V'). We also obtain augmented hyperedges E' by using the strategy described in Section 4.1. Then, we obtain node embeddings Z' E BRV] xd' as follows: = fo(" E').
Step 2: Decoding, We again mask the embeddings of the nodes that we sampled earlier with learnable embedding token m (F) E Rd obtaining the masked node embedding matrix Z" (i.e., Z"" ml() Vu; E V and Z 3 Vuz E VV V"). Then, we acquire reconstructed node features X E R [ by using a decoder HNN gø as follows: X = (7" E').
Step 3: Updating, After gaining reconstructed node features X, we maximize the similarity be- tween the original and reconstructed features of the masked nodes. To this end, we minimize the following loss function, which is proposed by Hou et al. (2022): L' = (1/D²) En,ev-(1 ( / X (I). Using gradient descent to minimize L', we update the parameters of the encoder HNN Se, the decoder HNN gv, the input token (1), and the embedding token (B)
EXPERIMENTAL RESULTS
We now evaluate the efficacy of HYPEBOY as techniques for (1) pre-training hypergraph neural net- works (HNNs) for node classification (Section 5.1) and (2) learning general-purpee purpose representations (Section 5.2). Then, we justify each of components through an ablation study (Section 5.3).
Datasets. For experiments, we use 11 benchmark hypergraph datasets. The hypergraph datasets are from diverse domains, expressing co-citation, co-authorship. computer graphics, movie-actor, news, and political membership relations. In Appendix D, we detail their statistics and descriptions.
Baselines methods. We utilize 16 baseline methods. They include (a) 10 (semi-) sxpprvisude HNNs, including 2 state-of-the-art HNNs (ED-HNN (Wang et al., 2023a) and PhenomNN (Wang et al., 2023b)), (b) 2 generative SSL strategies for ordinary graphs (GraphMAE2 (Hou et al., 2023) and MaskGAE (Li et al., 2023)), and (c) 4 SSL strategies for hypergraph (TriCL (Lee & Shin, 2023a), HyperGCL (Wei et al., 2022), HyperGRL (Du et al., 2022), and H-GD, which is a direct extension g graph SSL method (Zheng et al., 2022) to hypergraphs). We use UniGCNII (Huang & Yang, 2021)* and GCN (Kipf & Welling, 2017) as the backbone encoders for hypergraph SSL methods and ordinary graph SSL methods, respectively 3. In Appendix E, we provide their details, including their implementations, training, and hyperparameters.
of ordinary
HyPEBOY. We utilize UniGCNII encoder of HYPEBOY, which is the same as that of other hypergraph SSL methods. For both node- and set projection heads, we use an MLP. Further details of HYPEBOY, including its implementations and hyperparameters, are provided in Appendix E.4.
"HyperGRL utilizes a GCN as its hackhone encoder since its input is an ordinary graph. "Results for alternative encoders be found in Appendix F.4.
7

---

Published as a conference paper at ICLR 2024
Table 1: Efficacy as pre- pre-training techniques: AVG and STD of accuracy values in node classification under the fine-tuning protocol. The best and second-bes best performances low, respectively. R.G. and A.R. denote the random guess and average ranking among all methods, respectively. O.O.T. means that training is not completed within 24 hours. HYPEBOY outperforms all the baseline methods in 8 datasets, and overall, it obtains the best average ranking.
colored green and yel-
Method
Clseseer
Cora
Pubmed
Cor--CA DBLP-P
DBLP-A AMiner IMDB
MN-40
20News
House
|.R.
R.G.
18.1 (69) 17.4 (10) 36.0 (0.7) 17.8 (02) (8.9 25.6 (1.0) 10.2 (022) 33.9 (0z) 3.7 (0:2) (1.) 26.7 (0.3
17.8
MLP
(1.0)
27.9 (10) 62.1 (3.) 34,8 (5.) 73.5 (1.0) 56.0 (4.) 22.3 (1.7) 39.1 (24) 89.4 (1.5) 73.1 (144 72.2 (39)
14.2
(Semi- (Sap--iipeeddd
HGNN
(1.) 50.0 (12) (5.0
(5.0) (62) 85.3 (08) 67.1 (60) 30.3 (25) 42.2 (2.9 8800 (1.) 76.4 (1.9) (2.7
(38)
10.7
HyperGCN
(9.) 33.1 (102) 63.5 (1444 37.1 (91) 53.5 (11.) (144) 26,4 37.9 55.1 (18) (94) 49.8 (35)
15.6
HNHN
(8.7)
50.0 (19) 72.1 (54) 48.3 (62) 84.6 (436 30.0
42.3 861 (1.) (1.5 49.7 (2:)
12.6
UniGCN
(8.2
49.1 (84) 74.4 (39) 51.3 (63) 86,9 (06) (47) 32.7 (18) 41.6 (35) 89.1 (1.0) 77.2 (1.2) 51.1 (24)
9.6
UniGIN
(9.1) 47.8 (12) 69.8 (5.6) 48,3 (6.1) 83.4 (ox) 63.4 (5.)) 30.2 (1.4) 41.4 (27) 822 (1.) 70.6 (L.) 51.1 (3.0)
13.9
UniGCNI1
(2.) 48.5 (1.4) 74.1 (3.9) (7.5 87.4 (06) 65.8 32.5 (1.7) 42.5 (39) 90.8 (1.1) 70.9 (1.0) 50.8 (4:3)
9.4
AllSet
(3.0)
(42) 72.4 (4.5) (5.) 85.9 (06)
(393 29.3 (122) 42.3 (24) 02. (66) 71.9 (2.5) 54.1 (34)
10.4
ED-HNN
(8.0) 47.6 (12)) 72.7 (4:) 54.8 (54) 86.2
65.8 (4.3 30,0
41.4 90.7 (6.3
(37)
10.4
PhenomNN
49.8 (9.6 56.4 (96)
76.1 (3.5) 60.8 (6.2) 88.1
72.3 (4.3) 33.8 (2.0) 44.1 959 (1.0 70.4 (56)
3.9
GraphMAE2
41.1 (10.0) 49.3 (8:3)
72.9 (4:2) 55.4 (8.4) 86.6 (0.6)
69.5 (4.4) 32.8 (1.9) 43.3 90.1 (0.7) (1.9 52.8 (35)
8.9
Self Self-supervieel
MaskGAB
49.6 (10.)) 57.1 (83)
72.8 (4.3) 57,8 (59)
86,3 (0.5)
74,8 (3.1) 33.7 (1.0) 44.5 (2:5) 90.0 (0.9) O.O.T. 51.8 (333
7.7
TriCL
(9.8 60/2 (75)
76.2 (36) 64.3 (55)
88.0 (o.4)
79.7 (2.9) 33.1 (2:2) 46.9 (29) 90.3 (1.0) 77.2 (1.0) 69.7 (4:9
3.4
HyperGCL
(92) 60.3 (14)
76.8 (3) (2.0 87.6 (0.5)
79.7 (38) 33.2 (166 (3.
(3.) 91.2 (0.8) 77.88 (0.8 69.2 (49)
3.5
H-GD
(5.4 50.6 (82)
(3.) 58.8 (6.2 87.3 (0.5)
75.1 (5.) 32.6 (2:2) 43.0 90.0 (1.0) 77.2 (1.0) 69.7 (5.1)
6.1
HyperGRL
42.3
(83)
73.0 (43) 55.8 (88) 86.7 (0.6) 70.8 (3.) 33.0 (1:8 43.1 90.1 (0.) O.O.T. 52.5 (33)
9.2
HYPEBOY
56.7 (28) (12) (14) (46) 88.2 80.6 (2.) (2:2 47.6 (2.) 90.4 (0.9) 77.6 (0.9) 70.4 (48)
1.7
5.1 EFFICACY AS A PRE-TRAINING TECHNIQUE (HINE-TUNED EVALUATION)
Setup. Following Wei et al. (2022), we randomly split the nodes into traininn/validatiionness sets with the ratio of 1%/1 %/98%, respectively.* For reliability, we assess each method on 20 data splits across 5 random model initializations, following Lee & Shin (2023a). We report the average (AVG) and standard deviation (STD) of test accuracy values on each dataset. Specifically, for each SSL strategy, including HYPEBOY, we pre-train backbone encoder with the corresponding SSL scheme and then fine-tune the encoder in a semmi))supervise manner.
Results. As shown in Table 1, HYPEBOY shows the best average ranking among all 18 methods. Two points stand out. First, pre-training an HNN with HYPEBOY generally improves node clas- sification. Compared to the performance of UniGCNII (HyPEBOY'S s backbone encoder), HYPE- BOY obtains performance gains up to 12.5 points in 10 out of 11 datasets. Second, HYPRBOY out- performs all other SSL strategies. Specifically, compared to the second-best method (TriCI.,, the accuracy gap is up to 5.0 points. In addition, the suboptimal performance of the state-of-the-art generative SSL strategies for graphs (i.e. GraphMAE2 and MaskGAE) implies the importance of preserving higher-order interactions in learning hypergraph representations. In summary, HYPE- BoY serves as an effective SSL strategy to pre-train HNNs for node classification.
5.2 EFFICACY AS A GENERAI-PURPOSS EMBEDDING TECHNIQUE (LINEAR EVALUATION)
Setup. We assess the generalizability of learned representations from HYPEBOY in two downstream tasks: node classification and hyperedge prediction. Considering this objective, we limit the base- line methods to SSL strategies, which yield embeddings independent of downstream tasks, and the original node features (i.e., naive X). We use the linear evaluation protocol (i.e., the embeddings are used fixed inputs to the classifiers for each task). For node classification, we use the same settings described in Section 5.1. For hyperedge prediction. we split hyperedges into sets by the ratio of (0%/20%/20%/ For its evaluation, we obtain the same number of negative hy- peredge samples as that of the ground-truth hyperedges. We report the average (AVG) and standard deviation (STD) of test AUROC values on each dataset. Further experimental details about hyper- edge prediction, including negative hyperedge sampling, are provided in Appendix E.3.
Results. As shown in Table 2, HYPEBOY has the best average ranking in both node classification and hyperedge prediction. Specifically, in node classification, compared to the second- best method (TriCL), the accuracy gap up to 6.3 points. This demonstrates that HYPeBoY is more effective in learning general-purpose hypergraph representations than the other SSL strategies on hypergraphs.
We ensure that training includes least one node from each class.
8

---

Published as a conference paper at ICLR 2024
Table 2: Efficacy as general-punge purpose embedding techniques: AVG and STD of accuracy/AURoC values in node classification and hyperedge prediction tasks under the linear evaluation protocol. In each downstream task, the best and second-best performances are colored green and yellow, respectively. A.R. denotes the average ranking among all methods. completed within 24 hours. HYPEBOY obtains the best average ranking in both downstream tasks.
O.O.T. means that training is not
Method
Citeseor
Com
Pahmed
Cora-CA DRLP.P
DBLP-A
AMiner
IMDB
MN-40
20News
House
A.R.
Node clussitccation
Naive X
27.8 (1.0) 32. (288 (S. 69.4 (0.7) (4.7)
(12) 38.1 (1:) (1)) (1.6 (544
5.5
GraphMAE2
29.
(1.0 (.5) (8.1)
75.6 (1.7) (5.) 27.3 (27) 36.6
(9.1
62.3 (23) (3.)
MaskGAE
(11.) (63)
56.0 (5.6
84.8 (0:7) 75.1 (35)
(21) (2))
44,1 (39) (0)) O.O.T. 52.0
4.5
TriCI.
(303)
(k3)
(3.6 87.
80.9
35.00300
(3.0 (5.0(5)) 67.2 (ao) (.5)
266
HyperCCC
(8.6 (s.3 (8.6 (6.) (52) (38)
(22)
47.5 (2.9) (28) 71.2 (4) (5.4
H-GD
37.6 (68) (12)
48.6 (14)
73.3
(3.)
33.8 (50)
(29) (6.))))) (24 (8.3
5.5
HyperGRI.
(x.3
(k.6
(8.)
(k.)
78.7
(2.7
(2.0
34.8 (100
(1.(0.))
O.O.T. (2.0
HYPEBOY
(9.9
6945 (340
66.0 (4.0)
87.9
81.2
34.3 (3:)
48.8
89.2 (2.) 75.7 (20) (5.4
1.5
Hyperedge prediction
Naive
(2.3
68.3 (1.6)
55.0
90.0
72.1 (133
80.0
39.5
(0))
97.7 (19) (4.8
6,4
GraphMAE2
(2.3
(1.7
(1.1)
(1.3
85.2 (0.4) 68.3 (16)
80.7 (19)
53.7 (2.6)
99.5
90.1 (5:2 (2.9
6.1
MaskGAE
86.1
88.5 (1.4)
92.9 (0:)
81.8
93.2 (0:5)
79.3
84.6
58.1
O.O.T. (01)
(3.)
4.1
THCC.
(1.5
(1. 91.9 (0:5)
07.8
94.8 (0:2) 87.9
90. (6)) 58.9
(9.6
98.2 (10) (0.0
188
HyyeeGGC
(2.9
(1.4
(..6
(1.) 83.6 (04) (3.5
82.1 (76)
53.8
(o.)
96.7 (10) (633
5
H-GD
(2.0
(3.)
87.2 (02))
73.2 (4.0
91.6
(1.9)
84,9
53.1
99.5
(21)
(3.)
5.3
HyperGRI.
(k.0)
(1.)
78.3 (1.8 80.0 (3.)
88.2 (0.3)
80.8 (144
81.5 (0:7)
54.7 (2.9)
98.8 (0.4)
O.O.T.
88 (3.3
5.5
HYPuBoY
01.1
(1.) (0.) 88 104
95.5
89.8 (133
(0:)
59.4 (3.1) (01) (16) 87.0 (2.)
1.5
Table 3: The ablation study with four variants of HYPEBOY on node classification under the fine- tuning protocol. The best and second-best performances are colored green and yellow, respectively. F.R.. H.F., and P.H. denote Feature Reconstruction, Hyperedge Filling, and Projection Heads, re- spectively. A.R. denotes the average ranking among all methods. NA denotes pre-training. HYPEBOY outperforms others in most datasets, justifying each of its components.
F.R. H. H.H.
Citeseer
Cora
Pubmed Cons-CA DBLP-P DBLP-A AMiner
IMDB
MN-40
20News
House
A.R.
NA
X
(4.) (145 74.1 54.8
(06) 65.8 (19) (2.((1)) (1.5 90.8 (11) 70.9 (16) 50.8 (4:1)
5.5
VI
X
x
(1.2
60.7 (62) (662 63.5 (600 88.1 (o5) 78.5 (2.9 33.5 (2.) 46.8 90,0 (L.) 77.4 (29) 68.5 (4.5)
4.3
52.7 (16) 59.7 (91)
(o1) 76. (32) 63.5 (60) 88.2 19.1 (3:5 (33) (63) 90.6 (1.0 77.0 (09) (9.6(99)
(0.6
3.3
V
(230
58.9 (83) 74.1 (39) (x0) 87.8 (0.) 79.9 (3.9
(2.1 (2.3 91.4 (0.) 77.5 (09) 70.1 (4.5)
3.6
(6.0
(6.) 76.5 (3.1) (4.3) (8.0 80.3 (2.4) 34.0 (2.0) (2.3 90.8 (1.0) 77.4 (1.0) 69.3 (5:0)
2.5
Ours]
V
56.7 (9x) (1.3 (5.0 (4.3 88.2 (0:4) 80.6 (3.6 34.1 (1:3 (7.6 90.4 (0.9)
77.6 (0:) 70.4
1.4
5.3 ABLATION STUDY
We analyze the necessity for each component of HYPEBOY, specifically, (a) the hyperedge filling task (Section 3.1). (b) projection heads (Section 4.2), and (c) feature reconstruction warm-up (Sec- tion 4.4). To this end, we utilize four variants of HYPEBOY: (VI) without feature reconstruction warm-up and projection heads, (V2) without feature reconstruction warm-up S, (V3) without the hy- peredge filling process, and (V4) without projection heads. Here, projection heads are used only for methods with the hyperedge filling process. Note that V3 is a method that only utilizes the feature reconstruction process, which is described in Section 4.4.
As shown in Table 3, HYPEBOY, equipped with all of its components, outperforms the others in most datasets, demonstrating the effectiveness of our design choices. There are two other notable results. First, the necessity of projection heads is evidenced by the superior performance of V2 (compared to V1) and ours (compared to V4). Second, the advantage of the hyperedge filling task over feature reconstruction is manifested by the better average rank of V2 compared to V3.
CONCLUSION
In this work, conduct a comprehensive analysis of generative self-supervised learning on hyper- graphs. Our contribution is three-fold. First, we formulate the hyperedge filling task, generative self-supervised learning task on hypergraphs, and investigate the theoretical connection between the task and node classification (Section 3). Second, we present a generative SSL method HYPEBOY to solve the task (Section 4). Third, we demonstrate the superiority of HYPEBOY over existing SSL methods on hypergraphs through extensive experiments (Section 5).
STo mitigate issue of over-relying projection heads (Section 4.4), we have trained an encoder without projection heads the beginning, and after some epochs, we train the encoder with projection heads.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This work was supported by Samsung Electronics Co., Ltd. and Institute of Information & Commu- nications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2022-0-00871, Development of AI Autonomy and Knowledge Enhancement for AI Agent Collaboration) (No. 2019-0-00075, Artificial Intelligence Graduate School Program (KAIST)).
REFERENCES
Ali Behrouz, Farnoosh Hashemi, Sadaf Sadeghian, and Margo Seltzer. Cat-walk: Inductive hyper- graph learning via walks. In NewrlPS, 2023.
Austin R Benson, Rediet Abebe, Michael Schaub, Ali Jadbabaie, and Jon Kleinberg. Simplicial closure and higher-order link prediction. Proceedings of the National Academy of Sciences, 115
(48):E1 1221-E1 E1130, 1230, 2018.
Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer, 2006.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
Chien, Chao Pan, Jianhao Peng, and Olgica Milenkovie. You are allset: A multiset function framework for hypergraph neural networks. In ICLR, 2022.
Minyoung Choe, Sunwoo Kim, Jaemin Yoo, and Kijung Shin. Classification of edge-dependent labels of nodes in hypergraphs. In KDD, 2023.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NACCL, 2019.
Yihe Dong, Will Sawin, and Yoshua Bengio. Hnhn: Hypergraph networks with hyperedge neurons. In ICML Workshop on Graph Representation Learning and Beyond (GRL+), 2020.
Boxin Du, Changhe Yuan, Robert Barton, Tal Neiman, and Hanghang Tong. Self-supervised hyper- graph representation learning. In BigData, 2022.
Dheeru Dua, Casey Graff, et al. Uci machine learning repository. 2017.
Yifan Feng, Zizhao Zhang, Xibin Zhao, Rongrong Ji, and Yue Gao. Gvenn: Group-view convolu- tional neural networks for 3d shape recognition. In CVPR, 2018.
Yifan Feng, Haoxuan You, Zizhao Zhang, Rongrong Ji, and Yue Gao. Hypergraph neural networks. In AAAI, 2019.
Bean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. In NeurIPS, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. CVPR, 2022.
Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong. Hongxia Yang, Chunjie Wang, and Jie Tang. Graphmae: Self-supervised masked graph autoencoders. In KDD, 2022.
Zhenyu Hou, Yufci He, Yukuo Cen, Xiao Liu, Yuxiao Dong, Evgeny Kharlamov, and Jie Tang.
Graphmae2: A decoding-enhance masked self-supervised graph learner. WWW, 2023.
Jing Huang and Jie Yang. Unignn: a unified framework for graph and hypergraph neural networks. In ISCAI, 2021.
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. In ICLR, 2022.
10

---

Published as a conference paper at ICLR 2024
Kazi Zainab Khanam, Gautam Srivastava, and Vijay Mago. The homophily principle in social network analysis: A survey. Multimedia Tools and Applications, 82(6):8811-888 8854, 2023.
Sunwoo Kim, Fanchen Bu, Minyoung Choe, Jaemin Yoo, and Kijung Shin. How transitive are real-world group i and reproduction. KDD, 2023a.
Sunwoo Kim, Minyoung Choe, Jaemin Yoo, and Kijung Shin. Reciprocity in directed hypergraphs:
measures, findings, and generators. Data Mining and Knowledge Discovery, 37(6):230-22888 2023b.
Sunwoo Kim, Dongjin Lee, Yul Kim, Jungho Park, Taeho Hwang, and Kijung Shin. Datasets, tasks, and training methods for large-scale hypergraph learning. Data Mining and Knowledge Discovery, 37(6):2216-2254, 2023c.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In ICLR, 2017.
Michael Laakasuo, Anna Rotkirch, Max Van Duijn, Venla Berg, Markus Jokela, Tamas David- Barrett, Anneli Miettinen, Eiluned Pearce, and Robin Dunbar. Homophily in personality enhances group success among real-life friends. Frontiers in Psychology, 11:710, 2020.
Dongjin Lee and Kijung Shin. I'm me, we're us, and i'm us: Tri-directional contrastive learning on hypergraphs. In AAAI, 2023a.
Geon Lee and Kijung Shin. Temporal hypergraph motifs. Knowledge and Information Systems, 65 (4):1549-1586, 2023b.
Geon Lee, Fanchen Bu, Tina Eliassi-Rad, and Kijung Shin. A survey on hypergraph mining: Pat- terns, tools, and generators. arXiv preprint arXiv:240.00887, 2024.
Jintang Li, Ruofan Wu, Wangbin Sun, Liang Chen, Sheng Tian, Liang Zhu, Changhua Meng, Zibin Zheng, and Weiqiang Wang. What's behind the mask: Understanding masked graph modeling for graph autoencoders. In KDD, 2023.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In ICML, 2010.
OpenAl. Gpt-4 technical report. 2023.
Prasanna Patil, Govind Sharma, and M Narasimha Murty. Negative sampling for hyperlink predic- tion in networks. In PAKDD, 2020.
Ding Ruan, Shuyi Ji, Chenggang Yan, Junjie Zhu, Xibin Zhao, Yuedong Yang. Yue Gao, Changqing Zou, and Qionghai Dai. Exploring complex and heterogeneous correlations on hypergraph for the prediction of drug-target interactions. Patterns, 2(12):003990, 2021.
Khaled Mohammed Saifuddin, Briana Bumgardner, Farhan Tanvir, and Esra Akbas. Hygnn: Drug- drug interaction prediction via hypergraph neural network. In ICDE, 2023.
Ramit Sawhney. Shivam Agarwal, Arny Wadhwa, and Rajiv Ratn Shah. Spatiotemporal hypergraph convolution network for stock movement forecasting. In ICDM, 2020.
Ramit Sawhney, Shivam Agarwal, Amav Wadhwa, Tyler Derr, and Rajiv Ratn Shah. Stock selection via spatiotemporal hypergraph attention network: A learning to rank approach. In AAAI, 2021.
Zeen Song, Xingzhe Su, Jingyao Wang. Wenwen Qiang, Changwen Zheng, and Fuchun Sun. Towards the sparseness of projection head in self-supervised learning. arXiv preprint arXiv:2307.08913. 2023.
Hang Su, Subhramsu Maji, Evangelos Kalogerakis, and Erik Cearned-Milles Multi-view convolu- tional neural networks for 3d shape recognition. In 2015.
11

---

Published as a conference paper at ICLR 2024
Qiaoyu Tan, Ninghao Liu, Xiao Huang, Soo-Hyun Choi, Li Li, Rui Chen, and Xia Hu. S2gae: Self-supervised graph autoencoders are generalizable learners with graph masking. In WSDM, 2023.
Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang. Jun Yan, and Qiaozhu Mei. Line: Large-scale information network embedding. In WWW, 2015.
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang- Videomae: Masked autoencoders are data- efficient learners for self-supervised video pre-training. In NeurlPS, 2022.
Nate Veldt, Austin R Benson, and Jon Kleinberg. Combinatorial characterizations and impossibili- ties for oigher-order homophily. Science Advances, 9[))eabb33200 2023.
Petar Veličković, William Fedus, William Hamilton, Pietro Liō, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. In ICLR, 2019.
Peihao Wang, Shenghao Yang, Yunyu Liu, Zhangyang Wang, and Pan Li. Equivariant hypergraph diffusion neural operators. In ICLR,, 2023a.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In ICML, 2020.
Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S Yu. Hetcrogencous graph attention network. In WWW, 2019.
Yuxin Wang, Quan Gan, Xipeng Qiu, Xuanjing Huang, and David Wipf. From hypergraph energy functions to hypergraph neural networks. In ICMI,, 2023b.
Tianxin Wei, Yuning You, Tianlong Chen, Yang Shen, Jingrui He, and Zhangyang Wang. Augmen- tations in hypergraph contrastive learning: Fabricated and generative. In NeurlPS, 2022.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong
Xiao. 3d shapenets: deep representation for volumetric shapes. CVPR, 2015.
Lianghao Xia, Chao Huang, and Chuxu Zhang. Self-supervised hypergraph transformer for recom- mender systems. In KDD, 2022.
Naganand Yadati, Madhay Nimishakavi, Prateek Yadav, Vikram Nitin, Anand Louis, and Partha Talukdar. Hypergen: A new method for training graph convolutional networks on hypergraphs. In NeurlPS, 2019.
Nagamand Yadati, Vikram Nitin, Madhav Nimishakavi, Prateek Yadav, Anand Louis, and Partha Talukdar. Nhp: Neural hypergraph link prediction. In CIKM, 2020.
Se-eun Yoon. Hyungseok Song, Kijung Shin, and Yung Yi. How much and when do need higher-order information in hypergraphs? a case study on hyperedge prediction. In www, 2020.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In NeurlPS, 2020.
Junliang Yu, Hongzhi Yin, Jundong Li, Qinyong Wang. Nguyen Quoe Viet Hung, and Xiangliang Zhang. Self-supervised multi-channel hypergraph convolutional network for social recommenda- tion. In WWW, 2021.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander Smola. Deep sets. In NeurIPS, 2017.
Chuxu Zhang, Dongjin Song, Chao Huang, Ananthram Swami, and Nitesh V Chawla. Heteroge- neous graph neural network. In KDD, 2019.
Junwei Zhang, Min Gao, Junliang Yu, Lei Guo, Jundong Li, and Hongzhi Yin. Double-scale self- supervised hypergraph learning for group recommendation. In CIKM, 2021.
Yizhen Zheng. Shirui Pan, Vincent Lee, Yu Zheng, and Philip S Yu. Rethinking and scaling up graph contrastive learning: An extremely efficient approach with group discrimination. In NeurlPS, 2022.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
In this section, we provide a proof of each theorem. In addition, we extend the results from Sec- tion 3.2, which assume vi Ci is assumed, to include cases where E Co-
A.I PROOF OF THEOREM 1
Proof. We first analyze node o; that belongs to C1 (i.e., 8/ E C1). By definition, Eg [x))))))- Pa (f(a ,1; 1) > (a po; I)) holds. Thus, Ex [(e))=] is determined by the distribution of
fær p1, I) (ay No, I), where random variable is 2;. We provide the exact formula for this inequality:
f( pl, 1) > ((x Po: 1);
= eexp(-(2t ) (x: - 11)) > exp(- (o))((r; - 10)),
=(x; 11)² (a:; (I) < (x; ma0) (x: (4)),
-af w; 244 az pi in af x; 2 x; + Po no:
=(m - mo) T x. - po no - 2 un > 0 1" x, > 0, :. defiiiiiio of μn and po in Assumption 1.
Thus, Ex [1) is equivalent (T x; 0). In similar sense, E ((xx)==] is equivalent to Pa [I 2; > 0), since as mentioned in Section 3.2, z; is function of ax. Veg E V.
æ. L holds by (F2). To further elaborate on this Now, formalize 2,. Note that z, 2; Vv, equation, we first detail C:
(- -log
exp(x? (E Eggy #+))
V = O (- log(((X,,,0) (v: (is)))
En explx> (2 on Cves mx))
(6)
dx.
dx:
T (Eoneor 2xi)
I 0log Emeev exp(x" (Emens xx))
Ox;
exp(x? (E (Evyegn xx))
Σ Wx METg
Σ ØR Ex
(8)
Eneey (xp (Eusens xxx))
exp(xx (EoLen:: Rk))
Eney exp(x? (Err xb,))
Tb
(9)
t/e Eths
By Eq (9), z; can be expressed as a function of æh. Vou E V, as follows:
exp(æ? (Enren 21))
Σ (g
2i Bi + 7 Σ Eeev xxp (Eoseo 2x,))
(10)
(Term 1)
Let x', denotes Ee Eq., Øg. Furthermore, denote (Term 1) Eq (10) as f(x) E (0, 1). Then, we
rewrite Eq (10) as z; 0; 7 (1 f(x)) ayy We finally rewrite T z; as follows:
T²z = IT + 7(1 f(x))
(11)
Let 8 denotes I x.y- Note that by the statement of Theorem 1, IT x 0 = B> 0 holds. Thus, our main interest, which is Pa (1T 2, > 0), can be rewritten as follows:
Pa ["r z, > 0) Pa ((T' + + f(x)))) > >0).
(12)
13

---

Published as a conference paper at ICLR 2024
Note that when I'x; x; > 0 holds, 1" 24 > 0 holds well, since 7(1 f(x)){ 3 0 holds. Thus, Eq (12) is split as follows:
Po
I z, > 0) = Pr a 00,>00 0) Pa (->(1-1(x))8 < <1'as æ; < <0);
1 x
= =m [
Em [1(0)))
+ P -78 (1 f(x) <0
(13)
(a) Expected accurncy of naive
(b) Addittionl guin via hyperedge Gilling
Note that the gain term, which is the (b) term of Eq (13), is always greater than zero.
Generalization to Dt E Cg. Now, we analyze a node v, that belongs to Co (i.c., E/ E Co). this case, the previous condition I' xy > 0 = =3 > 0 becomes 1' I' æ'
B < In a similar sense, for the expected accuracy: Pa I z; > 0 is changed as Pe I 23 < 0). In this setting, we can directly extend the result Eq (12) follows:
Pe 1 1 2, <)) Pæ (1l f(z){))) < <0).
(14)
By employing the above proof, we can obtain the following result (note that 3 < 0 in this case);
Pa 1 <
Pa [" 0) Pa (0 <0<(1=<>>>11- <=<(1_((x)))
1 x
Em F(())(=)==)==
Eu [L
+ Pa -78 (1 f(x)) 0
(15)
Expected accuracy a;;
(b) Addittional gain via hyperedge filling
Thus, we can also derive the same result for vs E Co.
A.2 PROOF OF THEOREM
Remark, We have denoted the affinity parameter in Assumption 2 as P to distinguish it from the hyperedge filling probability P(x, In this proof. by allowing slight duplication notation, let P = since we do not use P(),E,8) in this proof. In addition, we let == and qij ==
Proof. We first derive the functional form of Pa.e (i' 2x > p). By vs E e, ni.
which is the condition of the theorem, the following holds:
Pa.c [iT (Emes
0, 200Ee
Pæ.e
1"
Σ Pr
o >
(16)
Pe
It is important to note that if the number of nodes that belong to both C1 and éj is decided, denoted by (i.c., s = |enCi], the distribution of 1 (Ercey x²s/ is automatically decided, since æ,, Vu, e V independently generated from Gaussian distribution, whose mean vector decided according to the class of vt- This is because of Enney ~ ~N((2s for given (i.e., |el -)). From this result, the following two results induced:
S) (S - 1)I), where S is size of
(2s - 1-11-54 (S 1)d
IT
Σ
N
(17)
Eq
d
P
IT
Σ
2b
>
0
8
+b
(2s S -11
(18)
4(S - 1)
EgEg
14

---

Published as a conference paper at ICLR 2024
where (()) is C.D.F. of the standard Gaussian distribution. Then, rewrite Eq (16) follows:
s Pa.e
[ [ (ueeg > 0,, EC)
(19)
Pe (Us Eel
s=0
8 Px
[ (Ennee Fik
> 8)
x (c(
(20)
Pe (0g p)
==0
By Assumption 2, each Pe(s, os E) is derived as follows :
E((s (s)
()
p¹(1 p)s-
(1 p)
N
(21)
prob. of at
prob, of
prob. os C1
By using Eq (21), we further detail Pe(re E e [p) as follows:
s
5
p²(1 -p)s-
(1 - (-p))ps-o
N-1
E.(
=
Σ
2 +
(22)
8
2
s=0
(S)
pp((1 - -p)s- (1 - p>
Σ
(23)
8
2
N
1
8
S
sp" (1 p) p)
s(1 - p)pp$~
(24)
N
2
2
8=0
snw B(S.))
8rw (S,1 111ppp))
1
Sp
S(1 p)
:. exexectattion Binomial distribution,
(25)
N
2
S
(26)
2N
With the result of Eq (26) and Eq (18), we rewrite Eq (20) as follows:
2N
Σ
8
8
(5)
pr -
-p)s-
2 + 2 N
(1
p)"ys-
(2s (28-1- 1
4(8 - 1)
d
(27)
1
Σ
9
(1 (c1-p)s- -)5- -))-=+(11--- +1- p)))))=--00)))
(28 120-11-) 4(S - 1))
(28)
3
Note that our main function, which is Pa.e [ (Em 2x
> p). is equual to Eq (28).
important to note that Eq (28) is symmetric w.r.t. 0.5 in p E [0, 1]. Thus, if we prove Eq (28) is strictly increasing on p E [0.5, 1]. the proof of the statement that Eq (28) is strictly decreasing on p E [0, 0.5) is straight forward. Hence, we prove the second statement of the thcorem by showing that Eq (28) is strictly increasing on p E [0.5, 1].
Now, we show the first and second statements of the thcorem. Here, we first show the second statement, and from the result, further prove the first statement.
Second statement. We first rewrite Eq (28) by adequately mixing two binomial functions. For simplicity, we denote ((2ss 5) 41$ 11) as ((s), since the value that changes in ((-) is only
6The term prob. indicates probability.
15

---

Published as a conference paper at ICLR 2024
and other terms fixed constants. From this, Eq (28) is rewritten follows:
1
(p" p)8- * + +1 p)pp)-))((s)
(29)
5
==0
9 + 5
1
8
sp (1 p)
s(1 p)"
(30)
8
(Term 1)
(Term 2)
We define s' : :S s). Due to the symmetric characteristic of the binomial function and standard Gaussian distribution, we can rewrite (Term 2) in Eq (30) as follows:
8
Σ s'=0
(1 (s-)))
)
(S-
( 28' -11
(S -5-26'
d
(31)
S
8
4(S -
Since we can regard & and s' as equivalent terms in our framework, we add (Term 1) of Eq (30) and Eq (31) follows:
5
p² (1
p)s-
d
d
[(S s) (S 2s 1/1 4(5 1)
+ s (2s- S DD A(S 1(S-11 - 1)
8
S
Torm (a)
(32)
We aim show that Eq (32) is strictly increasing function w.rt. p [0.5, 1]. For simplicity, let K dd/(4(S - 1)). We first discard a constant term 1/S from Eq (32) for simplicity. Note
that without Eq (32) Term (a), Eq (32) is equivalent to I, and this is sum of binomial probability mass functions.
That is, we can think of Eq (32) weighted average of the following values:
x) ((S 28 - K) + a ((2x S - 1)K), vs E 10,1,...,5 S).
(33)
Specifically, we let (1) - 5- as weight of [(S - 8) ((S - 2s - -) + 5h ((2s - - {) Note that values of Eq (33) and their weights are both symmetric about s S/2: which means and S - have the same weight value. Thus, we can rewrite Eq (32) for an odd number S as:
S
(S) (p'(1 ()) (1-))-)55- -1-10
Σ
- p1-)))p5-o)
1/2
X [S 5) ((S -2s
-2s -))K) + sa ((2s - 15-1)40 -))
(34)
While this formulation is for odd number S, we extend further results even number S. We first show that Eq (33) is an increasing function w.r.t s E [|S/2],
1,,.,, ,S. For two
cases where and k: + 1, their Eq (33) is expressed follows:
SS k: - 1) (S 24: 3) + (k + 1) (2k- - 5 + 1)]
(35)
[SS k))
- 2 - + (2k S 1)],
(36)
=k (4 112k--910 S - - 5 + (2k- -S+1)
(37)
>S 1b) (S -
1 (S 2k 3)) + 1b (S -22 -3
(38)
By the condition of > [S/2]. k tic of the CDF the standard normal distribution, ( (2k - S
S --1 k holds. In addition, by the characteris-
(2k - 5 - 1)) >
(& (S - 2k - 1) Tb (S 10--2 2k 3)) also holds. Moreover, since 2 [S/2]. $ (2k + 1) >
(SS - 2k - 3) also holds. In sum, Eq (35) > Eq (36) holds, and this result implies that Eq (33) is an increasing function w.r.t. s (1S/2), S/22| +1, 1.......... ,S).
Since greater s has a higher value of Eq (33), we can naturally conclude that Eq (34), which is goal function, is an increasing function w.r.t p if weights more assigned the terms
16

---

Published as a conference paper at ICLR 2024
related to the higher p increases. From this rationale, we show the following statement: Es* E {|s/2],.. .... .S), s.t. Ow(p, s)/Op= > 0, 2 8" and (w((,, s) < 0, I s", where w(p,s) 8) = ()) (x¹(1 - + - p))pp= -0 We first derive the first derivative of s))
aw(p, s)
(5) sp -1 (1 (1-p) - (S - s)sppp (1 p) -
(39)
Op 8
(a)
(6)
8(1 p)--1 11--1p 1 + (S 5)(1 p)'r""
1))11===+ --0-1)
(40)
(c)
(d)
Then, we show that: 3s* E [[S/2), ,.), s.t. ((a) (b)) > (()) Vs 2 8* and ((a)-(b)
<
((c) - (d)), Vs I 8*. We elaborate on (a) - (b) and (c) - (d) as follows:
(a) (b) p² (1 p) (s(1 p) - (S- s)p)
(41)
=p- (1 p) 1 sp Sp sp);
(42)
(c) - (d) = (1 - 10) 1- (S - 5) - p));
(43)
= (1 p)=-1) - (-sp (S - 5) - Sp + sp)
(44)
Remark. When p = 1/2, ((a) - (b)) - ((c) - (d) = 0 holds, which indicates that if Eq (32) turns out to be an increasing function w.r.f. p. Eq (32) has its minimum value p 1/2. Thus, we narrow down the range of p as p > 1/2.
We first formalize (a) (b) = (c) (d) equation. To this end, we utilize the logarithm function to simplify the computation (note that (*) is omitted).
= p" (1 - (0--p)S--11 - Sp) = (1 - (1-p)"-1p$-1
- S Sp),
(45)
(s 1) log (S - 1) log (1 - p) + log (s Sp),
= (s log (1 p) + (S - logp + log (s - S + Sp),
= (s 1)log
p ) + (S 8 - 1) log
1 P log $ S(1 - p)
Sp 0,
-
p
8 Sp 0.
(2s S) log
log
(46)
1
8 S(1 p)
Term (a)
Term
To show the required conditions, it is enough to show that (A) Eq (46) < 0 holds at S/2, and (B) Eq (46) > holds at 5 = S. This is because since Eq (46) is an increasing function w.r.t. 8, the above two conditions imply that there exists s*, which is our interest, between S/2 and S. When plugging S/2 to 8, 2s- S gets zero. Since p > 1/2 is given (see the above Remark), log s-S(1-)) becomes negative. Thus, (A) is proven. We now plug S into and obtain the following result:
Sp
+ log S- Sp
(2S - S) log
(47)
1 p
S-5(1 p)
S log
p
+ log
5(1 -ppp) p)
= =Slog
P log 1 p
(48)
p
Sp
1 1-p p
p
= (S - 1)log
> 0,;
(49)
1-p p
Thus, we can conclude that Eq (32), which is our interest, is an increasing function w.r.t p > 1/2, and has its minimum value at p = 1/2.
Now, we show that the above proof can also be applied to an even number S. Note that for an even number S, Eq (32) is equal to the addition of Eq (34) that starts with s = S/2 + 1 and Eq (50),
w(p,s) 5)
x
S
x
S+1
(1 = sيع
(50)
2
We derive @wi(, 0) as follows: Ow(p,s) 8)
S
S 9-2
1-p) p) 6-2 (1 (1-2p) 5 0, E (0.5,
Op x
x
2
p
(1
(51)
17

---

Published as a conference paper at ICLR 2024
Since (w(p, (p, s) < O holds for (S + 1)/2, our proof is still valid for number S. We finalize the proof of Second statement.
First statement. From the above proof, now show the lower bound of Eq (27). which is the first statement. Note that the value of Eq (27) (equivalent to Eq (32)) is minimized at p = 0.5. Thus, we prove the statement by finding the lower bound of Eq (28) at p 0.5. For simplicity, again use the following notation K == 1(8-13'
5
d
(p" (1 -p) + p1 +11- (1-p))p8--)1
(2s (2s-1-
5 4(S - 1))
(52)
8
5
s
28
+ ((28 - - -1)K).
(53)
==
First, consider an even number S. Then, for a((2s S 1)K), Vs (1,-.. S), consider the below relations:
( + +2 10-00-- (--5+33
(8 1) 1 -) +5 (
(54)
(al)
(b1)
(62)
(a2)
In Eq (54), (al) + (a2) holds, and (b1) + (62) holds also. In this manner, we can pair all terms in Eq (54). In general sense, the following holds: k11 + k21 such that kn + h21 S, k12 + k22 = 1, kn < k12, and h22 < K22 hold. Note that k + K21 k22 is lower-bounded by knttkn since distributing the weight of a larger value to smaller value will decrease the overall value. From this result, we can obtain lower bound of Eq (54) as follows:
5 Σ 28
1
+ (12s- 5-12
1 5
Σ
S
28
S
1
0.5.
(55)
2
2
2
Now, we consider an odd number S. We can extend the proof of an even number S case, while additionally considering the following term: other hand, since (()) = 0.5, we still ensure Eg (55) holds in this case as well. Thus, for both even- and odd-number S, the value of Eq (28) at p = 0.5 is lower-bounded by 0.5, which is a global lower bound of Eq (28). We finalize the proof of First statement.
,+ (0) that does have any pair. On the
Moreover, as described, Eq (28) is symmetric w.r.t. p 0.5 on p E 1). In conclusion, we have verified that Eq (28) is strietly decreasing at E [o, 0.5). by extending the result J E [0.5, 1.0).
Generalization to 1,- E Co. Note that the required condition for the improvement in 2 E Co is 1 Enreo 36 < O. Thus, we further investigate Pa.e (1'' Ix < 0 ()). where &. E e and
e {u;} (refer to Remark). First, let denote the number of nodes that belong to both e, and Co. Note that we aim to investigate the following probability:
Pa.ee (1"
< 0, E p))
Pa.c Tr
Σ E.
<
P
(56)
Ee p)
Here, the following holds:
Σ Ibk ~ ~N((ss - 1 11-5) (S 1)I),
(57)
N (- (2s - -1 S)d
Tr
Σ
Lk
(S 1)d
(58)
2
Eq
Pr
I
Σ
0
8
= +b
(2s S 1-11 1)1
(59)
4(S - 1)
LovEg
18

---

Published as a conference paper at ICLR 2024
Note that Eq. (59) is equivalent to Eq. (18). Moreover, the following also holds:
Ee(p) (S)
p" (1 p)s-
p)-
(60)
prob. of =c=0
prob, of at
proh. of oz E Co ne
Here again, Eq. (21) is equivalent to Eq. (60). Finally, we guarantee that Eq, (56) is expressed as follows:
Ps
(En Eq 2b
> 0
$) Pe(s,uEEe)
(61)
Pe (1: Ee p)
x=0
2N
8
(
10(1 - p) p)s 8-
(1 - 11-p)pps-
8 1
d
+
5 (2 4(S - -1)
(62)
$
8)
2
N
1
d
(v" (1 5-5 (1 p)" -)0 (28 (28-1- 5) 4(S - 1))
(63)
5
Importantly, Eq. (63) is equal to Eq. (28). This result implies that Theorem and its proof is generalizable the case of 2g E Co also.
A.3 EMPIRICAL DEMONSTRATION ON THE PROOF.
To empirically validate the proof provided above, we conduct toy experiments. We consider every combination of S E {2, 3, 4, 5, 6, 7) which is size of eg (i.e., le,), and d (2,3, 4,5, 6, 7, 8), which is dimension of input node feature E Rdd, Ve; E V. Then, we visualize how Eq (27) varies w.r.t. S, d, and P [0. 1]. Note that Xaxis each plot corresponds to P. As shown Figure 3, for all cases, we can verify that the value of Pate (I" (Em,e, > PP is strictly-increasing w.r.t. P E [0.5, 1] and -deceasingg w.r.t. T [0, (statement 2), and the value is lower bounded by 0.5 (statement 1). Thus, we can ensure that Theorem 2 is valid through empirical verification.
B ADDITIONAL THEORETICAL ANALYSIS
B. EXISTENCE OF REASONABLE SOLUTIONS IN THE HYPEREDGE FILLING TASK
In this section, investigate the existence of reasonable solution in the proposed hyperedge filling task. To this end, we first formalize the concept of the hyperedge filling task solved.
Definition 1 (Solved hyperedge filing task). For a hypergraph G (V,E), the hyperedge filling task is solved when the following holds for probabilistic model P(x.E,0)()):
P(x,g,e) (ws I e \ (us)) > PiX.E.e) (rk e (w;)), Ve, E ey, Voy E {u.: ( (v.)) & E,U, E V).,ee, E E.
Roughly, Definition 1 indicates that for all possible hyperedge filling cases in a hypergraph G, probabilistic model p always has the greater probability of filling a correct node for given query subset than that of filling wrong node. It is important to note that our goal is to identify whether there exists a reasonable solution for p(-), not trivial solution. Thus, we formalize the concept of reasonable solution the following definition:
Definition 2 (Reasonable solution). Let Q {ez \ {wi) Vo; E E Ve, E E}. For a hypergraph G (V,E), E) a reasonable solution of the hyperedge filling task is pair of node representations R and query set representattonss Q E RR 1@]x such that:
zT q; > z 9j
Voz Sy, Vv, E VV\ Sj, Vag Q, where 5 {v. ((v.) ngg EE) (vs & 9z). vs E V).
19

---

Published as a conference paper at ICLR 2024
S
2
S
4
S
S
S
S
1.0
1.0
1.0
1.0
1.0
1.0
1.0
2
0.8
0.8
0.8
0.8
0.8
0.8
0.8
d
0.4
0
0.5
188%
0.5
48.40
0
0.5
1.8%g
0
0.5
1.8
0.5
1.8
0.5
1.8 0
0.5
1.0
3
1.0
1.0
1.0
1.0
3.0
1.0
1.0
d d=3
0.8
00.8
0.8
0.8
0.8
0.8
0.8
0.4
0
0.5
1.8.9
0.5
1.8.4 0
0.5
1.8.9 0
0.5
1.8.4
0.5
1.8
0.5
1.8
0.5
1.
1.0
1..0
1.0
0.8
0.8
0.8
0.8
0.8
0.8
0.8
d
0.6
0.6
0.6
0.6
0.6
1.8.4
1.8
1.8
1.8
1.8 0.8
0.5
1.8
0.5
1.0
1.0
11.0/
11.0
0.8
0.8
0.8
0.8
0.8
d
0.6
0.6
0.6
0.6
0.6
0.6
0.6
1.84
1.8
6
188
188
1.8
1.8
1.0
1.0
1.0
0.8
0.8
d
0.6
0.6
0.6
0.4
0
0.5
1.8.
0.5
1.8.4
0
0.5
1.8.4
0
0.5
1.8.4 0
0.5
1.8.4
0.5
1.8
0.5
1.0
1.0
1.0
7.0
7.0
0.8
0.8
0.8
P
0.6
0.6
0.6
0.6
0.6
0.6
0.6
0.4
0.5
1.8%4
0.5
1.8.4
0.5
1.8.4
1.8.4
1.8%
0.5
1.8
0.5
1.0
1.0
1.0
1.0
11
0.8
0.8
0.8
2
0.6
0.6
0.4
0.5
1.8.4
0.5
0.5
1.84
0.5
4.4
0.5
1.8.4
0.5
1.84
0.5
1.0
X-Axis: P
theoretical lower bound
Px
1
Xk > 0 P
Y-Axis: Probability
PREqIj
Figure 3: Empirical demonstration of Theorem 2. Note that S denotes the size of a hyperedge, and d
denotes the dimension features. As stated, Pa.c. [I" (Emeg 3c
> 0 P strictly increasing
in P E [0.5, (statement 2), and lower bounded by 0.5 (statement
Now, we analyze whether there exists reasonable solution for any hypergraph G = (V,E).
Theorem 3 (Existence of reasonable solution). For any hypergraph G (V,E). there exists reasonable solution for some embedding dimension d.
Proof. Consider binary matrix B E (1, 1V where each column j (1,.... , rep- resents one-hot encoding of S, over V (i.c., B,, 1se, E Syl. where 1[:] is an indicator func- tion). Here, consider matrix decomposition of B, specifically, singular value decomposi- tion of B. Denote this singular value decomposition B ULVT, where U | Let Z = UE and Q == V. In such a case, the
V E (rank())) and V E R xmnk(B).
following holds:
[1 if 0; E Sj otherwise.
2 9j
(64)
20

---

Published as a conference paper at ICLR 2024
Thus, n q3 2 qj Ve, S,, Vog S Aaz Q always holds, which implies that Z U3 and Q = V are one of the reasonable solutions.
From Theorem 3, we demonstrate that there exists reasonable solution for any hypergraph G.
B.2 DIMENSIONAL COLLAPSE HYPEBOY WITHOUT PROJECTION HEADS.
In this subsection, we provide a theoretical amalysis of why HYPEBOY without projection heads may suffer from dimensional collapse. Jing et al. (2022) have suggested that one of the reasons for the dimensional collapse in contrastive learning lies in the gradient flow (spec., mechanism of how representations are updated via contrastive loss). Specifically, consider linear model that yields representations of number of data by Z XW. where X Rn are node features and is a learnable parameter matrix. When we update parameters W by using contrastive losses, k-th biggest singular value of weight matrices σą evolve proportional to its value (e.g., 6k + 0g x C, where 0% is a k-th biggest singular value of updated W and e is a positive constant). Due to this mechanism, small single values grow slowly or diminish, causing dimensional collapse.
IR dRd
In this analysis, motivated by the analysis of Jing et al. (2022), we track how singular values of node representations change
learning hyperedge filling task. Following the setring of Jing al. (2022), we consider linear model: Z XW. Note that X are given features of data points. Thus, the change singular values of embeddings Z depends on W.
We first define the loss of hyperedge filling as follows:
exp (2f (Ev, Eqis Fx)
6: -= EEEE NEER
Σ
Σ log
(65)
Eney exp (zf (Evreqrs [oz)
Now, we obtain the closed form of the following update equation, which is based on the gradient descent: W W VV L. where is fixed learning rate. We first derive (AC/OZ), and utilize the chain rule: (OC/OZ) x (öz/OW).
The derivative of numerators of Eq (65) can be written follows:
a (Eeree En,ee, x (Ev,eus 2z)) -EZ,
OZ
where E E VIXM We now derive the derivative of the denominators of Eq (65);
Egg 2 x Heek (ur, 0z} ex, Vea E E.H.
a (Enjee Evjee, log (Emeyexp x (Eveey 2z ))))
(66)
Oz
We denote multiset of every possible query subset Q (i.e., fe; \ fws The reason for multiset is that given subsets can be duplicated due the nature of overlapping hyperedges (e.g., when we omit node v1 from {vi, U2, v3) and from {ra, U2, va), remaining sets become identical).
is E E E E)).
We split the derivation into the node level. In addition, we let subset of Q such that includes node Uk as Qt- Then, for node tik its gradient is computed as follows:
cxp
xT
(Erreg 2x))
Σ 2o PMEE)
(67)
YEQ Ensey exp
2T (Eerg, Zz
exp
x7T (Evre 2s
Zi.
(68)
g. v.EV Ev,ey exp
(xT
(Evr 2s
21

---

Published as a conference paper at ICLR 2024
We define two more matrices to express Eq. (67) and (68) in simple matrix form. By assigning index k -1, (1,... [ to Øj E Q, we define binary matrix Q E [[[[ that represents Q:
{
if v E U/ otherwise.
Q2i
R that models probability filling each node V/ E V to each query subset dj E 2, which is defined as follows:
In addition, we denote score matrix E
(z." (Ev, Eo: =x))
exp
A En,Ev exp
(69)
xz] (Ev, Eo; zx)
By using A and Q, we can express Eq (67) and (68) for all nodes as follows:
Q'A
QJQ
Z.
(70)
Fig (67) of all modes
Fqg (68) of all nodes
Finally, we express (0C/OZ) as follows:
Oc
oz (-E+Q'A+A'Q) Z.
(71)
In addition, by using the chain rule and matrix derivative rules, OL/OW derived as follows:
ac
TW x
(-E+ xw.
(72)
We denote updated W as W. To sum up. update rule of W W Vw expressed as follows:
W = (I+ (E 7 ATQ x) W. Q'A++
(73)
Here, denote k-th biggest singular values of W and W ã. and dg, respectively.
Note that (I + x E Q + ATQ) x) a function of X, E, and W. Thus, we denote this term g X &, W) dddd
If we take a closer look at g(X, W)W, W is near-linear transformation of W itself". In such a case, from the min-max characterization rule, the following inequality holds:
0z of Tke where of is the biggest singular value of a matrix g(X,E,
(74)
Eq (74) indicates that ou is upper-bounded by the multiplication between the biggest singular value of gX, E, W) and σы- While it is not an exact equality, this result implies that the changed singular value likely be correlated with its previous singular value. Hence, small singular values are likely to grow slowly, similar to Jing et al. (2022) study, which may cause dimensional collapse.
C DISCUSSIONS
C.1 COMPARISON WITH HYPERGRL
In this section, we compare our HYPEBOY against HyperGRL el., 2022). Both methodologies utilize strategy that leverages node and subset of hyperedge for seff-supervised learning. Specifically, this involves using pair that consists of a node ez that belongs to hyperedge e,, and the subset of e, excluding v. (e; (v,)), for training purposes. However, HYPEBOY significantly differs from HyperGRL in various aspects.
II perfect linear transformatio, since function g(-) is also function of W.
22

---

Published as a conference paper at ICLR 2024
Motivation. The motivation of HYPEBoY is markedly distinct from that of HyperGRL. The hyper- edge filling task is a task that aims to correctly fill the missing node for a given (query) subset. The underlying motivation is to train machine learning model to generate hyperedges, thereby iden- tifying entities that are forming group interactions. This may enable the model to understand the complex, higher-order relationships within a given hypergraph. This objective aligns with the goals of guestion-answering approaches in NLP (Devlin et al., 2019). Conversely, the key motivation of HyperGRL, which performs node-level pretext task introduced by Du et al. (2022), described as follows: "the node inside one specific hyperedge should be distinguished from nodes outside this hyperedge given the context of node". Here, the motivations behind Du et al. (2022) are akin to those of proximity-based approaches, including LINE (Tang et al., 2015).
Method design. The design choices of HyPEBOY and HyperGRI. differ significantly. Below, we outline these technical distinctions.
Input: HYPEBOY receives the original hypergraph structure, while HyperGRL. receives a clique- expanded graph.
Backbone encoder: HYPEBOY utilizes hypergraph neural networks, while HyperGRL employs graph neural networks.
Augmentation: HYPEBOY augments node features and hyperedges, while HyperGRL does not utilize augmentation strategy.
Projection head: HYPEBOY utilizes different projection head parameters for mode-projection and query subset-projection, while HyperGRL utilizes the same parameters for both.
Negative samples: HYPEBOY utilizes the entire node as negative samples for each hyperedge filling case, while HyperGRL samples several nodes.
SSL signal: HyPEBOY utilizes all possible (node, query subset) pairs a given hypergraph, while HyperGRL samples single pair for each hyperedge.
Feature reconstruction: HYPEBOY utilizes node feature reconstruction warm-up, while Hyper- GRL utilizes hyperedge prediction loss as another SSL signal.
Performance comparison. important note that HyPEBOY is superior to HyperGRL as an SSL strategy for hypergraphs. We provide empirical evidence below:
Under the fine-tuning protocol, HYPEBOY outperforms HyperGRL in every dataset (refer to Sec- tion 5.1).
Under the linear-evaluation protocol with the node classification task, HYPEBOY outperforms HyperGRL in every dataset (refer to Section 5.2).
Under the linear-evaluation protocol with he hyperedge prediction task, HYPEBOY outperforms HyperGRL in 10 out of 11 datasets (refer to Section 5.2).
C.2 POTENTIAL EXTENSIONS OF HYPEBOY
While this work mainly assumes static and undirected hypergraphs, many real-world group interac- tions have directions and temporal dynamics. Directed hypergraphs (Kim al., 2023b) and temporal hypergraphs (Lee & Shin, 2023b) are utilized to include directional information and temporal infor- mation in hypergraphs, respectively. HYPEBOY can be extended such hypergraphs by adequately considering their unique characteristics. Moreover, while we assume transductive hypergraph set- ting, there is recent effort to perform representation learning on hypergraphs under an inductive case (Behrouz et al., 2023). Considering this, HYPEBOY can also be extended to an inductive case, incorporating the idea of Behrouz et al. (2023). Lastly, HYPEBOY can be enhanced to capture more complex characteristics of hypergraph (e.g., spperedge-dependeee of nodes (Choe et al., 2023)).
D DATASETS
In this section, we provide details of the used datasets. In experiments, utilize 11 hypergraph benchmark datasets. Following Lee & Shin (2023a), we remove nodes that do not belong to any hyperedge. Data statistics of each dataset reported in Table 4.
23

---

Published as a conference paper at ICLR 2024
Table 4: Statistics of 11 datasets we utilize in our experiments.
Citeseer
Cora
Pubmed Cora-CA DBLP-P DBLP-A AMiner IMDB MN-40
20News House
[V]
1,458
1,434
3,840
2,388
41,302
2,591
20,201
3,939
12,311
16,242
1,290
[&l
1,079
1,579
7,963
1,072
22,263
2,690
8,052
2,015
12,311
100
341
Deer le]
3,453
4,786
34,629
4,585
99,561
6,201
32,028
9,560
61,555
65,451
11,843
# Classes
6
7
4
12
40
2
# Features
3,703
1,433
500
1,433
1,425
334
500
3066
100
100
2
Source of each dataset. The sources of each dataset used in this work are as follows:
The Cora, Citeseer, Pubmed, Cora-CA, and DBLP-P datasets are from the work Yadati et al. (2019).
The DBLP-A and IMDB datasets are from the work of Wang et al. (2019).
The AMiner dataset is from the work of Zhang et al. (2019).
The Mondelnet-40 (MN-40) dataset is from the work of Wu et al. (2015).
The 20Newsgroups (20News) dataset is from the work of Dua et al. (2017).
The House dataset is from the work of Chien et al. (2022).
Co-citation datasets. We utilize three co-citation datasets: Cora, Citeseer, and Pubmed. In these datasets, each node represents publication and each hyperedge represents set of publications co- cited by publication. For example, if a publication has cited three publications that correspond to U4 U/, and Uk. respectively, these publications (nodes) are grouped as a hyperedge {er 2/ v&} E. Node features are bag-of-words of the corresponding publication (node). Node class indicates the academic category of the corresponding publication.
Co-authorship datasets. We utilize four co-authorship datasets: Cora-CA, DBLP-P, DBLP-A, and AMiner. In Cora-CA, DBLP-P, and AMiner, each node represents a publication, and set of publications that are written by the same author is grouped as a hyperedge. Node features are bag- of-words of the corresponding publication (node). Node class indicates the academic category of the corresponding publication. Conversely in DBLP-A, each node represents an author, and co- authors of a publication are grouped hyperedge. Node features bag-of-words regarding the research keywords of the corresponding author Node class indicates the primary research area of the corresponding author.
Computer graphic datasets. We utilize one computer graphical dataset: Modelnet-40 (MN-40). In this dataset, each node represents visual object, and hyperedges are synthetic hyperedges that have been created with k-NN graph constructed according to the features of each data point, following Feng et al. (2019) and Chien et al. (2022). Node features are embeddings (obtained via GVCNN (Feng et al., 2018) and MVCNN (Su et al., 2015)) of the corresponding visual object. Node class indicates the category of the corresponding visual object.
We utilize one movie-actor relational dataset: IMDB. In this dataset, each node indicates a movie, and the filmography of an actor is grouped as hyperedge. Node features are bag-of-we words of the plot of the corresponding movie. Node class indicates the genre of the movic.
Movie-Actor dataset.
News dataset. We utilize one news dataset: 20NewsGroups (20News). this dataset, each node represents document of 20 newsgroups and set of documents containing particular word is grouped as a hyperedge. Node features are TF-IDF representations of news messages of the corre- sponding news document. Node class indicates the category of the corresponding news document.
Political membership dataset. We utilize one political membership dataset: House. In this dataset, each node represents member of the US House of Representatives, and members helonging the same committee are grouped as hyperedge. Node features are generated by adding noise to the one-hot encoded label vector, following Chien al. (2022). Node class indicates the political party of the corresponding member.
PPlcase do not be confused with the academic corresponding author.
24

---

Published as a conference paper at ICLR 2024
E EXPERIMENTAL DETAILS
In this section, we provide our experimental details. We first describe the machines we use in our experiments. Then, we provide details of HYPEBOY and baseline methods, including their implementation and hyperparameter settings.
E. MACHINES
All experiments are conducted on a machine with NVIDIA RTX 8000 D6 GPUs (48GB memory) and two Intel Xeon Silver 4214R processors.
E.2 DETAILS OF BASELINE METHOODS
Neural networks. We utilize 10 (semi-Jsupervised baseline models: MLP, HGNN (Feng et al., 2019), HyperGCN (Yadati et al., 2019), HNHN (Dong et al., 2020), three unified hypergraph net- works, (UniGCN, UniGIN, UniGCNII) (Huang & Yang. 2021), AllSet (Chien et al., 2022), ED- HNN (Wang et al., 2023a), and PhenomNN (Wang al., 2023b). For all the methods, fix their hidden dimension and model layers as 128 and 2, respectively.
Graph SSL. To utilize two graph-generative graph SSL methods, which are GraphMAE2 (Hou et al., 2023) and MaskGAE (Li et al., 2023), and one hypergraph-generatie which is HyperGRL (Du al., 2022), transform the input hypergraph into graph by using the clique expansion, which converts each hyperedge into a clique of a graph (Dong et al., 2020). More- over, utilize 2-layer GCN (Kipf & Welling, 2017) 7) with hidden dimension 128 backbone encoder of these two SSL methods.
graph SSL method,
H-GD. One of baseline methods H-GD directly extends the group discrimination approach, an SSL technique designed on ordinary graphs (Zheng et al., 2022). We strictly follow the overall struc- ture suggested by Zheng (2022), while the feature augmentation and topology augmentation have been replaced into T₂ and Te that are described in Section 4.1, respectively.
E.3 DETAILS OF SETTINGS
Hyperedge prediction setting. Note that we need to obtain negative hyperedge samples to evaluate a model on the hyperedge prediction task. our experiments, we utilize negative hyperedge samples that are generated by the Size Negative Sample strategy (Patil et al., 2020). Specifically, we first sample the size of the current hyperedge from the real-world hyperedge size distribution of the corresponding dataset. Then, we sample nodes uniformly random and fill the corresponding hyperedge. By using the strategy, we obtain negative hyperedges with the same number of ground- truth hyperedges and utilize them for rraiiing/validationneesss
Hyperedge embedding. To perform hyperedge prediction, we require a representation of each hyperedge. On the other hand, since the output of utilized HNNs and GNNs are node embeddings, we need an additional technique to acquire hyperedge representation from node embeddings. To this end, we utilize a max-min pooling aggregation function (Yadati et al., 2020). For example, for hyperedge e; {u,, ve). whose constituent node embeddings are Zj, zk. Ze E Rd, respectively, its hyperedge embedding h is obtained as follows:
h
max se (j.k.))
min 2ts)d=1
(75)
Finally, we utilize h; representation of ej.
E.4 DETAILS OF IMPLEMENTATION
Feature reconstruction warm-up of HYPEBOY. As described in Section 4.4, before training an encoder HNN with HYPEBOY, we perform a feature reconstruction process, which directly extends GraphMAE (Hou et al., 2022) to hypergraph structure. This process mitigates the encoder's 's over- reliance on the projection heads, improving the performance in downstream tasks (Section 5.3). For the decoder HNN Dg. utilize 2-layer UniGCNII (Huang & Yang, 2021), which has the same
25

---

Published as a conference paper at ICLR 2024
Table Hyperparameter combination HYPEBOY that shows the best validation accuracy on each dataset. Pe E [0, 1] indicates the magnitude of feature augmentation, Pe indicates the magnitude of topological augmentation, and SS Lepochs indicates the training epoch of hyperedge filling task.
Citeseer Cora
Pubmed Cora-CA DELP-P DBLP-A AMiner
IMDB MN-40
20News
House
Pv
0.4
0.0
0.4
0.0
0.2
0.3
0,4
0.2
0.4
Pe
0.9
0.9
0.5
0.8
0.6
0.9
0.6
0.9
0.5
0.5
0.8
SSL epochs
120
200
180
200
180
120
100
180
180
80
140
architecture as the encoder HNN. We mask 50% of input nodes and set hyperedge augmentation
ratio Pe. as 0.2.
Projection heads of HYPEBOY. We utilize a two-layer MLP model with a ReIU (Nair & Hinton, 2010) activation function without dropout- and normalization-layyee for both node projection head f' and set projection head f (Section 4.2). Note that these projection heads are updated via the hyperedge filling task together with the encoder HNN.
Training details and hyperparameters. We train all models using the Adam optimizer (Kingma & Ba, 2015) with a fixed weight decay rate of 10-6. We fix the hidden dimension and dropout rate of all models 128 and 0.5, respectively. When training any neural network for downstream tasks, we train a model for 200 epochs, and for every 10 epochs, we evaluate the validation accuracy of the model. Then, we utilize the model's checkpoint that yields the best validation accuracy to perform the final evaluation of the model on the test dataset.
For linear evaluation protocol of node classification, we utilize logistic classifier, with learning rate 0.001. For a linear evaluation protocol of hyperedge classification, we utilize a two-layer MLP, with a learning rate of 0.001 and a dropout rate of 0.5.
Regarding hyperparameter tuning, for each model and dataset, we find the hyperparameter combi- nation that gives the best validation performance. Then, with the selected hyperparameters, we train the model on training dataset and evaluate the trained model on test dataset.
For all the supervised models, we tune the learning rate as hyperparameter within (0.05, 0.01, 0.005, 0.001, 0.0005, 0.00013. Moreover, especially for PhenomNN, which is state- of-the-art hypergraph neural network, we additionally tune their other hyperparameters. Specifically, along with the learning rate, additionally tune Åo and A within {0, 1, 10), which are weights of different message-passing functions, and a within {0, 10). which is the weight of a node's 's own feature.
For all the SSL baseline methods, we set broader search spaces, which are as follows:
TriCL (Lee & Shin, 2023a): We tune their feature augmentation magnitude within (0.2, 0.3, 0.4). hyperedge augmentation magnitude within (0.2, 0.3, 0.4). and learning rate of its all components within {0.01, 0.001, 0.0001}.
HyperGCL (Wei et al., 2022): We tune learning rate of an encoder within {0.01, 0.001, 0.0001). that of view generator within (0.01, 0.001, 0.0001 and the weight of contrastive loss 3 within (0.5,1,5 1,2).
HGD, variant of GD (Zheng et al., 2022): We tune their feature augmentation magnitude within {0.2, 0.3, 0.4), hyperedge augmentation magnitude within (0.2, 0.3, 0.4), and learning rate of its all components within (0.01, 0.001, 0.0001).
GraphMAE2 (Hou et al., 2023): We tune mask ratio within {0.25, 0.5, 0.75) and learning rate of its all components within (0.01, 0.001, 0.00013.
MaskGAE (Li et al.. 2023) We tune degree loss weight a within {0.001, 0.002, 0.003) and learn- ing rate of its all components within (0.01, 0.001, 0.0001).
HyperGRL (Du et al., 2022) We tune negative-sampling adjacency matrix order & within (1,2,3) and learning rate of its components within {0.01, 0.001, 0.00013.
In addition, we set self- superviedd learning epochs of the above methods as hyperparameters, search- ing within (50, 100, 150, ... ,450, 500).
26

---

Published as a conference paper at ICLR 2024
Table 6: Comparison between HYPEBOY (HyPeBoy (denoted by w/ Aug.)) and a variant of HY- PEBOY that does not use augmentation strategy (denoted by w/o Aug.) in the node classification task under two protocols. The best performance colored as green. In most of the settings, HYPEBOY outperforms its variant.
Protocol
Method
Citsseer
Pubmed CAra-CA
DBLPA AMiner
MN-40
20Newx
Fine
w/o Aug.
(K7) 58.6 75.5 (4.0) 62.9 (60) 87.8 (04) (2.) 33.9 (2.1) (252 90.7 (0.7)
77.5 69.9 (49)
tuning
w/ A/g.
36.7 (98) 62.3 (2.2) (3.4) 66.3 (4.) (0.4) (23) 34.1
90.4 (0.9)
77.6 (0.9) (48)
Linear
wlo Aug.
55.8 (9.0) 60.6 (1.3) 72.7 (5.4) 63.3 (4o) (7.6 81.4 (25) 34.2 (177 47.8 (21)
89.7 (19)
75.1 (1.6 67.5 (1.)
evaluation
w/ Aug.
(9.9) 63.5 (9:4) 75.0 (1.4) 66.0 (46) 87.9 (0.) 81.2 (2:7)
34.3 (1:2) 48.8 (18)
89.2 (2:3) 75.7 (2.1 (644
Citeseer
Pubmed
Cora
DBLP-P
DBLP-A
-4
-20
-25
60
120
-20
120
-18
With Projection head
Without projection heads
X-Axis: Singular value rank index Y-Axis: Log of singular values
Figure 4: Analyzing dimensional collapse of HYPEBOY with/without projection heads on five benchmark datasets. While HYPEBOY does not suffer from dimensional collapse, its variant that does not utilize projection heads, suffers from this issue.
For HYPEBOY, we tune feature augmentation magnitude within {0.0, 0.1, 0.2, 0.3, 0.4] and hy- peredge augmentaiton magnitude Pe within {0.5, 0.6, 0.7, 0.8, 0.9}. We fix the learning rate and training epochs of the feature reconstruction warm-up as 0.001 and 300, respectively.
Regarding the overall training process of HYPEBOY, we first train an encoder HNN with a feature reconstruction process for 300 epochs. Then, we train the encoder HNN with the hyperedge filling task. Specifically, the hyperedge filling training epoch is hyperparameter, which we search within {20, 40. 180, 200). We report a hyperparameter combination of each dataset that shows the best validation accuracy our fine-tuning protocol experiment (Section 5.1) in Table 5.
F ADDITIONAL EXPERIMENTAL RESULTS
In this section, we present additional experimental results that were excluded from the main paper due to space constraints.
F.1 COMPARISON AGAINST HYPEBOY WITHOUT AUGMENTATION
As described in Section 1. augmentation (spec., masking) has played a key role in various gener- ative SSL methods for obtaining effective representations. Motivated by these findings, we incor- porate augmentation process Tz and тe into HYPEBOY. To assess the impact of augmentation, we analyze a variant of HYPEBOY that omits the augmentation step. In this variant, the input to HY- PEBOY'S encoder HNN consists of ground-truth features and topology, (X, We then compare the performance of this variant against our proposed HYPEBOY in node classification tasks, under both fine-tuning and linear evaluation protocols. As shown in Table 6, HYPEBOY consistently out- performs the variant across most of the settings, demonstrating the necessity of augmentation the model's 's performance.
F.2 DIMENSIONAL COLLAPSE ANALYSIS
In Section 4.2, we discuss the role of projection heads in terms of dimensional collapse: projec- tion heads encourage an encoder HNN to avoid dimensional collapse. We further analyze this phenomenon in five benchmark hypergraph datasets: Citeseer, Pubmed, Cora-CA, DBLP-P, and DBLP-A. As shown in Figure 4, across all these datasets, we verify that certain singular values of embeddings achieved via HYPEBOY without projection heads drop to zero. This empirically demonstrates that dimensional collapse has occurred in such cases (refer to red lines). Notably, such issues are not observed in HYPRBOY (refer blue lines).
27

---

Published as a conference paper at ICLR 2024
Table 7: Performance of SSL methods in various HNNs under fine-tuning protocol. The best and second-best performances colored green and yellow, respectively. NA indicates the correspond- ing encoder without pre- pre-traiie training.... HYPEBOY outperforms other hypergraph SSL methods in most of the settings.
Method
Citeseer
Cora
Pubmed
Cora-CA
DBLP-P
NA
41.9 (7.8)
50.0 (7.2)
72.9 (5.0)
50.2 (5.7)
85.3 (0.8)
TriCL
49.3 (10.3)
56.9 (10.0)
74.3 (4.1)
57.2 (6.1)
87.4 (5.0)
HGNN
HyperGCL
47.2 (9.3)
59.5 (7.)
76.3 (45)
53.3 (6.9)
85.8 (o.4)
HYPEBOY
52.1 (9.1)
61.1 (9.8)
76.8 (4.3)
60.0 (6.0)
87.4 (0.5)
NA
39.9 (10.0)
48.5 (8.9)
72.4 (4.0)
50.6 (7.5)
85.7 (0.7)
TriCL
46.6 (8.3)
59.8 (8.9)
74.0 (3.8)
57.3 (4.8)
87.0 (5.0)
Mean DeannoollngConv
HyperGCL
43.2 (9.4)
59.6 (7.6)
74.2 (4.3)
55.0 (6.7)
85.7 (0.9)
HYPEBOY
54.5 (8.3)
61.2 (7.8)
77.3 (3.4)
63.0 (4.4)
86.8 (0.5)
NA
43.6 (6.3)
48.9 (9.2)
69.8 (4.5)
50.7 (6.7)
82.8 (1.0)
TriCL
48.9 (7.6)
57.7 (8.4)
72.0 (4.5)
57.8 (4.9)
85.1 (0.5)
SetGNN
HyperGCL
46.8 (8.5)
54.6 (7.5)
73.1 (3.9)
56.9 (5.3)
83.6 (0.7)
HYPEBOY
53.8 (9.1)
62.3 (6.4)
73.4 (3.4)
61.0 (3.9)
84.2 (0.7)
Overall Uniformity
Clves
Class
Cless
Cress
Clexs
Cles
Citeseer
0
Overall Laifornky
Class
Pubmed
Overall Uniformiy
Class 0
Coss
Class
Class 3
Class
Class
Coss
Cora-CA
Unioomiity
DBLP-P
0
Ovecall Uniformiy
C
Class 1
Class
Class
DBLP-A
Figure 5: Analyzing alignment and uniformity of representations obtained via HYPEBOY, In most cases, representations obtained by HYPEBOY achieve both alignment and uniformity.
F.3 ALIGNMENT AND UNIFORMITY ANALYSIS
In Section 4.3, discuss the role of our P(x,E,e) (-) design choice in promoting alignment and uni- formity of node representations. This characteristic is further analyzed across five datasets: Citeseer, Pubmed, Cora-CA, DBLP-P, and DBLP-A. As shown in Figure 5, node representations obtained from HYPEBOY achieve both uniformity (first column) and class alignment (other columns) in most of the cases.
28

---

Published as a conference paper at ICLR 2024
Table 8: Efficacy of pretext tasks under fine-tuning protocol. The best performances are colored as green. Among the three tasks, the hyperedge filling task shows superior performance.
Method
Citeseer
Cora
Pubmed
Cora-CA
DBLP-P
Naive contrastive learning
59.2 (6.7)
44.5 (10.2)
75.2 (4.0)
62.1 (5.3)
86.7 (0.5)
Naive feature reconstruction
58.6 (8.0)
51.5 (9.2)
74.7 (5.1)
61.9 (7:3)
87.3 (0.6)
Naive hyperedge filling
60.7 (8.2)
51.6 (11.8)
76.2 (3.6)
63.5 (6.0)
88.1 (0.5)
Table 9: Comparing HYPEBOY against other topological SSL methods under fine-tuning protocol. The best performances are colored as green. Among all the four methods, HYPEBOY shows the best performance.
Data
Method
Citeseer
Cora
Puhmed
Cora-CA
DBLP-P
Graph
Edge filling
44.1 (11.3)
55.7 (8.2)
73.6 (38)
55,5 (8.2)
86.9 (0.5)
Graph
Hyperedgeedge filling
42.8 (9.2)
56.4 (6.0)
73.6 (4.0)
55.0 (8.6)
86.7 (0.5)
Hypergraph
Hyperedge prediction
43.9 (74)
56.9 (9.7)
73.0 (4.9
59.1 (6.5)
85.8 (0.6)
Hypergraph
Hypereedge filling (HYPeBoY)
56.7 (9.8)
62.3 (2.7)
(3.4
66.3 (4.6)
88.2 (0.4)
F.4 ENCODER ANALYSIS
In this section, we experimentally verify that the efficacy of HYPEBOY extends beyond UniGC- NII (Huang & Yang. 2021). For this purpose, we employ three HNNs: HGNN (Feng et al., 2019), MeanPoolingConv (Lee Shin, 2023a), and SeIGNN (Chien et al., 2022). Notably, the latter two encoders, MeanPoolingCony and SeIGNN, serve backbone encoders in TriCL (Lee & Shin, 2023a) and HyperGCL (Wei et al., 2022), respectively. The experimental setting mirrors that of the fine-tuning experiments detailed in Section 5.1. As shown in Table 7, HYPEBOY outperforms other hypergraph SSL methods in most of the settings, validating its broad effectiveness across a diverse range of HNNs.
F.5 PRETEXT TASK ANALYSIS
In this section, we evaluate the efficacy of different SSL pretext tasks. Specifically, we assess our hyperedge filling task by comparing it with two other SSL tasks: contrastive learning and feature reconstruction. To ensure a fair comparison and minimize the impact of the underlying SSL method design choices, we streamline each method as described below:
Naive contrastive learning: This method a variant of TriCL (Lee & Shin, 2023a) that does not projection head parameters. Other settings are all the same that of TriCL (e.g.. contrastive losses and view generation process).
Naive feature reconstruction: This method is a variant of the hypergraph feature reconstruction method (Section 4.4), that does not use a decoder HNN.
Naive hyperedge filling: This method is variant of HYPEBOY that does not use feature recon- struction warm-up and projection heads. Specifically, this method is equivalent to V1, which is described Section 5.3.
As shown in Table 8, the naive hyperedge filling method outperforms the other two methods across all settings, highlighting the superior effectiveness of the hyperedge filling task.
F.6 TOPOLOGICAL GENERATIVE PRETEXT TASK ANALYSIS
In this section, we compare HYPEBOY with other topology generative SSL methods. To this end, adopt the following three baseline methods:
Graph edge filling: Utilizes edge filling on clique-expanded graph (Appendix E.2), employing the same method HYPEBOY used to fill size-2 hyperedges.
Graph hyperedge filling: Applics hyperedge filling on a clique-expanded graph (Appendix E.2), using the same approach as HYPEBOY for filling hyperedges.
Hyperedge prediction: Undertakes the hyperedge prediction task as an SSL task, leveraging size negative samples (Patil et al., 2020).
29

---

Published as a conference paper at ICLR 2024
Hyperedge Size
Hyperedge Size
Hyperedge Size
Cora Cite
AMiner
IMDB
242
230
0.20
224
616
041
Figure 6: Homophily analysis of three benchmark datasets: Cora-Cite, AMiner, and IMDB. Each color represents the class (label) of nodes. An upward trend in class indicates homophily, whereas a downward trend indicates nonhhomphilly. Cora-Cite demonstrates homophily, IMDB shows non- homophily, and AMiner falls in between, with some classes exhibiting homophily and others not.
As shown in Table 9, HYPEBOY outperforms the other three topological generative SSL methods in all settings, highlighting the effectiveness of the hyperedge filling task.
F.7 HOMOPHILY ANALYSIS
In this section, we demonstrate that HYPEBOY exhibits prominent node classification performance across various homophilic scenarios, including both homophilic and non-homophilic hypergraphs. To this end, we analyze both real-world hypergraphs and semi-synthetic hypergraphs.
Real-world hypergraphs. We analyze the homophily characteristic of three hypergraph benchmark datasets: Cora-Cite, AMiner, and IMDB, which are utilized in this work. Note that HYPEBOY shows the best performance among all the methods these datasets (refer Table 1).
For the homophily analysis, we utilize the method suggested by Veldt et al. (2023), which visualization-biia based approach. The analysis should conducted for each hyperedge size, with the results for each hyperedge size presented in separate plots. In these plots, the line color represents the node class. brief interpretation of the resulting plots is as follows:
An upward trend line indicates that nodes of the corresponding class demonstrate higher-order homophilic characteristics within hyperedges of the same size.
downward trend line indicates that nodes of the corresponding class display higher-order non- homophilic characteristics within hyperedges of the same size.
As shown in Figure 6, the three datasets (Cora-cite, AMiner, and IMDB) exhibit different homophilic patterns. Specifically, the Cora-Cite dataset demonstrates homophilic tendencies, the IMDB dataset shows a comohooppiiiic pattern, and the AMiner dataset falls between, with some classes being homophilic and others not. Given that HYPEBOY outperforms other methods in these datasets, can conclude that its efficacy valid for both homophilic and non-homophilic hypergraphs.
Semi-synthetic hypergraphs. We analyze how the performance of HYPEBOY varies the ho- mophilic extent hypergraphs varies. To this end, we corrupt the hypergraph topology of two homophilic hypergraph datasets: Cora and Citeseer. Specifically, method. Details are provided in Algorithm Note that the more we swap nodes a hypergraph, the more the hypergraph gets non-homophilic.
utilize the node-swapping
30

---

Published as a conference paper at ICLR 2024
Table 10: Analysis performance changes under variations in homophilic extent. The smallest drop from the original performance is highlighted in green. NA indicates UnIGCNII encoder without pre- training. As shown, HYPEBOY the most robust method against homophily violation among the used hypergraph SSL methods.
Dataset
Method
Original
T = 50
T = 100
T = 150
T = 200
NA
(-)
(-1.8)
44.7 (-3.8)
43.5 (-5.0)
40.9 (-7.6)
TriCL
60.2 (-)
59.1 (-1.1)
56.1 (-4.1)
55.7 (4.5)
53.6 (-6.5)
Cora
HyperGCL
60.3 (-)
57.4 (-2.9)
53.7 (-6.6)
52.0 (-8.3)
50.0 (-10.3)
HyPEBOY
62.3 (-)
61.7 (-0.6)
(-3.))
58.0 (-4.3)
56.7 (-5.6)
NA
(-)
41.4 (-2.8)
(-5.2)
35.9 (-8.3)
33.8 (-10.4)
TriCL
51.7
51.1 (-0.6)
47.5 (-4.2)
45.2 (-6.5)
41.8 (-9.9)
Citeseer
HyperGCL
(-)
45.8 (-1.2)
43.3 (-3.7)
40,9 (-6.1)
37.7 (-9.3)
HYPEBOY
56.7 (-)
56.5 (-0.2)
54.2 (-2.5)
50.8 (-5.9)
48.6 (-8.1)
Algorithm Node swapping algorithm
Input: A set of hyperedges E, Number of iterations
Output: Shuffled hyperedges E'
1: E' &
2: for 1 do
3:
Sample hyperedges eç and e/j from E' uniformly random
4:
Sample node v from e and node oj from e', uniformly random
5:
ef (e) N (v))
6:
+ (e, Me
7:
( (e) (e)
Return: Swapped hyperedges E
As shown in Table 10, HYPEBOY the most robust among the hypergraph SSL methods evaluated, exhibiting the smallest performance drop in scenarios where homophily is violated.
F.8 ADDITIONAL
SPLITS
In this section, we investigate the efficacy of HYPEBOY in various node-label-split settings. To this end, we sample a certain number of nodes for each class as training nodes and validation nodes. Specifically, we utilize the following two settings:
For each node class, use 5 nodes/5 nodes/the rest as train/validation/tess nodes, respectively.
For each node class, use 10 nodes/10 10 nodes/the rest as train/validdtionneess nodes, respectively.
Other experimental settings are the same as that of the fine-tuning experiments (Section 5.1). As shown in Table 11, HYPEBOY outperforms other hypergraph SSL methods in all the settings. Thus, we demonstrate that the efficacy of HYPEBOY is not restricted to a particular label split setting.
F.9 HETEROGENOUS HYPERGRAPH EXPERIMENT
In this section, we investigate the efficacy of HYPEBOY as pre-training strategy on the heteroge- neous hypergraph dataset, which contains hyperedges of various semanties. To this end, utilize the ACM dataset, where nodes correspond to publications. There are two types of hyperedges: authorship and subject relations.
Authorship: Publications authored by the same person are represented hyperedge.
Subject relations: Publications sharing subject are represented hyperedge.
Other experimental settings the same that of the fine-tuning experiments (Section 5.1). As shown in Table 12, HYPEBOY outperforms other hypergraph SSL methods in the ACM dataset, demonstrating its efficacy beyond homogeneous hypergraphs.
31

---

Published as a conference paper at ICLR 2024
Table 11: Results in the various label-split settings under the fine-tuning protocol. The best and second-best performances are colored green and yellow, respectively. NA indicates UniGCNII en- coder without pre-training. As shown, HYPEBOY outperforms other hypergraph SSL methods in all the settings.
Method
Citeseer
Cora
Pubmed
Cora-CA
DBLP-P
NA
53.7 (6.8)
63.7 (3.8)
71.6 (4.6)
60.7 (4.9)
77.8 (4.7)
TriCL
60.1 (5.2)
71.0
71.9 (3.8)
67,6 (2.9)
80.5 (2.5)
nodes
HyperGCL
57.0 (5.6)
70.3 (2.9)
72.2 (4.4)
61.5 (4.4)
79.5 (3.7)
HyPEBoY
63.1 I (4.6)
72.0 (2.4)
72.9 (3.2)
67.8 (2.4)
81.5 (2.)
NA
62.7 (3.8)
72.8 (2:2)
73.7 (3.0)
67.7 (3.0)
82.4 (1.9)
TriCL
66,3 (2.6)
75.7 (2.3)
74.5 (3.1)
69.7 (2.2)
83.6 (1.9)
10 nodes
HyperGCL
64.4 (3.9)
74.9 (2.1)
74,3 (3.1)
(344
83.6 (16)
HYPEBOY
67.7 (2.6)
75.8 (1.8)
74.9 (2.7)
70.9 (1.7) 84.6 (1.5)
Table 12: Results the ACM dataset under the fine-tuning protocol. The best performances are colored green. Among the four methods, HYPEBOY shows the best performance.
Method UniGCNII
TriCL
HyperGCL
HyPEBOY
Accuracy 40.0 (3.1) 40.3 (2.4) 40.8 (3.4)
41.7 (2.6)
G POTENTIAL APPLICATIONS OF HYPEREDGE FILLING
In this section, we provide two potential applications of the hyperedge filling task.
Email recipient recommendation: Given recipients of an email, recommend a user likely to be added as recipient of the email. Nodes in hypergraph indicate users, and each hyperedge indicates an email, consisting of a sender, recipients, and CCs.
Item recommendation: Given items in a shopping cart recommend an item to be co-purchased with the items. Nodes in hypergraph indicate items, and each hyperedge contains set of co- purchased items.
32


Reviews:
[{"review_id": "caIir2Re1J", "reviewer": "Reviewer_3LGX", "summary": "This paper introduces the first generative SSL methods, especially designed for hypergraph learning. The proposed self-supervised task is to fill in the missing node from an incomplete hyperedge. The authors show theoretically and empirically that, unde some conditions, this self-supervised task is well aligned with hypernode classification, achieving better results than other contrastive SSL methods from the hypergraph literature.", "questions": "Please see the Weaknesses section", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper identify and fill in a gap existent in the hypergraph literature: designing generative SSL tasks for hypergraph representation learning. The proposed hyperedge filling task is intuitive and well-suited for hypergraph representation learning. The theoretical finding enhances confidence in the approach.\n- The experimental section is comprehensive, demonstrating the individual contribution of each component.\n- The paper is generally well written.", "weaknesses": "- All the SSL-based results presented in the paper uses UniGCNII as a backbone. The authors motivate this decision by saying that this combinations achieves best overall performances. While I consider the comparison fair (since all the reported baselines uses UniGCNII as well), it is essential to conduct experiments that demonstrate the method's advantages when applied with various backbones. This additional experiments would clearly demonstrate the advantages of the proposed method. \n- While the authors did a good job in explaining the intuition behind this, it is somewhat discouraging to observe that the suggested approach performs noticeably worse when the feature reconstruction warmup is omitted (Table 3)\n- Minor: It would be useful to include the UNIGCNII baseline (without any of the 3 component) as a line in Table 3\n- Given the big improvement brought by the augmentation (masking) scheme (Table 6 in appendix), I am curious to know if the other baseline methods benefit from a similar augmentation step.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QwzwVMgSWi", "reviewer": "Reviewer_ctQb", "summary": "The paper introduces a generative self-supervised learning task called \"hypergraph filling\"  and explores generative self-supervised learning on hypergraphs. The author focuses on addressing issues related to overemphasized proximity, dimensional collapse, and non-uniformity/alignment problems in learned representations. To tackle these issues, the author proposes the \"HYPEBOY\" strategy for hypergraphs, both in theory and through empirical experiments. Furthermore, the author demonstrates the effectiveness of this approach in tasks such as node classification and link prediction.", "questions": "1. It's not straightforward to extend edge reconstruction methods to hyperedges in SSL. Considering this challenge, why did the author choose to employ SSL for hypergraphs without addressing the embeddings of hyperedges explicitly.\n2. Could you elaborate on the real-world applications of the hypergraph filling task? How is this task practically relevant?\n3. The author uses Gaussian distribution, Bernoulli sampling, and binomial distribution in the method. Could you explain the reasoning behind these choices and how they compare to more conventional methods like attention strategies and neural network approaches?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The author effectively identifies the issue of overemphasized proximity and demonstrates the beneficial impact of augmentation. Additionally, to address the problem of dimensional collapse, the author introduces a two-stage training scheme, which helps reduce the reliance on projection heads.", "weaknesses": "1. The rationale behind employing generative SSL for hypergraph representation is not convincingly established, and it confronts several challenges, including dimensional collapse.\n2. The author devises a SSL strategy for hypergraphs by utilizing existing encoders and decoders such as UniGCNII, HNN, and MLP, without introducing any novel model designs. The concept of the projection head for hypergraph encoding is inspired by Deep Sets[1].\n3.The author utilizes UniGCNII[2] as an encoder for HYPERBOY, and primarily focuses on homogeneous hypergraphs. However, it's important to note that heterogeneous hypergraphs are also prevalent, and the embedding method for hyperedges is not discussed.\n\n[1] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. NeurIPS 2017: 3391-3401.\n[2] Jing Huang, Jie Yang. UniGNN: a Unified Framework for Graph and Hypergraph Neural Networks.IJCAI 2021: 2563-2569.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZFL4FndF6A", "reviewer": "Reviewer_4MNM", "summary": "The paper makes an in-time contribution to studying the generative pretraining strategy for hypergraph neural networks. Specifically,  a novel generative SSL task on hypergraphs, hyperedge filling, is proposed, with the sound analysis demonstrating its effectiveness for node classification tasks. Extensive experiments are performed to support the claim.\n\nThe main intuition behind the magic of hyperedge filling is that the hyperedge, when it satisfies the homophily assumption, is indicative of the node membership and eventually helps node classification. This analysis supports the intuition. Although there might be some inconsistency between theory and practice, I appreciate the analysis part a lot.", "questions": "Please see Weaknesses", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Nice and reasonable intuition to motivate the algorithm: The forming of hyperedge indicates the nodes share the same (or similar) membership. This might be motivated by the stochastic block models. It is a great intuition and I think it fits many applications such as social networks.\n- Sound theoretical analysis to support the intuition. This is done by analyzing how the representations change to optimize hypergraph filling loss can improve node classification results.\n- Extensive empirical results are provided. More appreciatively, the numerical characteristics, such as proximity, are examined to reach certain conclusions.", "weaknesses": "I do not have major criticisms for this paper. I only have some questions regarding the analysis part, which might result from missing some points during reading.\n\n-  In the hyperedge filling analysis, it seems to not relate to the neural network architecture. How do authors think the choice of the hypergraph neural networks would affect the performance?\n- In the hyperedge filling process (F2), the representation is updated with a gradient w.r.t. the hyperedge filling loss and a step size $\\gamma$. Per my reading of the proof, the value of $\\gamma$ (only need to be greater than 0) seems to not affect the result, while in practice this might not be true. Do I miss some points here?\n- Is the edge filling optimal to perform as it is in hypergraphs? Considering I am gonna perform clique expansion to get a graph and perform edge filling. How would the result be different?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dIHNA2c4Dz", "reviewer": "Reviewer_EPYj", "summary": "This article proposes a novel self-supervised learning (SSL) task, hyperedge filling. The authors also give the relationship between hyperedge filling and node classification. Based on the SSL task, the authors propose HypeBoy, which is composed of 3 steps, hypergraph augmentation, hypergraph encoding, and hypergraph filling. The authors demonstrate the effectiveness of HypeBoy under multiple SSL tasks on multiple datasets.", "questions": "1. In BASIC SETTING, one of the assumptions is that the homophily ratio of each hyperedge is in [0.5, 1]. In graphs, the homophily rate is defined as the proportion of intra-class edges to all edges. So how does this definition apply to the hypergraph? Do all datasets in the experiments satisfy this assumption?\n2. I noticed that in section 5.1, the article uses a setting of 1% of the training set, which is different from most articles. Most other papers use fixed division or 5/10 nodes per category. I'm curious about what the considerations are for such an experimental setup?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The article proposes a novel SSL task, hyperedge filling, which is a supplement to the general node classification task.\n2. The hyperedge-filling task proposed in the article can obtain more accurate node features, thereby helping other tasks.", "weaknesses": "1. In the hypothesis, the premise of the article is that each hyperedge contains most of the same category vertices. However, there are hyperedges that do not satisfy this situation. For example, a hyperedge of size 10 contains 5 nodes of category A and 5 category B.\n2. One of the main contributions of the paper is hyperedge filling, but using only hyperedge filling (v1) in the ablation experiment is the worst among v1-v4. The effectiveness of using hyperedge filling alone is questionable, or it needs to be bound to feature reconstruction and projection heads.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)