SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- NDC Class [6]: The NDC Class dataset is a temporal higher-order network, in which each hyperedge corresponds to an individual drug, and the nodes contained within the hyperedges represent class labels assigned to these drugs. The timestamps, measured in days, indicate the initial market entry of each drug. Here, hyperedge prediction aims to predict future drugs.
- NDC Substances [6]: The NDC Substances is a temporal higher-order network, where each hyperedge represents an NDC code associated with a specific drug, while the nodes represent the constituent substances of the drug. The timestamps, measured in days, indicate the initial market entry of each drug. The hyperedge prediction task is the same as NDC Classes dataset.
- High School [6, 92]: The High School is a temporal higher-order network dataset constructed from interactions recorded by wearable sensors in a high school setting. The dataset captures a high school contact network, where each student/teacher is represented as a node and each hyperedge shows Face-to-face contact among individuals. Interactions were recorded at a resolution of 20 seconds, capturing all interactions that occurred within the previous 20 seconds. Node labels in this data are the class of students, and we focus on the node class "PSI" in our classification tasks.
- Primary School [6, 93]: The primary school dataset resembles the high school dataset, differing only in terms of the school level from which the data is collected. Node labels in this data are the class of students, and we focus on the node class "Teachers" in our classification tasks.
- Congress Bill [6, 94, 95]: Each node in this dataset represents a US Congressperson. Each hyperedge is a legislative bill in both the House of Representatives and the Senate, connecting the sponsors and co-sponsors of each respective bill. The timestamps, measured in days, indicate the date when each bill was introduced.
- Email Enron [6]: In this dataset nodes are email addresses at Enron and hyperedges are formed by emails, connecting the sender and recipients of each email. The timestamps have a resolution of milliseconds.
- Email Eu [6, 96]: In this dataset, the nodes represent email addresses associated with a European research institution. Each hyperedge consists of the sender and all recipients of the email. The timestamps in this dataset are measured with a resolution of 1 second.
- Question Tags M (Math sx) [6]: This dataset consists of nodes representing tags and hyperedges representing sets of tags applied to questions on math.stackexchange.com. The timestamps in the dataset are recorded at millisecond resolution and have been normalized to start at 0.
- Question Tags U (Ask Ubuntu) [6]: In this dataset, the nodes represent tags, and the hyperedges represent sets of tags applied to questions on askubuntu.com. The timestamps in the dataset are recorded with millisecond resolution and have been normalized to start at 0.
<sup>3</sup>https://www.cs.cornell.edu/~arb/data/
Table 3: Dataset statistics. HeP: Hyperedge Prediction, NC: Node Classification
Dataset	NDC Class	High School	Primary School	Congress Bill	Email Enron	Email Eu	Question Tags M	Users-Threads	NDC Substances	Question Tags U
|V|	1,161	327	242	1,718	143	998	1,629	125,602	5,311	3,029
|E|	49,724	172,035	106,879	260,851	10,883	234,760	822,059	192,947	112,405	271,233
#Timestamps	5,891	7,375	3,100	5,936	10,788	232,816	822,054	189,917	7,734	271,233
Task	HeP	HeP& NC	HeP & NC	HeP	HeP	HeP	HeP	HeP	HeP	HeP
- Users-Threads [6]: In this dataset, the nodes represent users on askubuntu.com, and a hyperedge is formed by users participating in a thread that lasts for a maximum duration of 24 hours. The timestamps in the dataset denote the time of each post, measured in milliseconds but normalized such that the earliest post begins at 0.
The statistics of these datasets can be found in Table 3.
F.2Baselines
We compare our method to eight previous state-of-the-art methods and baselines on the hyperedge prediction task:
- CHESHIRE [11]: Chebyshev spectral hyperlink predictor (CHESHIRE), is a hyperedge prediction methods that initializes node embeddings by directly passing the incidence matrix through a one-layer neural network. CHESHIRE treats a hyperedge as a fully connected graph (clique) and uses a Chebyshev spectral GCN to refine the embeddings of the nodes within the hyperedge. The Chebyshev spectral GCN leverages Chebyshev polynomial expansion and spectral graph theory to learn localized spectral filters. These filters enable the extraction of local and composite features from graphs that capture complex geometric structures. The model with code provided is here.
- HyperSAGCN [26]: Self-attention-based graph convolutional network for hypergraphs (HyperSAGCN) utilizes a Spectral Aggregated Graph Convolutional Network (SAGCN) to refine the embeddings of nodes within each hyperedge. HyperSAGCN generates initial node embeddings by hypergraph random walks and combines node embeddings by MEAN (.p) pooling to compute the embedding of hyperedge. The model with code provided is here.
- NHP [87]: Neural Hyperlink Predictor (NHP), is an enhanced version of HyperSAGCN. NHP initializes node embeddings using Node2Vec on the CE graph and then uses a novel maximum minimum-based pooling function that enables adaptive weight learning in a task-specific manner, incorporating additional prior knowledge about the nodes. The model with code provided is here.
- HPLSF [89]: Hyperlink Prediction using Latent Social Features (HPLSF) is a probabilistic method. It leverages the homophily property of the networks and introduces a latent feature learning approach, incorporating the use of entropy in computing hyperedge embedding. The model with code provided is here.
- HPRA [88]: Hyperlink Prediction Using Resource Allocation (HPRA) is a hyperedge prediction method based on the resource allocation process. HPRA calculates a hypergraph resource allocation (HRA) index between two nodes, taking into account direct connections and shared neighbors. The HRA index of a candidate hyperedge is determined by averaging all pairwise HRA indices between the nodes within the hyperedge. The model with code provided is here.
- CE-CAW: This model is a baseline that we apply CAW [33] on the CE of the hypergraph. CAW is a temporal edge prediction method that uses causal anonymous random walks to capture the dynamic laws of the network in an inductive manner. The model with code provided is here.
- CE-EvolveGCN: This is a snapshot-based temporal graph learning method that we apply EvolveGCN [90], which uses RNs to estimate the GCN parameters for the future snapshots, on the CE of the hypergraph. The model with code provided is here.
- CE-GCN: We apply Graph Convolutional Networks [91] to the CE of the hypergraph to obtain node embeddings. Next, we use MLP to predict edges. The implementation is provided in the Pytorch Geometric library.
Table 4: Hyperparameters used in the grid search.
Datasets	Sampling Number M	Sampling Time Bias α	SETWALK Length m	Hidden dimensions
NDC Class	4, 8, 16, 32, 64, 128	{0.5, 2.0, 20, 200} × 10-7	2, 3, 4, 5	32, 64, 128
High School	4, 8, 16, 32, 64, 128	{0.5, 2.0, 20, 200} × 10-7	2, 3, 4, 5	32, 64, 128
Primary School	4, 8, 16, 32, 64, 128	{0.5, 2.0, 20, 200} × 10-7	2, 3, 4, 5	32, 64, 128
Congress Bill	8, 16, 32, 64	{0.5, 2.0, 20, 200} × 10-7	2, 3, 4, 5	32, 64, 128
Email Enron	8, 16, 32, 64	{0.5, 2.0, 20, 200} × 10-7	2, 3, 4, 5	32, 64, 128
Email Eu	8, 16, 32, 64	{0.5, 2.0, 20, 200} × 10-7	2, 3, 4	32, 64, 128
Question Tags M	8, 16, 32, 64	{0.5, 2.0, 20, 200} × 10-7	2, 3, 4	32, 64, 128
Users-Threads	8, 16, 32, 64	{0.5, 2.0, 20, 200} × 10-7	2, 3, 4	32, 64, 128
NDC Substances	8, 16, 32, 64	{0.5, 2.0, 20, 200} × 10-7	2, 3, 4	32, 64, 128
Question Tags U	8, 16, 32, 64	{0.5, 2.0, 20, 200} × 10-7	2, 3, 4	32, 64, 128
For node classification, we use additional five state-of-the-art deep hypergraph learning methods and a CE-based baseline:
- HyperGCN [19]: This is a generalization of GCNs to hypergraphs, where it uses hypergraph Laplacian to define convolution.
- ALLDeepSets and ALLSetTransformer [25]: These two methods are two variants of the general message passing framework, Allset, on hypergraphs, which are based on the aggregation of messages from nodes to hyperedges and from hyperedges to nodes.
- UNI GCNII [28]: Is an advanced variant of UNI GNN, a general framework for message passing on hypergraphs.
- ED-HNN [124]: Inspired by hypergraph diffusion algorithms, this method uses star expansions of hypergraphs with standard message passing neural networks.
- CE-GCN: We apply Graph Convolutional Networks [91] to the CE of the hypergraph to obtain node embeddings. Next, we use MLP to predict the labels of nodes. The implementation is provided in the Pytorch Geometric library. [91]
For all the baselines, we set all sensitive hyperparameters (e.g., learning rate, dropout rate, batch size, etc.) to the values given in the paper that describes the technique. Following [60], for deep learning methods, we tune their hidden dimensions via grid search to be consistent with what we did for CAr-WALK. We exclude HPLSF [89] and HPRA [88] from inductive hyperedge prediction as it does not apply to them.
F.3 Implementation and Training Details
In addition to hyperparameters and modules (activation functions) mentioned in the main paper, here, we report the training hyperparameters of CAr-WALK: On all datasets, we use a batch size of 64 and set learning rate  $= 10^{-4}$ . We also use an early stopping strategy to stop training if the validation performance does not increase for more than 5 epochs. We use the maximum training epoch number of 30 and dropout layers with rate  $= 0.1$ . Other hyperparameters used in the implementation can be found in the README file in the supplement.
Also, for tuning the model's hyperparameters, we systematically tune them using grid search. The search domains of each hyperparameter are reported in Table 4. Note that, the last column in Table 4 reports the search domain for hidden dimensions of modules in  $\mathrm{CAT - WALK}$ , including SETMIXER, MLP-MIXER, and MLPs. Also, we tune the last layer pooling strategy with two options: SETMIXER or MEAN(). whichever leads to a better performance.
We implemented our method in Python 3.7 with PyTorch and run the experiments on a Linux machine with nvidia RTX A4000 GPU with 16GB of RAM.
Table 5: Performance on node classification: Mean ACC  $(\%)$  ± standard deviation. Boldfaced letters shaded blue indicate the best result, while gray shaded boxes indicate results within one standard deviation of the best result.
Methods	High School	Primary School	Average Performance
Inductive	CE-GCN	76.24 ± 2.99	79.03 ± 3.16	77.63 ± 3.07
HYPERGCN	83.91 ± 3.05	86.17 ± 3.40	85.04 ± 3.23
HYPERSAGCN	84.89 ± 3.80	82.13 ± 3.69	83.51 ± 3.75
ALLDEEPSETS	85.67 ± 4.17	81.43 ± 6.77	83.55 ± 5.47
UNI GCNII	88.36 ± 3.78	88.27 ± 3.52	88.31 ± 3.63
ALLSET TRANSFORMER	91.19 ± 2.85	90.00 ± 4.35	90.59 ± 3.60
ED-HNN	89.23 ± 2.98	90.83 ± 3.02	90.03 ± 3.00
CAT-WALK	88.99 ± 4.76	93.28 ± 2.41	91.13 ± 3.58
Transductive	CE-GCN	78.93 ± 3.11	77.46 ± 2.97	78.20 ± 3.04
HYPERGCN	84.90 ± 3.59	85.23 ± 3.06	85.07 ± 3.33
HYPERSAGCN	84.52 ± 3.18	83.27 ± 2.94	83.90 ± 3.06
ALLDEEPSETS	85.97 ± 4.05	80.20 ± 10.18	83.09 ± 7.12
UNI GCNII	89.16 ± 4.37	90.29 ± 4.01	89.73 ± 4.19
ALLSET TRANSFORMER	90.75 ± 3.13	89.80 ± 2.55	90.27 ± 2.84
ED-HNN	91.41 ± 2.36	91.74 ± 2.62	91.56 ± 2.49
CAT-WALK	90.66 ± 4.96	93.20 ± 2.45	91.93 ± 3.71
G.1 Results on More Datasets
Due to the space limit, we report the AUC results on only eight datasets in Section 4. Table 6 reports both AUC and average precision (AP) results on all 10 datasets in both inductive and transductive hyperedge prediction tasks.
G.2 Node Classification
In the main text, we focus on the hyperedge prediction task. Here we describe how CAr-WALK can be used for node classification tasks.
For each node  $u_0$  in the training set, we sample  $\max \{ \deg(u_0), 10 \}$  hyperedges such as  $e_0 = \{u_0, u_1, \ldots, u_k\}$ . Next, for each sampled hyperedge we sample  $M$  SETWALKs with length  $m$  starting from each  $u_i \in e_0$  to construct  $S(u_i)$ . Next, we anonymize each hyperedge that appears in at least one SETWALK in  $\bigcup_{i=0}^{k} S(u_i)$  by Equation 3 and then use the MLP-MIXER module to encode each  $\mathrm{Sw} \in \bigcup_{i=0}^{k} S(u_i)$ . To encode each node  $u_i \in e_0$ , we use MEAN(.) pooling over SETWALKs in  $S(u_i)$ . Finally, for node classification task, we use a 2-layer perceptron over the node encodings to make the final prediction.
Table 5 reports the results of dynamic node classification tasks on High School and Primary School datasets. CAt-WALK achieves the best or on-par performance on dynamic node classification tasks. While all baselines are specifically designed for node classification tasks, CAt-WALK achieves superior results due to  $①$  its ability to incorporate temporal properties (both from SETWALKs and our time encoding module), which helps to learn underlying dynamic laws of the network, and  $②$  its two-step set-based anonymization process that hides node identities from the model. Accordingly, CAt-WALK can learn underlying patterns needed for the node classification task, instead of using node identities, which might cause memorizing vertices.
G.3 Performance in Average Precision
In addition to the AUC, we also compare our model with baselines with respect to Average Precision (AP). Table 6 reports both AUC and AP results on all 10 datasets in inductive and transductive hyperedge prediction tasks. As discussed in Section 4,  $\mathrm{CAr - W_{ALK}}$  due to its ability to capture both temporal and higher-order properties of the hypergraphs, achieves superior performance and outperforms all baselines in both transductive and inductive settings with a significant margin.
G.4 More Results on RNN v.s. MLP-Mixer in Walk Encoding
Most existing methods on (temporal) random walk encoding see a walk as a sequence of vertices and uses sequence encoders like RNNS or TRANSFORMERS to encode each walk. The main drawback of these methods is that they fail to directly process temporal walks with irregular gaps between timestamps. That is, sequential encoders can be seen as discrete approximations of dynamic systems; however, this discretization often fails if we have irregularly observed data [125]. nper This is the main motivation of recent studies to develop methods on continuous-time temporal networks [34, 126]. Most of these methods are too complicated and sometimes fail to generalize [127]. In CAr-WALK, we suggest a simple architecture to encode temporal walks by a time-encoding module along with a MIXER module (see Section 3.4 for the details). In this part, we evaluate the power of our MIXER module and compare its performance when we replace it with RNNS [82]. Figure 7 reports the results on all datasets. We observe that using MLP-MixER with the time-encoding module in CAr-WALK can always outperform CAr-WALK when we replace MLP-MixER with a RNN, and mostly this improvement is more on datasets with high variance in their timestamps. We relate this superiority to the importance of using continuous-time encoding instead of sequential encoders.
Figure 7: The importance of using MLP-MIXER in CAT-WALK. Using an RNN instead of MLP-MIXER can damage the performance in all datasets. RNNS are sequential encoders and are not able to encode continuous time in the data.
Temporal hypergraph learning methods, such as CAr-WALK, benefit a wide array of real-world applications, including but not limited to social network analysis, recommender systems, brain network analysis, drug discovery, stock price prediction, and anomaly detection (e.g. bot detection in social media or abnormal human brain activity). However, there might be some potentially negative impacts, which we list as: ① Learning underlying biased patterns in the training data, which may result in stereotyped predictions. Since CAr-WALK learns underlying dynamic laws in the training data, given biased training data, the predictions of CAr-WALK can be biased. ② Also, powerful dynamic hypergraph models can be used for manipulation in the abovementioned applications (e.g., stock price manipulation). Accordingly, to prevent the potential risks in sensitive tasks, e.g., decision-making from graph-structured data in health care, interpretability and explainability of machine learning models on hypergraphs is a critical area for future work.
Furthermore, this work does not perform research on human subjects as part of the study and all used datasets are anonymized and publicly available.