SlowGuess/NeurIPS_2023_OCR huggingface dataset:
While simple concatenation does not suffer from this undesirable property, it is not permutation invariant. To overcome these challenges, we design an all-MLP permutation invariant pooling function, SETMIXER, that not only captures higher-order dependencies of set elements but also captures dependencies across the number of times that a node appears at a certain position in SETWALKs.
SetMixer. MLP-Mixer [44] is a family of models based on multi-layer perceptrons, widely used in the computer vision community, that are simple, amenable to efficient implementation, and robust to oversquashing and long-term dependencies (unlike RNNs and attention mechanisms) [44, 57]. However, the token-mixer phase of these methods is sensitive to the order of the input (see Appendix A). To address this limitation, inspired by MLP-Mixer [44], we design SetMixer as follows: Let  $S = \{\mathbf{v}_1,\dots ,\mathbf{v}_d\}$ , where  $\mathbf{v}_i\in \mathbb{R}^{d_1}$ , be the input set and  $\mathbf{V} = [\mathbf{v}_1,\dots ,\mathbf{v}_d]\in \mathbb{R}^{d\times d_1}$  be its matrix representation:
$$
\Psi (S) = \operatorname {M E A N} \left(\mathbf {H} _ {\text {t o k e n}} + \sigma \left(\text {L a y e r N o r m} \left(\mathbf {H} _ {\text {t o k e n}}\right) \mathbf {W} _ {s} ^ {(1)}\right) \mathbf {W} _ {s} ^ {(2)}\right), \tag {5}
$$
$$
\mathbf {H} _ {\text {t o k e n}} = \mathbf {V} + \sigma \left(\operatorname {S o f t m a x} \left(\operatorname {L a y e r N o r m} (\mathbf {V}) ^ {T}\right)\right) ^ {T}. \tag {6}
$$
Here,  $\mathbf{W}_s^{(1)}$  and  $\mathbf{W}_s^{(2)}$  are learnable parameters,  $\sigma(.)$  is an activation function (we use GeLU [80] in our experiments), and LayerNorm is layer normalization [81]. Equation 5 is the channel mixer and Equation 6 is the token mixer. The main intuition of SETMIXER is to use the Softmax(.) function to bind token-wise information in a non-parametric manner, avoiding permutation variant operations in the token mixer. We formally prove the following theorem in Appendix E.3.
Theorem 3. SETMIXER is permutation invariant and is a universal approximator of invariant multi-set functions. That is, SETMIXER can approximate any invariant multi-set function.
Based on the above theorem, SETMIXER can overcome the challenges we discussed earlier as it is permutation invariant. Also, it is a universal approximator of multi-set functions, which shows its power to learn any arbitrary function. Accordingly, in our anonymization process, we use  $\Psi(.)=\mathsf{SETMIXER}(.)$  in Equation 3 to hide hyperedge identities. Next, we guarantee that our anonymization process does not depend on hyperedges or nodes identities, which justifies the claim of inductiveness of our model:
Proposition 1. Given two (potential) hyperedges  $e_0 = \{u_1, \ldots, u_k\}$  and  $e_0' = \{u_1', \ldots, u_k'\}$ , if there exists a bijective mapping  $\pi$  between node identities such that for each SETWALK like  $\mathrm{Sw} \in \bigcup_{i=1}^{k} S(u_i)$  can be mapped to one SETWALK like  $\mathrm{Sw}' \in \bigcup_{i=1}^{k} S(u_i')$ , then for each hyperedge  $e = \{w_1, \ldots, w_{k'}\}$  that appears in at least one SETWALK in  $\bigcup_{i=1}^{k} S(u_i)$ , we have  $\mathrm{ID}(e, e_0) = \mathrm{ID}(\pi(e), e_0')$ , where  $\pi(e) = \{\pi(w_1), \ldots, \pi(w_{k'})\}$ .
Finally, we guarantee that our anonymization approach is more expressive than existing anonymization process [33, 45] when applied to the CE of the hypergraphs:
Theorem 4. The set-based anonymization method is more expressive than any existing anonymization strategies on the CE of the hypergraph. More precisely, there exists a pair of hypergraphs  $\mathcal{G}_1 = (\mathcal{V}_1, \mathcal{E}_1)$  and  $\mathcal{G}_2 = (\mathcal{V}_2, \mathcal{E}_2)$  with different structures (i.e.,  $\mathcal{G}_1 \not\cong \mathcal{G}_2$ ) that are distinguishable by our anonymization process and are not distinguishable by the CE-based methods.
3.4 SETWALK Encoding
Previous walk-based methods [33, 34, 78] view a walk as a sequence of nodes. Accordingly, they plug nodes' positional encodings in a RNN [82] or Transformer [83] to obtain the encoding of each walk. However, in addition to the computational cost of RNN and Transformers, they suffer from over-squashing and fail to capture long-term dependencies. To this end, we design a simple and low-cost SETWALK encoding procedure that uses two steps: ① A time encoding module to distinguish different timestamps, and ② A mixer module to summarize temporal and structural information extracted by SETWALKs.
Time Encoding. We follow previous studies [33, 84] and adopt random Fourier features [85, 86] to encode time. However, these features are periodic, so they capture only periodicity in the data. We add a learnable linear term to the feature representation of the time encoding. We encode a given time  $t$  as follows:
$$
\mathrm {T} (t) = \left(\omega_ {l} t + \mathbf {b} _ {l}\right) \| \cos \left(t \omega_ {p}\right), \tag {7}
$$
where  $\omega_{l},\mathbf{b}_{l}\in \mathbb{R}$  and  $\omega_{p}\in \mathbb{R}^{d_{2} - 1}$  are learnable parameters and  $\parallel$  shows concatenation.
MIXER Module. To summarize the information in each SETWALK, we use a MLP-MIXER [44] on the sequence of hyperedges in a SETWALK as well as their corresponding encoded timestamps. Contrary to the anonymization process, where we need a permutation invariant procedure, here, we need a permutation variant procedure since the order of hyperedges in a SETWALK is important. Given a (potential) hyperedge  $e_0 = \{u_1,\dots ,u_k\}$ , we first assign  $\mathrm{Id}(e,e_0)$  to each hyperedge  $e$  that appears on at least one sampled SETWALK starting from  $e_0$  (Equation 3). Given a SETWALK,  $\mathrm{Sw}: (e_1,t_{e_1})\to \dots \to (e_m,t_{e_m})$ , we let  $\mathbf{E}$  be a matrix that  $\mathbf{E}_i = \mathrm{Id}(e_i,e_0)\| \mathrm{T}(t_{e_i})$ :
$$
\operatorname {E N C} (\mathrm {S W}) = \operatorname {M E A N} \left(\mathbf {H} _ {\text {t o k e n}} + \sigma \left(\text {L a y e r N o r m} \left(\mathbf {H} _ {\text {t o k e n}}\right) \mathbf {W} _ {\text {c h a n n e l}} ^ {(1)}\right) \mathbf {W} _ {\text {c h a n n e l}} ^ {(2)}\right), \tag {8}
$$
$$
\mathbf {H} _ {\text {t o k e n}} = \mathbf {E} + \mathbf {W} _ {\text {t o k e n}} ^ {(2)} \sigma \left(\mathbf {W} _ {\text {t o k e n}} ^ {(1)} \operatorname {L a y e r N o r m} (\mathbf {E})\right). \tag {9}
$$
3.5 Training
In the training phase, for each hyperedge in the training set, we adopt the commonly used negative sample generation method [60] to generate a negative sample. Next, for each hyperedge in the training set such as  $e_0 = \{u_1, u_2, \ldots, u_k\}$ , including both positive and negative samples, we sample  $M_{\text{SETWALKs}}$  with length  $m$  starting from each  $u_i \in e_0$  to construct  $S(u_i)$ . Next, we anonymize each hyperedge that appears in at least one  $\text{SETWALK}$  in  $\bigcup_{i=1}^{k} S(u_i)$  by Equation 3 and then use the MIXER module to encode each  $\text{Sw} \in \bigcup_{i=1}^{k} S(u_i)$ . To encode each node  $u_i \in e_0$ , we use MEAN(. ) pooling over  $\text{SETWALKs}$  in  $S(u_i)$ . Finally, to encode  $e_0$  we use SETMIXER to mix obtained node encodings. For
hyperedge prediction, we use a 2-layer perceptron over the hyperedge encodings to make the final prediction. We discuss node classification in Appendix G.2.
We evaluate the performance of our model on two important tasks: hyperedge prediction and node classification (see Appendix G.2) in both inductive and transductive settings. We then discuss the importance of our model design and the significance of each component in CAr-WALK.
Baselines. We compare our method to eight previous state-of-the-art baselines on the hyperedge prediction task. These methods can be grouped into three categories: ① Deep hypergraph learning methods including HyperSAGCN [26], NHP [87], and CHESHIRE [11]. ② Shallow methods including HPRA [88] and HPLSF [89]. ③ CE methods: CE-CAW [33], CE-EVOLVEGCN [90] and CE-GCN [91]. Details on these models and hyperparameters used appear in Appendix F.2.
Datasets. We use 10 available benchmark datasets [6] from the existing hypergraph neural networks literature. These datasets' domains include drug networks (i.e., NDC [6]), contact networks (i.e., High School [92] and Primary School [93]), the US. Congress bills network [94, 95], email networks (i.e., Email Enron [6] and Email Eu [96]), and online social networks (i.e., Question Tags and Users-Threads [6]). Detailed descriptions of these datasets appear in Appendix F.1.
Evaluation Tasks. We focus on Hyperedge Prediction: In the transductive setting, we train on the temporal hyperedges with timestamps less than or equal to  $T_{\mathrm{train}}$  and test on those with timestamps greater than  $T_{\mathrm{train}}$ . Inspired by Wang et al. [33], we consider two inductive settings. In the Strongly Inductive setting, we predict hyperedges consisting of some unseen nodes. In the Weakly Inductive setting, we predict hyperedges with at least one seen and some unseen nodes. We first follow the procedure used in the transductive setting, and then we randomly select 10% of the nodes and remove all hyperedges that include them from the training set. We then remove all hyperedges with seen nodes from the validation and testing sets. For dynamic node classification, see Appendix G.2. For all datasets, we fix  $T_{\mathrm{train}} = 0.7T$ , where  $T$  is the last timestamp. To evaluate the models' performance we follow the literature and use Area Under the ROC curve (AUC) and Average Precision (AP).
Hyperedge Prediction. We report the results of CAT-WALK and baselines in Table 1 and Appendix G. The results show that CAT-WALK achieves the best overall performance compared to the baselines in both transductive and inductive settings. In the transductive setting, not only does our method outperform baselines in all but one dataset, but it achieves near perfect results (i.e.,  $\approx 98.0$ ) on the NDC and Primary School datasets. In the Weakly Inductive setting, our model achieves high scores (i.e.,  $>91.5$ ) in all but one dataset, while most baselines perform poorly as they are not designed for the inductive setting and do not generalize well to unseen nodes or patterns. In the Strongly Inductive setting, CAT-WALK still achieves high AUC (i.e.,  $>90.0$ ) on most datasets and outperforms baselines on all datasets. There are three main reasons for CAT-WALK's superior performance: ① Our SETWALKs capture higher-order patterns. ② CAT-WALK incorporates temporal properties (both from SETWALKs and our time encoding module), thus learning underlying dynamic laws of the network. The other temporal methods (CE-CAW and CE-EvolveGCN) are CE-based methods, limiting their ability to capture higher-order patterns. ③ CAT-WALK's set-based anonymization process that avoids using node and hyperedge identities allows it to generalize to unseen patterns and nodes.
Ablation Studies. We next conduct ablation studies on the High School, Primary School, and Users-Threads datasets to validate the effectiveness of CAT-WALK's critical components. Table 2 shows AUC results for inductive hyperedge prediction. The first row reports the performance of the complete CAT-WALK implementation. Each subsequent row shows results for CAT-WALK with one module modification: row 2 replace SETWALK by edge-dependent hypergraph walk [40], row 3 removes the time encoding module, row 4 replaces SETMIXER with MEAN(.p) pooling, row 5 replaces the SETMIXER with sum-based universal approximator for sets [97], row 6 replaces the MLP-MIXER
Table 1: Performance on hyperedge prediction: Mean AUC  $(\%)\pm$  standard deviation. Boldfaced letters shaded blue indicate the best result, while gray shaded boxes indicate results within one standard deviation of the best result. N/A: the method has numerical precision or computational issues.
Methods	NDC Class	High School	Primary School	Congress Bill	Email Enron	Email Eu	Question Tags	Users-Threads
Strongly Inductive
Inductive	CE-GCN	52.31 ± 2.99	60.54 ± 2.06	52.34 ± 2.75	49.18 ± 3.61	63.04 ± 1.80	52.76 ± 2.41	56.10 ± 1.88	57.91 ± 1.56
CE-EVOLVEGCN	49.78 ± 3.13	46.12 ± 3.83	58.01 ± 2.56	54.00 ± 1.84	57.31 ± 4.19	44.16 ± 1.27	64.08 ± 2.75	52.00 ± 2.32
CE-CAW	76.45 ± 0.29	83.73 ± 1.42	80.31 ± 1.46	75.38 ± 1.25	70.81 ± 1.13	72.99 ± 0.20	70.14 ± 1.89	73.12 ± 1.06
NHP	70.53 ± 4.95	65.29 ± 3.80	70.86 ± 3.42	69.82 ± 2.19	49.71 ± 6.09	65.35 ± 2.07	68.23 ± 3.34	71.83 ± 2.64
HyperSAGCN	79.05 ± 2.48	88.12 ± 3.01	80.13 ± 1.38	79.51 ± 1.27	73.09 ± 2.60	78.01 ± 1.24	73.66 ± 1.95	73.94 ± 2.57
CHESHIRE	72.24 ± 2.63	82.54 ± 0.88	77.26 ± 1.01	79.43 ± 1.58	70.03 ± 2.55	69.98 ± 2.71	N/A	76.99 ± 2.82
CAT-WALK	98.89 ± 1.82	96.03 ± 1.50	95.32 ± 0.89	93.54 ± 0.56	73.45 ± 2.92	91.68 ± 2.78	88.03 ± 3.38	89.84 ± 6.02
Weakly Inductive
Inductive	CE-GCN	51.80 ± 3.29	50.33 ± 3.40	52.19 ± 2.54	52.38 ± 2.75	50.81 ± 2.87	49.60 ± 3.96	55.13 ± 2.76	57.06 ± 3.16
CE-EVOLVEGCN	55.39 ± 5.16	57.85 ± 3.51	51.50 ± 4.07	55.63 ± 3.41	45.66 ± 2.10	52.44 ± 2.38	61.79 ± 1.63	55.81 ± 2.54
CE-CAW	77.61 ± 1.05	83.77 ± 1.41	82.98 ± 1.06	79.51 ± 0.94	80.54 ± 1.02	73.54 ± 1.19	77.29 ± 0.86	80.79 ± 0.82
NHP	75.17 ± 2.02	67.25 ± 5.19	71.92 ± 1.83	69.58 ± 4.07	60.38 ± 4.45	67.19 ± 4.33	70.46 ± 3.52	76.44 ± 1.90
HyperSAGCN	79.45 ± 2.18	88.53 ± 1.26	85.08 ± 1.45	80.12 ± 2.00	78.86 ± 0.63	77.26 ± 2.09	78.15 ± 1.41	75.38 ± 1.43
CHESHIRE	79.03 ± 1.24	88.40 ± 1.06	83.55 ± 1.27	79.67 ± 0.83	74.53 ± 0.91	77.31 ± 0.95	N/A	81.27 ± 0.85
CAT-WALK	99.16 ± 1.08	94.68 ± 2.37	96.53 ± 1.39	98.38 ± 0.21	64.11 ± 7.96	91.98 ± 2.41	90.28 ± 2.81	97.15 ± 1.81
Weakly Inductive
Inductive	HPRA	70.83 ± 0.01	94.91 ± 0.00	89.86 ± 0.06	79.48 ± 0.03	78.62 ± 0.00	72.51 ± 0.00	83.18 ± 0.00	70.49 ± 0.02
HPLSF	76.19 ± 0.82	92.14 ± 0.29	88.57 ± 1.09	79.31 ± 0.52	75.73 ± 0.05	75.27 ± 0.31	83.45 ± 0.93	74.38 ± 1.11
CE-GCN	66.83 ± 3.74	62.99 ± 3.02	59.14 ± 3.87	64.42 ± 3.11	58.06 ± 3.80	64.19 ± 2.79	55.18 ± 5.12	62.78 ± 2.69
CE-EVOLVEGCN	67.08 ± 3.51	65.19 ± 2.26	63.15 ± 1.32	69.30 ± 2.27	69.98 ± 5.38	64.36 ± 4.17	72.56 ± 1.72	68.55 ± 2.26
CE-CAW	76.30 ± 0.84	81.63 ± 0.97	86.53 ± 0.84	76.99 ± 1.02	79.57 ± 0.14	78.19 ± 1.10	81.73 ± 2.48	80.86 ± 0.45
NHP	82.39 ± 2.81	76.85 ± 3.08	80.04 ± 3.42	80.27 ± 2.53	63.17 ± 3.79	78.90 ± 4.39	79.14 ± 3.36	82.33 ± 1.02
HyperSAGCN	80.76 ± 2.64	94.98 ± 1.30	90.77 ± 2.05	82.84 ± 1.61	83.59 ± 0.98	79.61 ± 2.35	84.07 ± 2.50	79.62 ± 2.04
CHESHIRE	84.91 ± 1.05	95.11 ± 0.94	91.62 ± 1.18	86.81 ± 1.24	82.27 ± 0.86	86.38 ± 1.23	N/A	82.75 ± 1.99
CAT-WALK	98.72 ± 1.38	95.30 ± 0.43	97.91 ± 3.30	88.15 ± 1.46	80.45 ± 5.30	96.74 ± 1.28	91.63 ± 1.41	93.51 ± 1.27
Table 2: Ablation study on CAr-WALK. AUC scores on inductive hyperedge prediction.
Methods	High School	Primary School	Users in Threads	Congress bill	Question Tags U
1	CAT-WALK	96.03 ± 1.50	95.32 ± 0.89	89.84 ± 6.02	93.54 ± 0.56	97.59 ± 2.21
2	Replace SETWALK by Random Walk	92.10 ± 2.18	51.56 ± 5.63	53.24 ± 1.73	80.27 ± 0.02	67.74 ± 2.92
3	Remove Time Encoding	95.94 ± 0.19	86.80 ± 6.33	70.58 ± 9.32	92.56 ± 0.49	96.91 ± 1.89
4	Replace SETMIXER by MEAN(.)	94.58 ± 1.22	95.14 ± 4.36	63.59 ± 5.26	91.06 ± 0.24	68.62 ± 1.25
5	Replace SETMIXER by Sum-based Universal Approximator for Sets	94.77 ± 0.67	90.86 ± 0.57	60.03 ± 1.16	91.07 ± 0.70	89.76 ± 0.45
6
7	Replace MLP-MIXER by RNN	92.85 ± 1.53	50.29 ± 4.07	58.11 ± 1.60	54.90 ± 0.50	65.18 ± 1.99
8	Replace MLP-MIXER by Transformer	55.98 ± 0.83	86.64 ± 3.55	60.65 ± 1.56	89.38 ± 1.66	56.16 ± 4.03
9	Fix α = 0	74.06 ± 14.9	58.3 ± 18.62	74.41 ± 10.69	93.31 ± 0.13	62.41 ± 4.34
module with a RNN (see Appendix G for more experiments on the significance of using MLP-Mixer in walk encoding), row 7 replaces the MLP-Mixer module with a Transformer [83], and row 8 replaces the hyperparameter  $\alpha$  with uniform sampling of hyperedges over all time periods. These results show that each component is critical for achieving CAT-WALK's superior performance. The greatest contribution comes from SETWALK, MLP-Mixer in walk encoding,  $\alpha$  in temporal hyperedge sampling, and SETMixer pooling, respectively.
Hyperparameter Sensitivity. We analyze the effect of hyperparameters used in CAr-WALK, including temporal bias coefficient  $\alpha$ , SETWALK length  $m$ , and sampling number  $M$ . The mean AUC performance on all inductive test hyperedges is reported in Figure 4. As expected, the left figure shows that
Figure 4: Hyperparameter sensitivity in  $\mathrm{CAr - W_{ALK}}$ . AUC on inductive test hyperedges are reported.
Figure 5: Scalability evaluation: The runtime of (left) SETWALK extraction and (right) the training time of CAr-WALK over one epoch on High School (using different  $|E|$  for training).
increasing the number of sampled SETWALKs produces better performance. The main reason is that the model has more extracted structural and temporal information from the network. Also, notably, we observe that only a small number of sampled SETWALKs are needed to achieve competitive performance: in the best case 1 and in the worst case 16 sampled SETWALKs per each hyperedge are needed. The middle figure reports the effect of the length of SETWALKs on performance. These results show that performance peaks at certain SETWALK lengths and the exact value varies with the dataset. That is, longer SETWALKs are required for the networks that evolve according to more complicated laws encoded in temporal higher-order motifs. The right figure shows the effect of the temporal bias coefficient  $\alpha$ . Results suggest that  $\alpha$  has a dataset-dependent optimal interval. That is, a small  $\alpha$  suggests an almost uniform sampling of interaction history, which results in poor performance when the short-term dependencies (interactions) are more important in the dataset. Also, very large  $\alpha$  might damage performance as it makes the model focus on the most recent few interactions, missing long-term patterns.
Scalability Analysis. In this part, we investigate the scalability of CAr-WALK. To this end, we use different versions of the High School dataset with different numbers of hyperedges from  $10^{4}$  to  $1.6 \times 10^{5}$ . Figure 5 (left) reports the runtimes of SETWALK sampling and Figure 5 (right) reports the runtimes of CAr-WALK for training one epoch using  $M = 8$ ,  $m = 3$  with batch-size  $= 64$ . Interestingly, our method scales linearly with the number of hyperedges, which enables it to be used on long hyperedge streams and large hypergraphs.
5 Conclusion, Limitation, and Future Work
We present CAT-WALK, an inductive hypergraph representation learning that learns both higher-order patterns and the underlying dynamic laws of temporal hypergraphs. CAT-WALK uses SETWALKs, a new temporal, higher-order random walk on hypergraphs that are provably more expressive than existing walks on hypergraphs, to extract temporal higher-order motifs from hypergraphs. CAT-WALK then uses a two-step, set-based anonymization process to establish the correlation between the extracted motifs. We further design a permutation invariant pooling strategy, SETMIXER, for aggregating nodes' positional encodings in a hyperedge to obtain hyperedge level positional encodings. Consequently, the experimental results show that CAT-WALK ① produces superior performance compared to the state-of-the-art in temporal hyperedge prediction tasks, and ② competitive performance in temporal node classification tasks. These results suggest many interesting directions for future studies: Using CAT-WALK as a positional encoder in existing anomaly detection frameworks to design an inductive anomaly detection method on hypergraphs. There are, however, a few limitations: Currently, CAT-WALK uses fixed-length SETWALKs, which might cause suboptimal performance. Developing a procedure to learn from SetWalks of varying lengths might produce better results.
We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC).
Nous remercions le Conseil de recherches en sciences naturelles et en génie du Canada (CRSNG) de son soutien.
[119] Christopher Wei Jin Goh, Cristian Bodnar, and Pietro Lio. Simplicial attention networks. In ICLR 2022 Workshop on Geometrical and Topological Representation Learning, 2022. URL https://openreview.net/forum?id=ScfRNwKpec.
[120] Mustafa Hajij, Karthikeyan Natesan Ramamurthy, Aldo Guzmán-Sáenz, and Ghada Za. High skip networks: A higher order generalization of skip connections. In ICLR 2022 Workshop on Geometrical and Topological Representation Learning, 2022.
[121] Matthias Hein, Simon Setzer, Leonardo Jost, and Syama Sundar Rangapuram. The total variation on hypergraphs - learning on hypergraphs revisited. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013. URL https://proceedings.neurips.cc/paper_files/paper/2013/file/8a3363abe792db2d8761d6403605aab7-Paper.pdf.
[122] Sameer Agarwal, Kristin Branson, and Serge Belongie. Higher order learning with graphs. In Proceedings of the 23rd International Conference on Machine Learning, ICML '06, page 17-24, New York, NY, USA, 2006. Association for Computing Machinery. ISBN 1595933832. doi: 10.1145/1143844.1143847. URL https://doi.org/10.1145/1143844.1143847.
[123] Edmund Ihler, Dorothea Wagner, and Frank Wagner. Modeling hypergraphs by graphs with the same mincut properties. Inf. Process. Lett., 45(4):171-175, mar 1993. ISSN 0020-0190. doi: 10.1016/0020-0190(93)90115-P. URL https://doi.org/10.1016/0020-0190(93)90115-P.
[124] Peihao Wang, Shenghao Yang, Yunyu Liu, Zhangyang Wang, and Pan Li. Equivariant hypergraph diffusion neural operators. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=RiTjKoscnNd.
[125] Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equations for irregular time series. Advances in Neural Information Processing Systems, 33: 6696-6707, 2020.
[126] Giang Hoang Nguyen, John Boaz Lee, Ryan A. Rossi, Nesreen K. Ahmed, Eunyee Koh, and Sungchul Kim. Continuous-time dynamic network embeddings. In *Companion Proceedings of the The Web Conference* 2018, WWW '18, page 969-976, Republic and Canton of Geneva, CHE, 2018. International World Wide Web Conferences Steering Committee. ISBN 9781450356404. doi: 10.1145/3184558.3191526. URL https://doi.org/10.1145/3184558.3191526.
[127] Weilin Cong, Si Zhang, Jian Kang, Baichuan Yuan, Hao Wu, Xin Zhou, Hanghang Tong, and Mehrdad Mahdavi. Do we really need complicated model architectures for temporal networks? In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=ayPPc@SyLv1.
A Preliminaries, Backgrounds, and Motivations 21
A.1 Anonymous Random Walks 21
A.2 Random Walk on Hypergraphs 21
A.3 MLP-Mixer 22
B Additional Related Work 22
B.1 Learning (Multi)Set Functions 22
B.2 Simplicial Complexes Representation Learning 23
B.3 How Does CAI-WALK Differ from Existing Works? (Contributions) 23
C SETWALK and Random Walk on Hypergraphs 23
C.1 Extension of SetWALKs 24
D Efficient Hyperedge Sampling 25
E Theoretical Results 26
E.1 Proof of Theorem 1 26
E.2 Proof of Theorem 2 28
E.3 Proof of Theorem 3 28
E.4 Proof of Theorem 4 29
E.5 Proof of Proposition 2 30
F Experimental Setup Details 31
F.1 Datasets 31
F.2Baselines 32
F.3 Implementation and Training Details 33
G Additional Experimental Results 34
G.1 Results on More Datasets 34
G.2 Node Classification 34
G.3 Performance in Average Precision 34
G.4 More Results on RNN v.s. MLP-Mixer in Walk Encoding 35
H Broader Impacts 36
A Preliminaries, Backgrounds, and Motivations
We begin by reviewing the preliminaries and background concepts that we refer to in the main paper. Next, we discuss the fundamental differences between our method and techniques from prior work.
A.1 Anonymous Random Walks
Micali and Zhu [45] studied Anonymous Walks (AWs), which replace a node's identity by the order of its appearance in each walk. Given a simple network, an AW starts from a node, performs random walks over the graph to collect a sequence of nodes,  $W: (u_1, u_2, \ldots, u_k)$ , and then replaces the node identities by their order of appearance in each walk. That is: