SlowGuess/NeurIPS_2023_OCR huggingface dataset:
CAT-WALK: Inductive Hypergraph Learning via Set Walks
Ali Behrouz
alibez@cs.ubc.ca
Sadaf Sadeghian†
sadafsdn@cs.ubc.ca
Farnoosh Hashemi†
farsh@cs.ubc.ca
mseltzer@cs.ubc.ca
Temporal hypergraphs provide a powerful paradigm for modeling time-dependent, higher-order interactions in complex systems. Representation learning for hypergraphs is essential for extracting patterns of the higher-order interactions that are critically important in real-world problems in social network analysis, neuroscience, finance, etc. However, existing methods are typically designed only for specific tasks or static hypergraphs. We present CAT-WALK, an inductive method that learns the underlying dynamic laws that govern the temporal and structural processes underlying a temporal hypergraph. CAT-WALK introduces a temporal, higher-order walk on hypergraphs, SETWALK, that extracts higher-order causal patterns. CAT-WALK uses a novel adaptive and permutation invariant pooling strategy, SETMIXER, along with a set-based anonymization process that hides the identity of hyperedges. Finally, we present a simple yet effective neural network model to encode hyperedges. Our evaluation on 10 hypergraph benchmark datasets shows that CAT-WALK attains outstanding performance on temporal hyperedge prediction benchmarks in both inductive and transductive settings. It also shows competitive performance with state-of-the-art methods for node classification. (Code)
Temporal networks have become increasingly popular for modeling interactions among entities in dynamic systems [1-5]. While most existing work focuses on pairwise interactions between entities, many real-world complex systems exhibit natural relationships among multiple entities [6-8]. Hypergraphs provide a natural extension to graphs by allowing an edge to connect any number of vertices, making them capable of representing higher-order structures in data. Representation learning on (temporal) hypergraphs has been recognized as an important machine learning problem and has become the cornerstone behind a wealth of high-impact applications in computer vision [9, 10], biology [11, 12], social networks [13, 14], and neuroscience [15, 16].
Many recent attempts to design representation learning methods for hypergraphs are equivalent to applying Graph Neural Networks (GNNs) to the clique-expansion (CE) of a hypergraph [17-21]. CE is a straightforward way to generalize graph algorithms to hypergraphs by replacing hyperedges with (weighted) cliques [18-20]. However, this decomposition of hyperedges limits expressiveness,
These two authors contributed equally (ordered alphabetically) and reserve the right to swap their order.
Figure 1: The advantage of SETWALKs in walk-based hypergraph learning.
Figure 2: The advantage of SETWALKs in causality extraction and capturing complex dynamic laws.
leading to suboptimal performance [6, 22-24] (see Theorem 1 and Theorem 4). New methods that encode hypergraphs directly partially address this issue [11, 25-28]. However, these methods suffer from some combination of the following three limitations: they are designed for  $①$  learning the structural properties of static hypergraphs and do not consider temporal properties,  $②$  the transductive setting, limiting their performance on unseen patterns and data, and  $③$  a specific downstream task (e.g., node classification [25], hyperedge prediction [26], or subgraph classification [27]) and cannot easily be extended to other downstream tasks, limiting their application.
Temporal motif-aware and neighborhood-aware methods have been developed to capture complex patterns in data [29-31]. However, counting temporal motifs in large networks is time-consuming and non-parallelizable, limiting the scalability of these methods. To this end, several recent studies suggest using temporal random walks to automatically retrieve such motifs [32-36]. One possible solution to capturing underlying temporal and higher-order structure is to extend the concept of a hypergraph random walk [37-43] to its temporal counterpart by letting the walker walk over time. However, existing definitions of random walks on hypergraphs offer limited expressivity and sometimes degenerate to simple walks on the CE of the hypergraph [40] (see Appendix C). There are two reasons for this: ① Random walks are composed of a sequence of pair-wise interconnected vertices, even though edges in a hypergraph connect sets of vertices. Decomposing them into sequences of simple pair-wise interactions loses the semantic meaning of the hyperedges (see Theorem 4). ② A sampling probability of a walk on a hypergraph must be different from its sampling probability on the CE of the hypergraph [37-43]. However, Chitra and Raphael [40] shows that each definition of the random walk with edge-independent sampling probability of nodes is equivalent to random walks on a weighted CE of the hypergraph. Existing studies on random walks on hypergraphs ignore ① and focus on ② to distinguish the walks on simple graphs and hypergraphs. However, as we show in Table 2, ① is equally important, if not more so.
For example, Figure 1 shows the procedure of existing walk-based machine learning methods on a temporal hypergraph. The neural networks in the model take as input only sampled walks. However, the output of the hypergraph walk [37, 38] and simple walk on the CE graph are the same. This means that the neural network cannot distinguish between pair-wise and higher-order interactions.
We present Causal Anonymous Set Walks (CAT-WALK), an inductive hyperedge learning method. We introduce a hyperedge-centric random walk on hypergraphs, called SETWALK, that automatically extracts temporal, higher-order motifs. The hyperedge-centric approach enables SETWALKs to distinguish multi-way connections from their corresponding CEs (see Figure 1, Figure 2, and Theorem 1). We use temporal hypergraph motifs that reflect network dynamics (Figure 2) to enable CAT-WALKs to work well in the inductive setting. To make the model agnostic to the hyperedge identities of these motifs, we use two-step, set-based anonymization: ① Hide node identities by assigning them new positional encodings based on the number of times that they appear at a specific position in a set of sampled SETWALKs, and ② Hide hyperedge identities by combining the positional encodings of the nodes comprising the hyperedge using a novel permutation invariant pooling strategy, called SETMIXER. We incorporate a neural encoding method that samples a few SETWALKs starting from nodes of interest. It encodes and aggregates them via MLP-MIXER [44] and our new pooling strategy
Figure 3: Schematic of the CAT-WALK. CAT-WALK consists of three stages: (1) Causality Extraction via Set Walks ( $\S 3.2$ ), (2) Set-based Anonymization ( $\S 3.3$ ), and (3) Set Walk Encoding ( $\S 3.4$ ).
SetMixer, respectively, to predict temporal, higher-order interactions. Finally, we discuss how to extend CAT-WALK for node classification. Figure 3 shows the schematic of the CAT-WALK.
We theoretically and experimentally discuss the effectiveness of CAr-WALK and each of its components. More specifically, we prove that SETWALKs are more expressive than existing random walk algorithms on hypergraphs. We demonstrate SETMIXER's efficacy as a permutation invariant pooling strategy for hypergraphs and prove that using it in our anonymization process makes that process more expressive than existing anonymization processes [33, 45, 46] when applied to the CE of the hypergraphs. To the best of our knowledge, we report the most extensive experiments in the hypergraph learning literature pertaining to unsupervised hyperedge prediction with 10 datasets and eight baselines. Results show that our method produces  $9\%$  and  $17\%$  average improvement in transductive and inductive settings, outperforming all state-of-the-art baselines in the hyperedge prediction task. Also, CAr-WALK achieves the best or on-par performance on dynamic node classification tasks. All proofs appear in the Appendix.
Temporal graph learning is an active research area [5, 47]. A major group of methods uses GNNS to learn node encodings and Recurrent Neural Networks (RNNS) to update these encodings over time [48-55]. More sophisticated methods based on anonymous temporal random walks [33, 34], line graphs [56], GraphMixer [57], neighborhood representation [58], and subgraph sketching [59] are designed to capture complex structures in vertex neighborhoods. Although these methods show promising results in a variety of tasks, they are fundamentally limited in that they are designed for pair-wise interaction among vertices and not the higher-order interactions in hypergraphs.
Representation learning on hypergraphs addresses this problem [17, 60]. We group work in this area into three overlapping categories:
① Clique and Star Expansion: CE-based methods replace hyperedges with (weighted) cliques and apply GNNS, sometimes with sophisticated propagation rules [21, 25], degree normalization, and nonlinear hyperedge weights [17-21, 39, 61, 62]. Although these methods are simple, it is well-known that CE causes undesired losses in learning performance, specifically when relationships within an incomplete subset of a hyperedge do not exist [6, 22-24]. Star expansion (SE) methods first use hypergraph star expansion and model the hypergraph as a bipartite graph, where one set of vertices represents nodes and the other represents hyperedges [25, 28, 63, 64]. Next, they apply modified heterogeneous GNNs, possibly with dual attention mechanisms from nodes to hyperedges and vice versa [25, 27]. Although this group does not cause as large a distortion as CE, they are neither memory nor computationally efficient. ② Message Passing: Most existing hypergraph learning methods, use message passing over hypergraphs [17-21, 25-27, 39, 61, 62, 65, 66]. Recently, Chien et al. [25] and Huang and Yang [28] designed universal message-passing frameworks that include propagation rules of most previous methods (e.g., [17, 19]). The main drawback of these two frameworks is that they are limited to node classification tasks and do not easily generalize to other tasks. ③ Walk-based: random walks are a common approach to extracting graph information for machine learning algorithms [32-34, 67, 68]. Several walk-based hypergraph learning methods are designed for a wide array of applications [43, 69-78]. However, most existing methods use simple random walks on the CE of the hypergraph (e.g., [26, 43, 78]). More complicated random walks on hypergraphs address this limitation [40-42, 79]. Although some of these studies show that their walk's transition matrix
differs from the simple walk on the CE [40, 79], their extracted walks can still be the same, limiting their expressivity (see Figure 1, Figure 2, and Theorem 1).
Our method differs from all prior (temporal) (hyper)graph learning methods in five ways: CAr-WALK: ① Captures temporal higher-order properties in a streaming manner: In contrast to existing methods in hyperedge prediction, our method captures temporal properties in a streaming manner, avoiding the drawbacks of snapshot-based methods. ② Works in the inductive setting by extracting underlying dynamic laws of the hypergraph, making it generalizable to unseen patterns and nodes. ③ Introduces a hyperedge-centric, temporal, higher-order walk with a new perspective on the walk sampling procedure. ④ Presents a new two-step anonymization process: Anonymization of higher-order patterns (i.e., SETWALKs), requires hiding the identity of both nodes and hyperedges. We present a new permutation-invariant pooling strategy to hide hyperedges' identity according to their vertices, making the process provably more expressive than the existing anonymization processes [33, 34, 78]. ⑤ Removes self-attention and RNs from the walk encoding procedure, avoiding their limitations. Appendices A and B provide a more comprehensive discussion of related work.
3 Method: CAT-WALK Network
Definition 1 (Temporal Hypergraphs). A temporal hypergraph  $\mathcal{G} = (\mathcal{V},\mathcal{E},\mathcal{X})$ , can be represented as a sequence of hyperedges that arrive over time, i.e.,  $\mathcal{E} = \{(e_1,t_1),(e_2,t_2),\ldots \}$ , where  $\mathcal{V}$  is the set of nodes,  $e_i\in 2^{\mathcal{V}}$  are hyperedges,  $t_i$  is the timestamp showing when  $e_i$  arrives, and  $\mathcal{X}\in \mathbb{R}^{|\mathcal{V}\times f)}$  is a matrix that encodes node attribute information for nodes in  $\mathcal{V}$ . Note that we treat each hyperedge  $e_i$  as the set of all vertices connected by  $e_i$ .
Example 1. The input of Figure 1 shows an example of a temporal hypergraph. Each hyperedge is a set of nodes that are connected by a higher-order connection. Each higher-order connection is specified by a color (e.g., green, blue, and gray), and also is associated with a timestamp (e.g.,  $t_i$ ).
Given a hypergraph  $\mathcal{G} = (\mathcal{V},\mathcal{E},\mathcal{X})$ , we represent the set of hyperedges attached to a node  $u$  before time  $t$  as  $\mathcal{E}^t (u) = \{(e,t')|t' < t,u\in e\}$ . We say two hyperedges  $e$  and  $e^{\prime}$  are adjacent if  $e\cap e^{\prime}\neq \emptyset$  and use  $\mathcal{E}^t (e) = \{(e',t')|t' < t,e'\cap e\neq \emptyset \}$  to represent the set of hyperedges adjacent to  $e$  before time  $t$ . Next, we focus on the hyperedge prediction task: Given a subset of vertices  $\mathcal{U} = \{u_1,\dots ,u_k\}$ , we want to predict whether a hyperedge among all the  $u_{i}$ s will appear in the next timestamp or not. In AppendixG.2 we discuss how this approach can be extended to node classification.
The (hyper)graph isomorphism problem is a decision problem that decides whether a pair of (hyper)graphs are isomorphic or not. Based on this concept, next, we discuss the measure we use to compare the expressive power of different methods.
Definition 2 (Expressive Power Measure). Given two methods  $\mathcal{M}_1$  and  $\mathcal{M}_2$ , we say method  $\mathcal{M}_1$  is more expressive than method  $\mathcal{M}_2$  if:
1. for any pair of (hyper)graphs  $(\mathcal{G}_1, \mathcal{G}_2)$  such that  $\mathcal{G}_1 \neq \mathcal{G}_2$ , if method  $\mathcal{M}_1$  can distinguish  $\mathcal{G}_1$  and  $\mathcal{G}_2$  then method  $\mathcal{M}_2$  can also distinguish them,
2. there is a pair of hypergraphs  $\mathcal{G}_1' \neq \mathcal{G}_2'$  such that  $\mathcal{M}_1$  can distinguish them but  $\mathcal{M}_2$  cannot.
3.2 Causality Extraction via SETWALK
The collaboration network in Figure 1 shows how prior work that models hypergraph walks as sequences of vertices fails to capture complex connections in hypergraphs. Consider the two walks:  $A \to C \to E$  and  $H \to E \to C$ . These two walks can be obtained either from a hypergraph random walk or from a simple random walk on the CE graph. Due to the symmetry of these walks with respect to  $(A, C, E)$  and  $(H, E, C)$ , they cannot distinguish  $A$  and  $H$ , although the neighborhoods of these two nodes exhibit different patterns:  $A, B$ , and  $C$  have published a paper together (connected by a hyperedge), but each pair of  $E, G$ , and  $H$  has published a paper (connected by a pairwise link). We address this limitation by defining a temporal walk on hypergraphs as a sequence of hyperedges:
Definition 3 (SETWALK). Given a temporal hypergraph  $\mathcal{G} = (\mathcal{V},\mathcal{E},\mathcal{X})$ , a SETWALK with length  $\ell$  on temporal hypergraph  $\mathcal{G}$  is a randomly generated sequence of hyperedges (sets):
$$
\mathrm {S w}: \left(e _ {1}, t _ {e _ {1}}\right)\rightarrow \left(e _ {2}, t _ {e _ {2}}\right)\rightarrow \dots \rightarrow \left(e _ {\ell}, t _ {e _ {\ell}}\right),
$$
where  $e_i \in \mathcal{E}$ ,  $t_{e_{i+1}} < t_{e_i}$ , and the intersection of  $e_i$  and  $e_{i+1}$  is not empty,  $e_i \cap e_{i+1} \neq \emptyset$ . In other words, for each  $1 \leq i \leq \ell-1$ :  $e_{i+1} \in \mathcal{E}^{t_i}(e_i)$ . We use Sw[i] to denote the  $i$ -th hyperedge-time pair in the SETWALK. That is, Sw[i][0] =  $e_i$  and Sw[i][1] =  $t_{e_i}$ .
Example 2. Figure 1 illustrates a temporal hypergraph with two sampled SETWALKs, hypergraph random walks, and simple walks. As an example,  $\{A,B,C\}, t_6 \to (\{C,D,E,F\}, t_5)$  is a SETWALK that starts from hyperedge  $e$  including  $\{A,B,C\}$  in time  $t_6$ , backtracks overtime and then samples hyperedge  $e'$  including  $\{C,D,E,F\}$  in time  $t_5 < t_6$ . While this higher-order random walk with length two provides information about all  $\{A,B,C,D,E,F\}$ , its simple hypergraph walk counterpart, i.e.  $A \to C \to E$ , provides information about only three nodes.
Next, we formally discuss the power of SETWALKs. The proof of the theorem can be found in Appendix E.1.
Theorem 1. A random SETWALK is equivalent to neither the hypergraph random walk, the random walk on the CE graph, nor the random walk on the SE graph. Also, for a finite number of samples of each, SETWALK is more expressive than existing walks.
In Figure 1, SETWALKs capture higher-order interactions and distinguish the two nodes  $A$  and  $H$ , which are indistinguishable via hypergraph random walks and graph random walks in the CE graph. We present a more detailed discussion and comparison with previous definitions of random walks on hypergraphs in Appendix C.
Causality Extraction. We introduce a sampling method to allow SETWALKs to extract temporal higher-order motifs that capture causal relationships by backtracking over time and sampling adjacent hyperedges. As discussed in previous studies [33, 34], more recent connections are usually more informative than older connections. Inspired by Wang et al. [33], we use a hyperparameter  $\alpha \geq 0$  to sample a hyperedge  $e$  with probability proportional to  $\exp\left(\alpha(t - t_p)\right)$ , where  $t$  and  $t_p$  are the timestamps of  $e$  and the previously sampled hyperedge in the SETWALK, respectively. Additionally, we want to bias sampling towards pairs of adjacent hyperedges that have a greater number of common nodes to capture higher-order motifs. However, as discussed in previous studies, the importance of each node for each hyperedge can be different [25, 27, 40, 65]. Accordingly, the transferring probability from hyperedge  $e_i$  to its adjacent hyperedge  $e_j$  depends on the importance of the nodes that they share. We address this via a temporal SETWALK sampling process with hyperedge-dependent node weights. Given a temporal hypergraph  $\mathcal{G} = (\mathcal{V}, \mathcal{E}, \mathcal{X})$ , a hyperedge-dependent node-weight function  $\Gamma: \mathcal{V} \times \mathcal{E} \to \mathbb{R}^{\geq 0}$ , and a previously sampled hyperedge  $(e_p, t_p)$ , we sample a hyperedge  $(e, t)$  with probability:
$$
\mathbb {P} [ (e, t) | (e _ {p}, t _ {p}) ] \propto \frac {\exp (\alpha (t - t _ {p}))}{\sum_ {(e ^ {\prime} , t ^ {\prime}) \in \mathcal {E} ^ {t p} (e _ {p})} \exp (\alpha (t ^ {\prime} - t _ {p}))} \times \frac {\exp (\varphi (e , e _ {p}))}{\sum_ {(e ^ {\prime} , t ^ {\prime}) \in \mathcal {E} ^ {t p} (e _ {p})} \exp (\varphi (e ^ {\prime} , e _ {p}))}, \tag {1}
$$
where  $\varphi(e, e') = \sum_{u \in e \cap e'} \Gamma(u, e) \Gamma(u, e')$ , representing the assigned weight to  $e \cap e'$ . We refer to the first and second terms as temporal bias and structural bias, respectively.
The pseudocode of our SETWALK sampling algorithm and its complexity analysis are in Appendix D. We also discuss this hyperedge-dependent sampling procedure and how it is provably more expressive than existing hypergraph random walks in Appendix C.
Given a (potential) hyperedge  $e_0 = \{u_1, u_2, \dots, u_k\}$  and a time  $t_0$ , we say a SETWALK, Sw, starts from  $u_i$  if  $u_i \in \mathrm{Sw}[1][0]$ . We use the above procedure to generate  $M$  SETWALKs with length  $m$  starting from each  $u_i \in e_0$ . We use  $S(u_i)$  to store SETWALKs that start from  $u_i$ .
3.3 Set-based Anonymization of Hyperedges
In the anonymization process, we replace hyperedge identities with position encodings, capturing structural information while maintaining the inductiveness of the method. Micali and Zhu [45] studied Anonymous Walks (AWs), which replace a node's identity by the order of its appearance in each walk. The main limitation of AWs is that the position encoding of each node depends only on its specific walk, missing the dependency and correlation of different sampled walks [33]. To mitigate this drawback, Wang et al. [33] suggest replacing node identities with the hitting counts of the nodes based on a set of sampled walks. In addition to the fact that this method is designed for walks on simple graphs, there are two main challenges to adopting it for SETWALKs: ① SETWALKs
are a sequence of hyperedges, so we need an encoding for the position of hyperedges. Natural attempts to replace hyperedges' identity with the hitting counts of the hyperedges based on a set of sampled SetWALKs, misses the similarity of hyperedges with many of the same nodes. ② Each hyperedge is a set of vertices and natural attempts to encode its nodes' positions and aggregate them to obtain a position encoding of the hyperedge requires a permutation invariant pooling strategy. This pooling strategy also requires consideration of the higher-order dependencies between obtained nodes' position encodings to take advantage of higher-order interactions (see Theorem 2). To address these challenges we present a set-based anonymization process for SetWALKs. Given a hyperedge  $e_0 = \{u_1,\dots ,u_k\}$ , let  $w_{0}$  be any node in  $e_0$ . For each node  $w$  that appears on at least one SetWALK in  $\bigcup_{i = 1}^{k}S(u_{i})$ , we assign a relative, node-dependent node identity,  $\mathcal{R}(w,S(w_0))\in \mathbb{Z}^m$ , as follows:
$$
\mathcal {R} (w, \mathcal {S} (w _ {0})) [ i ] = | \{\mathrm {S w} | \mathrm {S w} \in \mathcal {S} (w _ {0}), w \in \mathrm {S w} [ i ] [ 0 ] \} | \quad \forall i \in \{1, 2, \dots , m \}. \tag {2}
$$
For each node  $w$  we further define  $\mathrm{Id}(w, e_0) = \{\mathcal{R}(w, S(u_i))\}_{i=1}^k$ . Let  $\Psi(\cdot): \mathbb{R}^{d \times d_1} \to \mathbb{R}^{d_2}$  be a pooling function that gets a set of  $d_1$ -dimensional vectors and aggregates them to a  $d_2$ -dimensional vector. Given two instances of this pooling function,  $\Psi_1(\cdot)$  and  $\Psi_2(\cdot)$ , for each hyperedge  $e = \{w_1, w_2, \ldots, w_{k'}\}$  that appears on at least one SETWALK in  $\bigcup_{i=1}^k S(u_i)$ , we assign a relative hyperedge identity as:
$$
\operatorname {I D} (e, e _ {0}) = \Psi_ {1} \left(\left\{\Psi_ {2} \left(\operatorname {I D} \left(w _ {i}, e _ {0}\right)\right) \right\} _ {i = 1} ^ {k ^ {\prime}}\right). \tag {3}
$$
That is, for each node  $w_{i} \in e$  we first aggregate its relative node-dependent identities (i.e.,  $\mathcal{R}(w_i, S(u_j))$ ) to obtain the relative hyperedge-dependent identity. Then we aggregate these hyperedge-dependent identities for all nodes in  $e$ . Since the size of hyperedges can vary, we zero-pad to a fixed length. Note that this zero padding is important to capture the size of the hyperedge. The hyperedge with more zero-padded dimensions has fewer nodes.
This process addresses the first challenge and encodes the position of hyperedges. Unfortunately, many simple and known pooling strategies (e.g.,  $\mathrm{SUM}(.)$ ,  $\mathrm{ATTN-SUM}(.)$ ,  $\mathrm{MEAN}(.)$ , etc.) can cause missing information when applied to hypergraphs. We formalize this in the following theorem:
Theorem 2. Given an arbitrary positive integer  $k \in \mathbb{Z}^+$ , let  $\Psi(\cdot)$  be a pooling function such that for any set  $S = \{w_1, \ldots, w_d\}$ :
$$
\Psi (S) = \sum_ {\substack {S ^ {\prime} \subseteq S \\ | S ^ {\prime} | = k}} f \left(S ^ {\prime}\right), \tag{4}
$$
where  $f$  is some function. Then the pooling function can cause missing information, meaning that it limits the expressiveness of the method to applying to the projected graph of the hypergraph.