SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yizhou Chen*
Shopee Pte Ltd., Singapore
Anxiang Zeng*
SCSE, Nanyang Technological University, Singapore
Guangda Huzhang*
Qingtao Yu
Kerui Zhang
Yuanpeng Cao
Kangle Wu
Zhiming Zhou
Temporal graphs offer more accurate modeling of many real-world scenarios than static graphs. However, neighbor aggregation, a critical building block of graph networks, for temporal graphs, is currently straightforwardly extended from that of static graphs. It can be computationally expensive when involving all historical neighbors during such aggregation. In practice, typically only a subset of the most recent neighbors are involved. However, such subsampling leads to incomplete and biased neighbor information. To address this limitation, we propose a novel framework for temporal neighbor aggregation that uses the recurrent neural network with node-wise hidden states to integrate information from all historical neighbors for each node to acquire the complete neighbor information. We demonstrate the superior theoretical expressiveness of the proposed framework as well as its state-of-the-art performance in real-world applications. Notably, it achieves a significant  $+9.6\%$  improvement on averaged precision in a real-world Ecommerce dataset over existing methods on 2-layer models.
Many real-world applications (such as social networks [28, 41], recommender systems [3, 29], traffic forecasting [18, 45] and crime analysis [11]) involve dynamically changing entities and increasing interactions, which can be naturally modeled as temporal graphs (also known as dynamic graphs) where nodes and edges appear and evolve over time. Compared with static graph solutions [13, 35, 9], dynamic graphs can be more accurate in modeling node states and inferring future events.
In modern graph representation learning, neighbor aggregation, i.e., generating higher-level node embeddings via referring to the embeddings of their neighbors [13, 35, 17], is a critical common ingredient that helps more effectively leverage the connectivity information in the graphs. Practically, however, relatively hot nodes may have rapidly growing lists of neighbors, which makes involving all neighbors in neighbor aggregation computationally expensive. In static graphs, SAGE [8] suggests randomly sampling  $d$  neighbors to approximate all neighbors. For temporal graphs, the prevalent approach is to consider only the most recent  $d$  neighbors [28, 33]. But such subsampling of neighbors
* The three authors contributed equally to the paper.
† Corresponding author. zhouzhiming@mail.shufe.edu.cn.
could lead to incomplete and biased information, as the neighbor information preceding the most recent  $d$  neighbors is permanently lost.
To mitigate the problem caused by neighbor subsampling, we propose an alternative solution to classic neighbor aggregation for temporal graphs, called recurrent temporal revision (RTR), which can be used as a standard building block (a network layer) for any temporal graph network. Each RTR layer incorporates a hidden state for each node to integrate the information from all its neighbors using a recurrent neural network (RNN). The hidden state will serve as the embedding of each node at that layer, embracing the complete neighbor information and having the flexibility to focus more on long-term or short-term behaviors depending on the characteristics of the problem / dataset.
The existence of such a hidden state implies that all historical neighbors have already been integrated through previous updates. So when a new edge event comes, we may build a message with the latest neighbor to update the hidden state through the RNN. On the other hand, taking the state updates of previous neighbors into account can also be beneficial, which could provide additional information including new interactions of these neighbors [28] as well as updates in the neighbor states due to the influence of their own neighbors. In other words, re-inputting neighbors would enable revising their previously provided information. Therefore, we propose a module called temporal revision (TR) to collect state updates of previous neighbors to update the hidden state together with its latest interaction. To better capture the state updates, besides their current states, we additionally input the revisions and state changes of neighbors into the temporal revision module. Subsampling of such state updates of neighbors can be less problematic, since the state of each neighbor has already been observed and integrated by the hidden state in the past.
Towards further enhancement, we propose to introduce heterogeneity in temporal revision, i.e., treating involved nodes non-uniformly, to boost its theoretical expressiveness. Heterogeneity can be naturally incorporated, since when a node collects revisions from its neighbors, the neighbors may, the other way around, collect the revision from the node (at a lower level), which is, however, directly accessible to the node itself. We thus introduce heterogeneity by recursively identifying and marking such self-probing nodes as specialties in revision calculation. We theoretically show that such heterogeneous revision leads to improved expressiveness beyond Temporal-1WL [33], and practically verify its potential to promote effective information exchange among nodes.
- We propose a novel neighbor aggregation framework for temporal graphs to tackle the challenges arising from the necessary subsampling of neighbors.
- We propose two expressiveness measures for temporal graph models in terms of temporal link prediction and temporal graph isomorphism test, and characterize their theoretical connection. We prove theoretically the superior expressiveness of our proposed framework, as well as provide several new results on the expressiveness of other baselines.
- We demonstrate in experiments that the proposed framework can significantly outperform state-of-the-art methods. In particular, we can consistently get improvements as the number of aggregation layers increases, while previous methods typically exhibit only marginal improvements [28, 38].
- An Ecommerce dataset is provided along with this submission, which can be valuable for evaluating temporal graph models in real-world settings.
2.1 Temporal Graph Modeling
A temporal graph can be represented as an ordered list of timed events, which may include the addition, deletion, and update of nodes and edges. For simplicity, we will only discuss the handling of edge addition events in this paper, where newly involved nodes will be accordingly added and initialized. It can be easily extended to handle other events [28]. We denote the set of nodes and edges up to time  $t$  as  $V(t)$  and  $E(t)$ , respectively.
Modeling of temporal graph typically generates node embeddings that evolve over time as graph events occur, which can be used by downstream tasks such as link prediction [2] and node classification [20]. The base node embeddings can have various sources, including direct node
features [8, 35, 41], learnable parameters [9, 36], generated by other modules such as an RNN [28, 38, 33] or specifically designed module [3, 16, 22]). Modern temporal graph methods [28, 5, 33] typically employ additional layers on top of the base embeddings to (further) leverage the connectivity information within the graph, generating higher-level node embeddings with a wider receptive field that encompasses relevant edges and nodes.
2.2 (Temporal) Neighbor Aggregation
Among various approaches, neighbor aggregation is one of the commonly used techniques in various graph models as a critical ingredient, where the embedding of each node is updated by considering the embeddings of its neighbors. In the context of temporal graphs, the neighbor aggregation layer can be defined as follows [28, 38, 33]:
$$
\mathbf {a} _ {u} ^ {k} (t) = \mathsf {A G G} ^ {k} \left(\left\{\left(\mathbf {h} _ {v} ^ {k - 1} (t), \Phi \left(e _ {u v, t ^ {\prime}}\right)\right) \mid e _ {u v, t ^ {\prime}} \in E (t) \right\}\right), \tag {1}
$$
$$
\mathbf {h} _ {u} ^ {k} (t) = \operatorname {C O M B I N E} ^ {k} \left(\mathbf {h} _ {u} ^ {k - 1} (t), \mathbf {a} _ {u} ^ {k} (t)\right), \tag {2}
$$
where  $t$  denotes the current time.  $e_{uv,t'}$  denotes the event between node  $u$  and node  $v$  at time  $t' (\leq t)$ .  $\Phi(e_{uv,t'})$  encodes the event feature, as well as its related time and sequence information, which may include the current time, the event time, the current number of interactions of  $u$  and  $v$ , etc.  $\mathsf{AGG}^k$  and  $\mathsf{COMBINE}^k$  denote arbitrary learnable functions, which are typically shared at layer  $k$ .  $\mathbf{a}_u^k(t)$  is the aggregated information from neighbors at the  $k-1$  layer, while  $\mathbf{h}_u^k(t)$  is the new embedding of node  $u$  at layer  $k$ , combining its current embedding at the  $k-1$  layer with the aggregated information  $\mathbf{a}_u^k(t)$ . This can be viewed as a straightforward extension of classic graph convolution layers [13, 35, 17] with additional time or sequence information.
2.3 Expressiveness of Temporal Graph Models
Expressiveness measure is crucial for identifying and comparing the capabilities of different models. While several studies have explored the expressiveness of (temporal) graph models [40, 5, 33, 39, 43], a widely accepted definition for the expressiveness of temporal graph models is currently lacking.
Towards addressing this gap and establishing a solid groundwork, we propose the following formal definition of expressiveness in terms of temporal graph isomorphism test and temporal link prediction:
Definition 1. (Temporal graph isomorphism indistinguishability). Given a pair of temporal graphs  $\langle G_1(t), G_2(t) \rangle$ , we say they are indistinguishable w.r.t. a model  $f$ , if and only if there exists a bijective mapping between nodes of  $G_1(t)$  and  $G_2(t)$ , such that, for each pair of nodes in the mapping, their embeddings generated by  $f$  are identical at any time  $t' \leq t$ .
Definition 2. (Temporal link prediction indistinguishability). Given a temporal graph  $G(t)$  and two pairs of nodes  $\langle (u_1, v_1), (u_2, v_2) \rangle$  therein representing two temporal link prediction problems, we say they are indistinguishable w.r.t. a model  $f$ , if and only if the embedding of  $u_1$  and  $u_2$ ,  $v_1$  and  $v_2$  generated by  $f$  are identical at any  $t' \leq t$ .
Definition 3. (Temporal graph isomorphism expressiveness). If a model  $f_{A}$  can distinguish any pair of temporal graphs that model  $f_{B}$  can distinguish, while being able to distinguish certain graph pairs that  $f_{B}$  fails, we say  $f_{A}$  is strictly more expressive than  $f_{B}$  in temporal graph isomorphism test.
Definition 4. (Temporal link prediction expressiveness). If a model  $f_{A}$  can distinguish any pair of temporal link prediction problems that model  $f_{B}$  can distinguish, while being able to distinguish certain pair of temporal link prediction problems that  $f_{B}$  fails, we say  $f_{A}$  is strictly more expressive than  $f_{B}$  in temporal link prediction.
We can similarly define equally expressive, more or equally expressive, and so on.
Through above, we have, for the first time, provided a consistent definition of expressiveness for both temporal graph isomorphism test and temporal link prediction. Besides, compared with existing definitions, ours is more concise and straightforward, avoiding the introduction of concepts such as the Identifiable Set as in [5] and Temporal Computation Tree as in [33].
Most existing results on the expressiveness of temporal graph models can be easily transferred to our definition. For example, message-passing temporal graph networks (MP-TGN) are strictly
(b)
Figure 1: (a) A comparison of our framework and other classic aggregations. Classic temporal aggregation typically involves subsampled neighbors in each step, which leads to incomplete and biased neighbor information, while our proposed RTR layer integrates the whole information of neighbors into node-wise hidden states. (b) An illustration of the inner structure of an RTR layer and our RTRGN, which builds  $k$  successive RTR layers on top of the base embedding. We provide the pseudocode for RTRGN embedding calculation in Appendix A.
more expressive in isomorphism test when the (temporal) neighbor aggregation layers are injective (referred to as IMP-TGN) [33]; and IMP-TGN would be as expressive as temporal 1-Weisfeiler-Lehman (Temporal-1WL) test [33, 39], if with sufficient many layers.
3.1 Recurrent Temporal Revision
In large graphs, having too many neighbors can pose practical computation challenges in neighbor aggregation. For static graphs, SAGE [8] suggests approximating neighbor information by randomly subsampling a fixed number  $(d)$  of neighbors. For temporal graphs, recent models [28, 33] typically involve the most recent  $d$  neighbors to reduce the computation cost of temporal neighbor aggregation, which often yields better performance compared with uniform sampling [28]. But such subsampling of neighbors fails to provide complete neighbor information and biases it towards short-term behaviors. The effects are illustrated in Figure I(a).
Towards addressing this problem, we propose to introduce a hidden state  $\mathbf{h}_u^k (t)$  in each aggregation layer  $k > 0$  for each node to integrate information from all its historical neighbors under a state transition model, typically recurrent neural network (RNN). Since events in the temporal graph, and hence the addition of neighbors, occur sequentially, such integration can be done naturally.
Formally, when an event  $e_{uv,t}$  occurs, we will need to update the hidden state  $\mathbf{h}_u^k$  of  $u$  (and also  $v$ ). A straightforward implementation would be updating the hidden state with a message of the new event. However, taking the state updates of previous neighbors into account can also be beneficial, which can provide additional information including: the new interactions of neighbors, their state evolutionary directions, updates in the neighbor states due to the influence of their own neighbors. To achieve this, we introduce a module called temporal revision (TR) to collect state updates of previous neighbors, which will be later fused with the latest event to update the hidden state:
$$
\mathbf {r} _ {u} ^ {k} (t) = \mathsf {T R} ^ {k} \left(\left\{\left(\mathbf {r} _ {v} ^ {k - 1} (t), \Delta \mathbf {h} _ {v} ^ {k - 1} (t), \mathbf {h} _ {v} ^ {k - 1} (t), \Phi \left(e _ {u v, t ^ {\prime}}\right)\right) \mid e _ {u v, t ^ {\prime}} \in E \left(t ^ {-}\right) \right\}\right), \tag {3}
$$
where  $\mathsf{TR}^k$  is an arbitrary learnable function shared at layer  $k$ ,  $t^{-}$  represents the time right before  $t$ . We collect not only the newest states of neighbors  $\mathbf{h}_v^{k-1}(t)$  at the  $k-1$  layer, but also their state changes  $\Delta \mathbf{h}_v^{k-1}(t) = \mathbf{h}_v^{k-1}(t) - \mathbf{h}_v^{k-1}(t^{-})$  and their revisions  $\mathbf{r}_v^{k-1}(t)$ , as well as the event information  $\Phi(e_{uv,t'})$ , to help the model better extract the state update information. We set  $\mathbf{r}_u^0(t)$  as zero vectors. When requesting the embedding of a node at the 0-th layer, we refer to its base embedding. Please refer to Appendix A for more detailed discussions of the base embedding.
In subsequent ablation studies, we show that retaining  $\Delta \mathbf{h}_v^{k - 1}(t)$  alone, while removing  $\mathbf{h}_v^{k - 1}(t)$ , will not lead to degeneration of performance, which means the effectiveness in aggregation can rely
solely on the changes of states, while not necessarily on the neighbor states themselves. Temporal revision is hence substantially different from classic temporal aggregations [38, 28, 5, 33].
To encode all neighbor update information, we build a state revising message (RMSG) via:
$$
\mathbf {m} _ {u} ^ {k} (t) = \mathsf {R M S G} ^ {k} \left(\mathbf {h} _ {u} ^ {k - 1} (t), \mathbf {r} _ {u} ^ {k} (t), \mathbf {h} _ {v} ^ {k - 1} (t), \Phi \left(e _ {u v, t}\right)\right), \tag {4}
$$
where  $\mathsf{RMSG}^k$  is an arbitrary learnable function shared at layer  $k$ , integrating the node embedding at the  $k-1$  layer  $\mathbf{h}_u^{k-1}(t)$ , the embedding of the new neighbor at the  $k-1$  layer  $\mathbf{h}_v^{k-1}(t)$ , the collected neighbor state update information  $\mathbf{r}_u^k(t)$ , as well as the event information  $\Phi(e_{uv,t})$ .
Then the proposed recurrent temporal revision (RTR) layer can be formulated as:
$$
\mathbf {h} _ {u} ^ {k} (t) = \mathsf {U P D A T E} ^ {k} \left(\mathbf {h} _ {u} ^ {k} \left(t ^ {-}\right), \mathbf {m} _ {u} ^ {k} (t)\right), \tag {5}
$$
where the state revising message is consumed to update the hidden state  $\mathbf{h}_u^k$  from time  $t^{-}$  to  $t$  via UPDATE $^k$ , a learnable function shared at layer  $k$  responsible for hidden state transition. For all RTR layers, we initialize  $\mathbf{h}_u^k(0)$  as zero vectors for all nodes.
3.2 Heterogeneous Revision
Heterogeneous message passing can improve the expressiveness of static GNN [43], which can be naturally integrated into our revision computation.
The term  $\mathbf{r}_u^k (t)$  recursively gathers revision from previously interacted nodes, and they may in turn gather the revision of node  $u$  (at a lower level), which is, however, directly accessible to  $u$  itself. Therefore, we may adopt a different treatment for such self-probing calculations during recursive aggregation. To achieve this, we may mark all nodes in the recursive computation path as specialties.
Formally, we define the heterogeneous revision as:
$$
\mathbf {r} _ {u} ^ {k} (t, \mathcal {S}) = \mathsf {T R} ^ {k} \left(\left\{\left(\mathbf {r} _ {v} ^ {k - 1} (t, \mathcal {S} \cup \{u \}), \Delta \mathbf {h} _ {v} ^ {k - 1} (t), \mathbf {h} _ {v} ^ {k - 1} (t), \Phi \left(e _ {u v, t ^ {\prime}}\right), \mathbb {1} [ v \in \mathcal {S} ]\right) \mid e _ {u v, t ^ {\prime}} \in E (t ^ {-}) \right\}\right), \tag {6}
$$
where we introduce an extra argument  $S$  for revision, which is a set that is initialized as empty  $\varnothing$  and recursively adds each  $u$  encountered along the computation path.  $\mathbb{1}$  stands for the indicator function, which triggers the difference between nodes in  $S$  and the rest, bringing about heterogeneity in the revisions. To switch to heterogeneous revision, we replace  $\mathbf{r}_u^k (t)$  in 4 with  $\mathbf{r}_u^k (t,\emptyset)$ . We set the base case  $\mathbf{r}_u^0 (t,S)$  as zero vectors for any  $u$  and  $S$ .
We show in the next section that incorporating such heterogeneous revision theoretically leads to enhanced expressiveness, which will be subsequently validated in our later experiments.
We illustrate the complete inner structure of an RTR layer in Figure II(b). The implementation details, including the design of all these learnable functions, are provided in Appendix A.
3.3 Model Expressiveness
For model expressiveness, we assume our RTR layers are built on top of a base RNN. For a fair comparison, it is assumed the base RNNs for all models are the same and of the best expressiveness, as suggested in [5, 33]. We focus on RTR layers whose (learnable) functions in (4) - (6) are injective, which can be achieved by adopting similar approaches as in [33, 39]. We refer to such a model as RTRGN. Besides, we assume node and edge attributes come from a finite set as [25, 5, 33].
We will mainly compare RTRGN with PINT [33], which is an instance of IMP-TGN that can be equally expressive as Temporal 1-WL in isomorphism test. PINT also offers augmented positional features (referred to as PINT-pos) designed to enhance its expressiveness.
Our first theoretical result demonstrates that the temporal graph isomorphism test is inherently more challenging than temporal link prediction. All proofs are deferred to Appendix B
Proposition 1. Being more expressive in temporal graph isomorphism test implies being more expressive in temporal link prediction  $(f_{A} > _{is}f_{B}\Rightarrow f_{A} > _{lp}f_{B})$  but not vice versa  $(f_{A} > _{lp}f_{B}\nRightarrow f_{A} > _{is}f_{B})$ .
3When there is no event at time  $t$  for  $u$  , but requesting the message, we set  $\mathbf{h}_v^{k - 1}(t) = \mathbf{0},\Phi (e_{uv,t}) = \mathbf{0}$  in 4.
Table 1: Summary of our theoretical results. Temporal graph isomorphism test is inherently more challenging than temporal link prediction. While Time-then-IMP and PINT are equally expressive as Temporal-1WL, our proposed RTRGN is strictly more expressive than the Temporal-1WL in both tasks. Relevant existing results are represented in gray for better reference.
Expressiveness (Link Prediction)	Expressiveness (Isomorphism Test)
fA >is fB ⇒ fA >lp fB (Prop.1)	fA ≈is fB ⇒ fA ≈lp fB (Corollary 1)
PINT-pos >lp PINT >lp MP-TGNs [33]	PINT ≈is Temporal-1WL [33]
Time-then-IMP ≈lp PINT ≈lp Temporal-1WL (Prop.2)	Time-then-IMP ≈is PINT ≈is Temporal-1WL (Prop.2)
RTRGN >lp PINT (Prop.3)	RTRGN >is PINT (Prop.3)
PINT-pos can be false positive (Prop.4)	PINT-pos can be false positive (Prop.4).
Most expressive: RNN + Temporal Relational Pooling (Prop.5 in Appendix B)
The first part of this proposition can be proved by reducing the temporal link prediction problem into a temporal graph isomorphism test of the current graph adding the two temporal links separately. The second part of this proposition is proved by constructing a pair of two models: one is more expressive than the other in temporal link prediction, while being not more expressive in temporal graph isomorphism test. The proof and the construction of the counter-examples inside suggest that temporal graph isomorphism test is a more challenging task than temporal link prediction, as it requires global distinguishability rather than just local distinguishability of the involved nodes.
Next, we demonstrate that Time-then-IMP, the strongest among the three classes of temporal graph models suggested in [5] with the graph module of its time-then-graph scheme being injective message passing (IMP) to ensure 1WL expressiveness [39], is equally expressive as PINT and Temporal-1WL in both temporal graph isomorphism test and temporal link prediction.
Proposition 2. Time-then-IMP is equally expressive as PINT and Temporal-1WL in both temporal graph isomorphism test and temporal link prediction.
The proof establishes that all these models are equally expressive as Temporal-1WL in both temporal link prediction and temporal graph isomorphism test. The proposition suggests that Time-then-IMP, although utilizes a distinct form of base RNN to encode all historical features of nodes and edges (which comes at the cost of significantly more space needed to embed each edge), its expressiveness is ultimately equivalent to the Temporal-1WL class, which includes models like PINT.
We next show that RTRGN is more expressive than PINT, as well as the Temporal-1WL class:
Proposition 3. RTRGN is strictly more expressive than the Temporal-1WL class represented by PINT in both temporal graph isomorphism test and temporal link prediction.
As we have shown, the expressiveness of all the previously mentioned models is bounded by the Temporal-1WL, which, although powerful, can fail in certain cases. To illustrate this, we construct a temporal graph isomorphism test task called Oscillating CSL (Figure 2) where all Temporal-1WL models, including Time-then-IMP, fail. The difficulty of the task stems from the regular structure of the graph snapshots and stacked graphs at each timestamp, which cannot be distinguished by Temporal-1WL tests. RTRGN excels in this task due to the heterogeneity introduced in its revision computation, which enables its expressiveness to go beyond the limit of Temporal-1WL.
Proposition 4. PINT-pos can be false positive in both temporal graph isomorphism test and temporal link prediction, e.g., may incorrectly classify isomorphic temporal graphs as non-isomorphic.