TGB-Seq Benchmark: Challenging Temporal GNNs with Complex Sequential Dynamics

Authors: Lu Yi, Jie Peng, Yanping Zheng, Fengran Mo, Zhewei Wei, Yuhang Ye, Yue Zixuan, Zengfeng Huang
Primary area: datasets and benchmarks
Keywords: datasets and benchmarks, temporal graph learning

Abstract:
Future link prediction is a fundamental challenge in various real-world dynamic systems. To address this, numerous temporal graph neural networks (temporal GNNs) and benchmark datasets have been developed. However, these datasets often feature excessive repeated edges and lack complex sequential dynamics, a key characteristic inherent in many real-world applications such as recommender systems and "Who-To-Follow" on social networks. This oversight has led existing methods to inadvertently downplay the importance of learning sequential dynamics, focusing primarily on predicting repeated edges.

In this study, we demonstrate that existing methods, such as GraphMixer and DyGFormer, are inherently incapable of learning simple sequential dynamics, such as "a user who has followed OpenAI and Anthropic is more likely to follow AI at Meta next." Motivated by this issue, we introduce the Temporal Graph Benchmark with Sequential Dynamics (TGB-Seq), a new benchmark carefully curated to minimize repeated edges, challenging models to learn sequential dynamics and generalize to unseen edges. TGB-Seq comprises large real-world datasets spanning diverse domains, including e-commerce interactions, movie ratings, business reviews, social networks, citation networks and web link networks. Benchmarking experiments reveal that current methods usually suffer significant performance degradation and incur substantial training costs on TGB-Seq, posing new challenges and opportunities for future research. TGB-Seq datasets, leaderboards, and example codes are available at https://tgb-seq.github.io/.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1729420676388

thread_type:
official_review

<thread_object>
creator: reviewer_umn2

<content>
summary:
The paper begins with an experimental investigation exploring the comparison of recent temporal GNN methods on the recommendation datasets, where the results are surprisingly lower than the classical methods in the recommendation. The paper then investigates the reason for the datasets and proposes a more comprehensive benchmark to compare the existing methods more effectively. This paper presents an important perspective, often overlooked by many in the field, to reevaluate whether the current approach to future link prediction tasks is reasonable. I believe this work is valuable for encouraging a rethinking within the field and can inspire future research efforts.

strengths:
1. This paper introduces a significant perspective that many in the field have often overlooked. It prompts a reevaluation of whether the current methods used for future link prediction tasks are truly appropriate.
2. This paper conducts insightful data analysis on existing datasets to examine the current evaluation standards.

weaknesses:
1. In the illustration of Figure 3, the node features are omitted. However, this omission is not appropriate given that incorporating node features is one of the key advantages of Graph Neural Networks (GNNs) compared to ID-based recommendation models. The illustration, without considering node features, fails to adequately demonstrate the full capability of GNN models, particularly in highlighting their ability to leverage rich feature information.
2. Recently, some efforts not mentioned in this submission have been made to benchmark and evaluate existing work in continuous time domains, such as in [1]. Although this paper has a different focus, the range of GNN methods that can be evaluated is more extensive.

[1] Chen C, Geng H, Yang N, et al. Easydgl: Encode, train and interpret for continuous-time dynamic graph learning[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.

questions:
See weakness for more details.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730491578673

thread_type:
official_review

<thread_object>
creator: reviewer_5t8y

<content>
summary:
In this work, the authors pointed out that existing methods are lacking in the learning of sequential dynamics while focusing primarily on predicting repeated edges. To this end, the authors first demonstrated that methods such as GraphMixer and DyGformer are unable to learn sequential dynamics in a toy dataset and then introduce a new benchmark, TGB-Seq, that contains a minimal amount of repeated edges. TGB-Seq contains large real-world datasets that spans diverse domains as well as both bipartite and non-bipartite networks.

strengths:
This paper focuses on examining how well current models can learn sequential dynamics (predict unseen edges). I believe the limitations of existing methods pointed out in this work is quite interesting. Here are the strengths of the paper:

- the idea of a benchmark containing datasets with very low amounts of repeated edges are interesting while opening up new research directions that design novel methods for such datasets. This is significant for the temporal graph learning community. 

- The paper is well-written and clearly presented. 

- The new datasets spans a diverse range of domains and contains both bipartite and non-bipartite networks. The authors also provided code and dataset access.

weaknesses:
Here are some weakness of the paper:

- **new but familiar idea**. The idea of measuring unseen edges and their effect in model performance is not entirely new. In the TGB work, the authors also mentioned the surprise index which measures the amount of test edges unseen during training (similar to the unseen edges idea in the paper though different).  The authors should further clarify any definition difference between the "unseen" edges as shown In Figure 2 versus the novel edges $E_{test} \ E_{train}$ as mentioned in the surprise index. 

- **lack of node, edge features for the datasets**. The authors mentioned that this work excludes datasets with node and edge features. Were there datasets that you considered that has node or edge features? Is it possible to include some features for the datasets in TGB-seq benchmark? Because node / edge features might be crucial for the model to learn sequential dynamics. For example, one user might enjoy coffee and this user will choose to purchase a new type of coffee beans (a categorical feature of the item node)) at a future time. With node features on the items, the model can learn this dynamics easier while it might be difficult to learn if no feature is provided. 


- **OOT errors are not intuitive**. In terms of running time, memory based models such as TGN and DyRep are usually more efficient than DyGFormer and CAWN (such as compared in the TGB work). It is not very intuitive to me why these models can not finish a single epoch in 24 hours as they are able to scale to the largest datasets on TGB. More clarification / explanation from the authors are appreciated. 

Considering that this work presents challenging new datasets, I am giving a positive score. However, I would like to know the answer to the concerns above.

questions:
See weakness for the questions as well.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730710894812

thread_type:
official_review

<thread_object>
creator: reviewer_gu3d

<content>
summary:
TGB-Seq presents a novel benchmark for the temporal link prediction task by emphasizing that the key characteristic of temporal link prediction is that the model should learn how to rank the most likely destination node for a given source node at the queried time, which aligns well with practical, real-world settings, such as recommender systems.

strengths:
1. Incorporates diverse datasets from different domains and different sizes (number of nodes/edges). 
2. Comprehensive report for experimental settings (hyperparameters search range, computational resources, etc.)  and assessment of SOTA Temporal GNNs baselines.
3. Clear presentation, and the paper is easy to read.

weaknesses:
1. For validation and test set, TGB-Seq only retains nodes that appear in the training set. I do not think this setting is practical for real-world temporal graphs, as they continuously evolve, and this setting is not comprehensive enough to assess the model’s ability to reason on new information that emerges in future timestamps. 

2. Previous work [1] (Poursafaei et al., 2022) also devised 2 other negative sampling strategies (NSS): (1) random NSS, which samples any possible node pairs, and (2) inductive NSS that samples edges that are unseen during the training stage. Therefore, previous works' inductive NSS also do not rely on historical edges, and random NSS samples are also from all possible nodes. How does TGB-Seq’s NSS differ from [1]’s inductive or random NSS?

3. TGB-Seq claims that the proposed evaluations of SOTA models show that achieving both efficiency and effectiveness in temporal GNNs remains an open problem, highlighting the distinctive feature of TGB-Seq. However, TGB [2] also presents large temporal graphs that could challenge any model’s efficiency, and some of their datasets have high surprise scores, which also challenge the performance of SOTA models that are listed in TGB-Seq. Compared to TGB [2], how distinctive is TGB-Seq’s capabilities in challenging Temporal GNNs?

Reference:

[1] Towards better evaluation for dynamic link prediction. 

[2] Temporal Graph Benchmark for Machine Learning on Temporal Graphs

questions:
Please refer to Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730790439145

thread_type:
official_review

<thread_object>
creator: reviewer_wmvb

<content>
summary:
This paper introduces TGB-Seq, a new benchmark designed to challenge temporal GNNs with complex sequential dynamics. Existing datasets often have excessive repeated edges, which fail to capture the sequential patterns present in many real-world applications like recommender systems and social networks. TGB-Seq minimizes repeated edges and includes diverse domains such as e-commerce, movie ratings, and social networks. The study reveals that current temporal GNN methods struggle with these datasets, highlighting the need for models that can better generalize to unseen edges. The paper contributes by providing new datasets that emphasize sequential dynamics and exposing the limitations of existing GNN approaches.

strengths:
1. This article highlights the issue of excessive repeated edges in existing dynamic graph datasets. The experiments presented in the text are very complete and convincing, and the presentation in Figure 2 is impressive. The motivation behind the construction of TGB-Seq is very sufficient, the problems it addresses are very clear, and it has significant practical implications.
2. The structure of this article is clear, and the expression is explicit. Overall, it is easy to follow.

weaknesses:
1. Section 3.2 is far from satisfactory. The conclusion of Section 3.2 is very interesting and quite important, but the entire section lacks formal expression, making the analysis of the limitations of existing dynamic graph models in this part very perfunctory.
2. The assumptions and premises in Section 3.2 are not clear. There is a simple description of premises starting from line 212 in the text, but it is very vague. Are the node features of the three types of nodes u, v, and i all blank? If so, this assumption lacks rationality. The subsequent discussions on the model's memory and aggregation parts are all based on this. If all nodes have the same features (all nodes have no features), it will inevitably lead to the indistinguishability of many nodes, which is similar to the discussion of graph isomorphism problems. In such an extreme case, the limitations of dynamic graph models are difficult to be widely recognized.
3. Continuing from the previous point, the authors should discuss why SGNN-HN can achieve such good results under the toy example and what problems it overcomes. Of course, these contents should be formally expressed in the form of expressions. The existing theoretical discussion in the text is difficult to understand.

questions:
Please see my comments for details
</content>
</thread_object>

</thread>
<thread>
creation date:
1730823495939

thread_type:
official_review

<thread_object>
creator: reviewer_pkns

<content>
summary:
The authors present a new, challenging benchmark for temporal graph modeling, specifically addressing scenarios with fewer or no repeated edges. They provide an analysis of the limitations of the existing methods and conduct experiments using both current temporal graph modeling methods and sequential recommendation methods.

strengths:
1. The motivation behind this work is compelling, as it is meaningful and reasonable to consider the repeat behavior in the temporal graph.

2. The proposed benchmark spans multiple domains with different data sizes and includes both bipartite and non-bipartite graphs, offering diverse resources for the research community to investigate issues around unseen edges.

3. The paper is well-structured and easy to follow.

weaknesses:
1. While the paper focuses on unseen edges in temporal graphs, the selected baselines are primarily general-purpose models within the graph or recommendation domains. There are specific works that address similar challenges. For example:

- In the temporal graph domain, there are models with the ability to handle the inductive nodes:

   - TREND: TempoRal Event and Node Dynamics for Graph Representation Learning. WWW’22.

    - On the Feasibility of Simple Transformer for Dynamic Graph Modeling. WWW’24

- In the recommendation domain, Models tailored to repeat and exploration behaviors, including:

    - RepeatNet: A Repeat Aware Neural Recommendation Machine for Session-Based Recommendation. AAAI’19
    - Repetition and Exploration in Sequential Recommendation. SIGIR’23
    - Right Tool, Right Job: Recommendation for Repeat and Exploration Consumption in Food Delivery. RecSys’24
    - To Copy, or not to Copy; That is a Critical Issue of the Output Softmax Layer in Neural Sequential Recommenders. WSDM’24

    - (the explore-only model) "Masked and Swapped Sequence Modeling for Next Novel Basket Recommendation in Grocery Shopping”, RecSys’23

Thus the review would like to see what the performance would be like using the specifically designed models on the new benchmark.

2. The toy example offers a controlled setting to assess sequential pattern recognition with the existing methods. However, it may oversimplify the problem, limiting its ability to reflect real-world dynamics. I think the authors should analyze the performance limitations in realistic settings to provide a more comprehensive understanding of the critical limitations of the existing methods, offering insights about the possible improvement areas.

3. The analysis in Tables 3 and 4 provides limited insights, focusing primarily on overall performance decreases with the new benchmark. Deeper insights should be included to inspire future model design for this benchmark. For example, evaluating models separately on seen and unseen edges to highlight how well they handle these distinct cases.

4. The authors claim that they include a state-of-the-art recommendation model named SGNN-HN. However, it was published in 2020, which is not the state-of-the-art recommendation model.

questions:
The authors only use one metric MRR to measure the performance, what about other ranking metrics (such as NDCG, Recall)? Do we need to design other metrics to measure the performance?
</content>
</thread_object>

</thread>


Metareview:
metareview:
The benchmark paper identified an important issue in existing graph datasets (repeated edges). The benchmark is diversified and extensive, spanning multiple domains. The writing and organization is clear and well executed. There is general consensus among the reviewers to accept this paper.

The reviewers also raised some issues, such as additional related work, evaluation setting, the formulation of section 3.2, which have been addressed satisfactorily in the rebuttal phase. 

The authors are suggested to go through the issues raised by each reviewer again, and make sure they are addressed in the final version.