Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper proposes two new techniques for the negative sampling of dynamic link prediction and two new visualization techniques and introduces five new dynamic graphs datasets from under-evaluated domains like politics, transport, and economics.

The paper is clear and understandable. The further examination resulted in several concerns. Addressing these concerns during the rebuttal may warrant an increased score.
strengths: The paper presents some new insights for evaluating link prediction on dynamic graph networks and exploration of properties of dynamic network datasets. The authors introduced new datasets concerning new domains (Politics, Transport, Economics), which may lead to methods that are more domain-concerned. The authors also provide detailed results in an appendix and discuss different performance metrics for link prediction.
weaknesses: * There are some concerns regarding the influence of the node/edges attributes, which are further discussed
* There are some concerns about the availability of datasets, which are further described
* The authors state that "if the number of available historical edges is insufficient to match the number of positive edges, the remaining negative edges are sampled by the random NS strategy." I would strongly recommend adding information about how many negative edges were sampled according to a random strategy for all evaluated datasets. These should be done in both Historical NS and Inductive NS 
correctness: There are some concerns about the submission correctness:

* The authors state that ”existing implementations have no collision check between positive and negative edges.” That is not generally valid, as some methods have already considered collision checking. E.g., FILDNE sampling procedure https://gitlab.com/fildne/fildne/-/blob/master/dgem/tasks/link_prediction/dataset.py
* The authors use attributed datasets. However,  in Table 1. with the datasets overview, there is no information about the node and edge features, which should be updated.
* In Appendix A.1. there for all models (except TGN), the authors do not state whether node/edge features are utilized. Also, that aspect is not covered in the discussion of the results.

clarity: Yes, the paper is well written and understandable.
relation to prior work: Relation to prior work is discussed.
additional feedback: Authors are advised to:

- Change the strength of some statements
- Provides more details on datasets' edge/node attributes and how models utilize them.
- Consider the aspect of dataset attributes to the presented results
- Improve availability of datasets, remove or fix redundant files (like files consisting of zero tensors)
review_title: Review

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors propose a more robust procedure for evaluating dynamic link prediction on graphs. By noticing the disconnect between existing benchmarks and real-world uses (easy negative edges unchecked and the strength of simple memorization), the authors propose three key contributions as part of a larger benchmark suite:
* Two novel sampling strategies on graphs: historical Negative Sampling (NS) and inductive NS, which provides more insight into the effects of such on dynamic link prediction benchmarks
* Two variants of EdgeBank, a novel but simple, memorization-based approach that beats many state-of-the-art methods on existing benchmarks, thus illustrating their weaknesses
* Finally, five new datasets that expand the domain of current benchmarks and two visualization tools (Temporal Edge Appearance Plot and Temporal Edge Traffic Plot) to help better understand the benchmark.
strengths: * Dynamic Link Prediction is an important problem, and the paper offers very clear answers with elegant baseline approaches, visualization techniques, and benchmarks.
* the authors have provided detailed analysis and experiments to illustrate the problems of current benchmarks, as well as the superiority of the new approach + baseline.
* the ideas of "Easy Negative Edges" and "Memorization Works Well" are very fresh & insightful to me.
weaknesses: * all five proposed datasets seem to have significantly fewer unique timestamps than existing ones & are sampled in longer frequencies, granted that the nature of some of these datasets (Politics) determines such outcome.
* the core contributions of the paper seem to be split into three sub-points that are slightly incoherent - thought individually interesting, the main text seems to also split evenly introducing each of them, therefore losing a bit of focus.
correctness: The construction of dataset, experiment design, and evaluation methods appear to be correct.
clarity: Apart from the focus point in weakness, this paper is very well written and easy to understand. The length of the main text and supplemental materials are also appropriate.
relation to prior work: This work is sufficiently different from previous contributions and well discussed.
additional feedback: ### Questions

* does the "near-perfect performance of dynamic link prediction methods" refer to the "Standard" setting in figure 1 and 5?
* is the new benchmark designed to be evaluated in a suite? If so, the additional five datasets don't seem to really solve the limited domain diversity problem as it only expands to politics, economics, and transportation. Domains such as "biological networks such as protein interaction" remain unaddressed.
* does LIWC in the appendix refer to https://www.liwc.app/help/howitworks?
* relating to the subject of backtesting in time series forecasting, is it possible to incrementally build $E_{\text{train}}$ and $E_{\text{test}}$ based on a series of time splits? Would this benchmark still be useful?
review_title: Review - Interesting Benchmark for an Important Problem

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The authors focus on representation learning and dynamic link prediction on continuous-time dynamic networks. The authors review the current evaluation procedures and datasets, summarize three main limitations, and then propose the corresponding solutions. 1. Most existing datasets are limited in the social interaction domain. So the authors provide 5 new datasets ranging from politics, economics, and transportation domains, and develop visualization techniques to examine different properties of datasets. 2. Current methods for dynamic link prediction have achieved near perfect performance due to the weak negative sampling-based evaluation. The authors thus propose 2 new sampling strategies. 3. Considering the re-occurring patterns in datasets, the authors propose a novel non-parameterized and memorization-based baseline EdgeBank.
strengths: 1. The proposed visualization techniques, TEA and TET plots, and the corresponding indices are useful to quickly examine datasets characteristics. 
2. New datasets and negative sampling strategies can facilitate the development of novel models.
3. The proposed EdgeBank will be a strong and widely-used baseline.
weaknesses: 1. Only the transductive setting is considered in the paper. Inductive temporal link prediction is also an important setting to test the model performance on new nodes.
2. Though 5 new datasets are proposed, their time granularity is too coarse. This makes the datasets more like discrete-time dynamic networks (sequences of snapshots).
correctness: The evaluation methods and experiment design seem correct to me.
clarity: The paper is well written and easy to follow.
relation to prior work: The authors clearly discuss how this work differs from previous papers in Related Work section. 
additional feedback: 1. (More on weaknesses #2) Most existing methods are not optimized for massive concurrent events [1]. For example, Jodie always processes events sequentially. Extra results on how the degree of concurrence of events will affect the models will be interesting.
2. Current methods have achieved near-perfect performance due to the weak negative sampling-based evaluation. Is it possible to show scores of MRR or Recall@k without sampling just like Jodie did? 

[1] Zhang et al., CoPE: Modeling Continuous Propagation and Evolution on Interaction Graph. CIKM'21.
review_title: Strong baseline and 5 new datasets

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: In this work, the authors point out problems with the evaluation of dynamic link prediction approaches. As approaches often achieve near-perfect results, the authors argue that a) datasets are too simple and b) current sampling techniques generate negative edges that lead to less differentiation of approaches. The authors offer novel tools for better evaluation of dynamic link prediction and advise others to use them: a ) They introduce new more robust negative sampling techniques, b) strong baseline method for comparing approaches against, c) 5 novel dynamic graph datasets and two new techniques for generating datasets analysis plots.
strengths: - Although the novel baseline method is fairly simple (which is not a bad thing) it achieves outstanding results, even outperforming complex models on specific settings and datasets.
- The proposed novel negative edge sampling technique allows the differentiation of approaches based on specific objectives.
- New datasets introduced in the paper are a good contribution to diversity (different domains, different topological structures, different temporal edge evolution patterns).

weaknesses: - The introduction is confusing
  - Maybe include in the introduction for people not deep into dynamic link prediction: What are negative edges and how are they used?
  - The term "easy negative edges" is used as it would be commonly known terminology, while afaik it is not. Please explain it shortly. What are easy negative edges? (From the context it is more obvious that easy refers to negative edges that are not very similar to the true edge and thus evaluation performance is higher, however, I would advise stating this explicitly)
  - Either move Fig 1. to Results
    - Fig 1. confused and surprised me a little at this point in the paper, I would move it rather to the results or remove it completely (as it is already contained in Fig 5)
  - or Rework the caption of Fig 1.
    - Caption should describe what can be seen in the figure
    - "The ranking of different methods changes in the proposed negative sampling settings, which eliminate easy negatives." -> This sentence is confusing, please rework it. Make it clear what eliminates easy negatives and, as previously said, what easy negatives are. 
    - Make it more clear that Historical and Inductive are your novel NS strategies.
  - What dataset was used for Fig 1.?
- Fig 3. is confusing
  - Fig 3. needs better description in text/caption, right now it is very confusing what can be seen in the plots. Here are some thoughts:
    - The X-Axis needs a less confusing label: With the current label "Percentage of observed edges" one would wrongly interpret the plot "(j) US Legis" at Timestamp 12 as that there are about 20% of observed edges in this timestamp which are orange? And then why is there another datapoint in Timestamp 12 that says about 27% of observed edges are orange? I think the X-Axis could be labeled in a more intuitive way.
    - Furthermore, is my assumption correct that a datapoint at a certain location X, represents the same edge at a different timestamp? This should be made more clear in the caption/text.

correctness: The technical aspects of the work seem correct.
clarity: The paper is well written and easy to read, however
- It irritates a little that reoccurrence and recurrence are used interchangeably throughout the text, maybe make that uniform where it makes sense.
- Typo L234: reocurring -> reoccurring
relation to prior work: No prior work reported.
additional feedback: N/A
review_title: Review of "Towards Better Evaluation for Dynamic Link Prediction"

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper aims at tackling the evaluation problem of dynamic link prediction. The authors propose two visualization techniques(TEA & TET plots) to illustrate the recurring patterns of edges over time, showing the difficulty of the datasets. They next show limitations of the current random negative sampling(NS) strategy and introduce two harder NS strategies, i.e. historical and inductive NS, to better test the generalization of DGNN models. A simple yet competitive memorization-based baseline EdgeBank is further proposed. 5 SOTA models are compared on 12 datasets in 3 different NS settings. It's found that performance and ranking of different models vary significantly. 
strengths: [+] 5 SOTA models are compared on 12 datasets in 3 different negative sampling settings. 

[+] TEA & TET plots, EdgeBank and random NS show the limitations of current evaluation of dynamic link prediction.

[+] 5 new datasets are introduced. 
weaknesses: [-]As this paper is a benchmark, it is better to include more DGNNs, like HTGN[1],EvolveGCN[2],DySAT[3],etc. 

[-]As the paper adopts transductive setting, i.e. all nodes can be seen during training, static GNNs seem also able to tackle the tasks. It's better to include several representative static GNNs(like GIN[4],GraphSage[5],etc) as baselines to show whether they are beated by DGNNs in different NS settings.

[-]It is better to include the training and inference time for each DGNN.

[-]It is better to modularize the codes for convenience of interested researchers.

[1]Yang M, Zhou M, Kalander M, et al. Discrete-time temporal network embedding via implicit hierarchical learning in hyperbolic space[C]//Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2021: 1975-1985.

[2]Pareja A, Domeniconi G, Chen J, et al. Evolvegcn: Evolving graph convolutional networks for dynamic graphs[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2020, 34(04): 5363-5370.

[3]Sankar A, Wu Y, Gou L, et al. Dysat: Deep neural representation learning on dynamic graphs via self-attention networks[C]//Proceedings of the 13th international conference on web search and data mining. 2020: 519-527.

[4]Xu K, Hu W, Leskovec J, et al. How powerful are graph neural networks?[J]. arXiv preprint arXiv:1810.00826, 2018.

[5]Hamilton W, Ying Z, Leskovec J. Inductive representation learning on large graphs[J]. Advances in neural information processing systems, 2017, 30.
correctness: Yes
clarity: Yes
relation to prior work: Yes
additional feedback: 1. Why use MLP instead of hadamard product operator[6] as downstream classifier?

[6]Grover A, Leskovec J. node2vec: Scalable feature learning for networks[C]//Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining. 2016: 855-864.
review_title: Review of "Towards Better Evaluation for Dynamic Link Prediction"

Meta-review: metareview: This paper presents multiple contributions towards understanding, evaluating, and developing better methods for dynamic link prediction. This includes:
 * New sampling strategies that better match real-world applications
 * A simple but surprisingly strong baseline, which already beats several more sophisticated methods
 * A new collection of 5 datasets for dynamic link prediction from different domains

Collectively, this is a significant advance supporting further research in dynamic link prediction. All reviewers were positive in their assessments of the paper, especially after the authors improved their paper in response to reviewer comments.

I recommend this for a spotlight because I think it could have significant impact within its area.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept