Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a self-supervised learning framework for dynamic graphs. Temporal edges are split into windows which are then used as sub-graph samples for self-supervised learning. The proposed model is efficient and obtains empirical improvements on several datasets.
strength_and_weaknesses: The idea proposed in this paper is interesting to me, and experiments show some empirical improvements and efficiency. However, I still have the following concerns and questions for the current version of this paper:

1. Section 4 is not easy to follow. In particular, the notation definitions are not clear enough. For example, at the beginning of Section 4, Eq (2) has M+1 intervals rather than M; What's the difference between stride S and window length W? Can the intervals defined in Eq (2) overlap?

2. Window size is essential to capture the temporal pattern and varies on different datasets (according to Figure 6 left). It is difficult to predefine, which hinders the practical use of the proposed approach.

3. The experimental results are mixed. The baseline CAW does better in many settings, especially on the inductive link prediction task (Table 5).

4. I'm curious about the efficiency comparison against CAW because CAW proposes an efficient link sampling algorithm and claims high efficiency. In addition to Figure 3, it would be better to give some formal time complexity analysis or profiling. This will make readers understand where the gap is.
clarity,_quality,_novelty_and_reproducibility: Please see the above comments.
summary_of_the_review: An interesting approach for dynamic graphs but further investigations are needed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method to improve representation learning for dynamic graphs using self-supervised learning (SSL) approach. Named DyG2Vec, the method is equipped with a SSL pre-training component that uses a non-contrastive loss objective, a window-based architecture as well as fine-tuning component that can tune the task-agnostic representations for the concerned dynamic graph learning tasks. As for the architecture, an encoder decoder pipeline is adopted that uses attention based graph neural network, as well as additional temporal edge encoding which captures degree centrality as well as common neighbors between two nodes. Experimental evaluation on multiple datasets show promise of using DyG2Vec.
strength_and_weaknesses: Strengths:
1. The paper contributes an approach that can be used to perform SSL pre-training on dynamic graphs using non-contrastive approach. This evidently improves the trained architecture for temporal graph tasks in a majority of the empirical evaluations shown.  
2. The background of the research topic is clearly written which follows the need to proposed DyG2Vec.  
3. The window-based training is interesting and from the experiments its observed that it is able to predict longer future events.  
4. Figure 3 study shows DyG2Vec is more efficient yet high performing compared to baselines.  
5. Ablation studies and experimental details are sufficiently mentioned that provides robustness of the studies as well as details for reproducibility later (after code release).  

Weaknesses and questions:  
1. There are couple of related works that could be discussed - SSL for dynamic graphs using contrastive loss [1], and provably expressive temporal networks [2], and where possible, their performance comparison with DyG2Vec can be mentioned.  
2. Page 6: How is temporal degree centrality is computed for an edge? It mentions "the current degrees of nodes u_p and v_p at time t_p." the degrees of u_p and v_p could be different; how are the two values combined to make the edge degree centrality?  
3. After reading the experimental discussion, the curiosity on what is the major contribution of DyG2Vec for the improve in performance -- SSL pre-training? window-based framework? temporal edge encoding? For instance, if its SSL pre-training can it be easily used in prior models as an augmentation or extension and whether it would provide an equivalent boost in performance? And for temporal edge encoding -- is it providing features to the architecture which otherwise cannot be learned by the temporal graph attention?  
4. Page 8: "Table 2 shows that DyG2Vec outperform all baselines significantly" - Although the performance of DyG2Vec is impressive, the quoted statement in Page 8 (as well as similar claims in other experimental sections) is misleading and incorrect as per the table of results. For example, in Table 2, DyG2Vec does not beat baselines in 2 out of 9 instances. Such precise numbers may be mentioned to correct the claims as quoted.  
5. For the performance versus Inference time graphs in Figure 3, how do the models compare in terms of parameters or sizes? Are all the models plotted in Figure 3 are of comparable model size (eg. number of layers or number of parameters)?  

[1] Jiang, L., Chen, K.J. and Chen, J., 2021. Self-Supervised Dynamic Graph Representation Learning via Temporal Subgraph Contrast.   
[2] Souza, A.H., Mesquita, D., Kaski, S. and Garg, V., 2022. Provably expressive temporal graph networks.
clarity,_quality,_novelty_and_reproducibility: The paper's clarity is good - most of the text would be understood conveniently to the readers.  In terms of quality, the paper presents a method that is intuitive and considers multiple factors when learning on dynamic graphs, such as temporal encodings, window based training, etc. Although there are a few issues (see weaknesses above), overall the clarity and quality is sufficient. In terms of originality, the paper presents a method that is motivated by existing works but used for SSL representation learning on dynamic graphs. In my evaluation, the SSL pre-training, window based training and temporal edge encoding are the new additions to the architecture-- some of which are motivated from prior works. Finally, there is no theoretical investigation of DyG2Vec -- refer [2] for a related work.
summary_of_the_review: The paper overall does great in presenting a self-supervised pre-training approach that can be used to generate task-agnostic node representations for dynamic graph learning, and is intuitively and empirically powerful than existing works. There are a few missing details (as asked in the weaknesses and questions section above) which can further be provided or answered.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a novel SSL pretraining architecture based on encoder-decoder model called DyG2Vec for dynamic graphs. It is a fixed window-based method to learn node embeddings for future predictions. Paper uses two views of temporal subgraphs in a non-contrastive SSL framework. The SSL objective consists of three terms, invariance criterion, variance term and covariance term. Two evaluation mechanisms are adapted for experiments. Authors performed two downstream tasks such as future link prediction and dynamic node classification on 7 benchmark datasets. 
strength_and_weaknesses: 1. The problem is challenging and impactful. 
2. Being an SSL framework, it has more advantages than supervised models. It can be used for a wide variety of tasks. 
3. Experimental results in Tables are not convincing although some of the results like Fig3 and Fig6 are interesting. 
4. Paper is easy to follow, and well-written. However, I see a major weakness in experimental results. 
5. Although there is a benefit of using DyG2Vec due to its inexpensiveness. However, it looks like CaW does a better job overall. 
6. This method might be useful for large graphs if this can make training faster at the cost of some % of accuracy but cannot be considered for all the scenarios. 
7. The authors mentioned in 5.1 that existing methods evaluate for K=1. However, this paper includes more results with K=200, 2000. What is the rationale for doing this? I see that results for K=1 are not good for DyG2Vec but better for K=200, 2000. Any intuition behind this is not explained. Why do other works have considered only K=1? 
8. Fig6 shows that MOOC and UCI datasets are sensitive against the value of W. 
9. I have seen a similar work [1] but for meshes in which the method takes a mesh and predicts a mesh for a future timestamp. Can you please comment on the similarity with this work? 
[1] LEARNING MESH-BASED SIMULATION WITH GRAPH NETWORKS 
clarity,_quality,_novelty_and_reproducibility: The paper is well-written, and the problem is interesting. The paper is well-placed in the literature, and relevant works are cited and referenced. However, the experimental results are not too convincing. The idea is somewhat novel. The work would benefit the GNN community if the code is available. 
 
summary_of_the_review: The work is presented well and has advantages being an SSL framework, inexpensive, etc. However, I have a few concerns that it is not outperforming in the majority of the settings. Rather CaW looks more like a generalized method but it is quite expensive. What is the rationale behind using the K>1 setting? There is similar work in a different domain (meshes)
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a SSL-compatible, efficient dynamic graph model named DyG2Vec, which uses a window-based mechanism to generate task-agnostic node embeddings. Experiments on benchmark datasets and two downstream tasks show that the method has stronger prediction ability and less inference time than SOTA baselines. The paper also introduces SSL pretraining and finetuning protocals for dynamic graphs.

strength_and_weaknesses: [+] A new SSL learning method for dynamic graphs by using a window-based mechanism.

[+] Experiments on benchmark datasets and two downstream tasks show that the method has stronger prediction ability and less inference time than SOTA baselines.

[+] The paper also introduces SSL pretraining and finetuning protocals for dynamic graphs.

[-] Limited novelty and unclear motivation. a) the proposed window-based mechanism seems straight-forward adaption from ssl learning for static graphs, and the view generators for ssl, i.e. edge dropping and edge feature masking in this paper, are commonly adopted in graph ssl literature. It is expected to design specific ssl methods by utilizing the dynamics which can not be considered in static graphs. b) It is not well explained why the window-based mechanism acts as a good ssl method for dynamic graphs. c) The motivation of SSL for dynamic graphs is also not well-explained. The authors state that "task labels are often scarce, costly to obtain, and highly imbalanced for large dynamic graphs", but it seems not supported in dynamic graph literature. The authors are expected to give support for the statement in detail. Moreover, in link prediction, which is common downstream task in dynamic graph literature, edges serve as training task labels (historic edges) and also testing task labels (future edges), and it does not support the statement "task labels are often scarce".

[-] Presentation can be largely improved. 

[-] Lack of deeper analyses. For example, it is not well-explained why the proposed method has better forecasting capability in larger horizon.

[-] Missing important related works. There are a few related works about self-supervised dynamic graph learning methods, e.g [1-4].

[1] Tian, Sheng, et al. "Self-supervised Representation Learning on Dynamic Graphs." Proceedings of the 30th ACM International Conference on Information & Knowledge Management. 2021.
[2] Cong, Weilin, et al. "Dynamic Graph Representation Learning via Graph Transformer Networks." arXiv preprint arXiv:2111.10447 (2021).
[3] Wang, Lu, et al. "Tcl: Transformer-based dynamic graph modelling via contrastive learning." arXiv preprint arXiv:2105.07944 (2021).
[4] Jiang, Linpu, Ke-Jia Chen, and Jingqiang Chen. "Self-Supervised Dynamic Graph Representation Learning via Temporal Subgraph Contrast." arXiv preprint arXiv:2112.08733 (2021).

(Minor)
1. typo in Figure 6.
clarity,_quality,_novelty_and_reproducibility: Clarity: Presentation can be largely improved. 
Quality: Extensive experiments but lack of deeper analyses.
Novelty: Propose an interesting SSL mechanism, but similar to related works and it is not well explained why the mechanism works for dynamic graphs.
Reproducibility: Datasets, settings, protocals, hyperparameters, and infrastructures are included in the appendix. 
summary_of_the_review: The paper proposes a new SSL learning method for dynamic graphs by using a window-based mechanism and extensive experiments show that the method has stronger prediction ability and less inference time than SOTA baselines. However, the reviewer concerns that 1) the motivation and presentation is not clear 2) the technical novelty is limited 3) importance related works are missing 4) deeper analyses of the proposed method is missing.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper introduces DyG2Vec, a new window-based encoder-decoder model for dynamic graphs by using an attention-based message-passing mechanism that utilizes hierarchical multi-head attention modules to encode node embeddings across time. Furthermore, the authors present a joint-embedding architecture for dynamic graphs in which two views of temporal sub-graphs are encoded to minimize a non-contrastive loss function. 

In general, the paper is well-written and easy to follow. The idea of the paper is interesting. However, the reviewers had some concerns about the contributions, related work, and experimental results and analysis of the paper. The authors addressed some of them but the reviewers are not fully convinced.
Decision: Reject