Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper introduces Memory Tower Augmentation (MeTA) --- an adaptive data augmentation approach for temporal graph networks (TGNs). The proposal aims to tackle overfitting and improve learning in TGNs. MeTA applies a hierarchical message-passing scheme to update TGNs' memory states. In addition, MeTA considers three different data augmentation strategies: i) perturb time, ii) remove edges, and iii) add edges with perturbed time.  The experiments show that MeTA achieves higher accuracy than existing TGNs (without data augmentation) on future link prediction and node classification tasks.

main_review: The proposed method (MeTA) is novel as it represents the first attempt to leverage adaptive data augmentation techniques for temporal graph networks. The results show that MeTA can boost the performance of the baselines. The main weakness is that MeTA scales linearly with the number of levels of memory modules ($L$). Also, I found section 3.2 particularly hard to follow, and the authors should consider rewriting it for clarity.

### Comments:
* Avoiding overfitting is one of the motivations (lines 3, 24). However, current TGNs are shallow models with 1 or 2 temporal aggregation layers. Is overfitting a problem in those methods? How much does the proposal alleviate it? Showing this would strengthen the paper. 
*The experiment with PerturbTime can change the ordering of the events. Does it have any relevant implications? Do the authors have any intuition about why this ordering change does not negatively impact the performance?
* Figure 4 shows that "large values of p1 and small p2 results in poor performances". The authors explain that a large p1 increases the risks of breaking the semantic information, while a small p2 cannot augment the features sufficiently. Based on that intuition, why do large values for p1 (=0.5) and p2 (=1.0) still lead to good performance (Figure 4)?
*It would be valuable to report numbers for different values of $L>2$ in the supplementary material.
*For some TGNs, a reasonable choice is to sample/aggregate temporal information that respects time. In this approach, neighbors are events in the past (e.g., see CAWNet). Is that the case for the evaluated TGNs? If so, both topological and temporal distances would convey the same information.
*The authors should cite this related work: CAWNet (https://arxiv.org/abs/2101.05974).

### Minor (inconsequential) issues
* In Table 2, denoting MeTA (L=1) in boldface seems misleading since there is no reason for MeTA (L=1) to be faster than the original approach (TGN).
* Some typos: Gaussion (line 184), tmax (line 188), $s^i$ (line 218), "more earlier edges" (line 220), "magnitudes , as" (line 53), "information , as" (line 42).
* I don't find the comparison against the refeed method relevant (section 4.3, and line 241). It is clearly much slower than MeTA. 

--------------------------------------
After the rebuttal phase, I have decided to increase my score to 7 (accept).

limitations_and_societal_impact: The authors adequately address the limitations and the broader impacts of the proposal.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes an adaptive data augmentation framework for dynamic graphs that improves the performances of two well-known baseline models on node classification and edge prediction on four datasets. Three data augmentations were used with the purpose of modifying temporal and topological features. These strategies inject the inductive bias that closer events in the dynamic graphs, whether in time or topology, are more informative. 
main_review: Temporal graph learning is relatively a new field of research in the graph machine learning community. Data augmentation has shown its merits in various fields and so adaptive augmentation for temporal graphs is an interesting and important topic of study. The problem and the approach are well-justified and discussed in the paper and generally, the paper is well written. However, there are a few comments and concerns I'd like to share:

First, although the algorithm is simple, efficient, and effective, it is not easy to fully understand it from the text initially. The authors can try and improve the description of the method in a more clarified way. Also, the main algorithm shouldn't be in the appendix. It would be helpful to re-structure the paper so that the pseudocode of MeTA doesn't end up in the appendix. Although the pseudocode also needs more clarification for a smooth reading by other researchers. 

Some previous papers are reporting average precision (AP) and accuracy for the studied datasets and this paper is doing the same. However, since in the discussion about the reasoning behind the DA strategies on page 5, the authors are showing the concern of real-world scenarios, it would be great if we can see the results for other metrics like mean reciprocal rank (MRR) or Recall (e.g. recall10) as proposed in JODIE.  In my view, these metrics are closer to real-world settings because most of the time there is no real or fake node to classify and you have to rank other nodes for a potential edge connection in the near future. (Consider recommending an object to a user in a social network based advertisement)

It is mentioned that DA is not applied during inference and it is a common thing to do, but what happens if DA is applied during inference too? 

The paper properly discussed the efficiency of the method in case of run time. Can you please compare the memory usage as well?
limitations_and_societal_impact: Limitations discussion is very limited and there is no social impact section.

For the social impact section, the authors can discuss employment of dynamic edge prediction in real-world recommendation systems.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper introduces Data Augmentation (DA) to the temporal graphs in order to offset the increased variance induced by complex GNNs. Three DA methods: perturb time, remove edges and add edges with perturbed time are considered. To make sure the DA effectively works for edges that are less informative, the author proposes a novel and efficient adaptive DA method based on recursion, which is named Memory Tower Augmentation (MeTA). The proposed MeTA can be added on top of RNN based temporal graph implementation. Experiments results on edge prediction and node classification tasks using public temporal graph data set suggest the effectiveness of the method. 
main_review: 1. contributions

This paper proposed an algorithm to implement the adaptive DA for the temporal graphs. The method (MeTA) first stacks layers of the memory module with varying DA parameters then uses the intermedia states from layers with more DA to update the RNN states of interest during training. This idea is both novel and interesting. On a high level, it is very similar to firstly stack temporal graphs with different DA strength and then uses recursion-based temporal graph neural networks (TGNNs) with shared weights across layers. So the time complexity only grows linearly with the number of layers in the MeTA and it works naturally with TGNNS. Overall, I think the idea is both neat and useful.

2. Presentation and quality

This paper is well organized. The authors do a good job explaining their design choice. However, for readers not very familiar with temporal graphs, it is hard to understand how the proposed method works. For example, I can only make sense of Figure 2 after I read and understand the whole section 3. 

The experiment setting is reasonable. The author compares the proposed method with the most state of temporal graph networks and the MeTA consistently performs better than baselines. 

3. Question

The paper claims the DA is adaptive. But in MeTA, the strength of DA only depends on the time and distance. While it is reasonable to believe the edges from past or distant edges are less informative, this is still heuristic-driven. Firstly the information here is not adequately defined and it is likely that edge/node might be highly useful yet remain distant from the target node. I don't whether the authors have thought about ways of making the DA truly adaptive. 

Proposition one assumes the edge distributions follow a homogeneous Poisson process. This is a very strong assumption. Even still considering the counting process, a more realistic setting for network events could be [Hawkes process](https://proceedings.neurips.cc/paper/2020/file/c5a0ac0e2f48af1a4e619e7036fe5977-Paper.pdf). Secondly, the proof of theorem 1 basically says any noise distribution won't affect the original edge distribution. But this also relies on the assumption of uniform occurrence. Wondering whether the authors have data to support these assumptions. In general, I don't think these propositions along with their proofs are the key contributions of this paper.



limitations_and_societal_impact: The author adequately addressed the issues.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes MeTA: multi-level module that processes the augmented graphs of different magnitudes on separate levels. Three DA strategies augmenting are adopted (perturbing the edge time to simulate time shifts; removing edges; adding edges). Experiments are conducted on edge prediction and node classification tasks.
main_review: Strong points: 
1.	Three DA strategies for the temporal graphs seem making sense, i.e. (i) perturbing time; (ii) removing edges; (iii) adding edges with perturbed time.
2.	Experimental results on four benchmarks show that MeTA not only improves accuracy on the baselines, but also reduces the number of memory updates per epoch.
3.	The robustness and sensitivity of the proposed method are verified by the soundness of the ablation study.

Weak points:
1.	The proposed method seems too simple without strong theoretical support. In fact the propositions and lemmas in my mind are a bit trivial and does not directly link to the augmentation performance. The current method only considers adaptive magnitude while in my mind the three augmentation operations shall also be adaptively applied if possible. There are many adpative data augmentation literature at least in vision whereby the different operations are adaptively applied.
2.	The setting in the experiment only compares with the original baseline. Author may compare MeTA with more recent works on DA. 
3.     The writing of the paper can be improved and there are some typos and grammatical errors in current form.

limitations_and_societal_impact: No.
review_title: null

Meta-review: After discussions between the reviewers and authors it seems as though a consensus has been reached amongst 3 of the reviewers that this paper is clearly worthy of acceptance.  The reviewer on the side of rejection has not engaged in discussions with the authors or in internal discussions on the paper.  The review against acceptance also lacks details in the objections raised to the paper.

I want to thank the authors for their detailed responses and for listening to and incorporating the feedback from the reviewers and thank those reviewers who did for engaging in discussions with the authors.
Decision: Accept (Poster)