Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper focuses on the problem of modeling interaction processes over dynamically evolving graphs and perform inference tasks such future interaction prediction and interaction classification. Specifically, the paper proposes a temporal point process based formulation to model the interaction dynamics where the conditional intensity function is parameterized by a recurrent network. With an occurrence of any event, the recurrent architecture updates the embeddings of the nodes involved in that event which then affects the intensity function and hence the likelihood of future events. The paper uses intensity based likelihood to train for future interaction prediction task while cross-entropy based loss for classification task. The paper demonstrates the efficacy of the method through experiments across multiple datasets and compare against representative baselines and further provides ablation analysis for the proposed architecture.

The paper demonstrates markedly improved empirical performance on multiple datasets and also performs the task of interaction classification which is not seen in recent works on evolving graphs, which are plus points. However, there are several concerns with the overall work that makes this paper weaker: (1) The main concern is with the novelty and more importantly the justification/analysis of the contributions proposed approach. (2) Further, while the ablation study provides some insights into architecture, it is not adequate (3) The paper misses comparison with a very important and recently proposed baseline, JODIE [1].


Main Comments:
--------------
- The paper leverages existing techniques built for learning over evolving graphs and augments it with three modifications: explicit use LSTM with time gate, stacked LSTM approach with fusion (Aggregation) and attention mechanism to select important neighbors to contribute to embedding update. The use of LSTM with time gate and fusion mechanism is very incremental contribution.  The attention mechanism proposed here is novel compared to existing works. However, there is very little justification or analysis provide or either of the contributions. This is big drawback of this contribution.
 
- For instance, the authors mention that stacked LSTM is used to capture multiple resolution. Can they provide some analysis or empirical demonstration that this actually happens? Also, the authors mention they use K in range of {1,2,3,4} but do not provide details what was useful for each dataset and how is it useful. How does the scaling parameter and alpha affect the performance and what are their roles? Also, what does superscript 'task' signify?
Similarly, they propose coattention mechanism with adaptive gate functions but does not provide any analysis of why they are useful and what characteristics they capture in the data that allows it to select most relevant neighbors. Is the attention mechanism temporally dependent?

- The authors perform ablation studies by switching off each component as a whole but considering the way this architecture is built, this is not a very useful exercise except knowing that each component contributes to the performance. A more detailed analysis and ablation is required. For instance, can the authors show performance with different K and  how it deteriorates/improves with it? Also, for stacked LSTM case, the authors show what happens when you use last layer, but what happens if the authors use only one layer (I guess this is K=1?) or don't use residual connections? When the time gate is switched off, does the authors also remove deltas from intensity function? what happens in this scenario? How does subgraph depth affect the quality of performance? What happens if authors don't sue adaptive gate functions?

- Figure 2 shows an example of bipartite graph, however, it seems datasets in experiments does have non-bipartite case? Is this true or the method only works for bipartite case?

- The use of proposed Algorithm 2 is not well justified. Why does the author need coloring and hashing mechanism instead of simpler BFS/randomwalk routine to collect previous interactions? Also, is this subgraph created for each event or it is computed offline during training? Further, the subgraph used for selection mechanism same as subgraph used for backtracking in LSTM? 

- Further, is it true that the training is done in order of ColorGraphSeq or is it done in order of dataset? How does the authors capture dependencies across dataset in later case?

- Do you also update cell states with selection mechanism? The DIP-UNIT equation in selection section does not show that update. Also, are the embeddings updated only during train or also during validation/evaluation?

- The authors only present the results as-is without any insights on the performance of DIP model vs others and why they are able to demonstrate good performance. It is highly desired that authors add discussion section for each set of results to provide such information

- The authors include support for new nodes for interaction classification task but remove them for interaction prediction task which is strange. Is there a specific reason for this? What is the effect on the performance if new nodes are allowed in test? Further, why is interaction classification not compared with temporal baselines? All baselines produce embeddings and the authors mention that classification for this paper is independent of marker history. While the temporal baselines do not train for the task, the authors can train a second stage classifier with learned embeddings to perform classification

- The authors do not compare with recently proposed JODIE [1] which is a big miss. The comparison is required as it also models interaction processes in a  novel way by actually predicting the next embedding directly instead of modeling the intensity. An empirical comparison and discussion of this method is required to compare with various state-of-art methods.

Minor:
-------

- The authors need to use better and consistent notations. Also, as the overall approach uses similar flow as previous papers such as DeepCoevolve, it is recommended that the authors make the presentation simpler to position it clearly with existing works. On page 3, section 3.2 both bold-face and normal letters are used as vectors. Is $\hat{x}_{u(t)}$ a vector?

- Please provide numbers to equations for better referencing

[1] Predicting Dynamic Embedding Trajectory in Temporal Interaction Networks, Kumar et. al. KDD 2019
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The paper is concerned with modeling continuous time-evolving graphs, for which it proposes to combine temporal point processes with a recurrent architecture to learn dynamic node representations. In addition, the paper proposes to stack multiple recurrent layers (to obtain node representations over multiple time scales) and use a temporal attention mechanism (to select relevant past interactions).

Modeling temporal and dynamic graphs is an important problem with many applications in ML and AI. The focus of the paper, i.e., to develop improved models by combining TPPs and representation learning, is a promising approach to this task and fits well into ICLR. Furthermore, the presented experimental results are promising.

However, I'm concerned about different aspects of the current version: The main contributions of the paper are a recurrent (LSTM-based) architecture to model the intensity function of a TPP, stacking multiple LSTM to form a deep architecture, and a temporal attention mechanism. However, none of these contributions on its own are particularly novel. For instance, prior work that introduces similar approaches include 
- Recurrent networks to parameterize intensity functions: (Dai, 2017), (Mei & Eisner, 2017), (Trivedi, 2019), ... 
- Temporal attention: (Trivedi, 2019) 

Hence, the main novelty seems to lie in the stacked architecture and the particular combination of modules (which is of limited novelty). The experimental results are certainly interesting, but it would be important to provide a more detailed analysis of the model to get insights into the causes for these improvements.

With regard to the model: The Log-likelihood function in Section 3.6.1 seems to be incorrect as the LL for a TPP would be L = \sum_{i:t_i \leq T} \log\lambda(t_i) - \int_0^T \lambda(s)ds, which is quite different from the equations in the paper. Is the LL in Section 3.6.1 the actual objective that has been optimized?

Regarding the experimental results: All models are trained on the same grid of embedding dimensions, but the proposed method is the only deep model. Hence, its maximum number of parameters can be up to 4x compared to the shallow models. How do the results look if all for models with comparable number of parameters (i.e., can the improvements be explained due to this difference)? It would also be good to get results on commonly used benchmarks (e.g. data used in DyRep or NeuralHawkes) to make the results of the new model comparable to prior experiments and datasets.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper considers modeling continuous time-evolving graphs using a temporal point process framework. It introduces a time gate in the LSTM to handle the temporal dependency and uses an attention mechanism to select relevant nodes to learn the underlying dynamics. 

Overall, this paper is not easy to understand in detail. Firstly, it is unclear how and why the temporal point process can deal with growing/shrinking graph nodes and changing interactions. Secondly, how does the DIP-UNIT handle the continuous graph changing? What if the graph changes with an uneven speed? Thirdly, how do all the small pieces work together to achieve the goal of the paper? An overview diagram or a toy example would greatly improve the readability of the paper. 

Besides, what is the computational cost of the proposed network? How large a graph could be and how fast its changes could be captured? 
review_title: Official Blind Review #1

Meta-review: All reviewers rated this paper as a weak reject.
The author response was just not enough to sway any of the reviewers to revise their assessment.
The AC recommends rejection.
Decision: Reject