Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
This paper introduces a model, Directed Acyclic Graph Neural Network (DAGNN), which processes information according to the flow defined by partial order. DAGNN can be regarded as a special case of previous GNN models, but specific to directed acyclic graph structures. The authors prove that the model satisfies the properties desired by DAG-based graph representation learning.Then they study topology batching on the proposed model to maximize parallel concurrency in processing DAGs. A comprehensive empirical evaluation is conducted on datasets from three domains to verify its effectiveness.


Reasons for score: Given the ubiquity of directed acyclic graphs, DAG-based graph neural networks have potential impacts on various fields. The authors propose an elegant and effective deep learning framework to learn node and graph representations on DAGs. My major concerns are the clarity of the model design, additional ablation tests, and sensitivity studies (see cons below). 

Pros: 
1) The problem is interesting. Directed acyclic graphs are very common in the real world. An efficient and powerful DAG-based graph neural network is expected to solve many unsolved problems in various domains.

2) The paper is well written. The authors introduce the framework based on the existing message passing neural network, which makes it easy to follow. The authors also present how this work handles special characteristics of DAGs. The techniques of the model are clearly stated. In particular, the comparison with highly relevant previous work is well explained. 

3) The authors provide sufficient experimental results, including comparative studies on four datasets with the state-of-the-art benchmarks and ablation tests, which show the effectiveness of the proposed framework.
 
4) This paper provides a theoretical analysis of properties of the proposed model, which demonstrates that graph representations extracted by the proposed model are  discriminative.  

Cons:
1) The framework needs more explanation. The authors introduce a recurrent neural network (Eq. 7) for updating node representation layer by layer. The input and past states are defined by the node representation at the last layer and the aggregated information from its predecessor nodes (i.e., message). As the authors introduced, these two arguments are switched in existing work. It would be better to provide more details about the design. 
-  One question is, how will the roles of two arguments affect the performance of the model?

2) Although the paper provides several ablation studies, I still suggest the authors to consider the following ablation studies to enhance the quality of the paper: 
	- What is the performance of the proposed model by changing the type of attention mechanism? Will it be relatively stable when using dot product attention which involves fewer parameters?
	- If we change the recurrent architecture to a feedforward neural network, what is the performance of the proposed model? Recurrent models are usually relatively slow in the training process and sometimes unnecessary.
	- How much does Bidirectional processing in recurrent neural networks help in improving the performance?
If the author can explain the above problems, it will be beneficial for people to understand this model.

3) No sensitivity analysis. The authors provide detailed model configurations, yet it is expected to see hyperparameter tuning results. For example, the proposed framework is a multi-layer graph neural network. It would be nice to test the sensitivity of the number of graph layers. Such knowledge of message-passing neural networks may not apply to the DAG-based framework. It would be better to provide some sensitivity studies or theoretical proof. 

4) A minor concern about the ablation study: why didn’t the authors report the results on the LP dataset?

review_title: Interesting work, needs some clarification

final_score: rating: 7: Good paper, accept
review_content: This work considers DAGNN for learning representations for DAGs. Compared with message-passing neural network, particularly D-VAE, there are three subtle and notable differences motivated by the properties of DAG: (i) attention for node aggregation, (ii) multiple layers for expressivity, and (iii) topological batching for efficient training. Some theoretic guarantees are established (similar to those in D-VAE). Experiments show an improved performance, and ablation studies validate the proposed modifications. I recommend a weak acceptance based on the present version and am happy to improve my rating if authors address my concerns.

1. For Eq. (3), authors claimed that 'An advantage is that DAGNN always uses more recent information to update node representations.' However, compared with the other modifications, I didn't see any place to validate this statement. Can the authors explain more here to validate it is indeed an advantage?

2. Theorem 2 and Corollary 3 are highly related and are suggested to be combined as one. Indeed, I did not see why 'Corollary 3' is a corollary.

3. 'Remark 2. Topological batching can be straightforwardly extended to multiple graphs for better parallel concurrency: one merges the Bi for the same i across graphs into a single batch.' Can author give more details here, when graphs have different lengths?

4. Authors proposed a parallel strategy for an efficient training. However, compared with D-VAE which is a sequential execution, the training time seems to be very close. Why? and what if DAGNN does not use the parallelization?

5. about the BN experiment: compared with RMSE, BIC is a more reasonable metric to evaluate the final performance of the obtained DAG, as a better RMSE may indicate spurious edges wrt. the true graph. Also, the authors use the BIC score as the criterion when finding an optimal DAG. So what are the BICs for the respective methods? How does it compare with the BIC of the true graph?

6. Also in the BN experiment part, 'Even though the DAG structure characterizes the conditional independence of the variables, they play equal roles to the BIC score and thus it is possible that emphasis of the target nodes adversely affects the predictive performance. In this case, pooling over all nodes appears to correct the overemphasis.' What does it mean by 'target nodes'? Can you also give more details about this reasoning and if possible, can you try other BN problems to validate this reasoning?

7. the overall writing is OK, but may be improved in several parts. For example, 1) the introduction part has many comparisons with other approaches when stating contributions; separating them into two parts may be more clear.  3) in appendix D, please use subsections for 'MODEL CONFIGURATIONS AND TRAINING' for an improved readability.


*** after reading rebuttal *** All my concerns are addressed, and I decide to improve my evaluation.
review_title: Improved modifications to learn representations for DAGs

final_score: rating: 7: Good paper, accept
review_content: The paper presents a graph neural network formulation specific for directed acyclic graphs. In this formulation, the aggregation function also considers the information about the current layer. The model considers nodes following a topological batching in order to have the information of the predecessors when calculating aggregation of a node.
The system has been compared with several competitors on three datasets. The results are good in terms of metrics and seem promising also in terms of training time.

The proposal is clearly defined, but I would be happy if the paper also contains a very simple example considering a small DAG and showing some steps of the computation.

The proofs seem to me correct and the datasets seem complex enough to make the tests significant. Graph Neural Networks have come to the fore in recent years as a promising approach to solve many tasks and this work opens a good line of research.

One possible cons is that it focuses only on directed graphs. But this can also be a pro. It is important to continue to test whether this choice will lead to a system that can achieve significantly better results than other systems that are more general and do not pose this limitation. To test this aspect, three datasets are not sufficient, but the results look promising.

As for the tests, unfortunately, I could not run the scripts. It seemed that I could not download/create the datasets.

I have only minor issues to highlight in the paper.
In formula (1), L is not defined, its definition is given later.
In figure 1 there are strange lines that should be removed.
On page 7, TargetInVocab baseline is considered but in the paper I can find only TargetInGraph. What is this baseline?
review_title: A good proposal defining GNNs for DAGs

Meta-review: This paper proposes a graph neural network architecture to learn representations for directed acyclic graphs. Specifically, the proposed method performs the aggregation of the representations from neighboring nodes in the topological order defined by the DAG, with a novel topological batching scheme, which allows to process the message passing operations in parallel. The authors propose theoretical analysis of the proposed methods, to show that it is invariant to node indexing and learns an injective mapping to discriminate between two different graphs. The proposed method is further experimentally validated on multiple tasks involving DAGs, and the results show that it outperforms existing GNNs, including existing methods that can capture DAGs such as D-VAE (encoder). 

The reviewers were unanimously positive about the paper. All reviewers find the performance improvements and time-efficiency obtained with the proposed method to be satisfactory or promising, and one of the reviewers (R4) mentions that the tackled problem is important and the paper is well-written. However, there were concerns regarding insufficient explanations, missing ablation studies, and missing details of some parts of the proposed method. Yet, most of the issues have been satisfactorily addressed during the interactive discussion period. I agree with the reviewers that the paper is tackling an important problem, find the paper well-written, and the proposed DAGNN as practically useful. Thus I recommend an acceptance. 

However, the contributions of the proposed work over D-VAE, which also deals with DAGs, should be better described, as also noted by R2. The DAGNN uses attention, and can stack multiple layers as it is a more general GNN framework while D-VAE is a generative model, but these seem like incremental differences over D-VAE, and it is not clear which contributes to DAGNN’s superior performance over D-VAE. Topological batching is a clear advantage of DAGNN over D-VAE, but the experimental results showing the advantage of it over D-VAE’s sequential training was missing in the original paper (while it was added later to the appendix). I suggest the authors to introduce D-VAE in the introduction, acknowledge that it also tackles DAGs, and clearly describe how the proposed method differs from D-VAE encoder in a separate section. Also, there needs to be an analysis on why the proposed DAGNN outperforms D-VAE, as well as time-efficiency comparison with the original D-VAE in the main text. 

Decision: Accept (Poster)