Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: 
Summary:

    The authors propose a new gating based recurrent graph attention networks for multi-relational graphs to capture long-range neighbor dependencies. The authors provide an interesting analysis of current gated GNN models (in the appendix + Figure 3) in light of their ability to capture long-range dependencies in graphs. Experimental results are reported for node classification with two synthetic datasets and two real-world datasets.  

——

Pros:

	(i) The work addresses an important problem of long-range dependencies with conventional graph neural networks. The work has an interesting backpropagation based explanation of the issue in the Appendix and Figure 3 provides a good illustration of the same. 
	(ii) The synthetic experiments are interesting and helpful to evaluate models for long-range dependencies 
	(iii) Impressive results on synthetic experiments 
	
——
Major Concerns:

	(i) Though the problem of interest is explained well in the appendix from the view of Gated GNNs. The paper's main section lacks a clear explanation of the problem; especially, there is no explanation of what it means by the horizontal vanishing gradient problem. 

	(ii) The model motivation is not clearly written in the main paper — why model both the inputs as states of a gated recurrent network. Eqn: 10 is not straightforwardly clear why the redundant combination of information is helpful. Overall, the primary contribution discussed in section 3.3.1 needs to be expanded and explained in contrast to GRU update to capture long-distant neighbor information. A similar backpropagation analysis for the proposed model will help us understand the power of the proposed model. Also, why is the model called 'symmetrically' GRU? 

	(iii) The r_x gate in Eqn: 9 is similar to the forget gates in LSTMs. How does it compare with the GraphLSTM updates? 

	(iv) Residual and Dense connections are not discussed and experimented. Like in JK-Nets, dense connections can be added to the each of the GCNs pertaining to different relations or can be added for the combined layer output from all relations. The baselines with highway connections need to be evaluated. 

	(v) There are too many components or design choices proposed/made, but there are no ablation studies on all the components. (a) Gated relational message, (b)Redundant usage of relational information for attention keys, (c) Concat-Ensemble of value vectors (d) symmetric GRU. Even in the current set of variations, would like to see, GR-GAT(SGRU) - value transformation, GR-GAT(SGRU) 


	(vi) Results are reported only for two real-world datasets. MUTAG and BGS can be added. No real-world datasets or tasks with potential long-range dependencies are experimented. Ex: Molecular graphs (MUTAG, ZINC, etc. ) and Protein graphs.

	(vii) Tree Max:
		- A height wise results+analysis of node-level task would be interesting.
		It is especially hard to understand why the GR-GAT(Ident)- value transform performs poorly. On the same note, how does GR-GAT(SGRU) - value transformation perform?
		- Why does GR-GAT(GRU) perform way poorer than GR-GAT(Ident) ? 
		- The text about model variations mentions GATE and SGRU to be the same but in Table: 2, there is both GR-GAT (SGRU) and GR-GAT(Gate). 
	
	(viii) WGCN results on AIFB ?

	(ix) Only node classification results reported, in which case the scope of the model and results studied should be explicitly mentioned to be restricted to node classification if that is the intent. If that is not the case, additional link prediction results like RGCN or other tasks should be reported. 

		Also, Additional results on single-relational homogeneous graphs can help disentangle the effect of the proposed relational module from the main contribution, the gating mechanism.
	

Overall Recommendation:

       The paper has interesting content, but the paper is not well organized and motivated well. There is sufficient merit if the analysis could be reformulated and generalized for all message-passing models — the horizontal long-term dependency. The backpropagation based vanishing gradient issue discussed is limited to Gated GNNs alone. It is essential to discuss residual and dense connections too. On the experimental front, adding more real-world datasets would strengthen the paper. 
——
Post Rebuttal 
Increased the score from 6 to 7. 
I would have strongly recommended the paper if it had more real-world datasets. 
review_title: Interesting analysis but writing needs improvement + Need more real world datasets

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors propose Gated Relational Graph Attention Nets (GR-GAT), a set of modifications to the GAT architecture in order to make them stronger under long-range relational reasoning, evaluating on various hand-crafted benchmarks as well as a real-world dataset previously explored in the area.

The GR-GAT tackles all aspects of the graph neural network pipeline:
- Message function that takes into account the edge type embedding, and features explicit gating over the sender node's past features, and usage of the CELU activation;
- Multi-head attentional aggregator which splits the value vectors into chunks, rather than replicating them;
- Symmetric approach to the GRU update rule, which factors both inputs into the gating.

The three approaches are potentially meaningful for ameliorating various issues with long-range reasoning, such as overfitting or vanishing gradients, and I think the methodology from this paper could be useful to GNN practitioners. However, the paper's current presentation and motivation does not feel suitable for a venue like ICLR, in my opinion.

My main concern is that the authors do not properly motivate most of their design choices, or properly ground them in a particular issue with (R)GATs or GGNNs. Many choices (to name a few: the gating message, the CELU activation, or the splitting value vector) are name-dropped in the paper, without properly explaining their significance in the architecture. The ablation studies and experimental discussions are also lackluster in this sense, in my opinion: the authors' discussion doesn't go much further than "the results demonstrate method X outperforms method Y", not managing to provide deeper insight into any of the design choices.

The best-motivated addition---the SGRU update---is in my opinion a pretty neat idea and should be more highlighted and motivated in the paper, perhaps with experiments specially designed to show its benefits. Gated attention as well as split messages have already been featured or attempted (in some form) by prior work: see, for example, the GaAN model from Zhang et al. (UAI 2018), hence the novelty of such proposals, in isolation, is limited.

While the GR-GAT model achieves some strong outcomes on synthetic benchmarks, the strength of these results is, in my opinion, insufficient to carry the weight of the paper, especially for a venue like ICLR. Especially considering that these graphs are designed with rather simple edge types (such as edge direction), the value of these results when transferred to real-world heterogeneous graphs is unclear. Further, the max-tree benchmark, where most of the interesting ablations are shown, is a bit concerning: it appears as if it could be quite easy to solve if using max-aggregation rather than attention (see Richter and Wattenhofer's "Normalized Attention Without Probability Cage" for some motivation on this), and maybe in this sense would not require any of the "heavy artillery" proposed here. 

As mentioned above, it would be interesting to create more targeted and diverse synthetic benchmarks, perhaps to specifically battle-test the SGRU component.

========= Post-rebuttal update:
I thank the authors for carefully addressing my comments, as well as other reviewers'.

Ultimately, this is a nice paper with a novel recurrent component, and I can see how it could perform well in practice.
However, the lack of stronger real-world experimentation (on datasets such as OGB) unfortunately renders the contribution insufficient -- the synthetic benchmarks being insufficient on their own to pull the weight of the paper.

I retain my score, but encourage the authors to carefully revise and resubmit for the next venue should the paper be rejected.
review_title: Promising long-range tweaks to the relational GAT, but lacks clear motivation

final_score: rating: 5: Marginally below acceptance threshold
review_content: #####Summary#####

This paper proposes a new GNN model (GR-GAT) for multi-relational graphs. The proposed method has better ability of capturing the long-range information. Essentially, the proposed GR-GAT is modified from GAT so that it can apply to the multi-relational graphs. Since the modifications are common and frequently used techniques, the novelty of this work is not enough. Also, why these modifications can help to capture long-range information is not well explained in this paper. Overall, this work is ok but not good enough for ICLR.

#####Pros#####

(1) The experiments on synthetic are well designed and can show the power of the proposed GR-GAT. 

(2) This paper studies a meaningful and challenging problem; that is capturing long-range information using GNNs.

#####Cons#####

(1) The novelty of the proposed method is not enough. Based on the description in Section 3, the proposed model is basically under the message passing framework (Eq (1)) and the concrete implementations of the three functions included in the message passing framework is not novel. For example, parameterizing message function using vector, aggregating neighborhood using attention, and updating using GRU are all popular used techniques in GNNs. Overall, this method is likely to be a combination of GAT and some basic techniques for multi-relational graphs.

(2) More importantly, the motivation is not explained clearly. The current version did not well explain why the proposed model can help to model long-range dependencies in Section 3. 

(3)  To show the ability of capturing long-range dependencies, the experiments are only conducted on small synthetic datasets and specific defined tasks. It is not enough to show the ability of capturing long-range information. More experiments and comparisons on large real-world datasets should be considered.

#####Suggestions for improvement#####

(1)	I think the main issue of the current version is that our readers cannot tell the contributions of this paper from Section 3. We cannot find what is the key proposal of this paper and how this proposal can help to capture long-range information. I think if this point can be improved, the novelty and motivation of this paper can be clearer.


######

Update: After looking at the revised version, I would like to raise my score to 5 since the motivation is clearer.

review_title: Recommendation to reject

final_score: rating: 2: Strong rejection
review_content: ## Summary
This paper presents a graph attention architecture that captures long-range interactions. The novelties in the architectures are (1) vector-based parameterization of edge type in modeling message, (2) slight modification of graph attention (Section 3.2), and (3) GRU-based node update function. The experiments are primarily on synthetic tasks. However, it is unclear if modeling such long-range interaction is useful in real tasks. The paper fails to demonstrate convincing results on the real tasks of entity classification in knowledge graphs.

## Pros
1. Detailed architecture explanation.
2. Good performance on synthetic tasks.
3. Careful design of synthetic tasks.

## Cons:
1. The novelty of architecture is limited as detailed below.
- Vector-based parameterization of edge type in the relational graph has been commonly adopted in GNNs for molecular graphs (e.g., Eq (1) in https://arxiv.org/pdf/1709.04555.pdf), and not novel. 
- The modifications of graph attention architecture are rather minor (removing a single linear transformation, adding edge type embedding in the key of the attention mechanism). 
- GRU-based node update function is not empirically shown to be beneficial although being highly complicated.
2. Experiments are largely synthetic, and no convincing results are provided for the real datasets on entity classification in knowledge graphs. It is unclear if modeling long-range interaction is useful in practice. One domain long-range interaction could be useful is molecule classification, where you can treat molecular graphs as multi-relational graphs and those graphs tend to have large graph diameters. Many datasets are readily available [here](https://ogb.stanford.edu/docs/graphprop/).
3. Details of the real knowledge graph datasets (AIFB and AM) are not provided in the main texts.

review_title: Limited architecture novelty, no convincing performance on real tasks.

Meta-review: This paper proposes a GNN architecture for multi-relational data to better address long-range dependencies in graphs. The proposed GR-GAT model is a variant of graph attention networks (GAT) with, among other modifications, vector-based edge type embeddings and GRU-type updates. Results are presented on AIFB, AM, and on synthetic benchmarks.

The reviewers agreed that this is an interesting contribution and that the results on the chosen synthetic benchmarks are insightful, but that experimental evaluation on real data and overall motivation of the architecture is lacking. In the rebuttal period, the authors have improved the writing and strengthened the motivation of the paper. However, given the limited amount of time, the authors were not able to sufficiently address the lack of experimental validation on real data (beyond AIFB & AM). I am inclined to agree with the reviewers that this paper needs significantly more work on the experimental evaluation, the overall presentation needs to be refined and it needs to more carefully analyse the effect of each individual architectural modification to meet the bar for acceptance.

Decision: Reject