Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose an AutoGNN model that introduces more design dimensions in both inter- and intro-layer. It is a decent attempt for authors to take into consideration the edge embedding in MPNN and adopt a stochastic differentiable search algorithm, making the model be able to better handle the link prediction problem and get more efficiently optimized. 
main_review: Strengths:

S1) The paper is well-written and organized, and the proposed model (AutoGEL) is technically sound. In AutoGEL, more design spaces are introduced to be learned both with intra- and inter-layer, which would intuitively make the trained model perform better than manually designed ones.

S2) In AutoGEL authors explicitly learns the edge embedding in the message passing framework to model the complex link information, making AutoGNN-like model be generalized to the LP task.

S3) Adopting differentiable search algorithm to optimize the GNN architecture parameters makes the model be able to be trained more efficiently and stably.

S4) Extensive experiments are well organized to show the effectiveness and competitiveness of AutoGEL.


Weaknesses:

W1) As authors mentioned in Section 4.2, the more the design dimensions, the longer the model will run, which indeed would be the case. Although the running times are still rather low, explicitly showing the scalable would be worth including.

W2) There are still critical structure components needing to be manually designed such as the number of layers of the model, which would have significant impact on the performance of model even after GNN operators being decided.

W3) The problem is not as novel, but Table 1 did help frame and clarify the distinction of this work compared to other recent related autoGNN methods.  


Other Comments/Questions/Typos:

1) In the paper, the authors attempt to add edge embedding in MPNN to handle the LP task and also improve the performance of the model on other tasks. However, in the experiments section, it seems in the specific models that are used in node classification and graph classification are missing the edge embedding component (as demonstrated in Figures 4 and 5 in the appendix).

2) In the ablation study, it seems better to add some experiments for testing the effectiveness of the edge embedding. (i.e., using model without the edge embedding component on LP tasks).

3) "The empirical comparison on the node classification and graph classification task is shown in Table 7." is mentioned, but there is no Table 7 and it also does not correspond to Table 7 in the supplemental file.

4) For selecting an operation in an operator, would it be necessary for the operators to converge to a one-hot vector? What if not and just take the real-valued operator vector as a mixed strategy?

5) How does the edge embedding matrix (i.e., h_e^x) update?

6) I was thinking more design dimensions can be similarly regarded as more parameters (but in terms of structure not weight), thus it would somewhat make it easier for overfitting.

Please note that I have gone through the author response and other reviews/comments, but respectfully keep my score of 6 (i.e., unchanged).

limitations_and_societal_impact: There does not appear to be any negative societal impact of this work. In fact, it helps alleviates some of the impracticalness of some prior autoGNN work regarding the computational running time needed for slight performance improvements (when thinking from an environmental standpoint).
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper introduces an automated graph neural network model with explicit link information. The model is tested on link prediction task (both homogeneous and heterogenous graphs) and node classification task. It shows improvement compared with the baseline models selected.
main_review: The paper mostly focuses on explaining how it incorporates the link information (edges) and it can generate edge embeddings so it would be suitable for link prediction in multi-relational graphs, such as knowledge graphs. The proposed method is primarily based on existing approaches with a few incremental tweaks and combining different components from different approaches (e.g., combining intra-layer message passing and inter-layer message passing). However, in the experiments, it seems that the model has yet being compared with the most up-to-date models in the link prediction task with FB15k-237 dataset (https://paperswithcode.com/sota/link-prediction-on-fb15k-237). So the overall contribution is unclear. Although link prediction is a hard task considering that there are now a lot of high baseline models to compare, the selling point of this paper seems to be using link information to improve upon link prediction task. The paper does not seem to convince the readers on this aspect. 
limitations_and_societal_impact: I don't see any potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes an automated Graph Neural Network (GNN) framework for link prediction tasks on graphs.
The experiment results suggest that the architecture search is useful.
main_review: This paper performs a GNN architecture search for link prediction tasks. 
The proposed search space is specialized for link prediction tasks.
The experiment results suggest that the architecture search is useful, however it seems that the performance gain is marginal.
In terms of novelty, the framework for the proposed search space follows paper [1], with some specialization on link prediction tasks.
Overall, the paper targets a useful problem in the graph learning community. 
My main concerns are over (1) if the technical novelty is enough; (2) if the experimental results are sufficient.

Additionally, I have these specific comments for the authors to stress.

(1) The search algorithm being used is very vague (Section 3.2).
The paper claims that "AutoGEL adopts its stochastic differentiable search algorithm".
However, for discrete architectural choice, e.g., sum, mean, max for AGG, it is unclear how that is differentiable.
The authors should elaborate more on the search algorithm; otherwise, the correctness/novelty of the algorithm cannot be evaluated.

(2) In paper [1], in fact, they have done an architecture search for link prediction tasks. That part of the results is shown in the Appendix (https://arxiv.org/pdf/2011.08843.pdf). The authors should correct the descriptions in Table 1 and corresponding discussions. Additionally, the author may want to emphasize more on the specialty of the proposed search space, compared to a standard GNN search space.

(3) Regarding the limitations of the paper, the authors mention that " In practice, GLP models run longer than Geometric and Bilinear models, which leads to the difficulty of tuning hyper-parameters". I think this is a non-trivial limitation. How difficult it is to use AutoGLP? 

(4) I think the authors should report one important metric, that is how many models have been searched to get each of the experiment result. The AutoML approach naturally spends much more computing resources, therefore directly comparing the numbers is not meaningful.

[1] Jiaxuan You, Zhitao Ying, and Jure Leskovec. Design space for graph neural networks. Advances in Neural Information Processing Systems, 33, 2020.
limitations_and_societal_impact: No.
Regarding the limitations of the paper, the authors mention that " In practice, GLP models run longer than Geometric and Bilinear models, which leads to the difficulty of tuning hyper-parameters". I think this is a non-trivial limitation. How difficult it is to use AutoGLP?
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a neural architecture search method for using GNNs for link prediction. The proposed method explicitly models edge information in the search process, and proposes several novel design and search space choices for AutoGNN. Through a DARTS-like differentiable search method, the proposed method achieves state-of-the-art performance on various link prediction and node/graph classification tasks.
main_review: Originality: Link prediction on graphs is an important task yet receives less attention than node/graph classification. This paper extends AutoGNN to link prediction and obtains SoTA results, which is highly appreciated for the community. The related work is sufficient, especially for including some latest arxiv papers on GNN link prediction.

Quality: The techincal part of the paper is ok. It is an application of an existing NAS algorithm SNAS with some novel design of search spaces. Nevertheless, the proposed search space choices are novel and reasonable to me.

Clarity: The paper is easy-to-understand and written in an intuitive way.

Significance: Through extensive experimental results, the proposed AutoGNN method demonstrates SoTA results across a number of tasks (not restricted to link prediction), outperforming manually designed models. This demonstrates the importance for explicit modeling edge information and link prediction in designing GNN architectures.
limitations_and_societal_impact: No. One concern is the search efficiency (in general) for NAS methods. The paper discusses this issue in Section 4.3. Another concern is that the search space does not incorporate different node labeling methods for GNN link prediction, such as distance encoding, which are shown to be crucial for success of GNNs on link prediction. Thirdly, I suggest demonstrating one or two common findings from the searched architectures: what components of a GNN are most important for link prediction?
review_title: null

Meta-review: This paper proposes a new autoGNN model that takes link information into considerations. The model has shown better performance compared to baselines. The reviewers had concerns on novelty and baselines, and suggested new experiments. The authors have done a good job in addressing reviewers’ concerns, which is reflected in the increase of some rating. The concerns regarding to KG link prediction tasks has been well addressed in the rebuttal, although the reviewer didn’t raise their rating. Overall, the paper extends an existing algorithm, by considering link prediction task and modeling edge information. Although the idea is incremental, the execution of this paper is solid. I thus recommend accept.
Decision: Accept (Poster)