Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper aims to construct an architecture for Knowledge distillation from GNN based methology to MLP for link Prediction task in specific.
As GNNs are more powerful and more accurate than vanilla MLP for link Prediction task primarily due to their ability to aggregate neighbot good information, these results in more model complexity and computational cost. Hence GNNs for link Prediction tasks can't be deployed directly in a production setting. Whereas MLPs had relatively less computational cost and complexity but don't take into consideration neighborhood information which is curcial for link Prediction tasks. Hence the authors propose a knowledge distillation architecture from teacher GNN to student MLP which utilises the advantages of accuracy of GNNs as well as less model complexity of MLPs for a production setting. 

strength_and_weaknesses: Strengths:

+ Good writing
+ New application of knowledge distillation 

Weaknesses

- Some parts of the experiments are not clear
- A clear motivation is lacking
- No readme file in the code--- this seriously constrains the reproducibility

clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is clearly written except some parts (see below). The paper seems to apply KD method directly into link prediction problem without a very clear motivation.
 
Quality: The paper is technically correct. Experiments are well conducted. I did not find any flaw.

Novelty: As mentioned before,  the paper plugs KD into a new task, which seems a new idea--- but I am not sure about the motivation

Reproducibility: No readme file present in the supplementary material. 

summary_of_the_review: Strengths

+ Motivation:  The authors have clearly outlined the motivation setting for such a proposal behind knowledge distillation.

+ Ablation studies confirm that their proposed model outperforms and close to GNN.

+ The have showcases on 2 large scale OG datasets, thereby confirming the effectiveness of their proposed model in large graphs.

+ The authors have  explains well the experimental setup in appendix. Well written. 



Weakness/questions for authors

 - Not clear in experiments what LLMatch metric is used currently.

- Uses logit based matching based on a very old paper. And also doesn't showcase the motivation of proposal of logit based matching. It should be clearly substituted with recent paper evidences.

- The authors have performed experiments in large OGB datasets which are typically used for link Prediction settings. But as the authors have mentioned in the start of the paper for applications in an industrial setting, did they also perform experiments in a realtime production setting like a recommendation system e.t.c. That would have been nice to have. An analysis on a large scale real production environment(rather than simulation) would tell much more about the utility of proposed method. If there are limitations, that should be clearly specified.

- Inference time graphs are plotted, however very less conclusive analysis done on much of a significant time change does KD provides over GNN w.r.t accuracy.

- Each of the experiment tables should also have clear notation what values they are showing. Is it showing HITs of AUC. That should specified under each table. Some tables don't have titles or caption ( example ablation study)
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes to distill GNN knowledge for link prediction into MLPs via distribution and rank based losses as opposed to prior work which only uses logit-matching or representation matching. The authors test their method against these prior knowledge distillation methods as well as stand-alone MLPs and the original teacher GNN. The authors further observe the difficulty with regards to cold-start nodes and observe significant performance improvements compared to some scaling GNN methods.
strength_and_weaknesses: Strengths:
 - The paper is well written and clear.
 - The experiments and models are described in a reproducible manner.
 - The proposed production setting and analysis of cold-start nodes is interesting and valuable to recommender systems.
 - The proposed method seems sensible.

Weaknesses:
 - The method is significantly outperformed on the only large link prediction graph provided (OGB-Citation2 Tab.1). This is problematic because the purpose of the method is to work on large-scale graphs, where GNNs might be too expensive to run. 
 - The comparison against GNN methods on cold-start nodes is not fair as known GNN techniques exist that might help in this setting (see [1], where a fully adjacent layer is proposed).
 - The evaluation method leaves me with some doubt as I think it is biased (if my understanding is correct) due to no ranking against all nodes in the graph (as is standard in KG link prediction). This is an unrealistic test setting that is also biased.
 - Knowledge graph completion is an important link prediction task that deserves to be mentioned in the introduction and related work, with a brief explanation why this setting was ignored.
 - The GNN scaling method comparison is a bit weak in that some key methods have not been considered as well as only considering inference time, but not performance. There will always be a trade-off. In general, the paper is too quick to assume that knowledge distillation to MLPs is needed and cannot be dealt with by accelerating GNNs, a point that needs further evidence before it can be accepted.

[1] https://arxiv.org/pdf/2006.05205.pdf
clarity,_quality,_novelty_and_reproducibility: Clarity:
 - Very good

Reproducibility:
 - Excellent

Quality:
 - Good

Novelty:
 - Medium
summary_of_the_review: The paper has excellent scientific communication and reproducibility. I want to highlight this as I believe papers in the field should be held to higher standards in this regard. So I would like to commend the authors on these points. The method is well motivated, but not technically sophisticated.

In my mind there is three barriers to acceptance:
1. I understand that there is a lack of large-scale homogeneous link prediction datasets out there. However, given the purpose of the knowledge distillation is to be applied on large scale graphs the performance in Table 1 on OGB-Citation2 is problematic. For acceptance, I would need to see a convincing analysis that shows that this is not due to the size of the dataset (the method works well on smaller citation graphs,e.g. Citeseer), but rather a peculariaty of this particular dataset. For instance, currently it is not unreasonable to hypothesis that MLPs will struggle to compress the information of many node comparisons efficiently, thus harming performance as the training graph gets larger, which would be a fundamental problem.
2. The argument that MLPs are better suited to cold-start nodes is interesting and gives this direction additional value. However, currently I  believe the comparison in table 3, needs adaption to be convincing. There are several GNN architectures that have proposed that do not only rely on the connectivity of the graph. The simplest example is [1], where a fully adjacent layer is used at the end, if the MLP can still out-perform such a GNN architecture, then I am willing to accept that KD into MLPs might be key for recommender systems, where cold-start nodes will indeed be common.
3. The comparison against GNN scaling methods needs a bit of improvement, namely I would like to see a discussion of the trade-off of inference time on the large-scale datasets (such as OGB-Citation2) and the final performance (as we can always predict faster while doing so less accurately). As well as including some more recent methods such as [2].

I appreciate that this is a lot of work for a rebuttal phase, but otherwise the paper remains too unconvincing for such a top-level conference. Should the authors be able to address the three points above, I am willing to raise my score to acceptance of the paper.

EDIT: The authors rebuttal has addressed my concerns. I have thus raised my score for the paper to be accepted.

[1] https://arxiv.org/pdf/2006.05205.pdf
[2] http://proceedings.mlr.press/v139/fey21a/fey21a.pdf
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a knowledge distillation framework for link prediction, Linkless Link Prediction (LLP). Unlike simple knowledge distillation methods that match outputs or representations of models, LLP distills relational knowledge of each link to the student MLP using proposed rank-based and distribution-based matching. These two matching strategies enable the student model to learn relational knowledge about context nodes. Experiments conducted on 9 benchmarks show the effectiveness and efficiency of the proposed framework.
strength_and_weaknesses: Strengths

- The proposed KD strategy for link prediction is interesting and more effective than the previously proposed KD strategy for node-level and graph-level tasks.

- The proposed method shows a faster inference time and even better performance than GNNs.
- The proposed method can be used meaningfully in a real-world scenario by showing excellent performance in cold start nodes, which have relatively few connections.

Weaknesses

- A lack of technical contribution. As mentioned by the author, from the perspective that link prediction is considered as a ranking task, the proposed method for knowledge distillation (margin-based ranking loss and distribution-based matching loss) were already proposed in RankDistill [1] which proposes knowledge distillation for the ranking task. 

- Knowledge Distillation becomes more significant as the scale of the graph increases due to the high latency of GNN, but the huge performance degradation in the largest scale graph (ogbl-citation2) raises questions about the necessity of KD. I wonder if it is a special case in ogbl-citation2 or if the performance drop of KD usually occurs in large-scale graphs.

[1] Reddi, Sashank, et al. "Rankdistil: Knowledge distillation for ranking." International Conference on Artificial Intelligence and Statistics. PMLR, 2021.
clarity,_quality,_novelty_and_reproducibility: The presentation of the paper is clear and easy to follow, but the novelty is limited in terms of the technical contribution. See the above section on strengths and weaknesses.

summary_of_the_review: This paper presents a knowledge distillation framework for link prediction. The presentation of the paper is clear and easy to follow, and the motivation is clear, but the technical novelty is limited.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies the knowledge distillation for graph link prediction. It proposes two novel loss functions based on ranking and distribution matching to distill knowledge from GNNs to MLPs. The motivation of using MLPs instead of GNNs is clear: in the production setting, GNNs cause severe delays due to neighbor explosion and complex computations. I agree with this motivation and am happy to see such a paper studying this problem. However, the following reasons prevent it from being accepted in the current form: 1) The performance drop from GNNs to the proposed LLP is too large on OGB datasets (especially on ogb-citation2, where the MRR drops from 82 to 53). This raises the concern of whether we still want to trust an MLP distilled from GNN, even it is much faster. I understand that on small datasets LLP even outperforms GNNs, but results on these datasets are far less convincing than those on OGB. 2) I have a strong concern on whether an MLP can really capture the important structural information learned by GNNs based only on the node features input. No matter how you modify the loss functions to match the outputs of GNN and MLP, the MLP does not take any structure as input---then it is impossible for MLP to capture similar structural features to those learned by a GNN, such as number of overlapped neighbors or distance between two nodes. This is a fundamental constraint of MLP. A possible solution is to let MLP additionally take some 1-hop neighbor statistics as input. 3) The technical contribution is limited. Only two heuristic matching losses are proposed without in-depth analysis. 4) Even before distillation, the GNN performance is far from those leading methods in the OGB leaderboard. This even worsens the performance after distillation. 

Overall, I like the studied problem but the above shortcomings prevent me from accepting the paper now. I encourage the authors to adequately address the above problems in their revision, which will make the work more solid and convincing.
justification_for_why_not_higher_score: I carefully read the paper and the reviews/responses. The reasons are listed in the meta-review.
Decision: Reject