Linkless Link Prediction via Relational Distillation
Authors: Zhichun Guo
William Shiao
Shichang Zhang
Yozen Liu
Nitesh Chawla
Neil Shah
Tong Zhao
Venue: Submitted to ICLR 2023 (2023)
Keywords: link prediction
knowledge distillation

Abstract: Graph Neural Networks (GNNs) have been widely used on graph data and have shown exceptional performance in the task of link prediction. Despite their effectiveness, GNNs often suffer from high latency due to non-trivial neighborhood data dependency in practical deployments. To address this issue, researchers have proposed methods based on knowledge distillation (KD) to transfer the knowledge from teacher GNNs to student MLPs, which are known to be efficient even with industrial scale data, and have shown promising results on node classification. Nonetheless, using KD to accelerate link prediction is still unexplored. In this work, we start with exploring two direct analogs of traditional KD for link prediction, i.e., predicted logit-based matching and node representation-based matching. Upon observing direct KD analogs do not perform well for link prediction, we propose a relational KD framework, Linkless Link Prediction (LLP). Unlike simple KD methods that match independent link logits or node representations, LLP distills relational knowledge that is centered around each (anchor) node to the student MLP. Specifically, we propose two matching strategies that complement each other: rank-based matching and distribution-based matching. Extensive experiments demonstrate that LLP boosts the link prediction performance of MLPs with significant margins, and even outperforms the teacher GNNs on 6 out of 9 benchmarks. LLP also achieves a 776.37x speedup in link prediction inference compared to GNNs on the large scale Citation2 dataset. 

Under review as a conference paper at ICLR 2023
LINKLESS LINK PREDICTION VIA RELATIONAL DISTILLATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Graph Neural Networks (GNNs) have been widely used on graph data and have shown exceptional performance in the task of link prediction. Despite their effec- tiveness, GNNs often suffer from high latency due to non-trivial neighborhood data dependency practical deployments. To address this issue, researchers have proposed methods based on knowledge distillation (KD) to transfer the knowl- edge from teacher GNNs to student MLPs, which are known to be efficient even with industrial scale data, and have shown promising results node classification. Nonetheless, using KD to accelerate link prediction is still unexplored. In this work, we start with exploring two direct analogs of traditional KD for link prediction, i.e., predicted logit-based matching and node representation-basee matching. Upon observing direct KD analogs do not perform well for link prediction, we propose a relational KD framework, Linkless Link Prediction (LLP). Unlike simple KD methods that match independent link logits or node representations, LLP distills relational knowledge that is centered around each (anchor) node to the student MLP. Specifically, we propose two matching strategies that complement each other: rank-based matching and distribution bbsed matching. Extensive experiments demonstrate that LLP boosts the link prediction performance of MLPs with signifi- cant margins, and even outperforms the teacher GNNs on 6 out of 9 benchmarks. LLP also achieves a 776.37 x speedup in link prediction inference compared GNNs on the large scale DGB-Citations dataset.
INTRODUCTION
Graph neural networks (GNNs) have been widely used for machine learning on graph-structured data (Kipf& & Welling, 2016a; Hamilton et al., 2017). They have shown significant performance in various applications, such node classification (Veličković et al., 2017; Chen et al., 2020), graph classification (Zhang et al., 2018; Ying et al., 201 2018b), graph generation (You et al., 2018; Shiao & Papalexakis, 2021), and link prediction (Zhang & Chen, 2018).
Of these, link prediction a notably critical problem in the graph machine learning community, which aims to predict the likelihood of any nodes forming link. It has broad practical applications such as knowledge graph completion (Schlichtkrull et al., 2018; Nathani et al., 2019; Vashishth al., 2020), friend recommendation on social platforms (Sankar al., 2021; Tang et al., 2022; Fan et al., 2022) and item recommendation for users on service and commerce platforms (Koren et al., 2009; Ying et al., 2018a; He al., 2020). With the rising popularity of GNNs, state-of-the-art link prediction methods adopt encoder- style models, where encoders are GNNs, and decoders are applied directly on pairs of node representations learned by the GNNs (Kipf & Welling, 2016b; Zhang& & Chen, 2018: 88; Cai & Ji, 2020; Zhao et al., 2022).
The success of GNNs is typically attributed to the explicit use of contextual information from nodes' surrounding neighborhoods (Zhang et al., 2020e). However, this induces heavy reliance on neighborhood fetching and aggregation schemes, which can lead to high time cost training and inference compared to tabular models, such as multi-layer perceptrons (MLPs), especially owing to neighbor explosion (Zhang et al., 2020b; Jia et al., 2020; Zhang et al., 2021b; Zeng et al., 2019). Compared GNNs, MLPs do not require any graph topology information, making them more suitable for new or isolated nodes (c.g., for cold-start settings). but usually resulting in worse general task performance encoders, which also empirically validate Section 4. Nonetheless, having

---

Under review as a conference paper at ICLR 2023
no graph dependency makes the training and inference time for MLPs negligible when comparing with those of GNNs. Thus, in industrial-scall applications where fast real-time inference is required, MLPs are still leading option (Zhang et al., 2021b; Covington et al., 2016; Gholami et al., 2021).
Given these
tradeoffs, several recent works propose to transfer the learned knowledge from GNNs to MLP using knowledge distillation (KD) techniques (Hinton al., 2015; Zhang et al., 2021b; Zheng et al., 2021; Hu et al., 2021), to take advantage of both GNN's performance benefits and MLP's speed benefits. Specifically, in this way, the student MLP can potentially obtaín the graph-context knowledge transferred from the GNN teacher via KD to not only perform better in practice, but also enjoy model latency benefits compared to GNNs, e-g. in production inference settings. However, these works focus on node- or graph-level tasks. Given that KD on link prediction tasks have not been explored, and the massive scope of recommendation systems contexts that are posed as link prediction problems, our work aims to bridge a critical gap. Specifically, we ask:
Can we effectively distill link prediction-relevant knowledge from GNNs to MLPs?
In this work, we focus on exploring, building upon, and proposing cross-model (GNN to MLP) distilla- tion techniques for link prediction settings. We start with exploring two direct KD methods of aligning student and teacher: (i) logit-based matching of predicted link existence probabilities (Hinton et al., 2015), and (ii) representation-bust
matching of the generated latent node representations (Gou et al., 2021). However, empirically observe that neither the logit-based matching nor the representation- based matching are powerful enough to distill sufficient knowledge for the student model to perform well on link prediction tasks. We hypothesize that the reason of these two KD approaches not performing well is that link prediction, unlike node classification, heavily relies on relational graph topological information (Martínez et al., 2016; Zhang & Chen, 2018; Yun et al., 2021; Zhao et al., 2022), which is not well-capture by direct methods.
To address this issue, we propose a relational KD framework, namely LLP: our key intuition is that instead of focusing on matching individual node pairs or node representations, we focus on matching the relationships between each (anchor) node with respect to other (context) nodes in the graph. Given the relational knowledge centered at the anchor node, i.e., the teacher model's predicted link existence probabilities between the anchor node and each context node, LLP distills it to the student model via two matching methods: (i) rank-based matching, and (ii) distribution-based matching. More specifically, rank-based matching equips the student model with a ranking loss over the relative ranks of all context nodes w.r.t the anchor node, preserving crucial ranking information that are directly relevant to downstream link prediction use-cases, c.g. cistr-contettual friend recommendation (Sankar al., 2021; Tang et al., 2022) or item recommendation (Ying et al., 201 8a; He et al., 2020). On the other hand, distribution-
matching equips the student model with the link probability distribution over context nodes, conditioned on the anchor node. Importantly, distribution- complementary to rank-based matching, as it provides auxiliary information about the relative values of the probabilities and magnitudes of differences.
matching is
To comprehensively evaluate the effectiveness of our proposed LLP, we conduct experiments on 9 public benchmarks. In addition to the standard transductive setting for graph tasks, we also design a more realistic setting that mimics realistic (on-line) use-cases for link prediction, which we call the production setting. LLP consistently outperforms stand-alone MLPs by 17.13 points on average under the transductive setting and 12.01 points under the production setting on all the datasets, and matches or outperforms teacher GNNs on 6/9 datasets under the transductive setting. Promisingly, for cold-start nodes, LLP outperforms teacher GNNs and stand-alone MLPs by 25.29 and 9,42 Hits @ 20 on average, respectively. Finally, LLP infers drastically faster than GNNs, e.g. 776.37 x faster on the large-scale OGB OGB-Citte LuB-Citationl dataset.
RELATED WORK AND PRELIMINARIES
We briefly discuss related work and preliminaries relevant to contextualizing our methods and contributions. Due to space limit, we defer more related work to Appendix A.
Notation. Let G (V,E) denote an undirected graph, where V denotes the set of N nodes and 3 C V x V denotes the set of observed links. A E (0, NJNN denotes the adjacency matrix, where Ai.j = 1 if exists an edge ei.j in ε and otherwise. Let the matrix of node features be denoted by X R NFF, where each a; is the F-dim raw feature vector of node i. Given both & and A have
2

---

Under review as a conference paper at ICLR 2023
the validation and test links masked off for link prediction,
a. (different from A.,) to denote the true label of link existence of nodes i and j. which may or may not be visible during training depending the setting. We use m (V x V) \ E to denote the no-edge node pairs that are used as negative samples during model training. We denote node representations by H E RRNDD, where D is the hidden dimension. In KD context with multiple models, we use h; and h; to denote node i's representations learned by the teacher and student models, respectively. Similarly, we use yi.j and Dt.j to denote the predictions for di,j by the teacher and the student models, respectively.
Link Prediction with GNNs. In this work, we take the commonly used encoder-decoder framework for the link prediction task (Kipf& & Welling, 20:6;; Berg et al., 2017; Schlichtkrull et al., 2018; Ying et al., 201 8a; Davidson et al., 2018: Zhu et al., 2021; Yun et al., 2021; Zhao et al., 2022), where the GNN-based encoder learns node representations and the decoder predicts link existence probabilities. Most GNNs operate under the message-passing framework, where the model iteratively updates each node i's representation h; by fetching its neighbors' information. That is, the node's representation in the 1-th layer is learned with an aggregation operation and an update operation:
1 = UPDATE (h!-1, AGGREGATE: (On les, E EE))),
(
where AGGREGATE combines or pools local neighbor features, UPDATE is a learnable transformation, and h2 = xy. The link prediction decoder takes the node representations from the last layer, i.e., h; for i E V, to predict the probability of a link between node pair and j:
yi = a (DEcODER(h,; hz)),
where a denotes Sigmoid operation. In this work, following most state-of-the-ard link prediction literature (Zhang et al., 202 Tsitsulin et al., 2018; Zhao et al., 2022; Wang et al., 2021). 1). we take the Hadamard product followed by a MLP as the link prediction DECODER for all methods.
Knowledge Distillation for GNNs, Knowledge distillation (KD) (Hinton et al., 2015) aims to transfer the knowledge from high-capacity and highly accurate teacher model to light-weight student model, and is typically employed in great promise in significantly reducing model complexity, while sometimes barely (or not) sacrificing task performance (Furlanello et al., 201 8; Park et al., 2019). As GNNs are known to be slow due to neighbor aggregation induced by data dependency, graph-based KD methods (Zhang et al., 2021b; Zheng et al., 2021) usually distill GNNs onto MLPs, which are commonly used in large-scale industrial applications due to their significantly improved efficiency and scalability. For example, Zheng et al. (2021) proposed KD>based framework to rediscover the missing graph structure information for MLPs, which improves the models' generalization of node classification task on cold-start nodes. Existing KD methods on graph data mainly focus node-level (Zheng et al., 2021; Zhang et al., 202 lb; Tian et al., 2022) and graph-level tasks (Ma & Mei, 2019; Zhang et al., 2020c; Deng& & Zhang, 2021; Joshi et al., 2021), leaving KD for link prediction yet unexplored. Our work focuses on distilling link prediction-relee relevant information from the GNN teacher to an MLP student, and investigates various KD strategies to align student and teacher predictions. Specifically, denoting the node representations for nodes and j learned by the student MLP as h, and h,, the link existence prediction by the student model can then be written as Jt.j = o (DECODER fh., h2)).
settings. KD methods have shown
CROSS-MODEL KNOWLEDGE DISTILLATION FOR LINK PREDICTION
In this section, we propose and discuss several approaches to distill knowledge from a teacher GNN to student MLP cross-model fashion, for the purpose of link prediction. In all cases, we aim to supervise the student MLP with artifacts produced by the GNN teacher, in addition to any available training labels (a.., w.l.o.g.) about link existence. We start by adapting two direct knowledge distillation (KD) methods: logit-matching and representation-matte
on link prediction tasks; we call these methods direct because they involve directly matching sample-wise predictions between teacher and student. Next, we motivate and introduce our proposed relational KD framework, LLP, with two matching strategies to distill additional topology-relate, structural information to the student. We call these methods relational because they call for preservation of relationships across samples between teacher and student (Park et al., 2019). We next discuss our proposals in more detail, which are also summarized in Figure 1.
3

---

Under review as a conference paper at ICLR 2023
Teacher GNN
Dec.
Rank
Distribution
Relational
Sample O + For
Graph Structure
Direct
fast
KD: LLP
KD
Rank
Distribution
Node Attr.
Student MLP
Dec.
Figure I: We explore KD methods for link prediction, which distill knowledge from a Teacher GNN to Student MLP encoder, each with their own decoder. We start by exploring direct KD methods: irvesentation-matcchi
and logit-matching. Upon observing their drawbacks not being able to distill relational information, we further propose a relational KD framework: LLP, which equips the student model with knowledge of each (anchor) node's relationships with other (context) nodes, via our proposed rank-based matching and iestribttion-based matching objectives.
3.1 DIRECT METHODS
Logit-matching one straightforward strategy to distill knowledge from the teacher to the student, where it directly aims to teach the student to generalize the teacher does the downstream task. It (Hinton et al., 2015) was originally proposed some time ago, but is still one of the most widely used KD methods in various tasks (Furlanello al., 2018; Yang et al., 2020b; Yan et al., 2020). Several works (Phuong & Lampert; Ji & Zhu, 2020) theoretically analyzed its effectiveness. Moreover, it had also been proved to be effective for knowledge transfer graph data (Yan et al., 2020; Yang et al., 2021; Zhang et al., 2021b) in recent years. For example, Zhang et al. (2021b) the soft logits generated by the teacher GNNs help supervise the student MLP and achieved strong performance on node classification tasks. In similar vein, we generate the soft score Yi. for the node pair or candidate edge (i. with the teacher GNN model, and train the student to match its prediction ji.j on this target:
LIM Σ
ACaup(ii. (ng) + (1 A) (Di.g Vi.j).
(3)
(KJ)EEUE-
where Lsup is the supervised link prediction loss (e.g., binary cross entropy) that directly trains the student model, Lmatch the loss for matching the student's prediction with the teacher's prediction, and À is a thper-parameter that mediates the importance of the ground-truth -tunt labels and gegttimatching signals. Note that multiple implementation choices exist for Lmatch- For example, mean- eqean-suurred error (MSE), Kullback-Leite (KI.) divergence, or cosine similarity. In the experiments, we opt for the empirical best choice for fair comparison across methods.
is another direct distillation method in which we aim to align the student's learned latent node embedding space with the teacher's. s. As this KD training signal only optimizes the encoder part of the student model, it must be used in conjunction with Loup so that the student decoder receives a gradient and can also optimized:
Representation-mattihn
CRM
Σ XErup(hh +(1 (1-A) A Ematch (fa, h4).
(L.))SUS-
Unlike logit-matching, representation- suareennttioonmmatciinng involves directly aligning node-level artifacts, which similar to object representation matching in computer vision (Romero et al., 2014; Kim et al., 2018; Wang al., 2020b; Chen et al., 2021a).
3.2 LINK PREDICTION WITH RELATIONAL DISTILLATION
The above direct methods ask the student model to directly match node-level link- level artifacts. However, one might ask: are matching these sufficient for link prediction tasks? This is especially relevant considering that most link prediction applications involve tasks where
Motivation.
4

---

Under review as a conference paper at ICLR 2023
ranking target nodes with respect to source, or anchor node, is the task of interest, i.e. ranking relevant candidate users or items with respect to a seed user (Huang et al., 2005; Trouillon et al., 2016). In other words, these contexts involve reasoning over multiple relations or link-level samples simultaneously, suggesting that matching across these relations could be more aligned with the target link prediction task, compared to the direct node-level or link-level methods.
Furthermore, several works (Zhang & Chen, 2018; Yun et al., 2021) suggest that graph structure information is critically important for link prediction tasks. For example, heuristic link prediction methods commonly show competitive performance compared to GNNs (Zhang & Chen, 2018) and have long-served as a comerstone for accurate link prediction even prior to neural graph methods (Martinez et al., 2016). Most heuristic methods measure the score of the target node pairs only based on the graph structure information (Barabási & Albert, 1999; Brin & Page, 2012), such as common neighbors and shortest path. In addition, several recent works (Zhang & Chen, 2017; 2018; Li et al., 2020; Zhao et al., 2022) also show that enclosing topology information such as local subgraph, distances with anchor nodes, augmented links can largely improve GNNs performance on link-level tasks. Observing that most successful methods in link prediction involve using relational information other than just the two nodes in question, also adopt this intuition in the distillation context, and propose our relational KD for link prediction. We elaborate next.
3.3 PROPOSED FRAMEWORK: LINKLESS LINK PREDICTION
In accordance with our intuition regarding preservation of relational knowledge, we propose novel relational distillation framework, called Linkless Link Prediction, or LLP. Instead of focusing on matching individual node pair scores or node representations, LLP focuses on distilling knowledge about the relationships of each node to other nodes in the graph; we call the former node an anchor node, and the latter nodes context nodes. For each node in the graph, when it serves the anchor node, we aim to equip the student MLP model with knowledge of its relationships with a set of context nodes. In practice, each node can serve as both an anchor node, as well as a context node (for other anchor nodes).
Let denote the anchor node and C. denote the corresponding set of context nodes of e. We denote the teacher model's 's predicted probabilities of v and each node in C, as V {Ve,li E C. Similarly, denote the student model's 's predictions on those as } ( i E ). To effectively distill the relational knowledge from Vo to Dd., we proposed two relational matching objectives to train LLP: rank-based matching and distribution
matching, which we introduce next.
Rank-based Matching. As aforementioned in Section 3.2, link prediction is often considered a ranking task, requiring the model to rank relevant candidates w.r.t. seed node, e.g. in user-item graph setting, the predictor must rank over a set of candidate items from the perspective of a user. Thus, we reason that unlike matching individual and independent logits, matching the ranking induced by the teacher can more straightforwardly teach the student relational knowledge about context nodes w.r.t. the anchor node, e.g. for specific user, item A should be ranked higher than item C, which should be ranked higher than item B. To adopt this rank-based intuition into a training objective, we adopt a modified margin-base based ranking loss that trains the student with the rank of the logits from the teacher GNN. Specifically, we enumerate all pairs of predicted probabilities in De and supervise with the corresponding pairs in Vo. That is,
LLL R
Σ
Σ
max(O, -T (Dv,i - (0.)) + +),
(5)
VEV
1, if Ye.i Yo.j > 6;
where
1, if Yo.i Yo.j < -6;
0, otherwise,
where 5 is the margin hyper-parameter, which is usually very small value (e.g. 0.05). Note that the above loss differs from the conventional margin-based ranking loss, because it has a condition for r r= 0 (inducing constant loss) on the logits pairs that the teacher GNN gives similar probabilities, ie., |yv. Vr.yi < 8. This design effectively prevents the student model from trying to differentiate minuscule differences in probabilities which the teacher may produce owing to noise; without this condition, the loss would pass binary information regardless of how small the difference is. We also empirically show the necessity of this design in Table 9 in Appendix D.
5

---

Under review as a conference paper at ICLR 2023
Distribution-based Matching. While the rank-based matching can effectively teach the student model relational rank information, we observe that it does not fully make use of the value information from V,. e.g. for a specific user, item A should be ranked much higher than item C, which should only be ranked marginally higher than item B. Although the logit-matching introduced in Section 3.1 might seem suitable here, observe that its link-level matching strategy only facilitates matching information on scattered node pairs, rather than focusing on the relationships conditioned on an anchor node- -mpiricalll, we also find that has limited effectiveness. Therefore, enable relational value-based matching centered on the anchor nodes, we further propose a distribution- based matching scheme which utilizes the KL divergence between the teacher predictions Ve and student predictions Dr, centered on each anchor node v. Specifically, we define it as
exp(Uos/T) log DEV IEC JEC, exp(Ues/T)
exp(0us/T) Ejec, exp(Ons/T))
CLPD = Σ
(6)
where 7 is temperature hyper-parameter which controls the softness of the softmaxed distribution. By also asking the student to match relative values within the probability distribution over context nodes conditioned on each anchor node, the distribution-batt matching scheme complements rank-based matching by providing auxiliary information about the magnitudes of differences.
Practical Implementation of LLP. In practical implementation, given the large number of nodes in the graph, it is infeasible for LLP to use all other nodes as the set of context nodes, especially for the rank-based matching which enumerates pairs of probabilities in D,. Hence, we opt for simplicity and adopt two straightforward sampling strategies for the constructing C.p. for each anchor node limit its size. First, to preserve the local structure around v. we sample p nearby nodes via fixed-length random walks, denoted as C.N. On the other hand, we also randomly sample q nodes from G (which are likely to be far-away from v)) to form CR which additionally preserves the global structure w.r.t. in the graph. p and q are hyper-pane papper-praamters.. Finally, we make C2 as the union of the nearby samples and random samples, i.e., C. C. U CR.. We note that LLP can easily adopt more sophisticated sampling strategies for further improvements; we leave this to future exploration.
While training LLP, we jointly optimize both the rank-based and distribution-baseee matching losses in addition to the ground coound-truthhaa label loss. Therefore, the overall training loss which LLP adopts for the student is
Loup + B. LLL n + Y LLLPD
where a, B, and are
which mediate the strengths of each loss term.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets. We conduct the experiments using 9 benchmark datasets, including 7 commonly used link prediction benchmarks (Cora, Citeseer, Pubmed, Computers, Photos, CS, and Physics) and 2 larger-scale link prediction benchmarks from OGB (Hu al., 2020) (DCB-Collab and DCB-Citation2), Further details and statistics of the datasets are summarized in Appendix B.
Evaluation Settings and Metrics. To comprehensively evaluate our proposed LLP and baseline methods on the link prediction tasks, we conduct experiments on both transductive and production settings. For the transductive setting, all the nodes in the graph can be observed for train/validationntestt sets. Following previous works (Zhang & Chen, 201 8; Chami et al., 2019; Cai et al., 2021) we randomly sample 5%/15% of the links with the same number of no-edge node pairs from the graph as the validation/test sets on the non-OGB datasets. And the validation/test links are masked off from the training graph. For the OGB datasets, we follow their official train/validationntest splits (Wang et al., 2020a; Mikolov et al., 2013). In addition to transductive setting, we also design more realistic setting that mimics practical link prediction use-cases, which we call the production setting. In the production setting, new nodes would appear in the test set, while training and validation sets only observe previously existing nodes. Thus, this setting entails three categories of node pairs (edges or no-edges) in the test set: existing existing, existing new, and new new, where the first category is similar to the test edges in the transductive setting, and the latter two categories together are similar to the inductive setting used in few recent works (Bojchevski & Günnemann, 2017; Hao et al., 2020;

---

Under review as a conference paper at ICLR 2023
Table 1:
Link prediction performance under transductive setting, For OGB-Collab and OCB-Citation2, we report Hits 55 and MRR respectively. For other datasets, we report @its@2 20. Best and second best performances are marked with bold and underline. respectively. A D AMPP. and AGNN represent differences between LLP and these methods.
GNN
ML.P
PLM
CRM
LLP
DireetKD
ANLP
ACNN
Cora
74.38 11.54
78.06
74.72-423
75.75
78.82
3.07
0.76
4.44
Citeseer
73.89
71.21
72.44-1.52
65.19a554
77.32=242
4,88
6.11
3,43
Pubmed
21.98
42.89
42.78-118
44.44
57.33
12.89
14.44
5.35
CS
59.51
34.01
40.69_3.12
61.10
68.622146
7.52
34.61
9.11
Physics
66,74
31.26:+9.12
52.11=244
52.34+:375
72.
19.67
40.75
5.27
Computera
31.66+3.00
20.19+1.58
12.81-1.0
21.75+1195
35,32+225
13.57
15.31
3.66
Photoa
51.50
27.83=2490
24.24-2.79
38.47
49.32
10.85
21.49
-2.18
OGB-Collab
48.69 #007
36.95
35.97=0.6
26.86
45,27
8.41
-3.42
0GB-Citation2
82.56+0044
40.63+0.00
38.42-001
42.50=000
53.20+120
10.70
12.57
-29.36
Chen et al., 2021b). Nonetheless, all three types of these edges appear with varying proportions in practical use-cases, e.g. growth of a social network or online platform, hence we evaluate on all three types. Note that we only conduct production setting experiments on non-OGB datasets, because the two OGB datasets are already temporally split in their public releases. We further elaborate the details of the production setting as well as the statistics in Appendix C.
For OGB datasets, we use their official metric (Hits @ 50 for OGB-Collab and Mean reciprocal Rank (MRR) for OGB-Citation2) following the public leaderboard', For other datasets, following previous works (Yun al., 2021; Zhang et al., 2021a; Zhao et al., 2022), use Hits @ 20 the main metrie, which is also one of the main metrics on OGB datasets. We also report AUC performance in Appendix D. For all experiments, validation) along with its standard deviation over 10 runs with different random initializations.
report the averaged test performance (with early-stopping on
Methods. In the remainder of this section: "GNN" indicates the teacher GNN that was trained with Laup: "MLP" refers to the stand-alone MLP that was trained with Crap: "LLM" " refers to MLP trained with logit matching (Equation
"LRM" refers MLP trained with node representation matching (Equation (4)); "LLP" refers to MLP trained with our proposed relational KD (Equation (7)). For the main experiments, we opt for simplicity and use SAGE (Hamilton et al., 2017) as the teacher GNN in all settings. We also include further experiments of different teacher GNN models in Figure 5 (in Appendix D.3).
4.2 LINK PREDICTION RESULTS
Transductive Setting. Table shows the link prediction performance of the proposed LLP with GNN, MLP, and the direct KD methods (as introduced in Section 3.1) in the transductive setting. We observe that LLP consistently outperforms MLP and direct KD methods across all datasets with large margins. Specifically, LLP achieves 17.13 points and 10.17 points improvements over MLP and direct KD methods averaged on datasets, respectively. On the Physics dataset, LLP achieves 40.75 points and 19.67 points absolute improvements over MLP and direct KD, respectively. Moreover, LLP achieves better performance than the teacher GNN model on 6 out of 9 datasets, demonstrating that proposed rank-based and distribution-based matching are able to effectively distill the relational knowledge for link prediction.
Production Setting, Table 2 shows the link prediction performance of the proposed LLP with GNN, MLP, and the direct KD methods in the production setting. For ease of comparison, we also stratify each method's performance still able to consistently outperform MLP and direct KD methods by large margins for all test categories. Specifically. LLP achieves 12.01 and 6.67 on Hits improvements over MLP and direct KD methods averaged over datasets, respectively. Moreover, LLP is at or above par with the teacher GNN on 3 out of the datasets. From the more stratified test performances, we observe that LLP can generally achieve similar performance with GNN on the existing existing category. but much worse on the other two categories that involve newly appeared nodes. We hypothesize that this is because GNN neighbor aggregation improves generalization for low-degree nodes. We also
the three different categories of the test edges. We observe that LLP
existing-cxisting
'https://ogb..... htttpe://obbsstafoord.com www.mfordddd///os////aeeee liieeer___inpppp linkprop/
7

---

Under review as a conference paper at ICLR 2023
Table 2: The performance measured by Hits@20 of production setting. Best and second best performances are marked with bold and underline, respectively.
GNN
MLP
LLM
LRM
LLP
DDirectD
AMLP
ACNN
Overall
27.80
22.9
22.65
22.24
27.87=124
5.22
4.97
0.07
Citeseer
38.78-239
31.21+375
29.35
26.23=100
34.75=224
5.40
3.54
-4.03
Pubmed
-2.71-18:
31.71
39.03=421
43.27=3112
53,48=1552
10.21
15.47
0.77
CS
60.69
10.1
48.07
58.90
60.74
1.84
22.59
0.05
Physics
55.82_141
29.99
2..74
36.32_2.0
S2.83
16.51
22.84
-2.99
Computers
34.38
19.43+0.82
12.79
20.28=1.0
24.58
4.30
5.15
-9.80
Photos
51.03
34.29
24.63
40.58=1.6
13.79-127
3.21
9.50
-7.24
Existing
Existing
Cora
28.81=100
28.00
27.662301
27.03
33.31
5.65
5.31
4.5
Ci teseeer
38.10
33.8813.00
2..24
27.52
37.501241
5.26
3.62
-0,60
Pubmed
52.67-178
41.58+1.61
42.57=4432
46.32=1.0
57.16=1134
10.84
15.58
4.49
CS
61.52-3.10
40.27==1.69
50.78
62.17=:145
63.99:1.36
1.82
23.72
2.47
Physics
56.56_144
32.32212.2
23.88
38.74
56.04-21.47
17.30
23.72
+0.52
Computers
35.13-1.45
21.46+1.08
+3.81
22.78=1.17
4.11
5.43
-8.24
Photos
51.90-6244
37.47=2.7
26.54
44.51=2.10
48.38
3.87
10.91
-3.52
Existing New
25.78
19.47
19.1
18.58
23.08 11.5
3.61
Citeseer
38.73
30.7714.7
28.7712.0
26.65
3,53
.4.3
Pubmed
53.98-129
23.70+22.0
24.91
32.21
38.94-244
15.24
-15.04
CS
56.78
29.25
36.60=2.17
45.28=093
47.05
1.77
17.80
-9.73
Physics
52.90_244
20.61
18.231.04
26.57
39.73
13.16
19.12
-33.77
Computers
31.07
11.00
1115
9.85 1005
14.88
5.03
3.88
-16.19
Photos
47.42_518
21.00
16.75
24.10=1.58
24.27=207
0.17
3.27
-3.155
New New
Cora
31.97
11.69
12.54
13.80
16. 16.90=5
3.10
5.21
11.07
Citeseer
42.74
18.71
16.29
17.26
21.94.1.3
4.68
3,23
-00.8
Pubmed
33.18-1224
5.45
4.55
11.36=443
15.00=265
3.64
9.55
-18.18
CS
64.10-3.55
26.27+
33.73++381
38.07=200
42.89
4.82
16.62
-1.211
Physics
48.96-151
13.20
12.56
18.88
32. 12.80
13.92
19.6
-16.16
Computers
32.61-118
5.55
6.72.1065
3.87
10.25 1141
3.53
4.7
-22.36
Photos
43.54
10.09
8.14
12,21
14.87=309
2.66
4.78
-28.67
D DGGB-collab
DGB-citation2
10000
57
2206.4
2209.1
32.88
46.72
51.25
54.74
54.47
54.82
54
1000
Inference Time
44.65
51.57
$4.85
54
85.85
56.30
134.3
128
146.0
100
10
-45
NYA
-42
SAGE
OSAGE
PSAGE
Neighor
LLP
Sample
10
15
20
q/p
Figure 2: Inference time comparison LLP with GNN inference acceleration methods in log scale.
Figure: 3: Link prediction performance measured by Hits of LLP on Pubmed with different number of samples for the context nodes.
observe that the performance gaps between the teacher GNN and stand-alone MLP new-new and existing-new new are much larger than that the cuisting-existingg category, which also evidences that GNNs have better inductive bias than ML.s graph data. Nonetheless, note that such significant and consistent performance improvements of LLP over MLP is valuable for large-scale industrial applications, given their popularity in practice.
4.3 INFERENCE ACCELERATION COMPARISON
We evaluate LLP in comparison other common GNN inference acceleration methods, which mainly focus on the hardware and algorithm to reduce the computation consuming, such as pruning (Zhou al., 2021) and quantization (Zhao et al., 2020; Tailor al., 2020). Following the experimental
8

---

Under review as a conference paper at ICLR 2023
settings in Zhang et al. (2021b), we measure the inductive inference time 10 randomly chosen nodes in the graph. We evaluate against 4 common GNN inference acceleration methods: (i) SAGE (Hamilton et al., 2017), (ii) Quantized SAGE (QSAGE) (Zhao et al., 2020; Tailor et al., 2020) from float32 to int8. (iii) SAGE with 50% weights pruned (PSAGE) (Zhou et al., 2021; Chen al., 2021c), and (iv) SAGE with Neighbor Sampling with fan-out 15. Figure 2 shows the results on the large-scale OGB datasets. We can observe that LLP can obtain 71.04x and 776.37x x speedup datasets, respectively. Comparing with the best acceleration method Neighbor Sampling (which reduces graph dependency, but does not eliminate it like LLP), LLP still achieves 15.12x and 50.51 x speedup on the two datasets, respectively. This is because all these inference acceleration methods still rely on the graph structure.
comparing with on SAGE on OGB-Collab and DGB-Citati
4.4 LINK PREDICTION RESULTS ON COLD START NODES
Dealing with cold start nodes (newly appeared nodes without edges) is a common challenge in recommendation and information retrieval appli- cations (Li et al., 2019; Zheng et al., 2021; Ding al., 102 Without these edges, GNNs cannot perform well as they rely heavily on neighbor information. On the other hand, MLPs, which do not make use of any graph topology information, arguably more suitable. Here, we simulate the cold-start setting by removing all the new edges during testing stage of the production setting, i.e. all the new nodes are isolated (see Appendix C for more details). Table 3 shows the performances of LLP, the stand-alone MLP, and the teacher GNN on the cold-start nodes. We observe that LLP consistently outperforms GNN and MLP by average of 25.29 and 9.42 on Hits@ 20, respectively.
Table 3: Link pred. Hits @ 20 on cold-start nodes.
GNN
MI.P
Ours
AMLP
Agna
6.39
17.92
22.01
4.09
15.62
Citeseer
11.04
29.33
32,09
2.76
21.05
Pubmed
4.63
22.74
37.68
14.94
33.05
CS
9.46
29.09
46.83
17.74
37.37
Physics
5.46
20.22
39.37
19.15
33,91
Computers
1.53
10.72
14.64
3.92
13.11
Photos
0.87
20.44
23.79
3.35
22.92
4.5 ABLATION STUDY
Effectiveness of LUP__ and LLLP_D- As our proposed LLP contains two matching strategies, rank-based and distribution-barn based matching, we evaluate their effectiveness by removing them from LLP. Moreover, we further evaluate by also removing Sup.. i.e., using only one of the match- ing losses as the overall loss for LLP. Table shows the results of these settings compared with the performances of full LLP, stand-alone MLP, and the teacher GNN Pubmed and CS datasets under both settings. We observe that both rank- based and distribution-based matching contribute significantly for the overall performance. In the transductive setting, both loss terms by themselves (the bottom two rows) can already outperform the teacher GNN. In the production setting, the matching losses alone outperform MLP and can achieve comparable performances with GNN after Csup is added. In conclusion, both rank-based and distribution-baa based matching can effectively distill the relational knowledge, and they achieve the best performance by complementing each other.
Table 4: Results measured by Hits @20 on differ- ent components of LLP.
Setting Datasel
Transductive Pubmed
Production Pubmed
CS
CS
GNN MLP
51.98
59.51
52.71
60.69
42.89
40.69
38.01
38.15
LLP
$7.33
68.62
53.48
60.74
w/o LLLP_R
55.35
66.61
53.40
60.53
w/o CLLP_ w/o CLLP Lenp
54.97
65.17
48.58
60.13
54.86
68.39
39,35
57.35
w/o CLL Lxup
53.30
68.30
41.43
55.63
Context Sampling Sensitivity to p and q. Figure 3 shows the link prediction performance of LLP on Pubmed under the transductive setting with different numbers of context node samples (p local samples, and q random samples). For the ease of hyper-parameter tuning, we make q a multiple of p. as shown in the x-axis of Figure 3. We observe that low number of random samples show poor link prediction performances, suggesting that preserving global relations are necessary for the proposed relation KD, Generally, the heatmap shows a clear trend, making the optimal values easy to locate.
CONCLUSION
Our work tackled problems related to applying GNNs for link prediction at scale. We note these models have high latency at inference time owing to non- non-trivial data dependency. In response, we explored applying model distillation methods from teacher GNN to student MLP models, which are advantaged in inference time. We first adopt two direct logit matching and representation matching KD methods the link prediction context and observe their unsuitability. In response, introduced
9

---

Under review as a conference paper at ICLR 2023
relational KD framework, LLP, which proposed rank-based matching and distribution-base match- ing objectives which complement each other to force the student to preserve key information about contextual relationships across anchor nodes. Our experiments demonstrated that LLP achieved MLP- level speedups (up to 776.37x over GNNs), while also improving link prediction performance over MIPs by 17.13 and 12.01 points in transductive and production settings, matching outperforming the teacher GNN in 6/9 datasets in transductive setting and 3/9 datasets in production setting, and with notable 25.29 on Hits 20 improvements on cold-start nodes.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our experiments, we provide the source code for LLP in the sup- plementary materials. Moreover, we will publically open-source our code later after we clean up our code package and add proper documentation for it. The hyper-parameters that are required to reproduce our experiments are provided in Appendix E.
ACKNOWLEDGEMENT
We appreciate Xiaotian Han from Texas A&M University, Wei Jin from Michigan State University, and Yiwei Wang from National University of Singapore for valuable discussions and suggestions.
REFERENCES
Lada A Adamic and Eytan Adar. Friends and neighbors the web. Social networks, 2003.
Uri Alon and Eran Yahav. On the bottleneck of graph neural networks and its practical implications. arXiv preprint arXiv:2006.05200 2020.
Albert-László Barabási and Réka Albert. Emergence of scaling in random networks. science, 1999.
Rianne van den Berg, Thomas N Kipf, and Max Welling. Graph convolutional matrix completion. arXiv preprint arXiv: /006 02266, 2017.
Beatrice Bevilacqua, Fabrizio Frasca, Derek Lim, Balasubramaniam Srinivasan, Chen Cai, Gopinath Balamurugan, Michael M Bronstein, and Haggai Maron. Equivariant subgraph aggregation networks. arXiv preprint arXiv:2110.02910, 2021.
Aleksandar Bojchevski and Stephan Günnemann. Deep gaussian embedding of graphs: Unsupervised inductive learning via ranking. arXiv preprint arXiv: 2017.
Sergey Brin and Lawrence Page. Reprint of: The anatomy of large-scale hypertextual web search engine. Computer networks, 2012.
Lei Cai and Shuiwang Ji. A multi-scale approach for graph link prediction. In Proceedings of the AAAI conference artificial intelligence, 2020.
Lei Cai, Jundong Li, Jie Wang, and Shuiwang Ji. Line graph neural networks for link prediction.
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Ines Chami, Zhitao Ying, Christopher Ré, and Jure Leskovec. Hyperbolic graph convolutional neural networks. Advances neural information processing systems, 2019.
Defang Chen, Jian-Ping Mei, Yuan Zhang, Can Wang, Zhe Wang. Yan Feng, and Chun Chen. Cross- layer distillation with semantic calibration. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021a.
Jiajun Chen, Huarui He, Feng Wu, and Jie Wang. Topology-aware correlations between relations for inductive link prediction in knowledge graphs. In Proceedings of the AAAI Conference Artificial Intelligence, 2021b.
Ming Chen. Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In International Conference on Machine Learning, 2020.
10

---

Under review as a conference paper at ICLR 2023
Tianlong Chen, Yongduo Sui, Xuxi Chen, Aston Zhang, and Zhangyang Wang. A unified lottery ticket hypothesis for graph neural networks. In International Conference on Machine Learning, pp. 1695-1706. PMLR, 2021c.
Paul Covington, Jay Adams, and Emre Sargin. Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM conference on recommender systems, pp. 191-198, 2016.
Tim Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub Tomczak. Hyperspherical variational auto-encoders. arXiv preprint UrXiv:/804.0000 2018.
Xiang Deng and Zhongfei Zhang. Graph-free knowledge distillation for graph neural networks. arXiv preprint arXiv:2105.075:1 2021.
Hao Ding, Yifei Ma, Anoop Deoras, Yuyang Wang, and Hao Wang. Zero-shot recommender systems. 2021.
arXiv preprint ar arXiv:2005.06
Wenqi Fan, Xiaorui Liu, Wei Jin, Xiangyu Zhao, Jiliang Tang, and Qing Li. Graph trend filtering networks for recommendation. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 112 121, 2022.
Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. arXiv preprint 1903.0040...000 2019.
Matthias Fey, Jan E Lenssen, Frank Weichert, and Jure Leskovee. Gnnautoscale: Scalable and expressive graph neural networks via historical embeddings. In International Conference Machine Learning, pp. 3294-3304 PMLR, 2021.
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Bomn again neural networks. In International Conference Machine Learning, pp. 1607-1616. PMLR, 2018.
Floris Geerts and Juan Reutter. Expressiveness and approximation properties of graph neural networks. arXiv preprint 4xXiv:200.006661, 2022.
Amir Gholami, Schoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference. arXiv preprint arXiv:2/03.. 13630, 2021.
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pp- 1263-1272. PMLR, 2017
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 2021.
Zhichun Guo, Chuxu Zhang, Wenhao Yu, John Herr, Olaf Wiest, Meng Jiang, and Nitesh Chawla. Few-shot graph learning for molecular property prediction. In WWW, 2021.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning large graphs. Advances in neural information processing systems, 2017.
Yu Hao, Xin Cao, Yixiang Fang, Xike Xie, and Sibo Wang. Inductive link prediction for nodes having only attribute information. arXiv preprint arXiv:2007.08055, 2020.
Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. Lightenn: Simplifying and powering graph convolution network for recommendation. Proceedings of the 43rd International ACM SIGIR conference research and development in Information Retrieval, pp. 639-648 2020.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in a neural network. arXiv preprint arXiv: 003513331, 2015.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong. Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 2020.
11

---

Under review as a conference paper at ICLR 2023
Yang Hu, Haoxuan You, Zhecan Wang, Zhicheng Wang. Erjin Zhou, and Yue Gao. Graph-mlpe mlp: node classification without message passing in graph. arXiv preprint arXiv:2106.0403 2021.
Zan Huang, Xin Li, and Hsinchun Chen. Link prediction approach to collaborative filtering. In Proceedings of the 5th ACM/IEEE-CS joint conference on Digital libraries, pp. 141-142, 2005.
Glen Jeh and Jennifer Widom. Simrank: measure of structural-context similarity. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, 2002.
Guangda Ji and Zhanxing Zhu, Knowledge distillation in wide neural networks: Risk bound, data efficiency and imperfect teacher. Advances in Neural Information Processing Systems, 2020.
Zhihao Jia, Sina Lin, Rex Ying, Jiaxuan You, Jure Leskovec, and Alex Aiken. Redundancy-free computation for graph neural networks. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020.
Chaitanya K Joshi, Fayao Liu, Xu Xun, Jie Lin, and Chuan Chuan-Sheng Foo. On representation knowledge distillation for graph neural networks. arXiv preprint arXiv:2111.0494,, 2021.
Jangho Kim, ScongUk Park, and Nojun Kwak. Paraphrasing complex network: Network compression via factor transfer. Advances in neural information processing systems, 2018.
Thomas Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv: (609.0000002000 2016a.
Thomas Kipf and Max Welling. Variational graph auto-encoders. arXiv preprint 1rXiv:1611.07331 2016b.
Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recommender systems. Computer, 2009.
Jure Leskovec and Christos Faloutsos. Sampling from large graphs. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, 2006.
Jingjing Li, Mengmeng Jing, Ke Lu, Lei Zhu, Yang Yang, and Zi Huang. From zero-shot learning to
cold-start recommendation. In Proceedings of the AAAI conference on artificial intelligence, 2019.
Pan Li, Yanbang Wang, Hongwei Wang, and Jure Leskovec. Distance encoding: Design provably more powerful neural networks for graph representation learning. Advances in Neural Information Processing Systems, 33:4465 4478, 2020.
Gang Liu, Tong Zhao Jiaxin Xu, Tengfei Luo, and Meng Jiang. Graph rationalization with environment-based augmentations. In Proceedings of the 28th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2022.
Xiaorui Liu, Wei Jin, Yao Ma, Yaxin Li, Hua Liu, Yiqi Wang, Ming Yan, and Jiliang Tang. Elastic graph neural networks. In International Conference on Machine Learning, pp. 6839. PMLR, 2021.
Jiaqi Ma and Qiaozhu Mei. Graph representation learning via multi-task knowledge distillation. arXiv preprint arXiv: /91 1.05700, 2019.
Yao Ma, Xiaorui Liu, Tong Zhao, Yozen Liu, Jiliang Tang, and Neil Shah. A unified view on graph neural networks as graph signal denoising. In Proceedings of the 30th ACM International Conference Information & Knowledge Management, 2021.
Haggai Maron, Heli Ben-Hamu. Hadar Serviansky, and Yaron Lipman. Provably powerful graph networks. Advances in neural information processing systems, 32, 2019.
Victor Martínez, Fernando Berzal, and Juan-Carlos Cubero. A survey of link prediction in complex networks. ACM computing surveys (CSUR), 49(4):1-33, 2016.
12

---

Under review as a conference paper at ICLR 2023
Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. Image-based recommendations on styles and substitutes. In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval, 2015.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 2013.
Deepak Nathani, Jatin Chauhan, Charu Sharma, and Manohar Kaul. Learning attention-based embeddings for relation prediction in knowledge graphs. arXiv preprint arXiv:1906.000 191..01995, 195, 2019.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
S Yu Philip, Jiawei Han, and Christos Faloutsos. Link mining: Models, algorithms, and applications. 2010.
Mary Phuong and Christoph Lampert. Towards understanding knowledge distillation. International Conference Machine Learning. PMLR.
Sashank Reddi, Rama Kumar Pasumarthi, Aditya Menon, Ankit Singh Rawat, Felix Yu, Seungyeon Kim, Andreas Veit, and Sanjiv Kumar. Rankdistil: Knowledge distillation for ranking. In International Conference Artificial Intelligence and Statistics. PMLR, 2021.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and
Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014.
Aravind Sankar, Yozen Liu, Jun Yu, and Neil Shah. Graph neural networks for friend ranking in large-scale social platforms. In Proceedings of the Web Conference, pp. 2535-2546, 2021.
Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. In European semantic web conference, pp. 593-607. Springer, 2018.
Oleksandr Shchur. Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. arXiv preprint arXiv:1811.05865 2018
William Shiao and Evangelos Papalexakis. Adversarially generating rank-constrained graphs. In 2021 IEEE 8th International Conference on Duta Science and Advanced Analytics (DSAA). IEEE, 2021.
Shyam A Tailor, Javier Fermandez-Marques, and Nicholas D Lane. Degree-quant: Quantization-aware training for graph neural networks. arXiv preprint arXiv:2008.0500 2020.
Xianfeng Tang, Yozen Liu, Xinran He, Suhang Wang, and Neil Shah. Friend story ranking with edge-contextual local graph convolutions. In Proceedings of the Fiffeenth ACM International Conference on Web Search and Data Mining, pp. 1007-1015, 2022.
Yijun Tian, Chuxu Zhang, Zhichun Guo, Xiangliang Zhang, and Nitesh Chawla. Nosmog: Learning noise-robust and structure-awwe mips on graphs. arXiv preprint arXiv:2208. 10010, 2022.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. Complex embeddings for simple link prediction. In International conference machine learning, pp. 2071-2080. PMLR, 2016.
Anton Tsitsulin, Davide Mottin, Panagiotis Karras, and Emmanuel Müller. Verse: Versatile graph embeddings from similarity measures. In Proceedings of the 2018 world wide web conference, pp. 539-548, 2018
Frederick Tung and Greg Mori. Similarity-preservvii knowledge distillation. In Proceedings of the JEEE/CVF International Conference on Computer Vision, 2019.
Shikhar Vashishth, Soumya Sanyal, Vikram Nitin, and Partha Talukdar, Composition- based multi- relational graph convolutional networks. 2020.
13

---

Under review as a conference paper at ICLR 2023
Petar Veličkovié, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arxiv: 2017.
Kuansan Wang. Zhihong Shen, Chiyuan Huang, Chieh-Han Wu, Yuxiao Dong, and Anshul Kanakia. Microsoft academic graph: When experts are not enough. Quantitative Science Studies, 2020a.
Xiaobo Wang, Tianyu Fu, Shengcai Liao, Shuo Wang, Zhen Lei, and Tao Mei. Exclusivity-consiiteni regularized knowledge distillation for face recognition. In European Conference Computer Vision, 2020b.
Zhitao Wang, Yong Zhou, Litao Hong, Yuanhang Zou, and Hanjing Su. Pairwise learning for neural link prediction. arXiv preprint arXiv:2112.029366 2021.
Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanic Jegelka. Representation learning on graphs with jumping knowledge networks. In International conference on machine learning, pp. 5453-5462. PMLR, 2018.
Bencheng Yan, Chaokun Wang, Gaoyang Guo, and Yunkai Lou. Tinygnn: Learning efficient graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020.
Cheng Yang, Jiawei Liu, and Chuan Shi. Extract the knowledge of graph neural networks and go beyond it: An effective knowledge distillation framework. In Proceedings of the Web Conference 2021, 2021.
Yiding Yang, Jiayan Qiu, Mingli Song, Dacheng Tao, and Xinchao Wang. Distilling knowledge from graph convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020a.
Ze Yang, Linjun Shou, Ming Gong, Wutao Lin, and Daxin Jiang. Model compression with two-stage multi-teacher knowledge distillation for web question answering system. In Proceedings of the 13th International Conference on Web Search and Data Mining, 2020b.
Zhilin Yang. William Cohen, and Ruslan Salakhudinov. Revisiting semi-supervised learning with graph embeddings. In International conference on machine learning. 2016.
Haoteng Yin, Muhan Zhang, Yanbang Wang, Jianguo Wang, and Pan Li. Algorithm and system co- design for efficient subgraph-based graph representation learning. arXiv preprint arXiv:2202.1 135388 2022.
Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William I. Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 974-983, 2018a.
Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. Hier- archical graph representation learning with differentiable pooling. Advances in neural information processing systems, 2018b.
Jiaxuan You, Rex Ying, Xiang Ren, William Hamilton, and Jure Leskovec. Graphrnn: Generating realistic graphs with deep auto-regressive models. In International conference on machine learning. PMLR, 2018
Seongjun Yun, Seoyoon Kim, Junhyun Lee, Jaewoo Kang, and Hyunwoo J Kim. Neo-gnns: Neigh- borhood overlap-aware graph neural networks for link prediction. Advances in Neural Information Processing Systems, 2021.
Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor Prasanna. Graph- saint: Graph sampling based inductive learning method. arXiv preprint 1.XXv:1907.04931 2019.
Chuxu Zhang, Huaxiu Yao, Chao Huang, Meng Jiang, Zhenhui Li, and Nitesh V Chawla. Few-shot knowledge graph completion. In Proceedings of the AAAI Conference on Artificial Intellgence. 2020a.
14

---

Under review as a conference paper at ICLR 2023
Dalong Zhang, Xin Huang, Ziqi Liu, Zhiyang Hu, Xianzheng Song, Zhibang Ge, Zhiqiang Zhang, Lin Wang. Jun Zhou, Yang Shuang, et al. Agl: a scalable system for industrial purposse graph machine learning. arXiv preprint arXiv:2003 2003:02444, 2020b.
Hanlin Zhang, Shuai Lin, Weiyang Liu, Pan Zhou, Jian Tang, Xiaodan Liang, and Eric P Xing. Iterative graph self-distillation. arXiv preprint arXiv:2010. 12609, 2020c.
Muhan Zhang and Yixin Chen. Weisfeiler-ehman neural machine for link prediction. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pp- 575-583, 2017.
Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. Advances in neural information processing systems, 2018.
Muhan Zhang, Zhicheng Cui, Marion Neumann, and Yixin Chen. An end-to-end deep learning architecture for graph classification. Proceedings the AAAI conference on artificial intelligence. 2018.
Muhan Zhang, Pan Li, Yinglong Xia, Kai Wang, and Long Jin. Labeling trick: A theory of using graph neural networks for multi-node representation learning. Advances Neural Information Processing Systems, 34:9061-9073, 2021a.
Shichang Zhang, Yozen Liu, Yizhou Sun, and Neil Shah. Graph-less neural networks: Teaching old mips new tricks via distillation. arXiv preprint 2XXiv:211000872:: 2021b.
Wentao Zhang, Xupeng Miao, Yingxia Shao, Jiawei Jiang, Lei Chen, Olivier Ruas, and Bin Cui Reliable data distillation on graph convolutional network. In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data, 2020d.
Ziwei Zhang, Peng Cui, and Wenwu Zhu. Deep learning graphs: A survey. IEEE Transactions Knowledge and Data Engineering, 2020e.
Lingxiao Zhao, Wei Jin, Leman Akoglu, and Neil Shah. From stars to subgraphs: Uplifting any gnn with local structure awareness. arXiv preprint 2 2021a.
Tong Zhao, Yozen Liu, Leonardo Neves, Oliver Woodford, Meng Jiang, and Neil Shah. Data augmentation for graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11015-11023 2021b.
Tong Zhao, Gang Liu, Daheng Wang, Wenhao Yu, and Meng Jiang. Learning from counterfactual links for link prediction. In International Conference Machine Learning, pp. 26911-26992 PMLR, 2022.
Yiren Zhao, Duo Wang, Daniel Bates, Robert Mullins, Mateja Jamnik, and Pietro Lio. Learned low precision graph neural networks. arXiv preprint arXiv:2009.09232. 2020.
Wenqing Zheng. Edward W Huang, Nikhil Rao, Sumeet Katariya, Zhangyang Wang, and Karthik Sub- bian. Cold brew: Distilling graph node representations with incomplete or missing neighborhoods. arXiv preprini arXiv:211 1..021001.0440 2021.
Hongkuan Zhou, Ajitesh Srivastava, Hanqing Zeng Rajgopal Kannan, and Viktor Prasanna. Acceler- ating large scale real-time gnn inference using channel pruning. arXiv preprint arXiv:2105.04528, 2021.
Zhaocheng Zhu, Zuobai Zhang, Louis-Pascal Xhonneux, and Jian Tang. Neural bellman-ford networks: A general graph neural network framework for link prediction. Advances in Neural Information Processing Systems, 2021.
15

---

Under review as a conference paper at ICLR 2023
A FURTHER RELATED WORK
In this section, we discuss other work related to LLP.
Graph Neural Networks (GNNs). Many GNN architectures have been proposed in recent years to model attributed graph data; most architectures follow the message passing (Gilmer et al., 2017; Guo et al., 2021; Ma et al., 2021; Liu et al., 2021; 2022) paradigm. Different GNN customizations include degree normalization (Kipf& & Welling, 2016a), neighbor sampling and neighbor separation (Hamilton et al., 2017; Zhao et al., 2021b), self- (Veličković et al., 2017), residual connections (Xu et al., 2018), and more. Alon & Yahav (2020) proposed to use a fully-adjacent layer at the end of GNN to deal with the bottleneck problem of GNNs. Moreover, researchers also proposed subgraph-hased methods (Bevilacqua et al., 2021; Zhao et al., 2021a), and tensor-based methods (Maron et al., 2019; Geerts & Reutter, 2022) for more expressive GNNs.
Link Prediction. Link prediction has achieved great attention from the research community, con- sidering its wide applications. Heuristic methods (Philip et al., 2010) were proposed to make the link prediction by measuring the link scores based on the structure information, such as the common neighbors and the shortest path. 2-order (Adamic & Adar. 2003) and high-order (Brin & Page, 2012; Jeh & Widom, 2002) heuristic methods were proposed to further improve the effectiveness. Link prediction with GNN is another important direction for link prediction (Yun et al., 2021), which is based on the learned embeddings. One line of work is the strategy we discussed in Section 2, where the GNN-based encoder learns node representations and the decoder predicts whether the link exists. It is worth mentioning that knowledge graph completion follows this strategy to predict not only the link existence but also the type of the link (Schlichtkrull et al., 2018: Nathani et al., 2019; Vashishth et al., 2020: Zhang et al., 2020a). These methods mainly use heterogeneous graph neural networks sensitive to different edge types. Another line of work casts link prediction tasks as classification problems on the enclosing subgraphs around each link (Zhang & Chen, 2018; Cai & Ji, 2020; Cai et al., 2021). Although these methods can improve task performance, they are usually extremely computationally expensive and cannot scale well in practical use-cases (Yin et al., 2022). Similarly, Zhu et al. (2021) proposed a GNN link prediction paradigm by encoding information of all paths between two nodes, which is also very expensive. Our work focuses on accelerating the more general and faster methods discussed in Section 2.
GNN Inference Acceleration. Pruming (Zhou al., 2021; Chen al., 2021c) and quantization (Zhao et al., 2020; Tailor et al., 2020) strategies were proposed for accelerating GNN inference. These methods do accelerate GNNs, but they rely on graph data for message passing and thus leave much room for speed improvement. We note that these approaches are complementary to cross-model distillation, and can be employed together with KD for additional inference time improvements. Other than the above acceleration methods, Hu et al. (2021) and Zhang et al. (2021b) accelerated GNNs by distilling them to MLP. These works focus on KD for node classification tasks, whereas we focus on link prediction tasks. GNNAutoScale (Fey et al., 2021) proposed effective method to accelerate the training process of GNNs. It also reduces the inference to a constant factor by directly using historical embeddings stored offline. However, in this case, all the methods in Figure 2 can share the same inference time benefits. Moreover, GNNAutoScale is not suitable for the production setting, where new nodes (without historical embeddings) appear frequently after the training process. So we did not include it as a bascline in this work.
Knowledge Distillation (KD). Logit-based (Hinton al., 2015; Furlanello et al., 2018: Zhang et al., 2021b) and representation-basee based (Romero et al., 2014; Gou et al., 2021) matching are two common KD methods, which match final-layer and intermediate- predicted logits between the teacher and the student, respectively. Our work is the first to adapt and evaluate these approaches in the link prediction setting, to the best of our knowledge.
For representation based KD, several work (Park et al., 2019; Tung & Mori, 2019; Joshi et al., 2021) proposed relational KD, which corresponds to instance-to-instance KD while preserving metrics among representations of similar instances. For GNNs Yang et al. (2020a) used knowledge of the neighboring nodes to teach the student to better classify the center node. In contrast, our KD strategies focus transferring relational knowledge between each pair of nodes from teacher to student. Both the rank- matchiin and wraabbutto--maaccin strategies help the student to better capture the relational graph topology information and make better link prediction.
16

---

Under review as a conference paper at ICLR 2023
Table 5: Statistics of all datasets used in the experiments.
Dataset
# Nodes
# Edges
# Features
Cora
2,708
5.278
1,433
Citeseer
3,327
4,552
3,703
Pubmed
19,717
44,324
500
CS
18,333
163,788
6,805
Physics
34,493
495,924
8,415
Computers
13,752
491,722
767
Photos
7,650
238,162
745
DGB-Collab
235,868
1,285,465
128
DGB-Citation2
2,927,963
30,561,187
128
Table 6: Detailed statistics data splits under production setting.
Nodes
Testing Edges
# Existing
# New
# Existing
Existing
# Existing New
# New New
Cora
1,896
812
765
675
142
Citeseer
2,329
998
673
568
124
Pubmed
15,774
3,943
5,648
2,858
358
CS
14,666
3,667
10,482
5,221
675
Physics
27,594
6,899
31,399
16,126
2,067
Computers
11,002
2,750
31,095
16,033
2,043
Photos 6,120
1,530
15,248
7,618
950
RankDistill (Reddi et al., 2021) is designed to transfer ranking knowledge from the teacher to the student. Different from our work which distill the relational information in graph context, it distills ranking in a non-graph context between teacher and student. We adopt different sampling and matching methods based
different motivations. Further analysis is shown in Appendix D.9.
KD on GNNs. Existing GNN-based KD work are mostly based on the logit-based KD (Hinton al., 2015) to obtain liigh models (Zhang al., 2020d; Zheng et al., 2021; Yang al., 2021). Yan et al. (2020) proposed to train a student GNN with fewer parameters using KD. Yang et al. (2021) improved the designed student model, which consists of label propagation and feature-based prior knowledge, using the pre-trained teacher GNN. Different from the above work, LSP (Yang al., 2020a) and G-CRD (Joshi et al., 2021) proposed structure preserving KD methods, which are specifically designed for GNN. Both of these work follow the original relational KD to preserve the metries among node representations and are applied on node classification tasks.
structure-preservingg
B ADDITIONAL DATASETS DETAILS
Here we present the details of the datasets used in the experiments. Cora, Citeseer, Pubmed (Yang et al., 2016) and citation network datasets, where the nodes and edges represent papers and citations, respectively. CS, Physics (Shchur et al., 2018) and OGB-Collab (Wang et al., 2020a) are all collaboration networks based on MAG. where the nodes represent authors and the edges indicate the collaboration for the paper. Computers and Photos (Shchur et al., 2018) are two well-known co-purchased graphs (McAuley al., 2015), where the nodes represent goods and the edges indicate two items were bought together. The detailed statistics of these datasets shown in Table 5.
( (Wang et al., 2020a; Mikolov et al., 2013) are all representative
17

---

Under review as a conference paper at ICLR 2023
C ADDITIONAL EVALUATION SETTING DETAILS
C.I TRANSDUCTIVE SETTING
The transductive setting standard setting for link prediction (Kipf & Welling, 2016b; Zhang Chen, 2017; 2018; Yun et al., 2021; Zhao et al., 2022), where the nodes in training/validaiioneesst all visible the training graph, but subsets positive links are masked out for validation and test sets.
C.2 PRODUCTION SETTING
In this work, we design a new production setting to resemble the real-world link prediction scenario. This setting mimics pratical link prediction use-cases. For example, user friend recommendation on social platforms where new users (nodes) and friendships (links) appear frequently. Under the production setting, the newly occurred nodes and edges that can not be seen during the training stage would appear in the graph at inference time.
Specifically, following the detailed procedures of splitting the datasets into the production setting:
Split all nodes: as the new nodes VN and remove them from the training graph. We denote the remaining nodes by y where superscripts E stands for Existing and N stands for New. Note that for Cora and Citeseer, we sample 30% nodes as new nodes because these two datasets are too small.
Given the graph G = (V, E), we randomly sample 10% of nodes from V
Split all edges:
We then split the edges & according to the node splits into three sets: gE-E g gE-N, and EN-N denoting the links between existing-existing existing new, and new-new node pairs, respectively.
Split edges in EE-E; For the existing-existing node pairs, we split it into three sets following an 80/10/10 splitting ratio: 80% as training edges, 10% as new visible edges for message passing, and 10% as testing edges. Note that validation set contains only existing passsagg...iing....... nodes ' as the nodes are not visible during training.
Split edges in E- and N-N:; We follow the same ratio and split these two sets following with 90/10 splitting ratio: 90% as newly visible edges (used only for message passing during testing inference), and 10% as testing edges.
Message passing edges during training: During training, the GNN model can only utilize the 80% exising-existing training edges for message passing.
Message passing edges for inferencing: During inference, the GNN model can conduct message passing on all exist except the testing ones. Specifically, the training and testing (total of 90%) sets of & EE-E, and gN
F and the 90% of newly visible message passing edges in 8 F
Testing edges: We all methods on the (10% of each) sampled from E - EE-N, and N-N respectively.
three separate testing edge sets
Table 6 shows the detailed statistic of different datasets under this setting.
C.3 COLD-START SETTING
Followed by the production setting, remove all the new edges appearing newly in the inference time. Then the new nodes will be the strict cold start nodes with no neighbor information for the model predict. The experimental results showed in Section 4.4 are conducted with this setting.
D ADDITIONAL EXPERIMENTAL RESULTS
D.1 AUC RESULTS ON TRANSDUCTIVE AND PRODUCTION SETTINGS
Here, we present AUC results on all the non-OGB datasets under the transductive setting in Table 7 and production setting in Table 8. In Table 7, we observe that our method outperforms both MI.P
18

---

Under review as a conference paper at ICLR 2023
Table Link prediction performance measured by AUC under transductive setting.
GNN
MLP
CLM
CRM
LLP
DirectKD
AMLP
AGNN
Cora
95.03 1033
94, 94.67..0.8
94.0510.77
95.23
0.56
0.20
Citeseer
95.15=003
13.11
94.1
22.88
95.32=001
1.21
2.21
0.17
Pubmed
97.89+007
97.82+006
00.0
97.90+a09
-0.06
0.01
4.06
CS
9024
97.61
98.0510.14
98.33
98.06.1004
-0.27
0.45
0.63
Physics 98.8010022
98.71
98.36.10.7
98. 1o.0c
99.10
0.14
0.39
0.30
Computers
98.76=003
98.46.2008
98.1 10.14
08.66220.0
98.84
0.18
0.38
0.08
Photos
98.98
98.71 +0.0s
98.51 +0006
98.95++0.0
99.03
0.08
0.32
0.05
57
52.5
51.06
52.21
52.80
52.38
51.29
38.61
54.85
54
-5.00
40.73
53.30
53.14
50.57
52.18
48.59
31.03
41.34
50.65
54.86
56.43
56.76
51
47.5
6.
34.91
45.69
49.24
51.94
50.79
48.84
0.
25.39
38.00
46.59
51.76
54.23
55.69
11.84
44.03
52.39
49.79
48.79
48.34
2.29
34.79
49.21
50.37
55.60
56.86
42.5
N/A
41.54
47.74
50.23
47.19
47.83
NIA
27.22
43.08
51.25
49.17
53.63
42
40.0
15
q/p
q/p
Figure Link prediction performance measured by Hits @2tts@20 of fuP-_ __ (left) and LLLP_D (right) on Pubmed with different and g.
and the teacher GNN all the datasets under the transductive setting. For the production setting, our method performs better than the teacher GNN on 4/7 datasets, as shown in Table 8.
D.2 SENSITIVITY ANALYSIS OF AND 9 FOR CLLP_ AND LLLP_D
To analyze the influence the context nodes on LLLP_ R and LLLP_D, we plot two heat maps to show their individual performance Pubmed under the transductive setting, shown in Figure 4. These two figures show different patterns with the context nodes. LLPPD (left figure) shows that the performance becomes better with nearby nodes (p) and higher random sampled rate(g/p). And random sampling rate can lead to a much better performance than nearby nodes. However, LLLP_ a- find that the results on the diagonal perform consistently better than those around, which means the random sampling rate should match the nearby nodes to work well. Besides, we can also observe that LLLP
sensitive with smaller number of context nodes than LLLP_D D- LLL R matches the performance by the relative ranking of the context nodes w.r.t. the anchor nodes. However, it becomes difficult for CLLP. n learn well when there are many context nodes. contrast, CLLP matches the distribution, and more context nodes provides a clearer picture about the link-related structure around the anchor node.
D.3 RESULTS WITH DIFFERENT TEACHERS
In our experiments, we use SAGE as the teacher GNN in both transductive and production settings. We further test LLP's performance with other GNNs, such GCN, GAT, and APPNP. In Figure 5, we find that our model always outperforms MLP with different GNN teachers. However, absolute performance is closely related to the teacher.
D.4 IMPORTANCE ANALYSIS LLLP_R R. LLLP_D AND Loup FOR LLP
We conduct the ablation study all non-OGB datasets to analyze the contributions of each compo- nent in Equation (7). In each ablation setting, we remove one component independently, as shown in Figure 6. We can observe that the performance drops when any of the three components (i.e.,
19

---

Under review as a conference paper at ICLR 2023
Table 8: Link prediction performance measured by AUC under production setting.
GNN
MI.P
LLM
CRM
D
AMLP
AGNN
Overall
Cora
12.59=163
+304
1..67
64.62
78.22
7.55
5.63
Citeseer
18
77.36:138
2.20
67.67
80.13:299
5.09
2.77
10.98
Pubmed
90.45=0a4
96.07 +0.13
96.13+0.26
96.74+0.0
94.30+034
-2.44
-1.77
3.85
97.08
95.96.11.19
96.59.0.0
87
-0.11
Physics
98.60=002
97.70=004
97.46=000
98. 20.000
98.75=aa11
0.75
1.05
0.15
Computers
98.67=005
97.85-2044
97.59
97.9 +0.95
97. 9708
-0.06
0.04
-0.78
Photos
98.78=0144
97.97=000
97.85
98. 180044
98.05
-0.13
0.08
-0.73
Existing
Existing
Cora
70.80
74.42
70.69
64.82.10.5
78.43
7.74
4.01
7.63
Citeseer
67.342181
34
13.79
68.00
18.36:1141
4.57
1.53
11.02
Pubmed
90.44=046
96.69
+6.22
97.24
95.17+0033
-2.07
-1.52
4.73
CS
97.01
96.08
96. 70=0.0
96.91 +0.00
97.00=aa3
0.09
0.92
-0.01
Physics
98 o00
97. 17.9
97.65
98. 18.20
98.76.1016
0.56
0.80
0.16
Computers
98.70=005
98.27:2005
27.95
98. 10.0
98.51
2:00
0.10
0.24
-0.19
Photoa
98.8
98.33+
98.20
08.57+0.7
98.61 +0004
0.04
0.28
-0.19
Existing
New
Cora
72.61 1:50
72.06
70.18
64.07
77.65 #1.12
7.47
5.59
5.04
Citeseer
69.90=1a8
77.58 +3.45
76.05+3.51
67. 1.71
81.23+ta71
5.18
3.65
11.33
Pubmed
90.82.1038
93.67 20:5
93.82.n054
94.8310.14
90.97.1074
-3.86
-2700
0.15
97.31=aa21 2020
95.46=1155
96. 8
18+0.00
96.31+a10
0.13
-1.00
Physics
98.57-100:
96.66 96.64-100s
96 2:009
97.17400.0
95.72
-1.45
-2.85
Computers
98.60=a05
96.23
96. +o.07
0.00
95.42++00s
-0.80
-3.18
Photos
18.6920044
96.53
96.4520.r
2:0.0
95.76.1016
-0.84
-0.77
-2.93
New
New
82,10=157
74.46:1
72.85
1.3
79.85
7.00
5.39
-2.25
Citeseer
11.48=167
79.23+308
77.13
68.36+2.7
84.68
7.55
5.45
9.20
Pubmed
84,30
87.971102
88.72.1.19
89,9510.0
83, $3.5411277
-6.41
.4.43
-0.76
CS
97.99
95.03
95.3911.4
95. 95.22.10.2
95.97
0.58
0.94
-2.02
Physics
98.84=012
95.72=n
96.43.200
96.80
94.78.2034
-2.02
-0.94
-4.06
Computers
98.07=0.1
93.22++a16
93.30
92.96+0.3
92.09 +0.73
-1.21
-1.13
-5.98
Photos
98.351011
94.21.1003
94.69100m
93.7910.43
92.091059
-2.60
-2.12
-6.26
20

---

Under review as a conference paper at ICLR 2023
0GNN GLP
60
56.8
54.9
52.1
Hits820
50
47.9
47.3
46.8
45
42.9
35
MLP
SAGE
GCN
GAT
APPNP
Model Architectures
Figure 5: Link prediction performance measured by Hits @ 20 on Pubmed under the transductive setting with different GNN teachers (SAGE, GAT, GCN, and APPNP).
2 Transductive
Production
65
59.12
55.99
56.09
57.07
Hits@20
50
40.65
41.51
42.55
40
w/o LLP_D
w/o LLP
w/o Sup
LLP
Figure 6: The averaged results of Hits @20 across all the datasets by dropping each component Equation (7).
LLP n. LUP D. and supp) is removed, which shows the importance of each component. It also demonstrates that LLLP_1 and CLLP indeed provide complementary link prediction-related infor- mation for the student. Other than these two components, also contributes, especially under the production setting. In the production setting, as the neighbor information sparse absent, the limited true label information becomes critically important.
find that the true link label information
D.5 NECESSITY ANALYSIS OF CLLP_R
To analyze the necessity of d in LLLP R. we conduct the experiments all non-OGB datasets to compare the results using LLLP _ with and without The results are shown in Table 9. We observe that the results of LLL & without always approach zero after several training epochs. It demonstrates the effectiveness of avoiding noise and transferring useful knowledge to the student.
D.6 COMPARISON OF TASK PERFORMANCE AND INFERENCE TIME USING DIFFERENT INFERENCE ACCELERATION METHODS
In Table 10, we compare both the task performance and inference time using different acceleration methods on the two large-scale OGB datasets. From this table, we can observe that LLP shares
Table 9: The performance measured by Hits @20 of LLP_R with and without
Method
Cora
Citeseer
Pubmed
CS
Physica
Computers
Photos
LLLP
76.52
75.23
53.30
68.30
60.28
25.98
33.33
CLLP_R R w/o 8
0.00
0.00
0.00
0.00
0.00
0.00
0.00
21

---

Under review as a conference paper at ICLR 2023
Table 10: Performance and inference time comparison of LLP with GNN inference acceleration methods. The performance is the averaged test result over 10 runs.
Neighbor Sample
Dataset
Metric
SAGE
QSAGE
PSAGE
MLP
LLP
Hits @ 50
48.69
45.36
48.34
31.50
36,95
45.27
OCB-Collab
Time (ms)
134.3
128.3
128.7
28.6
1.9
1.9
MRR
82.56
82.53
82.04
79.82
40.63
53.20
OGB-Citation2
Time (ms)
2243.7
2206.4
2209.1
146.0
2.9
2.9
the same inference time with MIP, which is 15.12x and 50.51 faster than the most efficient acceleration method Neighbor Sample on OGB-Collab and OGB-CC (GB-Citatioo2, respectively. Our method outperforms MLP with large margins both datasets. Although Neighbor Sample achieved certain speedup comparing to GNNs, and sometimes better prediction performance than LLP, it is still less competitive than LLP in production applications given the huge speed difference, which is critical for deployed models that require low latency.
Namely, our work considers KD strategies as viable option distill information from GNNs to MLPs owing to the large practical advantages that MLPs enjoy; these advantages (namely, latency) make them a leading choice for production systems compared to GNNs (Covington et al., 2016; Gholami et al., 2021; Zhang et al., 2021b), because GNNs suffer neighborhood explosion and data dependency downsides. Since violating tight latency constraints may not be feasible in production applications (e.g. "every 100ms of latency cost ... 1% in sales" 2, "a 100-millisecond delay in website load time can hurt conversion rates by percent" '), we may not be able to enjoy the strong performance of a GNN in such setting (despite knowing it may perform better). With this perspective (that we must deploy a fast MI.P model and cannot tolerate a slow GNN model), we can observe that our work proposes an effective KD strategy to significantly boost MLP performance with additional knowledge distilled from GNNs, offering significant advantages. Since in practice, is common to operate under such constraints, the consistent improvements in all datasets (Table 1 and Table 2) has strong considerations for production settings which are MLP-constrained regardless of the performance gap with respect to GNN. That said, we hope that future work can ideally bridge the gap between LLP and GNN results in all settings.
show over MLP
D.7 ANALYSTS OF THE RESULTS ON OGB-CITATION2
From Table 2, we can observe that LLP perform not well on datasets. To understand the the reason for worse performance (size, or peculiarity of the dataset), we reduced the size of the original dataset by sampling smaller graph from the original graph, and comparing the performance the original and downsampled graphs. We conduct this experiments on two larger-scale datasets, DGB-Collab and OGB-Citations Based on the conclusion of Leskovec & Faloutsos (2006), that sampling based on random walks best cest-preserving certain properties of the original graph, adopt this strategy generate the downsampled graphs. For OGB-Collab, sampled a graph with the similar size to Computers. For DGB-Cite tati on2, we sampled a graph with the similar size like OGB-Collab. For these 4 datasets (original and downsampled versions of DGB-Collss and OGB DGB-Citation2) we run GNN, MLP and LLP and report results in Table 1 1. This table yields 3 observations:
compared with other
The performance gap between GNN and LLP does not change significantly from the original graph to the downsampled graph both datasets.
The performance gap between GNN and LLP on DGB-Citation2 is always much larger than OGB-Collab: 29.36 3.42 the original graphs, and 10.76 vs. 2.07 the downsampled graphs.
http:///wwwwp.p.ooooooooooooooo. 1-in-slles critical-3004444998.8 html
22

---

Under review as a conference paper at ICLR 2023
Table 11: The statistics of the original and sampled graph of OGB-Collab and NGB-Citation2 and the experimental results on these graphs.
OGB-Collab
OGB-Citation2
Original
Sampled
Original
Sampled
#Nodes
235,868
23,891
2,927,963
122,440
#Edges
1,285,465
188,640
30,561,187
1,366,101
GNN
48.69
83.31
82.56
39.99
MLP 36.95
75.70
40.63
24.34
LLP
45.27
81.24
53.20
29.23
Table 12: The performance measured by Hits @0 of LLP and GNN+FA (Alon & Yahav, 2020) on the cold-start nodes.
Method
Cora
Citeseer
Pubmed
CS
Physics
Computers
Photos
GNN
6.39
11.04
4.63
9.46
5.46
1.53
0.87
GNN+FA
2.03
2.89
0OM
OOM
OOM
OOM
OOM
LLP
22.01
32.09
37.68
46.83
39.37
14.64
23.79
Although the downsampled graph from OGB-Citation2 is a similar size to the OGB-Collab original graph, the performance gap between GNN and LLP on the downsampled graph based on DGB-Citationl is still much larger than the gap on DGB-Collas original graph.
Summarily, point demonstrates that LLP 's performance is not significantly influenced by the size of the datasets. Both point 2 and 3 indicate that LLP worse performance on NCB-Citation2 is due to a peculiarity this particular dataset, rather than its size.
D.8 FURTHER COMPARISON RESULTS ON COLD START NODES.
We further compare LLP with another related work (Alon & Yahav, 2020), which does only rely on the connection relationship in the graph either. This paper identifies a bottleneck of graph neural networks, and proposes modify the last layer to be fully-adjacent layer (FA) as simple strategy to circumvent the bottleneck problem. To evaluate its performance on cold-start setting, we modify the last layer of GNNs with fully-adjacent layer following it. The results compared with LLP are shown in Table 12. We observe that this method is not suitable for large datasets it results in dense N adjacency matrix and results in each node receiving messages from other nodes (which is problematic when N is large). Most datasets can not into an NVIDIA 1000 GPU (40GB memory). The performance and Citeseer is even worse than vanilla GNNs. The potential reason is link prediction tasks do not heavily depend on long-range information, where the best results were found using only 2-3 layers observe that Alon & Yahav (2020) focuses evaluation smaller datasets which are sensitive to long-range dependencies, which seems to be mismatch with our intended setting.
utilized in our paper with fulle-adjacent layer GNN+FA on Cora
D.9 COMPARISON BETWEEN LLP AND RANKDISTIL (REDDI ET AL., 2021)
RankDistil (Reddi al., 2021) designed transfer ranking knowledge from the teacher to student, where the ranking is their ranking task training signal generated by the teacher. In LLP, we consider a link prediction task on graph data, which is standardly approached as a binary classification task (predicting link existence vs non-existence), and not ranking one. We proposed our KD framework to distill relational graph information from the teacher to the student. Different from RankDistil which distills the information in non- non-graph context, distill in graph context to keep the graph structure information. The difference in motivation across the methods leads to different choices in sampling and matching. For sampling method, method samples nodes which utilized teach the
23

---

Under review as a conference paper at ICLR 2023
Table 13: The performance measured by Hits @20 of LLP and RandDistil (Reddi et al., 2021).
Cora
Citeseer
Pubmed
CS
Physics
Computers
Photos
RankDistil
74.29
70.44
39.28
44.55
49.11
15.64
28.75
LLP
78.82
77.32
57.33
68.62
|
72.01
35.32
49.32
student based on the graph structure unlike RankDistil, which is very important for the link prediction on the graph data; RankDistil thereby struggles in effectively the student about this graph structure. Moreover, RankDistil samples nodes used to teach students based on the teacher's result, which differs from our method which samples independently of the teacher. The choice made by RankDistil here can potentially make it even harder for the student to learn graph structure in the case of teacher prieicredsss (since this also influences the sampling). For matching method, our method matches the order and the distribution of the sampled node pairs between teacher and student, which we show are both useful and complementary (Table 4) in preserving structure information distillation between the two. Different from our work, RankDistil keeps the order for "top-K" items and penalize high scores by the student for "bottom-K" items (since there too many candidate items). However, compared with keeping the order, penalizing the large scores is a "weaker" alignment method, which may be not beneficial for preserving graph structure information.
In addition to our discussion of conceptual differences and potential implications above, we also conduct some experiments to evaluate the impact of these differences in our link prediction task, by comparing the performance of LLP to RankDistil. We applied RankDistil on our task, where we take all the other nodes existing in the graph as the candidate "items" for each anchor node (as they would be considered in the RankDistil setup), and use RankDistil's 's matching methods to align the ranking results generated by teacher and student. Due the lack of guideline for the parameter settings for RankDistil, we conduct the hyperparameter search of the number of top items from [5, 10, 20, 50] and the number of bottom items from [10, 50, 100, 200]. The results are shown Table 13. We observe that LLP consistently outperforms RankDistil on all the datasets. We believe this demonstrates that sampling the context nodes w.r.t the anchor node based on the graph topology structure more effective in preserving relevant graph structure and link prediction-related knowledge than using all the other nodes in the graph to sample from a teacher-based ranking. Moreover, different matching methods we propose also help distill more resk-nelevant information than Distii in our setting.
E IMPLEMENTATION DETAILS
Transductive Setting. Inspired by GLNN (Zhang al., 2021b), we enlarge the size of the student MLP in our experiment. As suggested by GLNN this can significantly shorten the gap between the student MLP and the teacher GNN without greatly reducing the timing performance. We set the hidden dimension of student MLP two times larger than the teacher for Physics. ca, Computers, and Photos, and set it four times larger than the teacher for OGB DGB-Collab and DGB-Citation2. We examine the timing performance of the enlarged students by repeating the inference task ten times. The inference time of LLP increases from 1.9 to 7.1 seconds on OGB-Collab and from 2.9 to 15.2 seconds on DCB-Citation, but it is still 18.9x and 147x faster than SAGE, respectively.
Model Hyper-parammters. We take 2-layer SAGE (hidden size is set to 256) as the teacher for all the non-OGB datasets. For OGB-Collab and OGB-Citationl we follow their official implementation to set the layer size as 3. We take 3-layer MLP as the student on these two datasets. For LLP, we conduct the hyperparameter search of the weights for Sup LUP R and LLL D from [0.001, 0.01, 0.1, 1, 10, 100, 1000], the number of the nearby nodes p from [1,2.344.5] J. the random sampling rate q/p from [1, 3, 5, 10, 15], the learning rate from [0.001, 0.005] and the dropout rate from [0, 0.5].
Implementation and Hardware Details. Our code is implemented based on py Torch Geometric (Fey & Lenssen, 2019). We conduct experiments with NVIDIA V100 GPU(16GB memory). For OCB-Citation2, we run the experiments on NVIDIA 1 GPU with 40GB memory.
24


Reviews:
[{"review_id": "nleOJB-axB", "reviewer": "Reviewer_Z2kr", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a knowledge distillation framework for link prediction, Linkless Link Prediction (LLP). Unlike simple knowledge distillation methods that match outputs or representations of models, LLP distills relational knowledge of each link to the student MLP using proposed rank-based and distribution-based matching. These two matching strategies enable the student model to learn relational knowledge about context nodes. Experiments conducted on 9 benchmarks show the effectiveness and efficiency of the proposed framework.", "summary_of_the_review": "This paper presents a knowledge distillation framework for link prediction. The presentation of the paper is clear and easy to follow, and the motivation is clear, but the technical novelty is limited. \n", "strength_and_weaknesses": "Strengths\n\n- The proposed KD strategy for link prediction is interesting and more effective than the previously proposed KD strategy for node-level and graph-level tasks.\n\n- The proposed method shows a faster inference time and even better performance than GNNs.\n- The proposed method can be used meaningfully in a real-world scenario by showing excellent performance in cold start nodes, which have relatively few connections.\n\nWeaknesses\n\n- A lack of technical contribution. As mentioned by the author, from the perspective that link prediction is considered as a ranking task, the proposed method for knowledge distillation (margin-based ranking loss and distribution-based matching loss) were already proposed in RankDistill [1] which proposes knowledge distillation for the ranking task. \n\n- Knowledge Distillation becomes more significant as the scale of the graph increases due to the high latency of GNN, but the huge performance degradation in the largest scale graph (ogbl-citation2) raises questions about the necessity of KD. I wonder if it is a special case in ogbl-citation2 or if the performance drop of KD usually occurs in large-scale graphs.\n\n[1] Reddi, Sashank, et al. \"Rankdistil: Knowledge distillation for ranking.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Qg4MWuI3gW", "reviewer": "Reviewer_4BGr", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to distill GNN knowledge for link prediction into MLPs via distribution and rank based losses as opposed to prior work which only uses logit-matching or representation matching. The authors test their method against these prior knowledge distillation methods as well as stand-alone MLPs and the original teacher GNN. The authors further observe the difficulty with regards to cold-start nodes and observe significant performance improvements compared to some scaling GNN methods.", "summary_of_the_review": "The paper has excellent scientific communication and reproducibility. I want to highlight this as I believe papers in the field should be held to higher standards in this regard. So I would like to commend the authors on these points. The method is well motivated, but not technically sophisticated.\n\nIn my mind there is three barriers to acceptance:\n1. I understand that there is a lack of large-scale homogeneous link prediction datasets out there. However, given the purpose of the knowledge distillation is to be applied on large scale graphs the performance in Table 1 on OGB-Citation2 is problematic. For acceptance, I would need to see a convincing analysis that shows that this is not due to the size of the dataset (the method works well on smaller citation graphs,e.g. Citeseer), but rather a peculariaty of this particular dataset. For instance, currently it is not unreasonable to hypothesis that MLPs will struggle to compress the information of many node comparisons efficiently, thus harming performance as the training graph gets larger, which would be a fundamental problem.\n2. The argument that MLPs are better suited to cold-start nodes is interesting and gives this direction additional value. However, currently I  believe the comparison in table 3, needs adaption to be convincing. There are several GNN architectures that have proposed that do not only rely on the connectivity of the graph. The simplest example is [1], where a fully adjacent layer is used at the end, if the MLP can still out-perform such a GNN architecture, then I am willing to accept that KD into MLPs might be key for recommender systems, where cold-start nodes will indeed be common.\n3. The comparison against GNN scaling methods needs a bit of improvement, namely I would like to see a discussion of the trade-off of inference time on the large-scale datasets (such as OGB-Citation2) and the final performance (as we can always predict faster while doing so less accurately). As well as including some more recent methods such as [2].\n\nI appreciate that this is a lot of work for a rebuttal phase, but otherwise the paper remains too unconvincing for such a top-level conference. Should the authors be able to address the three points above, I am willing to raise my score to acceptance of the paper.\n\nEDIT: The authors rebuttal has addressed my concerns. I have thus raised my score for the paper to be accepted.\n\n[1] https://arxiv.org/pdf/2006.05205.pdf\n[2] http://proceedings.mlr.press/v139/fey21a/fey21a.pdf", "strength_and_weaknesses": "Strengths:\n - The paper is well written and clear.\n - The experiments and models are described in a reproducible manner.\n - The proposed production setting and analysis of cold-start nodes is interesting and valuable to recommender systems.\n - The proposed method seems sensible.\n\nWeaknesses:\n - The method is significantly outperformed on the only large link prediction graph provided (OGB-Citation2 Tab.1). This is problematic because the purpose of the method is to work on large-scale graphs, where GNNs might be too expensive to run. \n - The comparison against GNN methods on cold-start nodes is not fair as known GNN techniques exist that might help in this setting (see [1], where a fully adjacent layer is proposed).\n - The evaluation method leaves me with some doubt as I think it is biased (if my understanding is correct) due to no ranking against all nodes in the graph (as is standard in KG link prediction). This is an unrealistic test setting that is also biased.\n - Knowledge graph completion is an important link prediction task that deserves to be mentioned in the introduction and related work, with a brief explanation why this setting was ignored.\n - The GNN scaling method comparison is a bit weak in that some key methods have not been considered as well as only considering inference time, but not performance. There will always be a trade-off. In general, the paper is too quick to assume that knowledge distillation to MLPs is needed and cannot be dealt with by accelerating GNNs, a point that needs further evidence before it can be accepted.\n\n[1] https://arxiv.org/pdf/2006.05205.pdf", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DvLWwR6BK1", "reviewer": "Reviewer_Wrjf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper aims to construct an architecture for Knowledge distillation from GNN based methology to MLP for link Prediction task in specific.\nAs GNNs are more powerful and more accurate than vanilla MLP for link Prediction task primarily due to their ability to aggregate neighbot good information, these results in more model complexity and computational cost. Hence GNNs for link Prediction tasks can't be deployed directly in a production setting. Whereas MLPs had relatively less computational cost and complexity but don't take into consideration neighborhood information which is curcial for link Prediction tasks. Hence the authors propose a knowledge distillation architecture from teacher GNN to student MLP which utilises the advantages of accuracy of GNNs as well as less model complexity of MLPs for a production setting. \n", "summary_of_the_review": "Strengths\n\n+ Motivation:  The authors have clearly outlined the motivation setting for such a proposal behind knowledge distillation.\n\n+ Ablation studies confirm that their proposed model outperforms and close to GNN.\n\n+ The have showcases on 2 large scale OG datasets, thereby confirming the effectiveness of their proposed model in large graphs.\n\n+ The authors have  explains well the experimental setup in appendix. Well written. \n\n\n\nWeakness/questions for authors\n\n - Not clear in experiments what LLMatch metric is used currently.\n\n- Uses logit based matching based on a very old paper. And also doesn't showcase the motivation of proposal of logit based matching. It should be clearly substituted with recent paper evidences.\n\n- The authors have performed experiments in large OGB datasets which are typically used for link Prediction settings. But as the authors have mentioned in the start of the paper for applications in an industrial setting, did they also perform experiments in a realtime production setting like a recommendation system e.t.c. That would have been nice to have. An analysis on a large scale real production environment(rather than simulation) would tell much more about the utility of proposed method. If there are limitations, that should be clearly specified.\n\n- Inference time graphs are plotted, however very less conclusive analysis done on much of a significant time change does KD provides over GNN w.r.t accuracy.\n\n- Each of the experiment tables should also have clear notation what values they are showing. Is it showing HITs of AUC. That should specified under each table. Some tables don't have titles or caption ( example ablation study)\n", "strength_and_weaknesses": "Strengths:\n\n+ Good writing\n+ New application of knowledge distillation \n\nWeaknesses\n\n- Some parts of the experiments are not clear\n- A clear motivation is lacking\n- No readme file in the code--- this seriously constrains the reproducibility\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject