Geometric Knowledge Distillation: Topology Compression for Graph Neural Networks
Authors: Chenxiao Yang
Qitian Wu
Junchi Yan
Venue: NeurIPS 2022 Accept (2022)
Keywords: Graph Neural Networks
Knowledge Distillation
Geometric Deep Learning

Abstract: We study a new paradigm of knowledge transfer that aims at encoding graph topological information into graph neural networks (GNNs) by distilling knowledge from a teacher GNN model trained on a complete graph to a student GNN model operating on a smaller or sparser graph. To this end, we revisit the connection between thermodynamics and the behavior of GNN, based on which we propose Neural Heat Kernel (NHK) to encapsulate the geometric property of the underlying manifold concerning the architecture of GNNs. A fundamental and principled solution is derived by aligning NHKs on teacher and student models, dubbed as Geometric Knowledge Distillation. We develop non- and parametric instantiations and demonstrate their efficacy in various experimental settings for knowledge distillation regarding different types of privileged topological information and teacher-student schemes.

Geometric Knowledge Distillation: Topology Compression for Graph Neural Networks
Chenxiao Yang, Qitian Wu, Junchi Yan" Department of Computer Science and Engineering MoE Key Lab of Artificial Intelligence Shanghai Jiao Tong University {chr26195, echo770 yan junhii j@s edu cn
Abstract
We study new paradigm of knowledge transfer that aims at encoding graph topo- logical information into graph neural networks (GNNs) by distilling knowledge from a teacher GNN model trained on complete graph operating on smaller or sparser graph. To this end, we revisit the connection between thermodynamics and the behavior of GNN, based on which we propose Neural Heat Kernel (NHK) to encapsulate the geometric property of the underlying manifold concerning the architecture of GNNs. A fundamental and principled solution is derived by aligning NHKs on teacher and student models, dubbed as Geometric Knowledge Distillation. We develop non-parametric and parametric instantiations and demonstrate their efficacy in various experimental settings for knowledge distillation regarding different types of privileged topological informa- tion and teacher-student schemes.
student GNN model
Introduction
have shown remarkable performance in learning representations for structured instances. From the perspective of geometric deep learning [ 4 38]
Modern graph neural networks (GNNs) 25 49 3
part of the achievement of GNNs can be attributed to their implementation of the permutation
invariance property as geometric priors p iinto the architecture design. Nevertheless, in practice, GNNs highly rely graph topology, essential input information, to explore the relational knowledge implicit in interactions of instance pairs throughout the entire message passing process, termed geometric knowledge in this paper. As advances in generalized distillation [33 47 reveal the possibility of encoding input features into model construction, natural questions arise as to:
Is it possible, and so, how a GNN model, such that the model could precisely capture the underlying geometric knowledge even without full graph topology input?
encode graph topology as special type of 'geometric prior' into
In specific, we are interested in the following geometric knowledge transfer problem: a GNN model (with node-specific outputs for node-level prediction (23) is exposed with partial graph, which is subset of the complete graph. Formally speaking, we have notations:
(V,E) (partial graph), ğ {V,E) (complete graph), where C C.. S (V>V) Nε (1)
Our goal is to transfer or encode geometric knowledge extracted from & to the target GNN model that is only aware of G. Studying this problem is also of much practical value. As non-exhaustive
Junchi Yan is the correspondence author who is also with Shanghai AI Laboratory. "Geometric priors originally refer to the geometric principles naturally encoded in deep learning architectures, c.g., translational symmetry for CNNs, permutation invariance for GNNs.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Figure 1: Feature propagation the derlying manifold M. (a) Teacher: aware of the complete graph topology, and faith- fully explore geometric knowledge about the underlying manifold.
?
(b) Student be- fore GKD: only aware of partial graph topology, and estimate biased geometry property. (c) Student after GKD: able to propagate features on the same space teacher by alignment of NHKs.
a) Teacher
b) Student (before GKD)
Student (after GKD)
list of applications: improving efficiency without compromising on effectiveness for coarsened graphs [14 24 69]. privacy constrained scenarios in social recommenders or federated learning
where the complete graph unavailable 34 50 G8
promoting concentration on targeted community
to bring up economic benefits 557
Achieving this target is non-trivial for that we need to first find principled and fundamental way to encapsulate the geometric knowledge extracted by GNN model, which requires in-depth investigation on the role of graph topology throughout the progressive process of message passing. Therefore, we take a thermodynamie view borrowed from physics and propose a new methodology built upon recent advances revealing the connection between heat diffusion and architectures GNNs M 5 7 Specifically, we interpret feature propagation as heat flows on the underlying Riemmanian manifold, whose characteristies (that are dependent on graph topology and the GNN model) pave the way for principled representation of the latent geometric knowledge.
1.1 Our Contributions
New theoretical perspective for analyzing latent graph geometry. On top of the connection between heat equation and GNNs, we step further to inspect the implication of heat kernel for GNNs, and propose a novel notion of Neural Heat Kernel (NHK) with rigorous proof of its existence. Heat kernel intrinsically defines the unique solution to the heat equation and can fundamental characterization for the geometric property of the underlying manifold [8 19]. Likewise, NHK uncovers geometric property of the latent graph manifold for GNNs, and governs how information flows between pairs of instances, which lends us mathematical tool to encapsulate geometric knowledge extracted from GNN model and enables geometric knowledge transfer. This result alone might also be useful in broader contexts for understanding GNNs.
Flexible distillation framework with versatile instantiations. Based on the above insights, we treat NHK matrices as representation of the latent geometric knowledge, upon which we build flexible and principled distillation framework dubbed as Geometric Knowledge Distillation (GKD), which aims at encoding and transferring geometric knowledge by aligning latent manifolds behind GNN models as illustrated in Fig- T We also develop non-parametire and parametric versions of GKD, in terms of different ways to approximate NHK computation. Specifically, the former derives explicit NHKs via assumptions on latent space, and the later learns NHK a data-driven manner.
Applications for geometric knowledge transfer and conventional KD purposes. We verify the efficacy of GKD in terms of different geometric knowledge types (i.c., edge-aware and node-aware ones), and further show its effecttivnness for conventional KD purposes (c.g.. model compression, self distillation, online distillation) for broader applicability. We highlight that our methods consistently exceed teacher model and rival with the oracle model that gives the performance upper bound in principle.
1.2 Links to Related Works
Geometric Deep Learning. The study of geometric deep learning [5t 38 provides fundamental principles and methodology to generalize deep learning methods to Bucllielidean domains (e.g.. graphs and manifolds). From this perspective, architectures for off-the-shelf GNNs (28 49
5% 20
and graph Transformers 667 56 have naturally incorporated the geometric prior knowledge for graphs such as permutation invariance. Despite their remarkable success, they highly rely on graph topology. This work extends the idea of geometric deep learning by treating the global graph

---

topology as special type of prior knowledge, and attempts to encode it into GNNs themselves, such that the trained model would leverage information from the global graph topology even without explicitly taking it as input.
the outputs of a teacher model as alternative supervised signals to teach a student model, with various new paradigms including feature-based 442 66 and relation-based 64 65 ones. While some prior arts [261]] 500 69 63 60) attempted to combine KD and GNNs, i.., graph-based KD, they are nearly straight-forwant adaptations of KD without in-depth investigation the role of graph topology. also restricted by specific choice of GNN architecture or application scenarios. In contrast, we first formalize the problem of geometric knowledge transfer. theoretically answer the question "how to represent graph geometric knowledge and encode it into GNN models", and propose geometric distillation approach based on the theoretical results. More discussions on related works are deferred to Appendix G
Graph-Based Knowledge Distillation. Knowledge distillation (KD) 22
2 Preliminaries
We commence with brief detour to heat equation Riemannian manifolds, and its connection with modern GNN architectures. Moreover, we bring forth the notion of heat kernel to motivate this work.
Heat Equation on Manifolds. We are interested in heat equation defined on smooth k- dimensional Riemannian manifold M. Suppose the manifold is associated with scalar- or vector- valued function stu, () .M [0, oo) + IK" quantifying specific type of signals such as heal at point u E M and time I. Fourier's law of heat conductivity describes the flow of heat with respect to time and space, via partial differential equation (PDE) called heat equation 6 i.e.
Or(u.t) -c (),
(2)
Ot
where e is the thermal conductivity coefficient, and A is the natural Laplace-Belin Beltrami operator associated with M. Rewriting A the functional composition of the divergence operator V and gradient operator V. i.e., A = 4 o V, we can interpret the heat equation as: the variation of temperature within an infinitesimal time interval at a point is equivalent to the divergence between its own temperature and the average temperature on an infinitesimal sphere around it.
Implications on Graphs. A spatial discretisation of continuous manifold yields graph G = EV, whose nodes can be thought of embedded on the base manifold. In fact, the heat equation along with variants thereof (e.g., Schrödinger equation) have found widespread use in modeling graph More importantly, has been recently revealed be intimately related with the architectures of modern GNNs [1 [7): suppose X() = {æ(u, 0) heev E R denotes the initial condition for Eqn. 2 determined by input node features, then solving the heat equation under certain definitions of V* and V (i.e., definition of A) amounts to different architectures of GNNs. For instance:
dynamics [|| 25 3d
Example 1. 51 Define the discretised counterpart of A as the graph Laplacian matrix L = - D A 3. Numerically solving Eqn. @ using the forward Euler method with step size == yields the formulation of Simple Graph Convolution (SGC) (53) where O denote learnable transformation matrix
(D-) AD :)
((),, softmax (x())))
Define the gradient operator Vij as the difference of source and target node features, the divergence operator V as the sum of features of all edges for the node. Numerically solving Eqn. 2 using the explicit Euler scheme with step size T yields the following recursive formulation
Example
X
X + T) = (G 1) (() + X(t)
where G is diffusivity coefficient matrix in place of c.
Moreover, stacking non-linear transformation layer after each step yields the formulation of Graph Convolution Networks (GCN) 28 for Eqn. 3 Graph Attention Networks (GAT) 49 with residual connection for Eqn. 4 and even more GNN architectures by virtue of the flexibility of interpretion for heat equation graphs.
3

---

Heat Kernels. Intriguingly, it turns out that the initial value problem of heat equation on any manifold M has smallest positive fundamental solution depending on the Laplace operator A. known as the heat kernel [2]. It is denoted as a kernel function <(x, y, 1), such that
S.a -J_ k(u,, ug O)dp(u)
x(u;; = e (0
0) 1
(5)
where y is a non-negative measure associated with M. In physies, the heat kernel k( y, t) can be interpreted as a transition density that describes the asymptotic behavior of a natural Brownian motion on the manifold. Its formulation thus can be treated as a unique reflection or representation of the geometry of the underlying manifold. For example, if the manifold is a k-dimensional Euclidean Space Rº or a Hyperbolic Space IH the explicit formula of heat kernel is respectively given by,
x(us, uj t) (4TT)) exp
p² 41
and k(us, Uz
(-1) 1 2 m
- (6)
(4mt)²
sinh p ap
where p = d(u;; uz) denote geodesic distance. Heat kernel has also been adopted for graph-related applications such as community detection [11. graph clustering (58).
3 Extending Heat Kernel to GNNs
The starting point of this work is the development of neural heat kernel, built upon the previously- mentioned connection of GNNs and heat equation. As will be discussed later, this novel notion lends us a thermodynamic perspective to the intrinsic geometric property of the latent graph manifold embodied in GNNs, and hence paves the way for distilling geometric knowledge.
3.1 Neural Heat Kernel
Consider the graph signal (() at time t and node features ( at layer I as interchangeable notions. Consequently, feature propagation using one layer of GNN amounts to heat diffusion on the base manifold M within certain time interval T, leading the equivalences of X(I T) and H(++1;;
1++1) fo (o(H)))) X(t + r) = ((oo. x())))
where fo denotes an arbitrary GNN model with parameter 0, and (ffe. 9) denotes a generalization of Laplace-Beltami Beltrami operator defined over the base manifold M associated with graph G and the arbitrary backbone GNN model fo-
Remark. The equivalence of two equations in Eqn. is based on the recently established connection which reveal that the formulation of a GNN layer could be thought of as discretisations (that correspond to the left equation in Eqn. P of the continuous diffusion process (that correspond to the right equation in Eqn. 7 described by the heat equation. Furthermore, different definitions of Saplace-Beltrant operator A and schemes for solving Eqn. 2 could yield different GNNs (e.g., SGC [53] GAT 49), GRAND D While unclear whether there exists such definition of A for every GNN architecture, write the operator as ( G) to associate with model fe. and then use the analogy between GNN and heat equation as an analytical
between heat equation and GNNs 7 13
tool, in a similar manner with B3 46 5 45
g 7 for studying the geometry property of GNNs. See
more detailed justifications in Appendix
In light of this connection, we consider natural generalization of heat kernel for GNNs, termed as neural heat kernel (NHK) to highlight its difference with heat kernel in the thermodynamic context. In particular, single-layer NHK is defined as a positive definite symmetric kernel function denoted as x(() (v;, v;), where the sub-script 0 implies that it is associated with the architecture and parameters of the backbone GNN, and the super-script (I) implies that it is specific to each layer, analogous to the role of continuous time in Eqn.
Theorem 1. (Existence of Single-Layer NHK) Suppose two expressions in Eqn. are equivalent (see Appendix E for more discussions), then for any graph G and GNN model fo. there exist a unique (1) single-layer NHK function Kig (-) such that for any node Us V and 1 > 0.
h
Kg (v,, 1z)) h 11-1) u(0;))
(8)
VYEV

---

where h ( E IR denotes the feature of node at 1-th layer, and is a measure over verlices that could be specified as the inverse of node degree 1/d4.
To push further, we can generalize NHK across multiple layers of GNN, termed as a cross-layer NHK Ke(e%, Uj, 1 + 1 + k) (c.g.. from I-th layer to (I + k)-th layer of GNN). Its existence could be induced recursively by the semi-group identity property of NHK concerning consecutive GNN layers.
Theorem 2. (Semigroup Identity Property of NHK) The NHK satisfies the semigroup identity property: Ve,, Vj E V and > 0, there exists cross-layer NHK across two consecutive layers
ko(u, Ug 2)
Σ
Kig
(1+1) (0,, u4)Ki
(1+2) U2 Uy)du((r)
OLEV
This theorem indicates that stacks of multiple GNN layers also constitute a valid kernel, i.e.,
h
(1+R)
Ko (os 0,,1++++++++ k)) h5) p(v)). Σ VYEV
(10)
Analogous to heat kernel an unique characterization of the underlying space, NHK characterizes the geometric property of the latent graph manifold for GNNs. Additionally, NHK is dependent on GNN models through the definition of the associated Laplace-Beltte Belttrmmi operator (ffe. 9), inheriting the expressiveness of neural networks and varying through the course of training. Intuitively, NHK can be thought of as a model-driven encoding for topological information, encapsulating the geometric knowledge learned by GNNs into a tractable functional form.
3.2 Application in Geometric Distillation
Consider the problem of distilling geometric knowledge, which involves an intelligent teacher model fe., which is exposed to and pre-trained over the (relatively) complete graph 9 -V,, and student model fo that is exposed to the partial graph G = (V, where V S V and & S (V x nE. Our target is to train a student model (with the help of teacher model) that operates on G be as competitive as models operating on during inference. Since 9 a sub-graph of G, they should lie in the same space (i.e., latent manifold) governed by the underlying mechanism of data generation, and hence we expect student and teacher models to capture the same geometric property of this shared space. This leads to the principle of Geometric Knowledge Distillation (GKD); transfer the geometric knowledge the intelligent teacher to the student such that the student can propagate features as if is aware of the complete graph topology (see the example in Fig.
To this end, we resort to NHK matrices on the teacher (resp. student) model over the complete (resp. partial) graph as instantiations of their geometric knowledge, denoted as
(Teacher) Kg. (G, + k) = (150 (wi, Ug 1 ++)++)))m)))) 1) (Student) Kg - 1 k:) = {nO(US, U5, + 1 1>)|11
written compactly as (()+)((g) when A: = 1. The NHK matrix is a positive semi-definite symmetric matrix, and alike K, is dependent on the GNN model fo and graph G. Denote K (G) R the sub-matrix of K (9) with row and column indices in V. The distillation loss for GKD) is
(iTKo-vv(,,1 Lasi (Ko v. Ko + 1 + k) d(Ko ( - 1 + k), -
(11)
where .() is a similarity measure, for which we choose Frobenius distance as implementation,, i...
1 if (v), E E {
d(k d(Kgr,v, Kg) = ([(Ka Kg) @ W W
(12)
if (vi, U3) É E.
where W E XM is a weighting matrix to trade-off distillation loss with respect to different node ((++1 pairs depending on their connectivity. For & 1, the loss can be re-written as L (Ko Ko). Note that one can also specify different & for teacher and student models Eqn. in case when the teacher model is deeper.
5

---

Instantiations for Geometric Knowledge Distillation
Unfortunately, deriving explicit formulas for NHKs is prohibitively challenging due introduction of non-linearity. To circumvent it, we propose two types of instantiations for GKD, i.e., con-aaamettric and parametrie. The former considers explicit NHKs by making assumptions on the underlying space, and the latter learns NHK in data-driven manner.
4.1 Non-Parametric Geometric Distillation
Deterministic Kernel. One instantiation of NHK is a Gauss- Gauss-Weiersiee Weierstrass kernel in the form of Eqn. G by assuming the underlying space is Euclidean space. Since the distillation loss in Eqn.
a homogeneous function, we can remove its scaling factor and define NHK as
1
h
12
(Gauss-Weierstree Weierstras NHK) Ke(U;, uj,
exp
(13)
4T
where T denotes the estimation of the accumulated time interval. Alternatively, we can use Sigmoid kernel and define non-parametric NHK as:
(Sigmoid NHK) Ko(U;, 1,, tanh (a (h (h.2, )+))
(14)
where a, are positive constants depending / and k. is natural and intuitive choice similarity measurement and empirically found as-well effective, albeit does not correspond to any named manifold to our knowledge.
Randomized Kernel. We can also define Randomized kernel based on the following theorem.
Theorem (Expansion of NHK) Let {spw) = be orthonormal basis of eigenfunctions of -A(fe, G) with eigenvalues do S λ1 A2 ..., NHK allows the expansion:
Au yw (vx) "px (vz).
(15)
Based on this result, we resort to the approximation of NHK by defining a randomized kernel in similar form Eqn. (15,, leading to the following formulation of randomized NHK:
(Randomized NHK) Ki 11,,,,,,1+
AgT T a (Wwh))) a (Wwry)). (16)
K==
&.. is transformation where (Wirh:) is used to proximate sp (v:). WE I ..........
matrix, N (0, Iu) is a d-dimensional random variable from Gaussian distribution. In fact, under certain choice of activation function Eqn. (6)) could approximate diversity of kernels [I] 10 This design essentially enforces the alignment of teacher and student for arbitrary underlying manifold.
Training Scheme. We follow the standard training paradigm in KD literature is pre-trained by supervised prediction loss involving all labeled nodes in V. After teacher is well-trained, we fix f² and train the student model according to
22
17
the teacher
0 = arg min Spre Yo. Y)
Σ
( dis (Ko Ko),
(17)
L
where Y denotes ground-truth labels of labeled nodes in V, and Y, denotes the predictions of student model fo G. Lais is the distillation loss defined by Equ. denotes the total number layers.
4.2 Parametric Geometric Distillation
Inheriting the similar spirit of auto-encoding Bayes 27 we introduce a variational inverse-NHK that is independently parameterized, denoted N. whose existence is guaranteed by the invertibility of

---

NHK matrices. Together with Ni, they define a symmetric form characterizing feature propagation:
(Forward) h ((++k)
Σ
Ko Uj 1,,,+++++
hy) ((vz),
(18)
BEEV
(Backward)
h Σ "yEV
(o Uy, + U,
h (1++) ((ez).
(19)
practice, we follow existing kernel learning approaches
22 and parameterize the inverse-NHK as R (1, 11, + k + 1) g (+*)) (h (b))+)))
(20)
where Go : Rd R the associated learnable non- non-linear mapping. Given a pre-trained teacher model, distilling geometric knowledge boils down to 1) establishing equivalence of Eqn. (88 and Eqn. 19p. and 2) matching pseudo-inverse NHK matrices for teacher and student models (respectively denoted as K- and K) with clear meanings), leading to the training scheme as follows.
Training Scheme. Based on Eqn. 19 can define reconstruction loss with respect to the teacher model (similar applies to the student model) as
Crea ( (H() H
K H [
H
(21)
Then, minimizing the reconstruction loss with fixed GNN model parameter 9 amounts optimizing the variational parameter Φ, and minimizing prediction and distillation losses given fixed e amounts optimizing the student model parameter 0:
(H ((+b)) H!!)) ) + Crea (H(+)) H
Φ arg min
Cree
(22)
0 - arg min
Spre
Yo, Y) + aLdis (K) v. K or + +k++))
(23)
Applying two steps iteratively adds up to an EM-like algorithm for training the student model. In practice, we set + k as the last layer, and I the first layer to use as much information as possible. We justify the parametric GKD approach in Appendix. Diyy showing it essentially explores the true NHK behind GNN.
Experiments
We conduct experiments to validate the efficacy of our method on graph-structured data in terms of various types of privileged geometric knowledge, combinations of teacher-student GNN architectures and potential application scenarios. We use three benchmark datasets Cora [35,. Citeseer [4], Pubmed (39], and larger dataset OGBN-Arxiv [ for node classification tasks. More imple- mentation details and experimental results are deferred to Appendix. The codes are available at https://gi thtp//.ttttuuu... cww/ hhtpc1h/221/1// GKD
Implementation and Competitors. We consider the following variants of the proposed GKD. 1) GKD-G: non non-parametric Gaussian NHK; 2) GKD-S: non non-paraetrric Sigmoid NHK; 3) GKD- R: randomized NHK; 3) PGKD: parametric NHK. We choose KD methods that is representative in its own category for comparison, including KI) 22 FSP
64 ISP (3) We also report the performances of teacher and student model trained with the standard classification loss, short as Teacher and Student. The teacher model is trained using the complete graph ś. and, to calibrate with all other methods, tested using the partial graph G. Besides, is both trained and tested G, which naturally takes advantaged place given more information during inference. Since our method is compatible with the vanilla KD paradigm (22. we report the performance delivered by their combinations (i.e., GKD+KD and PGKD+KD).
consider an Oracle model which
Experiment Settings. We investigate on various experimental settings according to different types of privileged geometric knowledge. In the case of edge-aware geometric knowledge, the teacher model has access to additional edge information. i.c., & C E and V V. In the case of node-aware geometric knowledge, the teacher model has access to additional node information, i.e.. V C Í and = (V x V}. We also consider other conventional KD settings including model compression, self-distillation and online distillation, which will be illustrated in detail. The backbone so set as 3- 3-layer GCN 28 for both student and teacher models, unless otherwise stated.
7

---

Cora
CiteSeer
PubMed
Cora
CiteSeer
PuhMed
Onccle
63 +
73.64 0.48
87.16 0.19
Oracle
88.63 0.48
73.64 + 0.48
87.16 + 0.19
Teacher
81.61 + 0.37
70.88 + 0.62
$1.12 + 0.52
Teacher
87.27 + 0.51
72.92 - 0.90
85,98 + 0.23
Student
83.84 + 1.32
69.94 + 0.76
85.35 0.43
Student
84.84 + 1.61
70.32 + 1.12
84.74 + 0.27
KD
81.81 + 1.19
70.01 0.37
85.58
0.32
KD
86.71 + 0.77
71.96 + 1.10
85.55 + 0.45
FitNets
83.72 + 1.45
09.99 +: 0.56
85.06
0.27
FaNets
86,09 1: 1.12
72.00 + 0.78
85.78 + 0.26
FSP
$3.55 + 2.19
71.43 1.26
$5.16
+ 0.31
FSP
85.83 + 1.66
70.92 + 1.46
85,20 + 0.45
LSP
83.90 +
70.23
0.79
85.37 0.49
LSP
85.67 1.22
70.66 4: 1.01
85.71 0.50
GKD-G
87.68 + 1.07
73.04
0.70
85.74
0.38
GKDDG
88.66 0.85
73.18 + 0.88
86.07 + 0.45
GKD-S
88.01 +: 0.79
72. 46
0.52
85.94 +: 0.43
GKD-S
88.54 +: 0.52
72.85 + 0.57
86, 10 0.42
GKD-R
T2.97
86.19
0.55
GKD-R
88.98 0.39
72.80 0.22
86.16
PGKD
$8.41 + 0.62
73.12
1: 0.58
86.41
0.24
PGKD
89.15
0.45
73.33 - 0.36
86,09 - 0.54
GKD+KD
88.95 +: 00.30
73.21
+: 0.53
86, 29 0.28
GKD+KD
89.10 + 0.44
72.94 + 0.80
86,24 + 0.26
PGKD+KD
89.09
0.40
73.45
0.48
86.48
0.52
PGKD+KD
89.23 0.61
73.41 + 0.60
86.20 + 0.36
Table 1: Results of node classification accuracy for the edge-aware knowledge setting,
Table 2: Results of node classification accuracy for the node-aware setting.
Setting
Oracle
Teacher
Student
KD
GKD
PGKD
Edge-Aware
71.46 + 0.41
67.96 0.78
66.41 0.45
68.63 + 1.21
70.90 0.80
71.38 1.01
Node-Aware
71.46 + 0.41
69.35 + 0.72
67.49 + 0.65
68.86 + 0.66
71.31 + 0.83
71.27 0.70
Table 3: Results of testing accuracy on OGBN-Are Arxiv dataset.
5.1 Main Results
Edge-Aware Geometric Knowledge. We report results for the edge-aware geometric knowledge setting. To quantify the privileged information, we set the quantity (Iε) | called privileged information ratio (PIR), as 0.5. As shown in Tab. D all variants of GKD outperform other KD baselines both datasets, and significantly exceeds both Student and Teacher models. Further, GKD and its variants rival, if not surpass, the Oracle model. In other words, the student model trained using GKD could use far less graph topological information achieve very close performance to competitors that are aware of the full graph topology during inference. Furthermore, the parametric PGKD performs better than its non-parametric counterparts in most cases, and GKD-R is the most effective non- -on-paraeetric method in general. Despite that, GKD-G and GKD-S are also effective while being simpler. We presume that the performance variation of different GKD realizations stem from the different geometric property governed by the feature of datasets.
Node-Aware Geometric Knowledge. We further investigate the node-aware geometric knowl- edge setting where the teacher model has access to more labeled nodes and their relations with the rest nodes. We set the PIR w.r.t. labeled nodes, defined as ([Vtreix | |Vtrass 1)/ D/rataai I to 0.5. A unique challenge of this setting compared to the edge-ware counterpart is that, apart from graph topological information, the student model has less labeled training samples. As we can see from Tab. 2 The proposed GKD and its variants again consistently outperform KD baselines throughout all the cases, surpasses both Student and Teacher models, and are even as competitive as Oracle.
Larger Dataset. Table presents results on large graph, i.c., OGBN-Arxiv. We use the same PIR setting as citation networks, and choose the best variant of GKD to report in the table. Since the space complexity for GKD is O(n²), we randomly draw a mini-batch of nodes for computing the distillation loss in practice. Note that the loss function is unbiased as long as all nodes are evenly covered. It could also be seamlessly integrated with the original mini-batch method (by sampling ego-grape ggo-graphs) used for training large graphs without further modification. Again, we found our methods consistently outperform Teacher and Student models, and are close to the performance of the Oracle model, which suggests the effectiveness of GKD in large graphs.
Performance Variation with Privileged Ratio. The results with respect to varying privileged (edge-aware and node-ward information ratio are given respectively in Fig. 2 and Fig. The performance of Oracle model is invariant as it is trained and tested on the same (complete) graph. In general, for Teacher model, Student model, and vanilla KD, their performance drops with increasing PIR quantifying the information loss. In contrast, our method significantly more robust, only showing slight performance deterioration, exceeding the Teacher model, and approaching the Oracle model. Besides, we find an interesting phenomenon that in the edge-aware setting on Pubmed dataset, the performance of Teacher model is the worst, presumably because that the Teacher is trained using
8

---

672
Test ACCUFRNC
0.00
Exudet
12.2
22
03
84
35
07
66
14
64
87
0.3
63
C4
05
37
8.
30
(a) Cora
(b) Citeseer
(c) Pubmed
Figure 2: Performance variation with increasing PIR for the edge-aware knowledge setting.
425
400
Taater Oniee
Burre
Orde
Oede
Ratio
Ratio
Ralio
(a) Cora
(b) Citeseer
(c) Pubmed
Figure 3: Performance variation with increasing PIR for the node-aware knowledge setting.
Cora
Citeseer
Pubmed
13
-0
Time Epoch (mc)
25
30
Time/ (Cooch (mm)
88
1.00
Timee (Coch
30
15
15
0.90
16
-o
0.85
i
10
20
Urocl
Racchrr
student
KD1
FaNe
GND GI
GKD-S SF
6K0 R
POKD
Drccla
Teacher
student
Ko
FAN
FS1
GKD-GE
GID-S
GKD-B
PGKD
Reacher student
Ko
Fibtess
FSP
LSP
GKD nOXO
40
80
Epoch
120
160
200
Figure 4: (Left three panels) Training time per epoch (ms) for GKD and baselines on citation networks.
(Right panel) Convergence curves different methods on Cora, where the circle denotes the epoch when the best validation accuracy has been reached.
fully observed graph, and may perform poorly once the privileged part of graph topology becomes unavailable at test time.
5.2 Scalability Test
Training Time. We report the the training time per epoch of different methods in the left three panels of Fig. where we use the whole graph for training on Cora and Citeseer, and set the batch size as 5, 000 Pubmed. While the computation complexity for distillation loss of GKD (in both non non-parann non-parametric and parametric cases) is (dn²), we find that in practice their training time is not significantly worse than other KD baselines. specific, all variants of GKD take less than two times of the training time of vanilla KD while yielding better performance, and the simplest GKD variant using Sigmoid NHK even more efficient than baselines in some cases. mentioning that the distilled student models are equally efficient by using sparser graph structure (in edge-aware setting) or smaller graph (in node-aware setting), which are suitable for being deployed in latency-constrained scenarios.
also worth
Convergence Speed. We also compare the convergence speed of different variants of GKD and other methods in the right panel of Fig. where we use the same model architecture and optimizer for different methods and finetune other
to ensure fair comparison. Since the loss function is different for different methods, we report the convergence of training accuracy. As shown by the figure, GKD variants using Sigmoid and Gaussian NHKs can converge as fast as the vanilla KD. While GKD-R and PGKD in general take more epochs converge, they can converge within 200 epochs (which is the default setting recommended in (28) and the best validation accuracy is achieved within the first 100 epochs. We also find that EM-style algorithm used for training PGKD does not cost too much extra epochs for convergence than the non can-aaamettric GKD which is
reasonable since the learnable mapping de used Eqn. (22 is an independent module that is not involved in GNN's 's feed-forwart computation.

---

Oracle
Teacher
Student
GKD-G
GKD-S
GKD-R
PGKD
Offline Online
88.63 0.48
84.61 + 0.37
83.84
1.32
87.68
1.07
88.01 + 0.79
88.43 0.59
88.41 0.62 88.50 0.33
88.63 + 0.48
84.61 t 0.37
63.84
1.32
87.75 0.65
87.63 t 0.65
88.28 + 0.80
Table 4: Comparison of offline and online distillation on Cora in edge-aware setting.
5.3
Other Settings
Model Compression. We report the per- formance of GKD in the conventional model compression setting [22 where the teacher and student are different-sized
Setting
Teacher
Student
GKD
Compression
SGC
SGC
SGC
86.32
0.32
87.15 +
GNN models. In this setting, we train and test both teacher and student models on the complete graph. We use GCN as the backbone for teacher with relatively large hidden size 64, and use SGC or GCN (with hidden size 8 and 16) the student model. We report the best result among all vari- ants of GKD. As shown in the first section of Tab. S our method achieves notable im- provements over the student model, render- ing it as useful KD approach for model compression.
GCN-64
GCN-8
GCN-8
GCN-8
88, 76 +: 0.34
84.52 + 1.34
87.50 + 1.04
88.30 +: 0.46
GCN-16
GCN-16
GCN-16
87.24 +: 0.43
88.09 +: 0.79
88.62 d: 0.50
GCN-32
GCN-32
GCN-32
Soff-Disdl
58.63 0.48
88.98 +: 0.34
89.23 = 0.52
GCN-16
GCN-16
GCN-16
87.21 + 0.43
87.62 +: 0.52 88.56 +: 0.40
Table 5: Node classification accuracy on Cora in set- tings: 1) model compression: 2) self-distillation.
Self-Distillation. We further report results for the setting of self-distillation 15 37 special case of KD when the teacher's and student's architectures are identical, often used for refining their own performance. The results as shown in Tab. 5 validate that GKD could also be used to effectively boost GNN's own performance.
which is
Online Distillation. Table. shows the performance of different variants of GKD for online distillation where both the teacher model and the student model are trained in an end-to-end manner, in contrast to offline distillation where the teacher model is pre-trained. The results demonstrate the potential usage of GKD in this setting.
Conclusion, Current Limitations and Future Works
This paper formalizes the problem of graph topological knowledge transfer for GNNs. We investigate on the implication heat kernel in GNNs and propose the novel notion of neural heat kernel. We leverage it to characterize the geometric property of the underlying manifold for graphs, and propose the framework of geometric knowledge distillation to transfer geometric knowledge from teacher GNN model student GNN. Experimental results validate the effectiveness of our approach in various practical settings.
Despite that the proposed GKD is effective in various tasks and possesses decent training efficiency in practice, its theoretical space and time complexities are O( and (dn?) and the parametric instantiation may take some extra time converge compared to the pure non-KI counterpart. The algorithmic complexity can be reduced by using mini-batch training, and there also exist ways to reduce the overhead such as pre-computing teacher's NHK matrix using low-rank approximation. Finally, we do not foresee any direct negative societal impacts this work.
Acknowledgement
This work was partly supported by National Key Research and Development Program of China 2020AAA01176000, National Natural Science Foundation of China (61972250, 72061 127003), Shanghai Municipal Science and Technology (Major) Project (2251 105100. 2021SHZDZZ0102)
and
10

---

References
[1] Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? NeurIPS, 27, 2014.
[2] Nicole Berline, Ezra Getzler, and Michele Vergne. Heat kernels and Dirac operators. Springer Science & Business Media, 2003.
[3] Cristian Bodnar, Francesco Di Giovanni, Benjamin Paul Chamberlain, Pietro Liò, and Michael M Bronstein. Neural sheaf diffusion: A topological perspective on heterophily and oversmoothing gnns. arXiv preprint arXiv:2202. 04079, 2022.
[4] Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličkovié. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2 13478 2021.
[5] Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geometric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine, 2017.
[6] John Rozier Cannon. The one-dimensional heat equation. Number 23. Cambridge University Press, 1984.
[7] Benjamin Chamberlain, James Rowbottom, Davide Eynard, Francesco Di Giovanni, Xiaowen Dong, and Michael Bronstein. Beltrami flow and neural diffusion on graphs. NewrIPS, 2021.
[8] Benjamin Paul Chamberlain, James Rowbottom, Maria Gorinova, Stefan Webb, Emanuele Rossi, and Michael M Bronstein. Grand: Graph neural diffusion. ICML, 2021.
[9] Yuzhao Chen, Yatao Bian, Xi Xiao, Yu Rong, Tingyang Xu, and Junzhou Huang. On self- distilling graph neural network. arXiv preprint arXiv:2011.022:: 2020.
[10] Youngmin Cho and Lawrence Saul. Kernel methods for deep learning. NeurlPS, 22:342-350, 2009.
[11] Fan RK Chung and Fan Chung Graham. Spectral graph theory. Number 92, American Mathematical Soc., 1997.
[12] Francesco Di Giovanni, James Rowbottom, Benjamin P Chamberlain, Thomas Markovich, and Michael Bronstein. Graph neural networks as gradient flows. arXiv preprint ar0iv:2206. 10991, 2022.
[13] Moshe Eliasof, Eldad Haber, and Eran Treister. Pde-gen: Novel architectures for graph neural networks motivated by partial differential equations. Advances Neural Information Processing Systems, 34:3836-3844 2021.
[14] Matthew Fahrbach, Gramoz Goranci, Richard Peng, Sushant Sachdeva, and Chi Wang. Faster graph embeddings via coarsening. In ICML, 2020.
[15] Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In International Conference on Machine Learning, pages 1607- 1616. PMLR, 2018
[16) Vikas K. Garg, Stefanie Jegelka, and Tommi S. Jaakkola. Generalization and representational limits of graph neural networks. In ICML, pages 3419-3430, 2020.
[17] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 1999-1998,1 2021.
[18] Alexander Grigoryan. Heat kernel and analysis on manifolds, volume 47. American Mathemat- ical Soc., 2009.
[19] Alexander Grigor'yan yan. Estimates of heat kernels on riemannian manifolds. London Math. Soc. Lecture Note Ser, 773:140-225, 1999.
[20] William Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NeurIPS, pages 1025-1035, 2017.

---

[21] Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. A comprehensive overhaul of feature distillation. In ICCV, pages 1921-1930, 2019.
[22] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprini arXiv: mee../2533....1 2015.
[23] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. NeurIPS, 33:221 18-22133, 2020.
[24] Yu Jin, Andreas Loukas, and Joseph JaJa. Graph coarsening with preserved spectral properties. In AISTATS, 2020.
[25] Matthias Keller and Daniel Lenz. Unbounded laplacians on graphs: basic spectral properties and the heat equation. Mathematical Modelling of Natural Phenomena, 5(4):198 224, 2010.
[26] Jangho Kim, SeongUk Park, and Nojun Kwak. Paraphrasing complex network: Network compression via factor transfer. Advances in neural information processing systems, 31, 2018.
[27] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In International Conference Learning Representations, 2014.
[28] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.029000 2016.
[29] Johannes Klicpera, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. arXiv preprint arXiv:1810.05997, 2018.
[30] Johannes Klicpera, Stefan Weißenberger, and Stephan Günnemann. Diffusion improves graph learning. arXiv preprint arXiv: 1.05485, 2019.
[31] Kyle Kloster and David F Gleich. Heat kemnel based community detection. In KDD, pages 1386-1395, 2014.
[32] Zenan Li, Qitian Wu, Fan Nie, and Junchi Yan. Graphde: A generative framework for debiased learning and out-of-distriuttion detection on graphs. In Advances in Neural Information Processing Systems, 2022.
[33] David Lopez-Paz, Léon Bottou, Bernhard Schölkopf, and Vladimir Vapnik. Unifying distillation and privileged information. ICLR, 2016.
[34] Hao Ma, Dengyong Zhou, Chao Liu, Michael R. Lyu, and Irwin King. Recommender systems with social regularization. In International Conference on Web Search and Web Data Mining, pages 287-296, 2011.
[35] Andrew Kachites McCallum, Kamal Nigam, Jason Rennie, and Kristie Seymore. Automating the construction of internet portals with machine learning. Information Retrieval, 2000.
[36] Georgi S Medvedev. The nonlinear heat equation on dense graphs and graph limits. SIAM Journal Mathematical Analysis, 46(4):2743-2766 2014.
[37] Hossein Mobahi, Mehrdad Farajtabar, and Peter L Bartlett Self-distillation amplifies regular- ization in hilbert space. arXiv preprint arXiv:2002.05715. 2020.
[38] Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodola, Jan Svoboda, and Michael M Bronstein. Geometric deep learning on graphs and manifolds using mixture model cnns. In CVPR, pages 5115-5124, 2017.
[39] Galilco Namata, Ben London, Lise Getoor, Bert Huang, and UMD EDU. Query-driven active surveying for collective classification. In International Workshop on Mining and Learning with Graphs, 2012.
[40] Wonpyo Park. Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 3967-3976, 2019.
12

---

[41] Ali Rahimi, Benjamin Recht, et al. Random features for aaree-ccal kernel machines. In NIPS.
[42] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprini arXiv: 1412.6550, 2014.
[43] Laurent Saloff-Coste Aspects of Sobolev-type inequalities, volume 289. Cambridge University Press, 2002.
[44] Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi- Rad. Collective classification in network data. Al magazine, 2008.
[45] Matthew Thorpe, Tan Minh Nguyen, Hedi Xia, Thomas Strohmer, Andrea Bertozzi, Stanley Osher, and Bao Wang. Grand++: Graph neural diffusion with a source term. In International Conference Learning Representations, 2021.
[46] Jake Topping, Francesco Di Giovanni, Benjamin Paul Chamberlain, Xiaowen Dong, and Michael M Bronstein. Understanding over-squashing and bottlenecks graphs via curvature. arXiv preprint urXiv:2111.145222 2021.
[47] Vladimir Vapnik, Rauf Izmailov, et al. Learning using privileged information: similarity control and knowledge transfer. JMLR, 16(1):00332049, 2015.
[48] Dmitri V Vassilevich. Heat kernel expansion: user's manual. Physics reports, 388(5-6):279-366) 2003.
[49] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. ICLR, 2018
[50] Shuai Wang, Kun Zhang, Le Wu, Haiping Ma, Richang Hong, and Meng Wang. Privileged graph distillation for cold start recommendation. arXiv preprint S.Kiv:2005.19975 2021.
[51] Yifei Wang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Dissecting the diffusion process in linear graph convolutional networks. NeurlPS, 2021.
[52] Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning. In Artificial intelligence and statistics, pages 370-378. PMLR, 2016.
[53] Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Simplifying graph convolutional networks. In ICML, pages 6861-6877 2019.
[54] Qitian Wu, Chenxiao Yang, and Junchi Yan. Towards open-world feature extrapolation: An inductive graph learning approach. Advances in Neural Information Processing Systems, pages 19435-19447, 2021.
[55] Qitian Wu, Hengrui Zhang, Junchi Yan, and David Wipf. Handling distribution shifts on graphs: An invariance perspective. In International Conference on Learning Representations, 2022.
[56] Qitian Wu, Wentao Zhao, Zenan Li, David Wipf, and Junchi Yan. Nodeformer: A scalable graph structure learning transformer for node classification. In Advances in Neural Information Processing Systems, 2022.
[57] Shu Wu, Menggi Zhang, Xin Jiang, Xu Ke, and Liang Wang. Personalizing graph neural networks with attention mechanism for bession-based recommendation. TKDE, 2019.
[58] Bai Xiao, Edwin R Hancock, and Richard C Wilson. Geometric characterization and clustering of graphs using heat kernel embeddings. Image and Vision Computing, 2866)1003-1001, 2010.
[59] Keyulu Xu, Jingling Li, Mozhi Zhang, Simon S. Du, Ken-ichi Kawarabayashi, and Stefanie Jegelka. How neural networks extrapolate: From feedforward to graph neural networks. In ICLR, 2021.
(60] Bencheng Yan, Chaokun Wang, Gaoyang Guo, and Yunkai Lou. Tinygon: Learning efficient graph neural networks. In KDD, pages 1848-1856, 2020.
13

---

[61] Cheng Yang, Jiawei Liu, and Chuan Shi. Extract the knowledge of graph neural networks and go beyond An effective knowledge distillation framework. In WWW, pages 1227-1237, 2021.
[62] Chenxiao Yang, Junwei Pan, Xiaofeng Gao, Tingyu Jiang, Dapeng Liu, and Guihai Chen. Cross-task knowledge distillation multi-task recommendation. AAAI, 2022.
[63] Yiding Yang, Jiayan Qiu, Mingli Song. Dacheng Tao, and Xinchao Wang. Distilling knowledge from graph convolutional networks. In CVPR, 2020.
[64] Junho Yim, Donggyu Joo, Jihoon Bac, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In CVPR, 2017.
[65] Shan You, Chang Xu, Chao Xu, and Dacheng Tao. Learning from multiple teacher networks. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1285-1294 1294, 2017.
[66] Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. ICLR, 2017.
[67] Jiawei Zhang. Haopeng Zhang, Congying Xia, and Li Sun. Graph-bert: Only attention needed for learning graph representations. CoRR, abs/2001 005 140, 2020.
[68] Ke Zhang, Carl Yang, Xiaoxiao Li, Lichao Sun, and Siu Ming Yiu. Subgraph federated learning with missing neighbor generation. NeurlPS, 34, 2021.
[69] Shichang Zhang. Yozen Liu, Yizhou Sun, and Neil Shah. Graph-less neural networks: Teaching old mips new tricks via distillation. arXiv preprint I 2021.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're rou're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "iaLQQyWgsh_", "reviewer": "Reviewer_1r32", "summary": "In this paper, the authors formalize the problem of graph topological knowlege transfer for GNNs. A new notion of neural heat kernel is proposed to characterize the geometric property of th eunderlying manifold for graphs. And a framework of geometric knowlege distillation to transfer geometric knowledge from a teacher GNN to student GNN. ", "questions": "1. What's the effect of the proposed methoc on the social network dataset, e.g., Reddit?\n2. There are many distilling strategies. Is it possible to give more comparison between different distilling strategies? And different parameters?\n", "limitations": "The paper produces good experimental results and careful theoretical deriviation. I hope to see the effect on the dynamic graph of online distilling tasks. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper proposes to distilling structural knowledge from teacher model of large graph to student model of small graph, which sounds novel and useful. \n2. NHK is presented to encapsulate the geometric property of the underlying manifold concerning the architecture of GNN.\n3. The experimental results demonstrate the efficacy of the proposed method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "bzmAVODXdix", "reviewer": "Reviewer_w36o", "summary": "This paper focus on the knowledge distillation of GNN, where the teacher GNN has access to the complete version of a (sometimes large) graph, and the student GNN is only exposed to parts of it. The authors introduce the concept of heat kernels into the GNN formulation (termed \"neural heat kernels\"), given that it has been utilized in graph applications before GNN and has demonstrated its experimental effectiveness in the context of graph distillation.", "questions": "I am unfamiliar with this research topic; therefore, I would like to see the discussions on the following points.\n\n**[Why the two expressions in Eq.7 could be equivalent, i.e., Spatial vs Time]**\n\n**[Depth vs Width, in the formulation of NHK]**\n\n**[Computation Complexity]**\n", "limitations": "The authors claimed to address the limitations of their work in the checklist, however, I can't find its location. Adding the pointer to describe the limitations and future directions would be better.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n- ***Originality***: The methods are newly formulated, the existed task of grpah distiallation is well defined.The related work is adequately cited. \n\n- ***Quality***: The work is complete, my concerns on the experimental settings and results are put at the *clarity* and **Questions**.\n\n- ***Clarity***: The submission is clearly written and well organized. My concern is why NHK is especially helpful for the setting of graph distillation compared with other methods that describe the latent structure of the graphs. Also:\n\n  **[unify the notations]** e.g., in line 124 and eq.7, $\\Delta(\\theta, G)$ and $\\Delta\\left(f_{\\theta}, \\mathcal{G}\\right)$ are essetially representing the same entity, a Laplace-Beltrami operator.\n\n- ***Significance:*** With the code provided in the supplementary, other researchers or practitioners can use or build on the ideas. However, the performance gains are insignificant since the benchmark (Cora, Citeseer, and PubMed) are well studied and have less space for improvement.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "AU0FzlVB-Mb", "reviewer": "Reviewer_1tL3", "summary": "This paper proposes a new method of knowledge distillation for graph neural\nnetwork, which takes into account the geometry of the network architecture, in\nparticular edge- and node-awareness. The objective of knowledge distillation in\nthe context of teacher-student graph networks is to try to make the student\nnetworks, trained on partial graph (hence much smaller scale), mimic the best\nclassification performance of the teacher (trained on complete graph).\n\nTo do this, the authors propose a generalization of the heat kernel for GNNs,\ndubbed Neural Heat Kernel (NHK), based on the previous works that established\nconnection between heat kernels and different GNNs architecture. They establish\ntheoretical arguments for the existence of a single-layer NHK (theorem 1), and\na semigroup identity property (theorem 2) which proves that the operator\nresulting from stacking multiple later of NHKs remains a valid kernel.\n\nIn terms of modeling, they proposed the non-parametric and parametric form of\nNHKs, together with corresponding loss functions and training schemes. The\nnon-parametric distillation compose of three different forms of deterministic\nneural heat kernels, while parametric is based on an EM-like training\nscheme. All of these are demonstrated to have better classification\nperformances from related methods and even oracle networks.\n", "questions": "- In theorem 1 regarding the existence of NHK, I see the assumtion to have the\n  equivalence between graph signal X and node features layer H. Can the authors\n  comment more on this assumption, for example, will this ever be violated?\n  \n- In Section 5.4 on performance with privileged ratio, maybe I'm missing\n  something but why do we have the oracle in edge-aware setting perform quite\n  bad in only Pubmed dataset (i.e. I don't get your point on the\n  justification)?\n", "limitations": "Limitations have not been discussed in the main text.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- Overall I find the paper well-written, with clear structure -- although a bit\n  dense and undefined notations in the beginning (comments below in the\n  weakness).\n\n- The paper propose a more rigorous formalization of Knowledge Distillation in\n  the context of Geometric Deep Learning, with theoretical justification borrow\n  from PDE and differential geometry literature.\n  \n- The authors deliver some theoretical arguments to their proposed methods, and\n  empirical results show they all perform better than existing solutions on\n  even large-scale dataset.\n\nWeaknesses:\n\n- Major: I see absolutely no discussion on the computational time of the NHKs\n  in both parametric and non-parametric settings. Since now the training loss\n  need to take into account additional information of the underlying\n  topological space, the training schemes look quite complicated, which require\n  heavier part on hyperparameters-tuning. The choice of hyperparameters are\n  addressed in the Appendix, which resort to grid-search, so if I'm not wrong\n  this does not scale well in terms of computational time. I insist the authors\n  provide justification on this part, and maybe even better to add it in the\n  main text.\n\n- Major: related to first point, even though the performance of parametric NHK\n  (denoted by VGKD in the experimental results section) is generally the best,\n  I see no discussion on the convergence speed EM-like scheme for training the\n  parametric NHK.\n\n- Minor: The matching loss $\\mathcal{L}_{pre}$ is nowhere to be found in the\n  paper, which render missing equation in algorithm box 2. Could the author\n  provide the form of this loss somewhere, as it is quite important part as\n  well?\n\n- Minor: I'm a bit confused about the notation of the Example 1 in Section 2 --\n  what is $\\sigma$ fuction and what is $\\mathbf{\\Omega}$? Also is this the same\n  $\\sigma$ to denote activation function, which is used in Eq.(16)?\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-FySdZecYcE", "reviewer": "Reviewer_emui", "summary": "The authors formulate a new knowledge distillation problem: making a GNN trained on a subgraph emulate the output of another GNN trained on a larger graph. To tackle the problem, they draw inspiration from thermodynamics to propose a kernel over GNN embeddings. Then, distilling knowledge is reduced to a task of matching kernel matrices --- or between a network's embeddings an another's smoothed by the kernel matrix.", "questions": "* Both parametric and non-parametric approaches depend on kernels, which might be a computational bottleneck. How do they scale with the number of nodes?", "limitations": "Paper lacks a proper discussion of its own limitations. There is also no statement regarding possible societal impacts.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n* To the best of my knowledge, authors draw a novel connection between graph convolutions and heat kernels;\n* Authors attack an original problem: condensing the geometry of a large graph into a GNN trained on a subgraph;\n\n## Weaknesses\n* While novel, the motivation for the problem is weak. In most interesting cases, we will only observe a subgraph and there is no oracle capable of outputting embeddings based on the whole graph. This seems to limit the applicability of this sort of method. it would be more reasonable just to have some other graph(s) independently drawn from the same data-generating process;\n* The connection to thermodynamics seems very thin, as it just reduces to computing kernels over embeddings.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept