Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper focus on the knowledge distillation of GNN, where the teacher GNN has access to the complete version of a (sometimes large) graph, and the student GNN is only exposed to parts of it. The authors introduce the concept of heat kernels into the GNN formulation (termed "neural heat kernels"), given that it has been utilized in graph applications before GNN and has demonstrated its experimental effectiveness in the context of graph distillation.
strengths_and_weaknesses: 
- ***Originality***: The methods are newly formulated, the existed task of grpah distiallation is well defined.The related work is adequately cited. 

- ***Quality***: The work is complete, my concerns on the experimental settings and results are put at the *clarity* and **Questions**.

- ***Clarity***: The submission is clearly written and well organized. My concern is why NHK is especially helpful for the setting of graph distillation compared with other methods that describe the latent structure of the graphs. Also:

  **[unify the notations]** e.g., in line 124 and eq.7, $\Delta(\theta, G)$ and $\Delta\left(f_{\theta}, \mathcal{G}\right)$ are essetially representing the same entity, a Laplace-Beltrami operator.

- ***Significance:*** With the code provided in the supplementary, other researchers or practitioners can use or build on the ideas. However, the performance gains are insignificant since the benchmark (Cora, Citeseer, and PubMed) are well studied and have less space for improvement.
questions: I am unfamiliar with this research topic; therefore, I would like to see the discussions on the following points.

**[Why the two expressions in Eq.7 could be equivalent, i.e., Spatial vs Time]**

**[Depth vs Width, in the formulation of NHK]**

**[Computation Complexity]**

limitations: The authors claimed to address the limitations of their work in the checklist, however, I can't find its location. Adding the pointer to describe the limitations and future directions would be better.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a new method of knowledge distillation for graph neural
network, which takes into account the geometry of the network architecture, in
particular edge- and node-awareness. The objective of knowledge distillation in
the context of teacher-student graph networks is to try to make the student
networks, trained on partial graph (hence much smaller scale), mimic the best
classification performance of the teacher (trained on complete graph).

To do this, the authors propose a generalization of the heat kernel for GNNs,
dubbed Neural Heat Kernel (NHK), based on the previous works that established
connection between heat kernels and different GNNs architecture. They establish
theoretical arguments for the existence of a single-layer NHK (theorem 1), and
a semigroup identity property (theorem 2) which proves that the operator
resulting from stacking multiple later of NHKs remains a valid kernel.

In terms of modeling, they proposed the non-parametric and parametric form of
NHKs, together with corresponding loss functions and training schemes. The
non-parametric distillation compose of three different forms of deterministic
neural heat kernels, while parametric is based on an EM-like training
scheme. All of these are demonstrated to have better classification
performances from related methods and even oracle networks.

strengths_and_weaknesses: Strengths:

- Overall I find the paper well-written, with clear structure -- although a bit
  dense and undefined notations in the beginning (comments below in the
  weakness).

- The paper propose a more rigorous formalization of Knowledge Distillation in
  the context of Geometric Deep Learning, with theoretical justification borrow
  from PDE and differential geometry literature.
  
- The authors deliver some theoretical arguments to their proposed methods, and
  empirical results show they all perform better than existing solutions on
  even large-scale dataset.

Weaknesses:

- Major: I see absolutely no discussion on the computational time of the NHKs
  in both parametric and non-parametric settings. Since now the training loss
  need to take into account additional information of the underlying
  topological space, the training schemes look quite complicated, which require
  heavier part on hyperparameters-tuning. The choice of hyperparameters are
  addressed in the Appendix, which resort to grid-search, so if I'm not wrong
  this does not scale well in terms of computational time. I insist the authors
  provide justification on this part, and maybe even better to add it in the
  main text.

- Major: related to first point, even though the performance of parametric NHK
  (denoted by VGKD in the experimental results section) is generally the best,
  I see no discussion on the convergence speed EM-like scheme for training the
  parametric NHK.

- Minor: The matching loss $\mathcal{L}_{pre}$ is nowhere to be found in the
  paper, which render missing equation in algorithm box 2. Could the author
  provide the form of this loss somewhere, as it is quite important part as
  well?

- Minor: I'm a bit confused about the notation of the Example 1 in Section 2 --
  what is $\sigma$ fuction and what is $\mathbf{\Omega}$? Also is this the same
  $\sigma$ to denote activation function, which is used in Eq.(16)?


questions: - In theorem 1 regarding the existence of NHK, I see the assumtion to have the
  equivalence between graph signal X and node features layer H. Can the authors
  comment more on this assumption, for example, will this ever be violated?
  
- In Section 5.4 on performance with privileged ratio, maybe I'm missing
  something but why do we have the oracle in edge-aware setting perform quite
  bad in only Pubmed dataset (i.e. I don't get your point on the
  justification)?

limitations: Limitations have not been discussed in the main text.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The authors formulate a new knowledge distillation problem: making a GNN trained on a subgraph emulate the output of another GNN trained on a larger graph. To tackle the problem, they draw inspiration from thermodynamics to propose a kernel over GNN embeddings. Then, distilling knowledge is reduced to a task of matching kernel matrices --- or between a network's embeddings an another's smoothed by the kernel matrix.
strengths_and_weaknesses: ## Strengths
* To the best of my knowledge, authors draw a novel connection between graph convolutions and heat kernels;
* Authors attack an original problem: condensing the geometry of a large graph into a GNN trained on a subgraph;

## Weaknesses
* While novel, the motivation for the problem is weak. In most interesting cases, we will only observe a subgraph and there is no oracle capable of outputting embeddings based on the whole graph. This seems to limit the applicability of this sort of method. it would be more reasonable just to have some other graph(s) independently drawn from the same data-generating process;
* The connection to thermodynamics seems very thin, as it just reduces to computing kernels over embeddings.
questions: * Both parametric and non-parametric approaches depend on kernels, which might be a computational bottleneck. How do they scale with the number of nodes?
limitations: Paper lacks a proper discussion of its own limitations. There is also no statement regarding possible societal impacts.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors formalize the problem of graph topological knowlege transfer for GNNs. A new notion of neural heat kernel is proposed to characterize the geometric property of th eunderlying manifold for graphs. And a framework of geometric knowlege distillation to transfer geometric knowledge from a teacher GNN to student GNN. 
strengths_and_weaknesses: Strengths:
1. The paper proposes to distilling structural knowledge from teacher model of large graph to student model of small graph, which sounds novel and useful. 
2. NHK is presented to encapsulate the geometric property of the underlying manifold concerning the architecture of GNN.
3. The experimental results demonstrate the efficacy of the proposed method.
questions: 1. What's the effect of the proposed methoc on the social network dataset, e.g., Reddit?
2. There are many distilling strategies. Is it possible to give more comparison between different distilling strategies? And different parameters?

limitations: The paper produces good experimental results and careful theoretical deriviation. I hope to see the effect on the dynamic graph of online distilling tasks.
review_title: null

Meta-review: metareview: In this paper the authors propose the first distillation method for GNN to promote  the transfer of knowledge from a large graph to a smaller graph. The authors propose an extension of Neural Heat Kernel to encode information in the GNN and use it to propose  Geometric Knowledge Distillation. Numerical experiments are done with parametric and non-parametric versions of the method and show its interest on several tasks such as node classification, and more classical distillation tasks such as model compression and self distillation.

The contribution was appreciated and all reviewers agree about the novelty of the connection between tangent kernel and GNN, and its possible use in other applications in the future. The experiments are interesting and often show a large gain in performance wrt classical distillation methods. Some concerns were raised by reviewers: numerical complexity, limited practical interest, lack of discussion about limits.

Most of the concerns have been answered quite well by the authors during the discussion which was much appreciated.  The precise tables with numerical complexity and the limitation discussion now in appendix are very interesting. So the consensus among reviewers was that the paper deserves publication but the numerical complexity and complexity discussion MUST be integrated in the main paper instead of being in supplementary.
confidence: Certain
Decision: Accept