Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper applies KD in the context of the graph. It aims to distill the teacher output of a GNN model into a simple MLP model. Empirically, they show that this simple KD design is able to improve the student MLP model by a large margin and can match the results coming from a teacher GNN model. Besides, it shows empathically, the inference time can be greatly improved.
main_review: # Interesting point
- The MLP with a simple knowledge distillation design applied on top of a teacher GNN does show huge potentials in terms of the inference time. The aspect of bypassing fully the neighborhood fetch process during inference time is interesting.

- Empirically, they show this simple design has competitive performance as the teacher GraphSAGE model while enjoying hundreds of times faster inference time and ~30x faster than other inference acceleration methods. It does show the potential of this direction.


# Main concern

**Methodology aspect**

-  The novelty of the work is too limited, they only apply the original distillation formulation from Hinton et al. 2015.  There is no interesting distillation design that tailors to the unique characteristics of the dependency complication introduced by the graph.

- The literature discussion is not thorough, missing out on quite some important related works which discuss the knowledge distillation in the context of graph-structured data [1] [2] [3].

[1] Yang, Y., Qiu, J., Song, M., Tao, D. and Wang, X., 2020. Distilling knowledge from graph convolutional networks. In Proceedings of CVPR.

[2] Deng, X. and Zhang, Z., 2021. Graph-Free Knowledge Distillation for Graph Neural Networks. arXiv preprint arXiv:2105.07519

[3] Xu, Y., Zhang, Y., Guo, W., Guo, H., Tang, R. and Coates, M., 2020, October. Graphsail: Graph structure aware incremental learning for recommender systems. In Proceedings of CIKM.

- Some claim discussed in the motivation section is not accurate. The authors claim that "To infer for a single node with a L-layer GNN on a graph with average degree R will require $O(R^L)$ fetchings", which is not accurate considering when working on large graphs, we will not use node wise sampling technique. People tend to use efficient batching and layerwise or graph-wise sampling techniques [4] [5] to greatly reduce the node fetching process. 

[1] Chiang, W.L., Liu, X., Si, S., Li, Y., Bengio, S. and Hsieh, C.J., 2019. Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks. In Proceedings of KDD.

[2] Zeng, H., Zhou, H., Srivastava, A., Kannan, R. and Prasanna, V., 2020. GraphSAINT: Graph Sampling-Based Inductive Learning Method. In Proceedings of ICLR.

- Do we really have a scenario that we can establish the existing efficient batching and graph-wise sampling techniques are unable to meet the inference time constraint? Can authors provide an intuitive example in practice?

**Experimental result**

- The proposed approach doesn't show competitive performance on  Arxiv and Product dataset, which are the larger datasets we care more about the inference efficiency potentially. 

- This simple distillation design didn't show competitive performance when the teacher model is more advanced GNN architecture, as shown in Figure 5 (c), which makes the technique less interesting. I believe establishing that distilling a more competitive teacher GNN into a student model with less performance drop will make the work much stronger.

- To add on top of my argument above, the top model performance (which also uses a fashion of KD) on Arvix and Product is 76.2% and 86.4% respectively. Compared to the GLNN performance on Arvix and Product which is 63.5% and 68.9%. I'm not too sure about the accuracy and computation complexity trade-off make sense. 

- The baseline comparison of the paper is not thorough as I point out above regarding missing out on quite some important related works about KD in the context of graphs.  For an empirical-based paper, I do expect the experiment section to be much more thorough than what is being provided.

summary_of_the_review: The novelty of the work is limited, there is no interesting distillation design that tailors to the unique characteristics of the dependency complication introduced by the graph. Besides, the literature discussion is not thorough, missing out on quite some important related works (KD on graph-structured data). In the experimental study, this simple distillation design didn't show competitive performance when the teacher model is more advanced GNN architecture. Overall, I recommend rejecting this paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: To overcome inference latency of GNN models, this paper proposes a GLNN framework that trains a student MLP with supervision from a teacher GNN model. GLNNs significantly improves performances of pure MLPs and are comparable with traditional GNNs in many cases. The paper explores different real-world settings including transductive and inductive learning. It also discusses when GLNNs will fail, which is a significant question, and address that it is rare in practice.  
main_review: * Pros:

1. The motivation of training student MLPs with teacher GNNs is very important in practice. This paper explores a novel path that is separate from typical acceleration techinique in GNNs, such as sampling and canceling nonlinear transformation. Related discussion in Section 5.5 is well made.
2. The careful discussion on transductive and inductive learning is significant, with comprehensive experiment results in Table 3. I thought it would not work well in inductive cases, but it can.
3. It is an intriguing discovery that GLNNs with larger widths can achieve considerable performance boost.
4. The teacher GNN architecture is flexible, as shown in Section 6. 

* Concerns:

1. To clarify, is the training loss of GLNNs in Figure 4 the same as the first label-loss term in (1)?
2. There is a related work [1] that also trains MLPs on graphs instead of GNNs. They train MLPs with contrastive loss based on graph structure to guide MLPs to find good parameters in the MLP space, which is sharing similar spirits. It would be good to have a discussion on that.
3. I am wondering whether it will enhance the performance further if the input feature is combined with 1-hop neighorhood, such summation of features in 1-hop neighneighorhood, which is very simple. (Or compare with pure 1-hop GA-MLPs.) This stands between GLNNs and GA-MLP models. The motivation is from the fact that usually low-order neighborhood is the best choice and GNNs cannot handle well with long-range information. In this sence, it will be interesting to see how GLNNs work with simply combined features from low-order neighborhood, under the supervision of GNNs.

4. Since MLPs are trained to fit GNNs in different architectures, is it possible to explore the bias of different GNNs via analyzing learned parameters/functions in MLPs?

[1] Graph-MLP: Node Classification without Message Passing in Graph. Yang Hu et al. ArXiv:2106.04051.
summary_of_the_review: I would like to recommend to accept this paper for its practical intuition, satisfying performance and comprehensive discussion on limitations.
review_title: null

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: The paper proposes to distill knowledge from GNNs to MLPs so that at inference time it requires less burdensome. The idea is neat and the implementation is flawless. 
main_review: The paper shows that it's possible to distill knowledge from GNNs to MLPs without sacrificing performance, while reducing inference time by orders of magnitude. The approach is very simple, which copies a classical KD approach proposed by Hinton et al. At inference, it shows significant reduction of time compared to baseline GNNs and existing approaches. The paper is well-written, motivated, without unnecessary theory, and does enough ablation studies. 

The only concern is the datasets used to verify the approaches. While there are OGB ones, cora/citeseer/pubmed are quite small for this type of approaches to motivate enough KD. I would include other datasets, for example, coming with heterogeneous node features [1] (where MLP and GBM can work quite well) and heterophilous datasets [2] (where LP can work well). Results there would give additional insights onto the use cases of KD for GNNs. It's also interesting to compare performance of this approach compared to graph-agnostic baseline using GBM [1] (it should give even less inference time than MLPs and as such could be more amenable for this type of KD). 

I may update my score based on the authors' responses.

[1] Boost then Convolve: Gradient Boosting Meets Graph Neural Networks https://arxiv.org/abs/2101.08543
[2]  New Benchmarks for Learning on Non-Homophilous Graphs  https://arxiv.org/abs/2104.01404
summary_of_the_review: Practical limitations of GNNs in industry setup are quite severe for existing GNN models and this approach offers an easy way to overcome them and leverage graph topology at the same time.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a Knowledge Distillation (KD) based approach for producing MLPs able to achieve comparable performance to Graph Neural Networks (GNNs) on node-wise classification tasks. The MLPs receive as input only the features of each target node v and it is trained to imitate the behavior of a pre-trained GNN on an extended version of the training set consisting of:

1) the original labeled set where the GNN was trained;
2) an additional set of unlabeled nodes which are equipped with soft labels produced by the GNN.

As the GNN (which operates as a teacher in this scenario) has a good inductive bias, it can learn a robust decision boundary even from a rather small training set. Such a decision boundary can then be taught (at least partially) to the simpler MLP (which operates as a student) by training the MLP on a larger training set where it learns to capture patterns in the target nodes features that correlate well with the soft labels produced by the GNN. If the GNN shows good performance over unseen condition, the MLP trained in this way ends up being more robust wrt a MLP trained only on the original labeled set as it can observe a potentially significantly wider variety of conditions at training time.  

As in this scenario the MLP doesn’t receive any descriptor representing the behavior of the neighborhood of the target nodes, the overall approach doesn’t work in all possible settings (e.g. in situations where the label of a node depends exclusively on the behavior of the neighbors). Nonetheless, the authors showed in the paper that in classic scenarios (i.e. node classification tasks on citation or product networks) it is possible to effectively train a MLP able to achieve comparable performance to a GNN, while requiring a fraction of the computational cost as no convolution is realized on the given graph.
main_review: The paper is overall well written and rather easy to read, with a good introduction that describes the setting of the work, several references to prior art and an ablation study that analyzes the potential failure points of the approach (which I believe provides a good understanding of the pros and cons of the solution). To the best of my knowledge, the idea presented in the manuscript is overall novel and I believe generally interesting to the community given the performance and speed up the authors managed to achieve with their approach in the analyzed settings (e.g. 273x while losing only 2% in accuracy over OGB Products). 

The experimental evaluation proposed in the manuscript is generally sound, although I do have some points I would like to highlight in my review on this matter. Unless I’m misunderstanding the content of Section 5.2, the experiments for the transductive setting presented in Table 1 I believe boil down to a sanity check. As presented in the end of Section 5.2 indeed, the MLP in the transductive setting is trained to reproduce the logits of the GNN on *all* nodes of the given graph, irrespectively on whether these belong to the training, validation or test set. If the nodes thus all show different features, a sufficiently large and deep MLP should always be able to overfit on the features of the target nodes and replicate the predictions of the GNN. As such, the experiments presented in Table 1 don’t look particularly meaningful to describe the quality of the proposed approach and I would ask the author to please clarify in their rebuttal whether they avoided training the MLP on the logits produced by the GNN for validation / test nodes and, if I’m right, to better clarify the relevance of these results in the text.

Following the previous comment, in Table 4 the authors present two aggregated measures of the quality of the model (\Delta_{MLP} and \Delta_{GNN}) that should describe a typical production setting consisting of 80% unlabeled nodes that have been observed at training time and 20% of new unlabeled nodes the model has never seen before. As for the above, I believe the most interesting result here is represented by the performance of the model in the inductive scenario and I would thus suggest the authors to express the improvements in performance in the table for all the 3 different settings they described (i.e. transductive, inductive and production) instead of just the production one (it would better highlight the performance of the approach in different conditions and thus enhance clarity).

Finally, Figure 5 middle, it is possible to observe how the GNN actually improves in performance over the inductive set if we increase the size of this and thus restrict the size of the transductive one (i.e. the number of nodes the model is actually able to observe at training time). While the improvement seems to be marginal from the picture, this is somehow a surprising and counterintuitive result and I wonder if the authors have an explanation for this. Additionally, I would recommend adding standard deviation bars in all the plots of Figure 5 to highlight the stability of the performance of the models over the different datasets and potentially have a different picture (or a series of pictures) in the supplementary material showing the curves for each dataset independently. 

Besides the points highlighted above, I believe the manuscript is in good shape and I would recommend the acceptance of the paper to the conference.

summary_of_the_review: The paper proposes a novel approach for effectively training MLPs for node-wise classification tasks on graph structured data. The proposed approach allows to train architectures able to approach in most scenarios the performance of a reference GNN (SAGE) while requiring a fraction of the computational cost as no convolution is realized on the given graph. The experimental evaluation presented in the paper is generally sound and well highlights the benefits of the idea proposed in the manuscript.
review_title: null

Meta-review: This paper proposes a very simple procedure to accelerate the inference time of graph-structured Neural Networks, by distilling knowledge of a GNN into a node-wise MLP. 
Despite some concerns about the novelty of the methodology (which borrows heavily from previous KD works), reviewers generally found this empirical work well executed and providing a potentially useful baseline for large-scale applications. Therefore, the AC recommends acceptance.
Decision: Accept (Poster)