Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes to distill a better MLP from the GNNs via reliable knowledge distillation. In particular, the authors introduce a meta-policy to filter out those unreliable soft labels. The authors conduct extensive experiments on multiple graph datasets.
strength_and_weaknesses: Strengths:

(+) How to learn a better MLP from GNNs is an interesting question. Considering GNNs might suffer from the scalability issue, how to learn an MLP as a replacement deserver some effort.

(+) This paper conduct experiments on nine datasets. In addition, the authors report mean and standard deviation for most of the experiments, which is nice.

Weaknesses:

(-) Vague and confusing motivation. The authors claim that “directly leveraging soft labels from the GNN teacher is suboptimal” but don’t have a reasonable explanation. Why “a large portion of unlabeled nodes will be incorrectly predicted by GNNs”? What “limited generalization ability”? The “100% accuracy on the training set, yet their test accuracy is merely around 80%” seems like an over-fitting problem. How is this performance gap between training and testing relevant to “leveraging soft labels from the GNN teacher is suboptimal”?  

(-) Lack of novelty. The authors simply introduce a meta-policy to filter out unreliable soft labels based on node representations. The idea of distilling confident labels is not new [1, 2, 3], and methods using reinforcement learning meta-policy to determine confidence also exist [4, 5]. The author needs to reconsider the novelty of the proposed method and make new contributions, not just a combination of existing techniques.

[1] Müller, Rafael, Simon Kornblith, and Geoffrey E. Hinton. "When does label smoothing help?." NeurIPS, 2019

[2] Li, Yuncheng, et al. "Learning from noisy labels with distillation." ICCV. 2017

[3] Yang, Cheng, Jiawei Liu, and Chuan Shi. "Extract the knowledge of graph neural networks and go beyond it: An effective knowledge distillation framework." WWW, 2021

[4] Kostas, James, et al. "High Confidence Generalization for Reinforcement Learning." ICML, 2021

[5] Jordan, Scott, et al. "Evaluating the performance of reinforcement learning algorithms." ICML, 2020

(-) some technical details are unclear and incorrect. For example, Figure 1 (middle) is really confusing. GLNN contains an MLP student as the yellow line, what is the meaning of the red line MLPs-student? Are they different or are they the same? How is the MLPs-student implemented? Why the MLPs-student can perform much better (~0.78 accuracy) than the SAGE teacher when the noise ratio is zero? In the leaderboard of ogb-arxiv (https://ogb.stanford.edu/docs/leader_nodeprop/#ogbn-arxiv), the best model to date can only achieve 0.7719 accuracy, while MLPs-student already outperforms the state-of-the-art. I could not find any supporting claims/experimental results in the introduction or in the later experiment sections. The only time Figure 1 (middle) gets mentioned is in the preliminary section without any experimental details. I highly doubt the correctness of the figure. In addition, for the statement “GLNN-label is a variant of GLNN by excluding unlabeled nodes” - should it be GLNN-without-unlabel? All three models in Figure 1 (left) utilized labels, and GLNN-label cannot clearly distinguish the model.

(-) More supporting details are needed for some experiments. For example, In RQ2, the authors claim that “unlike small datasets, it is hard to train the MLP student on large graphs due to soft-label noises”. However, the soft-label noise difference between small and large datasets are not shown. In addition, what are the performances of SW, Entropy, and Cluster methods on large datasets?

(-) Lack of theoretical explanations or analyses. The motivation of this paper and the designs of the model is based on the “noisy soft labels”. However, why the accounting of the noises helpful is overlooked? How is the proposed model capture the noises and eliminate them? Why does elimination work? And how it works? How helpful can it be? For example, for Figure 4, the authors need to provide explanations for why RKD-MLP can perform well under noisy graph topology and noisy features. How are the graph topology and features relevant to the noisy soft labels? 

(-) Poor readability. A lot of figures and claims in this paper are ambiguous and hard to understand, as I mentioned in the previous comments. For example, Figure 6 is really confusing. What do different colors indicate? What is the difference between different datasets? What is the meaning of the proposed model and GNN having the same predictions? In Figure 2, what is the meaning of different colors?

(-) Copy and paste Table 5 in RQ6 from the GLNN paper without reference. Table 5 in the paper is exactly the same as Table 4 in the GLNN paper. No reference is given. It seems the proposed method will not introduce any computational requirements in the inference stage. Will it introduce a heavy computation burden in the training stage?

[6] Zhang, Shichang, et al. "Graph-less neural networks: Teaching old mlps new tricks via distillation." ICLR, 2022.
clarity,_quality,_novelty_and_reproducibility: This paper has poor clarity and does not have sufficient quality and novelty. The code is released.
summary_of_the_review: The current version of the draft needs some work. The motivation is unclear and the paper lacks novelty in terms of its approach. The paper also needs to 1) double-check the correctness of some results and statements, 2) provide supporting details for the experiments, 3) provide explanations and analyses, and 4) improve the readability. Many parts of the paper are ambiguous and missing important details.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new semi-supervised KD framework (RKD-MLP) to filter out unlabeled nodes whose soft labels are likely to be incorrectly predicted by teachers. In this framework, they use a simple reinforcement learning framework to learn whether nodes have reliable soft labels, thus improving performance and maintaining inference efficiency. The experiments show the benefits of RKD-MLP.
strength_and_weaknesses: Strength:
1. They study the impact of unlabeled nodes in GNNs distillation and demonstrate the importance of reliable pseudo labeling.
2. They provide a simple RL-based method to filter out unlabeled nodes with unreliable predicted soft labels.
3. Experiments show that the new method can improve the accuracy of different datasets.

Weakness:
1. Including more analysis on whether the meta-policy can filter out unreliable nodes would be better. Figure 6 shows some results. But the GNN prediction is used as the ground truth, which may be incorrect. One possible way is to test the policy on some labeled nodes in the testing set.
2. I would like to know how many unlabeled nodes are removed by the meta-policy. If too many nodes are removed, we may lose too much information from the data. The meta-policy uses 0.5 as the threshold, have you tried other thresholds?
3. This framework is doing something similar to out-of-distribution detection. It would be great to provide some discussion on connections or performance comparisons between the two.

clarity,_quality,_novelty_and_reproducibility: The paper has good clarity, quality, novelty, and reproducibility.

Minor:
1. More explanation and discussion about Figure 4 would be desirable.
2. It would be interesting to discuss or compare the inference speed between pruning, quantification, and your method.
3. There are some typos, for example in the related work section “However,s it only”.

summary_of_the_review: Overall, this paper has a clear motivation and proposes an effective method. But in the experiment, most of the results are about the performance of GNNs students. It would be nice to have more results about the meta-policy.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper aims to design a model which can achieve better accuracy and efficiency. The proposed method RKD-MLP utilizes knowledge distillation techniques and the idea similar to noise-label learning to achieve the mentioned goals. RKD-MLP aims to learn the 'reliable' soft label (which is called meta-policy in this paper) to guide the student network in learning a better MLP network.

Different from existing works, this paper is using the joint training approach to train the network under the reinforcement learning framework i.e., policy gradient with advantage. 
strength_and_weaknesses: Strengths:
1. It is an interesting approach that aims to achieve better accuracy and efficiency for GNN.
2. The idea is intuitionistic and easy to follow and understand.

Weaknesses:
1. As outlined in the title, the proposed method aims to boost accuracy and efficiency. However, the crucial experiments on the efficiency are placed in the appendix which is not reader-friendly. If I only go through the main context of this paper, I will have no idea how the proposed method can boost efficiency. There is no clue that why the proposed method can achieve good efficiency. It takes too many spaces to report the accuracy results in your main text, and some of them should be swapped with the efficiency results in your Appendix.
2. As the authors mentioned in the paper, the meta-set is the validation set. How the dataset is split? How does the size of the "meta-set" (i.e., validation set) affect your performance in terms of accuracy and efficiency? I think this question is worth investigating and quite important. If the proposed model can achieve a better performance than existing methods by using a small "meta-set", it can provide some evidence of the efficiency.

clarity,_quality,_novelty_and_reproducibility: The proposed method is quite straightforward, and it is incremental work following the GLNN. I think the novelty is limited.

Table 1 and Table 2 is not very friendly for readers. The best performance is not bold, and the second best is not indicated. Which may mislead readers in terms of the improvement. Moreover, the caption is saying that the improvement is calculated by comparing with GLNN only, which does not very make sense to me. I understand that as a follow-up work, compare with GLNN is necessary. But when we say the improvement, we normally will report the improvement achieved compared with the 2nd best performance.

As it is clear that on Table 1, the cluster approach is very close to the proposed RKD-MLP (i.e., 80.66 vs 80.97, 0.38% improvement).

I do not agree to place the main experiments into the Appendix especially when you place the keyword on your title. 
summary_of_the_review: Overall, considering the above points, especially the novelty and experiments, I think this paper is not good enough for acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed Knowledge Distillation (KD) method for Graph Neural Networks. 
The proposed method is motivated by a recent study that shows that  GNNs can be compressed to inference-friendly multi-layer perceptrons (MLPs) by training MLPs using the soft labels of labeled and unlabeled nodes from the teacher. 
However, leveraging the soft labels of all unlabeled nodes may be suboptimal since the teacher model would inevitably make wrong predictions.
This paper's proposed method, called: Reliable Knowledge Distillation for MLP optimization, is the first noise-aware knowledge distillation framework for GNNs. Its core idea is to use a meta-policy to filter out those unreliable soft labels. For training a meta-policy, the authors propose a reward-driven objective based on a meta-set and adopt a policy gradient to optimize the expected reward. 
Afterward, the authors use the meta-policy to the unlabeled nodes and select the most reliable soft labels for distillation. 
The extensive experiments across various GNN models,  7 small graphs, and 2 large-scale datasets demonstrate the superiority of the proposed method to the prior art.

strength_and_weaknesses: *Strength*

The paper is well-written, and the method is well-explained. 
The evaluation of the proposed is very impressive, including various datasets under both transductive and inductive settings
The authors performed an extension ablation study of the proposed dataset, showing how different model parameter choices impact performance. 

*Weakness*
My major concern is why the authors haven't evaluated their method with SOTA architectures GNNs as teachers.
Please see the paperwithcode SOTA benchamrks. 
 



clarity,_quality,_novelty_and_reproducibility: The proposed method is well-explained, and all the implementations detail, including source code, are included.
summary_of_the_review: This paper proposed Knowledge Distillation (KD) method for Graph Neural Networks. 
The proposed method is motivated by a recent study that shows that  GNNs can be compressed to inference-friendly multi-layer perceptrons (MLPs) by training MLPs using the soft labels of labeled and unlabeled nodes from the teacher. 
The method is well-explained, and evaluations are solid. My main concern is that there are still hand-crafted GNN-based architectures that outperform the proposed method's performances. See the SOTA result on paperwithcodes. I would like the authors to elaborate on this regard. Specifically, the authors conclude from the results of table 10 that they manage to achieve better performance of the student rather than the performance of the teacher. Are the same phenomena would preserve also using SOTA backbones as a teacher? 

In overall, I am positive about the novelty and the impact of the proposed method, looking forward receiving responce on the raised issue.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a distillation approach (RKD-MLP) to distill MLPs from GNNs, for the semi-supervised node classification task. Their approach uses reinforcement learning (RL) to filter out noisy soft labels from the teacher (GNN) model. 

In general, reviewers find the paper to be clearly written, and well-motivated, and appreciate the simple and intuitive RL-based approach. However, there are a number of main issues raised by reviewers:

- Experimental comparisons: performance improvements of the proposed method over the Entropy and Cluster baselines is small in many cases (and seem to be not statistically significant in some cases). This makes its benefits less clear considering that the RL approach seems more complex with likely higher training time than these simple heuristics, and also requires significantly more hyperparameter tuning (with around 5 hyperparameters to tune, with up to 10 choices each); it is also unclear on what data the tuning was done (considering that the validation set is already used for the RL algorithm). Reviewers also note that it is unclear why the proposed method is mainly compared against GLNN instead of Entropy and Cluster (for calculating the percentage improvement) when GLNN has usually weaker performance than 1 or both of these. These baselines are also seem to be not considered in some of the other experiments, such as the large-scale graphs (Fig 3) and the added experiments on the CPF teacher. 

- Technical novelty: some reviewers note that the paper can be seen as mainly combining some existing ideas of GLNN (knowledge distillation) and reinforcement learning. The author response points to the novelty of the RL formulation for filtering. They also state that the RL method avoids needing a hyperparameter for the threshold, though this is less convincing considering that it introduces a few other hyperparameters.

- Some other points were noted but satisfactorily addressed by the author response, including the use of validation set, SOTA teacher model, and improvements to paper presentation. I thank the authors for their efforts in addressing concerns and improving the paper.

In the end, reviewers and AC agree that while the work has promising merits, due to these issues, the work is not yet ready for publication at ICLR. The reviews offer a number of helpful suggestions for improvement, so I encourage the authors to continue improving the paper based on the reviews for future submissions.
justification_for_why_not_higher_score: The main reason was the experimental issues (e.g. small or insignificant improvements, and issues with comparison with simple baselines as mentioned in meta-review), while the novelty issue was also a significant consideration.
Decision: Reject