Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: To further improve the performance of few-shot graph neural networks (GNNs), the authors investigate the effect of node importance and theoretically demonstrate the effect of node importance on the lower bound of model performance. Then, based on the proposed theory, the authors propose a new method, named Node Importance Meta-Learning (NIML), for the few-shot node classification task. The empirical study shows the effectiveness of NIML. 
strength_and_weaknesses: Strengths:
S1, this paper is well presented and easy to understand and follow.
S2, although there are some assumptions, the proposed method is based on a complete theoretical foundation.
S3, the empirical study can demonstrate the effectiveness of NIML.

Weakness:

W1, the theory is based on one assumption that node distance is known. To solve this issue, the authors propose a simple but straightforward solution. It would be better if authors can conduct more experiments to show its eligibility.

W2, as mentioned in Sec. 1, GPN considers the importance of nodes, but it is lack theoretical analysis. The authors are expected to explain why NIML outperforms GPN in Tab. 1 since it also introduces an importance mechanism. (Sec. 6.2 misses such analysis).

W3, authors may need to explain where the results of baselines come from. For example, in the original paper, the acc of GPN in 5-way 5-shot at Reddit, Amazon-Elec, and DBLP are 68.4, 70.9, and 80.1 respectively. But in this paper, these results are 66.6, 70.3, and 78.6. Besides, why another benchmark data Amazon-Clothing is discarded in this paper?

W4, it seems the results of the first part in Sec. 6.3 are copied from Tab. 1. Thus, it is redundant to mention them again. Especially, these baselines use different frameworks. It may be unfair to show the effectiveness of different important mechanisms built on different frameworks. Instead, the authors can implement various mechanisms under one framework, then report the findings of existing important mechanisms.

clarity,_quality,_novelty_and_reproducibility: This paper is well polished and easy to understand and follow. It is a novel idea to introduce the theoretical analysis into the importance mechanism.
summary_of_the_review: Although this paper introduces a clear idea, it lacks more discussions on its core idea and some key experimental results.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper mainly proposes NIML, a few-shot graph node classification method to predict nodes with novel labels. The method considers the node importance among different nodes in a task and neighborhood relationships for a given node. The paper theoretically analyzes the influence of the node importance and verifies the effectiveness of NIML in the experiment part.
strength_and_weaknesses: Strength:  

1.The paper theoretically searches for the node importance in few-shot graph node classification problems and demonstrates that the node importance can help the model get a higher accuracy lower bound. 

2.The experiment shows that the proposed NIML performs better than other baselines.

Weakness:

1.I think the work is lack of novelty as the work GPN[1] has already proposed to add node importance score in the calculation of class prototype and the paper only give a theoretical analysis on it.

2.The experiment part is not sufficient enough. 
(1) For few-shot graph node classification problem to predict nodes with novel labels, there are some methods that the paper does not compare with. For example, G-Meta is mentioned in the related works but not compared in the experiments. A recent work TENT[2] is not mentioned in related works. As far as I know, the above two approaches can be applied in the problem setting in the paper.
(2) For the approach proposed in the paper, there is no detailed ablation study for the functionalities of each part designed. 
(3) It is better to add a case study part to show the strength of the proposed method by an example.

Concerns:

1.The paper consider the node importance among nodes with same label in support set. In 1-shot scenario, how node importance can be used? I also find that the experiment part in the paper does not include the 1-shot paper setting, but related works such as RALE have 1-shot setting, why?

2.The paper says that the theory of node importance can be applied to other domains. I think there should be an example to verify that conclusion.

3.In section 5.3, ‘we get access to abundant nodes belonging to each class’. I do not think this is always true as there might be a class in the training set that only has few samples given the long-tailed distribution of samples in most graph datasets. 


[1] Ding et al. Graph Prototypical Networks for Few-shot Learning on Attributed Networks

[2] Wang et al. Task-Adaptive Few-shot Node Classification


clarity,_quality,_novelty_and_reproducibility: Clarity, Quality: The paper is clear and easy to follow.

Novelty: I think the paper is lack of novelty as the method relevant to node importance has already been used in existing works. I do not think a theoretical analysis is novel enough.

Reproducibility: I think most of the experiment results in the paper can be reproduced.
summary_of_the_review: [Reject] I reject the paper because the main idea of the paper (node importance) has already been proposed in existing works(GPN) and the experiment for the method is not sufficient enough.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a meta-learning approach for few shot node classification on a graph. In particular, a ProtoNet metric-based approach is adopted. However, unlike traditional ProtoNet where the class prototypes are computed from class samples in an unweighted manner, this paper assigns different importance scores to different samples. The importance scores are derived from a distance prediction function, which is in turn based on an attention mechanism. The paper also analysed the theoretical properties, in which a weighted scheme can improve the lower bound on the accuracy, compared to the unweighted scheme. Experiments are conducted to evaluate the efficacy of the proposed approach.
strength_and_weaknesses: Strengths:
S1. The paper is well balanced between model design, theoretical analysis and empirical results.

S2. The paper is well written and well motivated.

S3. The empirical results are promising.

Weaknesses:
W1. The theoretical analysis is based on the lower bound of the accuracy, which shows that a weighted strategy can increase the lower bound. However, there is still a gap between the lower bound and the true accuracy. Without analysing how tight the bound is, an increase in the lower bound may not necessarily imply the impact on the accuracy. Some discussion on the tightness of the bound can be beneficial. 

W2. Ablation study on the importance score: Compared to Proto-GCN, Proto-GCN+GAT can already improve the performance quite a lot, and the marginal benefit brought by the proposed method NIML is relatively small (note the y axis starts from 60. So do we really need the proposed importance calculation? What is the computational overhead compared to just using attention as the importance score?
clarity,_quality,_novelty_and_reproducibility: Overall clarity is good. Idea is novel with theoretical support.
summary_of_the_review: Overall, I think this is a good paper where the strengths overweigh the weaknesses.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper investigates the problem of few-shot node classification on graphs. To improve the performance, the authors find that the node importance in each task is very important and should be taken into consideration when generating the prototype of each class. In particular, the authors theoretically and empirically demonstrate their viewpoints.

strength_and_weaknesses: Strengths:

1. Few-shot learning on graphs is an important and hot topic.

2. Theoretical evidence is welcomed to verify the necessity of considering the node importance in the calculation of the prototype.


Weaknesses:

1. One of my main concerns is that, the idea of considering node importance in each task is not novel. The authors also mentioned that some work (e.g., Ding et al., 2020) has been proposed based on this point for few-shot classification. I feel this limits the contribution of this paper. Though theoretical analysis for this point is provided, in my opinion, only theoretical analysis is not sufficient enough for a paper. A better organizational form of a paper is to propose a novel/interesting model which is associated with the corresponding theoretical analysis. 

2. For the calculation of neighborhood weights, why need to fix the number of neighbors? Is it possible to use all neighbors, since neighborhood sampling may result in information loss?

3. Eq.(15) does not employ $\mu_c$ as input. So why it can calculate the distance between node $v$ and center $c$?

4. From my view, I feel the experiments are not quite sufficient. It is better to provide more model analysis for demonstration.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to follow. The source code is provided.
summary_of_the_review: In summary, this paper provides the theoretical analysis to an existing simple approach, which I think is not quite sufficient enough, and not quite novel. The experiments are not sufficient, either.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This manuscript proposes NIML to investigate the node importance in node classification meta-learning tasks. Specifically, it theoretically demonstrates the node importance between neighbors can increase the lower bound of the model accuracy. Then it proposes the node importance meta-learning architecture to learn the importance score of each node and further train a weighted prototype for meta-learning. Extensive experiments on three benchmark datasets validate the superiority of NIML.
strength_and_weaknesses: Strength:

1. This work provides detailed theoretical proof to demonstrate that assigning higher weights to the samples that have closer distances to the prototype expectation will increase the lower bound of the model accuracy.

2. In general, this manuscript is well written with clear motivations, detailed related works, rigorous proofs, and relatively sufficient experiments, which is easy to follow.

Weakness:

1. There are some gaps between the theoretical analysis and the model design. The theoretical conclusion is that higher weights of samples will increase the lower bound of the model performance, which demonstrates that the model design of existing work Proto-GCN and the proposed model NIML are solid. This work proposes the category-irrelevant node attentions and further designs the weighted prototype based on the importance scores. However, the motivations of the proposed category-irrelevant node attentions (different from Proto-GCN) and the weighted prototype are not clearly discussed in this work. 

2. It claims that Proto-GCN as the input does not meet the expectations for distance predictor. But it does not provide a detailed explanation. Why is it necessary to design a category-irrelevant node information input? Why Proto-GCN is not suitable for your framework? What’s the main contribution of your model design?

3. The model design is rather simple and not novel enough. The main idea of the model design is to calculate the attention scores between nodes and further learn a weighted prototype. The theoretical analysis is to prove that node importance can improve the lower bound of the model performance. From my perspective, there are some gaps between the model design and the theoretical proof. You should provide the theoretical analysis of your model design and validate that your model design is better than Proto-GCN.

4. The discussion of related works is comprehensive, but some discussion about related works is a little bit vague and does not show the key idea of the related works. For instance, in section 2.3, “MetaHG presents a heterogeneous graph learning model for automatically detecting illicit drug traffickers on Instagram.” The main idea of this work is to design a meta-learning model on heterogeneous graphs to address the problem of the limited labeled drug traffickers on Instagram.

5. The presentations of most equations need to be revised. For instance, there should have a comma between the equation and ‘where’ (e.g., equations 3, 4,6, and 10). There should have a period after the equation (e.g., equations 1,2,5).
clarity,_quality,_novelty_and_reproducibility: Generally speaking, this work is well-organized and easy to understand. However, due to the limited novelty of the model design, the gaps between the theoretical proof and the model design, and some presentation mistakes, I would suggest that the model design should be improved and some explanation about the model design should be clearly clarified.
summary_of_the_review: This manuscript is well-written and easy to follow. Besides, it provides a detailed theoretical analysis of the influence of node weights on model performance, which is impressive. However, due to the limited novelty of the model design, gaps between theoretical proof and the model design, and unclear clarification of the model, I would suggest this paper further enhance the model design.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Adding weights to learning samples have been used in many machine learning tasks, and this paper extends the same idea to the scenario of few-shot node classification. The proposed method starts from a theoretical analysis which shows assigning higher weight to the data point that has closer distance to the prototype expectation will increase the lower bound of accuracy. Then, the authors propose a framework for node classification by introducing an importance score predictor. Experiments results show the proposed method is promising.

Strength
- This paper is well presented and easy to understand and follow.
- Empirical results are promising

Weakness
- Methodology design is not very novel, given existing works already consider sample weights (especially in the domain of few-shot graph learning).
- The usage of the theoretical results is not clear. There are concerns on how distance is estimated, how the lower bound indicate the practical performance. Besides, as authors also explained, the results are not specified to graph learning, it is suggested to have graph properties in the results or adding extra experiments outside graph domain.
justification_for_why_not_higher_score: The motivation and theoretical results are not well explained and motivated.
Decision: Reject